AWS Glue
Glue configuration for optimal logging and cost efficiency
Custom Log4J for Spark on Glue to decrease CloudWatch cost!
Latest
AWS
How AWS Budgets Can Prevent Costly Surprises
Or: how to prevent your CFO from turning 50 shades of red
System Integration Test
System Integration Test for Data Products
Or how to do Data Engineering like Metallica!
great-expectations
Custom dashboard for Great Expectations
Enable advanced dashboarding on Great Expectation results using metrics
thinnest-viable-platform
Why You Should Never Mandate The Adoption Of Your Data Platform
Welcome to the first part of our series on data platforms! In this series, we are going to dive into data platforms. In part 1 of the series, we start out by going over the main principles for platform success. In our unique role as a data consultanc...
PySpark
FP-Growth Algorithm and How to Avoid Its Dark Side!
Context A few weeks ago we were contacted by FrieslandCampina to help them with a problem they faced on their recommendation engine. Being one of the biggest dairy companies in the world they sell hundreds of dairy products to millions of customers a...
software development
Speed up dbt workflow with Task
Do you remember the last time you opened your laptop and thought: "I can't wait to spend half my day writing configs in YAML."? Yeah, neither do I. But if you use dbt heavily, I'm afraid you and YAML are in for the long run. Even with the help of dbt...
Software Engineering
Software Engineering for Data Engineers: Introduction
Transitioning from a web development background to data engineering, I've encountered a significant cultural shift. One of the most striking differences is the apparent lack of established software engineering practices within many data engineering t...
Structured Streaming
Mysterious Spark Checkpoints Behaviour
It all started from a change in the checkpoint path of our Spark applications. We use Spark Structured streaming and AWS S3 buckets to maintain checkpoints. Let’s say we were using s3://bucket/spark/t
Machine Learning
No Code, All Insight: SageMaker Canvas Connects Data Analysts to Machine Learning
What is No-Code ML? As a data scientist, I was always skeptical of no-code solutions since they usually provide so little flexibility that makes them practically useless or tries to provide too much flexibility that makes their UI/UX impossible to na...
SupplyChainAutomation
The Future of Logistics: AI's Impact on Supply Chain Management
Introduction Artificial Intelligence (AI) has come a long way since its early days as an academic curiosity in the mid-20th century. From intelligent assistants to autonomous cars, AI is reshaping the world as we know it. But besides its fancy applic...
generative ai
Amazon Bedrock: Simplifying Generative AI with AWS for 2023
Introduction Generative AI has seen a surge in interest over the past few years, with a plethora of applications ranging from content creation to complex reasoning tasks. Recognizing the growing demand, AWS has introduced its managed service, Amazon ...