AWS
Stop burning cash on AWS crawlers
Real-time partition discovery at a fraction of the cost
Latest
AWS Glue
Our learnings on AWS Glue Jobs Billing
How AWS Glue can bill Streaming jobs at termination (if you are not careful)
apacheiceberg
Apache Iceberg Compaction
Compact your Iceberg tables using Spark or AWS Athena
Navigating the Kafka Summit London 2024
Unveiling the Future of Data Architecture
AWS Glue
Glue configuration for optimal logging and cost efficiency
Custom Log4J for Spark on Glue to decrease CloudWatch cost!
AWS
How AWS Budgets Can Prevent Costly Surprises
Or: how to prevent your CFO from turning 50 shades of red
System Integration Test
System Integration Test for Data Products
Or how to do Data Engineering like Metallica!
great-expectations
Custom dashboard for Great Expectations
Enable advanced dashboarding on Great Expectation results using metrics
thinnest-viable-platform
Why You Should Never Mandate The Adoption Of Your Data Platform
Welcome to the first part of our series on data platforms! In this series, we are going to dive into data platforms. In part 1 of the series, we start out by going over the main principles for platform success. In our unique role as a data consultanc...
PySpark
FP-Growth Algorithm and How to Avoid Its Dark Side!
Context A few weeks ago we were contacted by FrieslandCampina to help them with a problem they faced on their recommendation engine. Being one of the biggest dairy companies in the world they sell hundreds of dairy products to millions of customers a...
software development
Speed up dbt workflow with Task
Do you remember the last time you opened your laptop and thought: "I can't wait to spend half my day writing configs in YAML."? Yeah, neither do I. But if you use dbt heavily, I'm afraid you and YAML are in for the long run. Even with the help of dbt...
Software Engineering
Software Engineering for Data Engineers: Introduction
Transitioning from a web development background to data engineering, I've encountered a significant cultural shift. One of the most striking differences is the apparent lack of established software engineering practices within many data engineering t...