AWS
Apache Iceberg Compaction
Compact your Iceberg tables using Spark or AWS Athena
Latest
kafka
Navigating the Kafka Summit London 2024
Unveiling the Future of Data Architecture
AWS Glue
Glue configuration for optimal logging and cost efficiency
Custom Log4J for Spark on Glue to decrease CloudWatch cost!
AWS
How AWS Budgets Can Prevent Costly Surprises
Or: how to prevent your CFO from turning 50 shades of red
data-engineering
System Integration Test for Data Products
Or how to do Data Engineering like Metallica!
great-expectations
Custom dashboard for Great Expectations
Enable advanced dashboarding on Great Expectation results using metrics
Data platform
Why You Should Never Mandate The Adoption Of Your Data Platform
Welcome to the first part of our series on data platforms! In this series, we are going to dive into data platforms. In part 1 of the series, we start out by going over the main principles for platform success. In our unique role as a data consultanc...
spark
FP-Growth Algorithm and How to Avoid Its Dark Side!
Context A few weeks ago we were contacted by FrieslandCampina to help them with a problem they faced on their recommendation engine. Being one of the biggest dairy companies in the world they sell hundreds of dairy products to millions of customers a...
software development
Speed up dbt workflow with Task
Do you remember the last time you opened your laptop and thought: "I can't wait to spend half my day writing configs in YAML."? Yeah, neither do I. But if you use dbt heavily, I'm afraid you and YAML are in for the long run. Even with the help of dbt...
Software Engineering
Software Engineering for Data Engineers: Introduction
Transitioning from a web development background to data engineering, I've encountered a significant cultural shift. One of the most striking differences is the apparent lack of established software engineering practices within many data engineering t...
spark
Mysterious Spark Checkpoints Behaviour
It all started from a change in the checkpoint path of our Spark applications. We use Spark Structured streaming and AWS S3 buckets to maintain checkpoints. Let’s say we were using s3://bucket/spark/t
Machine Learning
No Code, All Insight: SageMaker Canvas Connects Data Analysts to Machine Learning
What is No-Code ML? As a data scientist, I was always skeptical of no-code solutions since they usually provide so little flexibility that makes them practically useless or tries to provide too much flexibility that makes their UI/UX impossible to na...