Legacy Migration Starts with Understanding, not Inventory
An alternative to scale and direct legacy migration activities
Shahin
🧑🍳 Cooking data as a chef de partie at DataChef.co by day! 🪄 Avid coder inventing magic by night!
System Integration Test
System Integration Test for Data Products
Or how to do Data Engineering like Metallica!
great-expectations
Custom dashboard for Great Expectations
Enable advanced dashboarding on Great Expectation results using metrics
Software Engineering
Software Engineering for Data Engineers: Introduction
Transitioning from a web development background to data engineering, I've encountered a significant cultural shift. One of the most striking differences is the apparent lack of established software engineering practices within many data engineering t...
Structured Streaming
Mysterious Spark Checkpoints Behaviour
It all started from a change in the checkpoint path of our Spark applications. We use Spark Structured streaming and AWS S3 buckets to maintain checkpoints. Let’s say we were using s3://bucket/spark/t
spark
Run Spark Applications on AWS Fargate
Overview In this post, we are going to review how to run a Spark application, as a single node Fargate task. If you are familiar with Spark and its potential workload, you might wonder: “why would any
spark
How to Handle Null in Spark
Overview In this article, we will talk about the second-ugliest exception in the history of programming and attempt to handle it in our Spark apps. If you’ve ever worked with Spark in its native language, you’ve probably faced this bizarre, hard to d...
AWS
AWS VPC With Public and Private Subnets
Overview Virtual Private Clouds (or VPC) as you all probably know, is one of those services, which would be a lifesaver when you know how to use them. The isolation and service integrations provided by VPCs, suppose to reduce the common cloud managem...