Legacy Migration Starts with Understanding, not Inventory
An alternative to scale and direct legacy migration activities
π§βπ³ Cooking data as a chef de partie at DataChef.co by day! πͺ Avid coder inventing magic by night!
An alternative to scale and direct legacy migration activities
System Integration Test
Or how to do Data Engineering like Metallica!
great-expectations
Enable advanced dashboarding on Great Expectation results using metrics
Software Engineering
Transitioning from a web development background to data engineering, I've encountered a significant cultural shift. One of the most striking differences is the apparent lack of established software engineering practices within many data engineering t...
Structured Streaming
It all started from a change in the checkpoint path of our Spark applications. We use Spark Structured streaming and AWS S3 buckets to maintain checkpoints. Letβs say we were using s3://bucket/spark/t
spark
Overview In this post, we are going to review how to run a Spark application, as a single node Fargate task. If you are familiar with Spark and its potential workload, you might wonder: βwhy would any
spark
Overview In this article, we will talk about the second-ugliest exception in the history of programming and attempt to handle it in our Spark apps. If youβve ever worked with Spark in its native language, youβve probably faced this bizarre, hard to d...
AWS
Overview Virtual Private Clouds (or VPC) as you all probably know, is one of those services, which would be a lifesaver when you know how to use them. The isolation and service integrations provided by VPCs, suppose to reduce the common cloud managem...