Search for a command to run...
Articles tagged with #spark
Introduction In the data engineering world, bigger usually means better. More power, more scalability, and more buzzwords! PySpark, for instance, is typically our trusty steed when tackling massive da
Sparkle is revolutionizing the way data engineers build, deploy, and maintain data products. Built on top of Apache Spark, Sparkle is designed by DataChef to streamline workflows and create a seamless experience from development to deployment. Our go...
Custom Log4J for Spark on Glue to decrease CloudWatch cost!
It all started from a change in the checkpoint path of our Spark applications. We use Spark Structured streaming and AWS S3 buckets to maintain checkpoints. Let’s say we were using s3://bucket/spark/t
Overview In this post, we are going to review how to run a Spark application, as a single node Fargate task. If you are familiar with Spark and its potential workload, you might wonder: “why would any
Overview In this article, we will talk about the second-ugliest exception in the history of programming and attempt to handle it in our Spark apps. If you’ve ever worked with Spark in its native language, you’ve probably faced this bizarre, hard to d...
