spark
Run Spark Applications on AWS Fargate
Overview In this post, we are going to review how to run a Spark application, as a single node Fargate task. If you are familiar with Spark and its potential workload, you might wonder: “why would any
Latest
emr
How to run Great Expectations on EMR
What is Great Expectations Great Expectations is a great tool to validate the quality of your data and can be configured against a number of data sources, including BigQuery, MySQL, Snowflake and Athena. It helps you to test data instead of code. Let...
Style Transfer
Neural Language Style Transfer With StyleTransfer
StyleTransfer is a natural language processing solution developed in-kitchen by the machine learning team at DataChef. Stylometry The Da Vinci Code, Angels & Demons, The Lost Symbol, Inferno. They all start with Robert Langdon finding himself in a m...
spark
How to Handle Null in Spark
Overview In this article, we will talk about the second-ugliest exception in the history of programming and attempt to handle it in our Spark apps. If you’ve ever worked with Spark in its native language, you’ve probably faced this bizarre, hard to d...
AWS
3 ways to send custom metrics to AWS Cloudwatch
Amazon CloudWatch is a monitoring and observability service. CloudWatch monitors your resources and the applications you run on AWS in real-time. You can use CloudWatch to collect and track metrics, which are variables you can measure for your resour...
kafka
Deserialzing Confluent Avro Records in Kafka with Spark
Introduction Kafka producers and consumers are already decoupled in the sense that they do not communicate with one another directly; instead, information transfer happens via Kafka topics. But they are coupled in the sense that consumers need to kno...
AWS
AWS VPC With Public and Private Subnets
Overview Virtual Private Clouds (or VPC) as you all probably know, is one of those services, which would be a lifesaver when you know how to use them. The isolation and service integrations provided by VPCs, suppose to reduce the common cloud managem...