Software Engineering
Transitioning from a web development background to data engineering, I've encountered a significant cultural shift. One of the most striking differences is the apparent lack of established software engineering practices within many data engineering t...
Structured Streaming
It all started from a change in the checkpoint path of our Spark applications. We use Spark Structured streaming and AWS S3 buckets to maintain checkpoints. Let’s say we were using s3://bucket/spark/t
Machine Learning
What is No-Code ML? As a data scientist, I was always skeptical of no-code solutions since they usually provide so little flexibility that makes them practically useless or tries to provide too much flexibility that makes their UI/UX impossible to na...
SupplyChainAutomation
Introduction Artificial Intelligence (AI) has come a long way since its early days as an academic curiosity in the mid-20th century. From intelligent assistants to autonomous cars, AI is reshaping the world as we know it. But besides its fancy applic...
generative ai
Introduction Generative AI has seen a surge in interest over the past few years, with a plethora of applications ranging from content creation to complex reasoning tasks. Recognizing the growing demand, AWS has introduced its managed service, Amazon ...
authentication
Introduction In the process of developing software applications, each project necessitates the creation of a mechanism for granting the correct users access (i.e., authentication) and defining their permissions (i.e., authorization). This combination...
container orchestration
Introduction In the world of containerized applications, Amazon Web Services (AWS) offers two prominent services for orchestrating containers: Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). Both have their unique ...
generative ai
Introduction Last week, an AI painting created by the Stable Diffusion model gained so much attention on social media because of its eye-catching nature. MrUgleh, the creator of this image, shared his workflow in this Reddit post so other people can ...
Machine Learning
Introduction If you’ve ever wondered how computers, which fundamentally understand only numbers, can grasp the intricacies of human language filled with words carrying complex meanings, you’re not alone. In this article, we dive deep into the world o...
What does Infrastructure as Code (IAC) mean? Infrastructure as Code (IAC) refers to the practice of managing and provisioning technology infrastructure using code and automation techniques. Instead of manually configuring servers, networks, and other...
Koalas
Introduction Pandas is one of the most widely used data manipulation libraries in Python, known for its ease of use and powerful functionality. However, as the data size grows, Pandas can become slow and memory-intensive. In this blog post, we will c...
Docker
Introduction The importance of efficient and scalable application deployments in the ever-changing technology landscape cannot be overstated. Docker and Kubernetes have emerged as leading solutions to meet these demands, providing businesses with str...