great-expectations
Custom dashboard for Great Expectations
Enable advanced dashboarding on Great Expectation results using metrics
Latest
thinnest-viable-platform
Why You Should Never Mandate The Adoption Of Your Data Platform
Welcome to the first part of our series on data platforms! In this series, we are going to dive into data platforms. In part 1 of the series, we start out by going over the main principles for platform success. In our unique role as a data consultanc...
PySpark
FP-Growth Algorithm and How to Avoid Its Dark Side!
Context A few weeks ago we were contacted by FrieslandCampina to help them with a problem they faced on their recommendation engine. Being one of the biggest dairy companies in the world they sell hundreds of dairy products to millions of customers a...
software development
Speed up dbt workflow with Task
Do you remember the last time you opened your laptop and thought: "I can't wait to spend half my day writing configs in YAML."? Yeah, neither do I. But if you use dbt heavily, I'm afraid you and YAML are in for the long run. Even with the help of dbt...
Software Engineering
Software Engineering for Data Engineers: Introduction
Transitioning from a web development background to data engineering, I've encountered a significant cultural shift. One of the most striking differences is the apparent lack of established software engineering practices within many data engineering t...
Structured Streaming
Mysterious Spark Checkpoints Behaviour
It all started from a change in the checkpoint path of our Spark applications. We use Spark Structured streaming and AWS S3 buckets to maintain checkpoints. Let’s say we were using s3://bucket/spark/t
Machine Learning
No Code, All Insight: SageMaker Canvas Connects Data Analysts to Machine Learning
What is No-Code ML? As a data scientist, I was always skeptical of no-code solutions since they usually provide so little flexibility that makes them practically useless or tries to provide too much flexibility that makes their UI/UX impossible to na...
SupplyChainAutomation
The Future of Logistics: AI's Impact on Supply Chain Management
Introduction Artificial Intelligence (AI) has come a long way since its early days as an academic curiosity in the mid-20th century. From intelligent assistants to autonomous cars, AI is reshaping the world as we know it. But besides its fancy applic...
generative ai
Amazon Bedrock: Simplifying Generative AI with AWS for 2023
Introduction Generative AI has seen a surge in interest over the past few years, with a plethora of applications ranging from content creation to complex reasoning tasks. Recognizing the growing demand, AWS has introduced its managed service, Amazon ...
authentication
The Ultimate Guide to User Authentication: Building vs. Third-Party Services
Introduction In the process of developing software applications, each project necessitates the creation of a mechanism for granting the correct users access (i.e., authentication) and defining their permissions (i.e., authorization). This combination...
container orchestration
ECS vs. EKS: Choosing the Right AWS Container Orchestration Solution
Introduction In the world of containerized applications, Amazon Web Services (AWS) offers two prominent services for orchestrating containers: Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). Both have their unique ...
generative ai
The Rise of Controlism AI Art: Exploring Stable Diffusion and ControlNet
Introduction Last week, an AI painting created by the Stable Diffusion model gained so much attention on social media because of its eye-catching nature. MrUgleh, the creator of this image, shared his workflow in this Reddit post so other people can ...