Transitioning from a web development background to data engineering, I've encountered a significant cultural shift. One of the most striking differences is the apparent lack of established software engineering practices within many data engineering teams at various companies. This discrepancy, while noticeable, is not without reason. A majority of online resources offering guidance on becoming a data engineer tend to emphasize the following:
SQL
Spark, with a particular focus on Python
These are undoubtedly excellent starting points. However, it's important to remember that adhering to fundamental software development practices is essential regardless of the software type. These practices, which have evolved over the years, play a pivotal role in enhancing various aspects of software quality, complementing the core business requirements of the products.
Data products are no exception to this rule. The landscape for creating data products today is diverse, with some solutions overlapping and others introducing innovative approaches to longstanding challenges in software development.
My key criterion in selecting or designing a solution is its ability to maintain a "boring" software development experience. Throughout my years of experience, I've understood that a certain level of predictability in being a data engineer is advantageous. Maintaining a routine, especially in the development of data products, is crucial to ensure that the final products are:
Long-term reliable
Easily adaptable by new engineers
Flexible to accommodate new use cases
Without marketing teams to glamorize our work, our data products must stand on merit. They must prove their worth through reliability and effectiveness.
To achieve this, we must embrace the best software engineering practices, which guide us through various stages, including design, implementation, and maintenance.
In this series of articles, I aim to delve into these topics. My goal is to assist the community in enhancing the quality of the products we develop by leveraging the wisdom and experience of the greats in our field.
P.S. Living in the age of A.I., where much of our work could be automated, it's wise to shift our focus towards quality. This will help us stay relevant in the future of this industry and prepare us for what it has in store.