Stopping the nightmares of building a recommender system

Stopping the nightmares of building a recommender system


8 min read

As a data science developer, I have seen several unsuccessful recommender system projects through my career. To be honest, developing a recommender system from scratch is not a simple project. Fortunately, all these failed projects show similar situations and areas of improvement. In this blog post, I am going to share my experiences of these common challenges in building a recommender system and tell you the story on how we decided to fix them at DataChef.

A disaster in building recommender systems

Whether you are in a company that offers several services to its customers or you are in an e-commerce business that sells a wide variety of products, you probably know that you need a recommender system to personalize the experience of your users and boost up your business revenue. While recommender systems are promising machine learning (ML) solutions for maximizing the chances of grabbing your client’s attention, in reality, only a fraction of these systems are implemented in an appropriate way to benefit the business.

If you are to manage a team of engineers to build a recommender system in your company, you will be overwhelmed by the numerous challenges and complexities that will arise during the implementation of the system. These challenges include time management of the project, communication issues between team members and higher-level managers, orchestrating the results of each person’s work, technical complexities, etc. Meanwhile, time is passing by and you start to reach the project’s deadline and everything has become a headache. At this point, lots of time and money have been spent and decision-makers are starting to lose interest in the outcome of the project. Consequently, the chances of dropping the project are increasing.

Given the number of failed recommender system projects, it is a hard decision for a company’s decision-makers to start a machine learning journey. Actually, I can’t disagree with them. Developing a recommender system from scratch may not be a reasonable thing to do. I wouldn’t invest a significant amount of money on something that needs to be built from the very beginning simply because it needs to be tailored to specific needs. For example, who would invest in a car factory that wants to extract iron first to build a car?

It would be a miracle for the decision-makers to just push a button, skipping all of these costs and risks, and see whether a baseline recommender system is suitable for their business or not. It is only then that they can decide if they want to invest in developing a specific recommender system.

The dark sides of building recommender systems

The dark side of building recommender systems

There are several reasons why the development of a recommender system has a tendency to fail. Here are a few of them I have encountered.

1. Not just about data scientists

In companies, people usually think that, when it comes to recommender systems or ML projects in general, data scientists are all they need. Data scientists spend most of their time on developing and testing recommender models [1], although this is only one component of the entire recommender system pipelines. There are other elements to be considered when building a recommender system. We need data engineers to develop data transformation pipelines and data warehouses, DevOps engineers to provide governance, networking, and infrastructure orchestration, and software engineers to build APIs, log systems, and databases. Developing all of these components is a tedious task for a data scientist alone.

In one of my experiences in developing a recommender system, the data science team spent several weeks developing various models and evaluating them. They were so invested in that process that they forgot to consider the company’s biggest expectation: the speed at which the company could develop a solution and the value it adds to the business.

Since a company needs a working product rather than several laborious and possible solutions, the decision-makers agreed that the team had failed to accomplish the requirements, and the project was canceled. Incomplete comprehension of the project’s scope was one of the main reasons for its failure. In other words, there were no data engineers and machine learning engineers assigned to the project from the very beginning.

2. Hard to find the right people

Another difficulty of developing a recommender system is finding the appropriate people for the required roles. The reason is mainly the lack of data-related experts in the industry. Let us consider the decision-makers’ point of view in this situation. Supposing they are aware of the scope for developing a recommender system, the minimum number of roles required are data engineers, machine learning engineers, and data scientists. It is necessary to identify the right candidates, conduct multiple interviews with them, assign them some related toy projects, evaluate their level of expertise, and finally hire them.

If you were involved in hiring people for technical roles, you probably have been in this situation as well. In my experience, out of dozens of candidates, only one of them is suitable for the job. Even after we hire the right staff, it takes weeks for them to be appropriately onboarded and get familiar with the company’s context.

At this point, if everything goes well, we, as decision-makers, are ready to spend months waiting for the team to develop a proof-of-concept recommender system. This entire process is full of uncertainty and risk, and costs a lot of time and money.

3. Time consuming tasks

We now have a team of data scientists, ML and data engineers that have a good understanding of the project’s scope, prepare its setup, and are ready to develop a recommender system. The goal of a recommender system is pretty straightforward; a system that suggests suitable products to customers based on their behaviors. Nonetheless, developing a production-ready recommender system takes a lot of time since it is a complex system with multiple components depending on each other.

According to Deloitte research [2], developing a successful recommender system project takes an average of twelve months. That includes developing all the phases of a machine learning pipeline. Members of the team are each responsible for a specific component of the system. Here are some examples of components that need to be developed:

  • Data engineers develop data ingestion pipelines that reliably gather data from different sources and store them into data stores.
  • Data scientists develop pipelines that perform preprocessing and feature engineering on the data and store them in the feature stores, build and train recommender system models, evaluate models and select the best performing ones.
  • ML engineers are responsible for orchestrating and automating the entire system and developing feedback loops.

Obviously, developing all of these components from scratch in a timely manner is a hard task to do. Currently, there are no frameworks to help teams develop their own recommender system and help them with all the recommender system components. As a result, it takes months to develop a working baseline recommender system.

There is hope for recommender systems

There is hope for recommender systems!

With all that being said, should we stop creating custom-made recommender systems for our clients? At first glance, it may seem that building a recommender system is not worth the time and money. The good news is that most of the recommender system components perform repetitive tasks that are common in every business domain in which the system will be used. Only a small fraction of the system components needs to be modified to have a custom recommender system for each business.

In the software world, people are constantly creating general solutions as building blocks on top of which specific applications are built. These building blocks may be viewed as templates that encapsulate repetitive procedures that others can modify in order to fit their specific use-cases. We can adopt the same approach in the case of recommender systems.

Most of the ML operations (MLOps) of a recommender system pipeline can be implemented in the form of templates. Such templates reduce the risk of failure by providing building structures that are highly customizable and independent of specific recommender system procedures.

If we have a functional MLOps environment for the recommender system as a template, people can focus on optimizing the system according to their domain of expertise, rather than struggling to develop a production-ready baseline recommender system.

To resolve these issues, at DataChef we decided to leverage our experiences and come up with a recommender system template that serves as a baseline system and can be deployed at day one. So on the very first day, the baseline recommender system is working and gathering feedback data by recommending products and services to end-users. Besides that, everyone can play their own role to improve the system further.

For instance,

  • Business strategists and managers can focus on the real product-related challenges of the business and marketing strategies, rather than worrying about orchestrating the recommender system project and questioning its success. They can design better strategies and KPIs to optimize the business performance by utilizing all the business data and taking advantage of data science’s full potential.
  • Data scientists can focus on improving the recommender system and tailoring it to fit specific business needs. They can perform many feature engineering methods with various data sources, try state-of-the-art recommender system algorithms, A/B test various models and employ machine learning in its excellent way.
  • Data Engineers can focus on developing challenging data pipelines and best practices that serve the required data in a live and reliable manner.
  • Machine learning engineers can focus on improving the entire system by automating as much as possible and designing feedback loop strategies according to MLOps principles.

With these considerations, we developed DataChef RecSys that can be used as a baseline recommender system. In addition to that, we created a simple dashboard for the decision-makers to have a clear view of the business KPIs. You can deploy RecSys from AWS Marketplace for free. We’d love to hear how you used RecSys, and what we can do to make it even more useful for you and future generations of developers.

DataChef RecSys on AWS Marketplace

DataChef RecSys Documentation

We at DataChef are always happy to help companies deploy, customize, and further expand their recommender system to address their specific needs on their behalf. So keep in touch with us through, LinkedIn, or Twitter.

Stay in touch with us.


[1] 76% Of Enterprises Prioritize AI & Machine Learning In 2021 IT Budgets

[2] Business impacts of machine learning