Any machine learning project can be divided into several stages. These are:

• Machine learning problem definition: defining the problem, possible solutions, and resources.
• Dataset selection: deciding on the data needed, looking for sources, gathering.
• Data preparation: cleaning and normalizing the data, understanding its properties and distributions. 
• Model design: deciding on the appropriate model, features, algorithms, and parameters. 
• Model training: building, training, and evaluating the model.
• Operationalize: deployment of the model, keeping it up to date, monitoring its performance. 

Unfortunately, many projects fail at various stages, without even getting to production. There are multiple pitfalls to be aware of. However, with the right approach, it is possible to avoid or quickly resolve most machine learning challenges.

In this article, we will talk about the last stage – operationalization. This part of the project is probably the longest (in fact, it never ends) and requires constant attention. But it is crucial to identify and prevent any possible machine learning problems in this phase to ensure the successful development of the product. We will concentrate on the most common machine learning challenges one can encounter during the post-deployment phase.

Programming Languages Conflict

While most software products are usually written in one language, this is not the case for machine learning applications. Models are often designed in machine learning-oriented languages like Python or R. The production systems usually use faster languages like C++ or Java. These languages do not handle models well. 

The issue arises when joining machine learning models with production components. Incorrect merging may result in reduced performance of the original model. Speed and accuracy may greatly suffer and even make the model unusable. Luckily, there are several solutions available.

The most obvious solution would be to simply write everything in one language. This is, however, not always possible. Many languages do not support machine learning training. On the other hand, Python and R may not always be the best production options due to their speed limitations. 

This brings up another solution – containerization, which can solve incompatibility and portability problems by deploying a model into a container and running it as a microservice. The most popular container system is Kubernetes, which is designed to automate application deployment, scaling, and management. Unfortunately, containers still do not allow for automatic dependency checking and error testing. 

Other solutions include serializing the model, creating an API or DSL (Domain Specific Language). But containerization is by far the most widely used to avoid such machine learning challenges.

Long Response Time

The time needed to generate an inference rises with the increase of requests. This challenge is especially critical when there are multiple calls to the model at the same time. Again, there are several things you can do.

First of all, a smartly written API can significantly optimize the processes, simultaneously diminishing the response time. Multithreading and multiprocessing allow for scaling the API. Another option is batch processing, which can work with several inputs at the same time and, thanks to the nature of mathematical operations, this approach turns out to be faster and more efficient. 

There is also a possibility to increase hardware capabilities. Faster CPUs, GPUs, and more quick-access memory also allow for scaling the application. This method, however, has its limitations and is rather expensive. Of course, cloud services can make it cheaper and more manageable, but a nicely designed API is still preferable.

Quality Issues

Quality is crucial for machine learning. Data scientists often say: “Garbage in, garbage out.” This basically means that without good data, there will be no good model. 

Quality issues arise at different stages of an ML project, but the post-deployment phase is probably the most critical. It is harder to track machine learning problems here, and the cost of a mistake may be too high. A famous example is a sticker on a stop sign. It can fool the model into thinking that it is a different road sign. This may result in a car crash. 

An obvious solution would be to train the model on different examples, including bad ones like the sign with a sticker. There is even a name for this approach – adversarial machine learning. This technique transforms the initial data to make the machine learning model immune to any future transformations. As a result, you get higher accuracy and fewer mistakes. 

Another important thing is monitoring the performance of the model. Either a  human or another machine learning model will be responsible for ensuring the correctness of predictions. Any mistakes must be reported immediately and taken into account to avoid them in the future. The model can then be updated with the new data.

High Cloud Costs

Due to a variety of benefits, cloud services are being adopted by more and more companies. However, incorrect implementation of the application may lead to unexpected results. One of them is high costs. Fortunately, there is something you can do.

First of all, make sure to monitor the pricing plans and your cost strategies. If you encounter unacceptably high prices, you can transfer your services to another provider or simply decrease used resources. 

Resource monitoring is also essential. Any unused or not fully used resources may result in unexpected costs. For example, you do not need a high-performance GPU to run several mathematical calculations. 

Also, many cloud providers allow you to see the resource usage statistics. Special heatmaps are available for this purpose. You may decrease or increase the utilization of your resources in high- and low-demand periods. 

Finally, many companies now have their own data centers to carry out the operations. This option does not suit everyone, but it can provide benefits in certain cases.

Testing the ML Pipeline

The machine learning pipeline is the technical infrastructure used to manage and automate the ML processes. It includes several stages: data collection and preprocessing, feature extraction, model training and evaluation, model deployment, and monitoring. Basically, it is the entire workflow used to provide and support machine learning services. Needless to say, it is vital to ensure that all the components of the pipeline are working seamlessly together. 

Usually, most of the attention is paid to separate stages: data verification, model evaluation, etc. These are called unit tests. However, overlooking pipeline testing, as a whole (integration-testing), is fraught with consequences. The components may work fine separately but not work together at all. For example, data predictions may have the wrong format, and the user-end part of an application will not accept it, or data preprocessing may be inappropriate to a particular model and delete essential features. 

It is, therefore, vital to carry out integration-tests and verify that all the components are jointly compatible.

Toilet and door may work well separately (successful unit tests), but terribly together (failed integration-test) image source

Model Poisoning

Model poisoning is a problem related to quality issues. In this case, the difference is that bad data may seem fine, but they gradually decrease the model’s accuracy in reality. That’s why it’s called “poisoning.” 

These data samples may have a little noise that will create a bias in the model. The more such data is used, the worse the model will perform. Unfortunately, it is usually tough to detect such problems, which is why it is vital to control the quality beforehand and use only verified examples. 

An approach called robust monitoring is used to check the data and receive early warning signals. It allows you to take appropriate actions and prevent model poisoning. Thus, it is a good idea to research this approach and use it in the project.

Inconsistent Metric Results

It is confusing to see that metrics pipelines have been completed successfully, but the model’s performance is insufficient. This can usually happen when metrics calculations start before the model finishes its work. Such a mistake may lead to a time-consuming debugging process.

To avoid getting inconsistent results, appropriate dependencies must be present. It is, again, a question of optimization and correct system design. Data engineers should carefully plan the process of metric calculations and ensure the existence of all necessary dependencies.

Conclusion

In the article, we have discussed some of the most frequently faced issues in a machine learning project. Some of them are pretty easily solved, but others require a lot of time and effort. The most difficult machine learning challenges arise due to data quality problems. One must carefully monitor the data used and the methods for their processing. Data validation, pipeline testing, optimization – all of these are vital for machine learning application management and scaling.