4 minute read
You’ve designed and built a machine-learning model that predicts customer demand for one of your most popular products, at different prices. The model is extremely accurate––leveraging years of historical data on sales, customers, seasonality, and other variables. Therefore, the decision-makers in your organization are excited to start implementing it to enhance their targeting and improve their ROI. However, because they operate in fluid environments, machine-learning (ML) models require continuous monitoring to maintain their predictive power. This article will explain why models become less accurate over time, how you can track their performance to determine when they need to be updated, and strategies for addressing model degradation.
Organizations use machine learning for artificial intelligence (AI) and to discern patterns within large amounts of data. Given a sufficient data set representing the actionable population (AKA “training data”), a computer learns to recognize such patterns and can then make predictions about previously-unseen data. Despite the name, machine learning doesn’t constantly learn. Learning occurs in the first process of building a model: the training stage. Once deployed, most models are “static” and may start to worsen over time due to a variety of reasons. All reasons, however, are rooted in the fact that the algorithm receives no new information or feedback after deployment.
The phenomenon of a model losing its predictive power is known as concept drift. Concept drift happens due to unforeseen changes in the statistical properties of the problem that the model solves. The drift––and the model’s inability to self-correct (since it receives no new data)––causes model degradation. Here are some of the most common reasons this drift occurs:
Models have different shelf lives depending on the application and the level of accuracy needed when making predictions. One of the easiest ways to determine the level of degradation is to record predictions and compare them against the actual, measured results. This will allow you to spot any sudden decreases in performance and evaluate the effectiveness of the model. This is particularly effective if there is a process set up to continuously monitor accuracy after each implementation has “fully-baked” results. Keep in mind that the process used to build the model is reflective of the time period in which it was built. Understanding why model degradation occurs and measuring the performance of your model against real-world results allows everyone to address loss of model power and make more informed decisions.
As your model exhibits signs of concept drift or degradation, you can take steps to repair it. Which strategy you choose depends on the type of data included, what the model attempts to predict, the variables involved, and what’s at stake for your business if the model starts to lose power. Below are some solutions, but contact us if you’d like help making a strategy based on your data. We give great insights.
Your model’s predictive power depends on the quality of the data it’s built upon, including how much of that data is still relevant today. (“Garbage in, garbage out,” as they say.) As your data changes, your model may need tweaking.
If your model’s performance is still suffering after being refreshed with recent data, consider the second solution: Build a new model entirely. Rebuilding the model is more work, but it will allow for a reassessment of variables. As the variables may no longer be relevant, this option may end up being even more beneficial than you imagined. This assessment may also reveal where to incorporate new sources of data, which may make your modeling efforts more effective.
It’s necessary to maintain machine-learning models in order to sustain their predictive power. Although this process can be automated, the kinds of complex problems that cause your model to degrade require the competence of a trained data scientist. How you decide to keep your models in good condition will depend on a variety of factors, each unique in its application, rate of depreciation, and required level of accuracy. As part of the modeling process, it’s imperative that you consider how you will monitor the performance of your models to determine when they’re ripe for re-evaluation.
Need support creating your strategy or assistance in gleaning insights from data? Contact us. We’d love to help.
Ready to send better messages?
Sign up for our news, resources and updates. The inbox is our favorite place after all. We’ll make sure it’s worth it. (You can unsubscribe at any time, but you probably already knew that.)