Drift Monitoring

Introduction to MLOps and Model Drift

The advancement of AI is rapidly intensifying across various sectors. Yet, challenges in implementing machine learning have hindered AI systems' peak performance. MLOps, when it comes to creating machine learning models, face challenges reminiscent of those encountered by software before the rise of DevOps Monitoring. A pivotal aspect of MLOps Monitoring is identifying model drift.

Understanding Model Drift

At its core, model drift describes the deviation of a factor from its baseline. Data drift, which can lead to model drift, represents the difference between the current production data and the original dataset, often the training set. This gap indicates the model's alignment with the task it's designed for. When the production data diverges from the foundational data due to evolving real-world conditions, this indicates a possible concept drift or data integrity issues.

Types of ML Drift

Machine learning drift can be categorized based on the data distribution being analyzed:

• Prediction drift – Changes in the model's predictions. For instance, seeing more credit-worthy applicants after introducing your product to a wealthier region.
• Label drift – Shifts in the model's outcomes.
• Feature drift – Variations in the distribution of input data. A scenario might be all applicants' incomes rising by 2%, with no change in economic fundamentals.
• Concept drift – Alterations in the crucial connections between model inputs and outputs. An example could be changes in economic conditions influencing lending risks and hence, the criteria for loan eligibility.
• It's worth highlighting that concept drift zeroes in on the difference between a true and a learned decision boundary. To maintain the desired accuracy, re-learning the data becomes essential. Changes in prediction and feature distributions can be signs of significant real-world shifts if real-time ground truth isn't available.

Causes and Detection of Model Drift

Model drift can stem from a myriad of reasons, such as genuine changes in data distribution from external influences, shifts in input data distribution due to evolving consumer preferences or new market introductions, or challenges tied to data integrity.

Monitoring drift in predictions and features can be intricate without the right tools. Those in charge of upkeeping production models should routinely contrast live traffic with a baseline using one of the mentioned types.

Practical Steps for Drift Identification

To pinpoint the source of drift:

• Performance Analysis: Examine the performance of the affected traffic segment to glean deeper insights.
• Distribution Comparison: Contrast data distributions to discern the genuine shift and decide if model re-training is warranted.
• Real-time Drift Detection: Rapidly identify prediction drift in real-time outputs against a training or baseline set.
• Feature Examination: Investigate specific periods for feature drift. Utilize explainability to assess the significance of the drifting features, centering attention solely on impactful ones to eliminate false drift alarms.