Model drift monitoring: watching an AI system after go-live

Reference article · 5 min read

An AI system that passes acceptance testing has proven one thing: it worked on a particular set of inputs, at a particular moment, with the staff trained on it that month. None of those conditions stay fixed. The model is frozen at the point it was trained; everything around it keeps moving.

Monitoring is usually the last part of an AI project planned and the first cut when the schedule tightens. It also determines whether the system is still doing what you bought it for a year later. This article covers what degrades after go-live, what a small team can sustain, and the conditions that should force retraining or a pause.

What degrades after launch

• Input data shifts. Forms get redesigned, a field becomes required, a partner agency sends records in a new format, a scanner replacement changes text extraction. The model sees inputs that no longer resemble its training data.
• Staff behavior changes. Early on, reviewers check everything. As trust builds, they approve faster and look less closely. Overrides fall, which looks like improving accuracy but is thinning oversight.
• The vendor updates the underlying model. Hosted models get retrained, upgraded, or repointed to a new base model. Behavior can change without anything changing on your side, and without a release note you would read as material.
• Seasonal patterns the model never saw. Tax deadlines, enrollment windows, storm season, fiscal year end. A model trained on a few months of history has not seen your annual cycle, and the first pass through it will surprise you.
• The underlying rules change. A program adds a category, eligibility criteria are amended, a form version is retired. What the model learned no longer maps to the decisions staff make now.

What to measure weekly

Weekly monitoring should fit on one page and take under an hour to review. Five measures cover most of it.

• Override and correction rate. How often a human changed the output, by decision type. Track direction as well as count: corrections that loosen a decision and those that tighten one fail differently.
• Confidence distribution. Not the average, which hides everything, but the shape. Watch the share of cases in your low, middle, and high bands. A band that swells or empties is a signal even when accuracy looks stable.
• Queue volumes and aging. Cases routed to review, cases held, cases aging past your service target. A queue that grows quietly is often the first symptom of a model that has begun abstaining or misrouting.
• Volume and mix of inputs. Counts by channel, program, and language. A mix shift explains many apparent accuracy changes.
• Operational errors. Timeouts, failed calls, empty responses, records that fell out of the pipeline. Not model quality, but the public experiences them as such.

What to measure quarterly

Quarterly work needs real hours, so put it on a calendar with a named owner.

• Blind spot-check samples. Pull a stratified sample including low confidence cases and cases handled with no human touch, have a reviewer decide independently, then compare. This is your only read on cases nobody flagged.
• Late-arriving ground truth. Appeals, audits, reversals, field verification. Outcomes arriving months after a decision are the closest thing to truth you will get.
• Subgroup breakdowns. Recompute the weekly measures by geography, language, and program. Aggregate stability can hide a subgroup drifting since launch.
• Training baseline comparison. Ask the vendor to describe the data the model was trained on, then compare your current inputs against it.

Trigger conditions worth writing down

Set thresholds against your own baseline from the pilot and first stable months, not figures from a vendor deck, and write them into the operating procedure before anything goes wrong. Reasonable triggers: an override rate outside its normal range for several consecutive cycles, a shift in the confidence distribution, any vendor version change, a change to an upstream form or policy, a subgroup diverging from the overall pattern, or a serious incident.

Pair each trigger with a graded response rather than one switch: raise the sampling rate, route more cases to review, narrow scope to case types still performing, require retraining, or suspend automated handling and fall back to manual work. A fallback only works if rehearsed, so run it once while nothing is broken.

What to ask vendors and put in the contract

• Version identity. Every prediction should carry a model and version identifier visible in your own logs.
• Advance notice of changes. Written notice before any model, version, or prompt change that could alter behavior, and the ability to test before cutover.
• Your data, exportable. Inputs, outputs, confidence values, and human dispositions in a format you can analyze without the vendor's dashboard, retained per your records schedule.
• Re-baselining and retraining. An obligation to re-run agreed evaluations after a model change and share results, plus terms naming who retrains, on whose data, at whose cost, and how often.
• Independent evaluation. Your right to run a holdout set or hire a third party evaluator without breaching the agreement.

Confirm separately how the system fits your state's AI inventory and disclosure requirements. These vary by state and are amended often, so check current text with counsel or your oversight body. Voluntary frameworks such as the NIST AI Risk Management Framework help structure monitoring, but they are guidance, not a compliance checklist.

How to start

Begin before go-live. Log every prediction, confidence value, and human action with a timestamp and version identifier. Capture your baseline during the pilot. Build the weekly page from data you already hold and name who reads it. Put the quarterly review on the calendar with its sample size and reviewer named. Write the pause procedure, then test it.

HQT works with state and local agencies on data and software delivery, including the logging and monitoring practices that keep a deployed model observable.

Frequently asked questions