MLOps drift loop

A model that notices the world changed, retrains itself, and only ships the new version if it wins.

TL;DR

Problem
Code doesn't rot; models do. When traffic drifts, a classifier quietly gets worse until someone notices.
What I built
A closed loop for a network-intrusion classifier: label-free drift detection, automatic retraining, a champion/challenger gate, and a hot reload of the serving API — with no human in the path.
Result
After a synthetic shift that included a new attack type, realized F1 recovered automatically from 0.27 to 0.78, measured on unseen traffic.

Batch 20F1 0.78v2 · Promoted challenger

Tap a point or use the arrow keys to read a batch.

The loop

Realized F1 on each unseen batch, as the real serving process and controller replay the synthetic shift.Synthetic dataSource

View data

Realized F1 per unseen batch · commit 0a0b95f, exported 20 Oct 2026
Batch	Median t	F1	Model	Event
1	12,199.5	0.265	v1 · Stale champion
2	12,599.5	0.226	v1 · Stale champion
3	12,999.5	0.252	v1 · Stale champion
4	13,399.5	0.245	v1 · Stale champion
5	13,799.5	0.352	v1 · Stale champion	Drift detected; challenger promoted, at the controller tick after this batch
6	14,199.5	0.803	v2 · Promoted challenger
7	14,599.5	0.780	v2 · Promoted challenger
8	14,999.5	0.790	v2 · Promoted challenger
9	15,399.5	0.763	v2 · Promoted challenger
10	15,799.5	0.796	v2 · Promoted challenger
11	16,199.5	0.774	v2 · Promoted challenger
12	16,599.5	0.781	v2 · Promoted challenger
13	16,999.5	0.771	v2 · Promoted challenger
14	17,399.5	0.788	v2 · Promoted challenger
15	17,799.5	0.788	v2 · Promoted challenger
16	18,199.5	0.796	v2 · Promoted challenger
17	18,599.5	0.750	v2 · Promoted challenger
18	18,999.5	0.761	v2 · Promoted challenger
19	19,399.5	0.772	v2 · Promoted challenger
20	19,799.5	0.780	v2 · Promoted challenger

A FastAPI service serves predictions and logs every request.
Evidently checks the features for drift; it doesn't wait for labels, which arrive late.
On drift, the system retrains, and MLflow registers a challenger.
The challenger is promoted only if it beats the champion by at least 0.01 F1 and clears a 0.50 floor.
Serving hot-reloads the new model, and Prometheus and Grafana track it all.

Data: 20,000 synthetic rows over time.
The shift: the drift period moves 40% of the features and adds an attack type the old model has never seen.
The replay: the real serving process and controller replay it in 20 batches of 400 rows.
The result: realized F1 sits around 0.27 while the stale champion serves drifted traffic. Drift is caught and the challenger promoted on the first allowed check, after which F1 holds around 0.78.

Easy detection. The synthetic shift trips every feature, so detection was never borderline.
Optimistic promotion check. The promotion test overlaps the retraining data, so it's optimistic. That's why the number I quote comes from the per-batch series on unseen batches.
Label lag. In production, the F1 line would lag by the label delay.
The model isn't the point. A plain logistic regression actually beat the random forest at baseline (F1 0.688 vs 0.576). The deliverable is the loop, not the model.

Deploy targets: a Docker Compose stack, Kubernetes manifests for minikube, and an Oracle Kubernetes Engine overlay.
GitHub Actions:
- CI with a model-quality gate;
- multi-architecture images pushed to GHCR;
- a scheduled daily retrain.

Code doesn't rot; models do. The model took an afternoon; the honest evaluation took weeks.

How this was built

I wrote the execution spec and owned the evaluation design and its honesty checks; Claude Code wrote much of the implementation.