Model promotion fails when “latest pickle on a shared bucket” is the release process. Production needs a registry, immutable versions, evaluation evidence, and a rollback path.
Define environments clearly: experiment, staging with offline/online eval, and production with canary or shadow traffic. Attach metrics for accuracy, latency, drift, and cost to every promotion decision.
Wire serving through the same delivery platform as apps—GitOps or pipelines that update manifests, not manual kubectl edits after a notebook export.
MLOps is platform engineering for models. When SRE and ML share the same promotion language, incidents get shorter.
Align ML promotion with the same change windows and incident process as application releases.
Monitor training/serving skew and data freshness as reliability signals, not only offline accuracy.
Give platform and ML teams a shared vocabulary for environments: dev, staging, prod—no special snowflake clusters without owners.
Key takeaways
- Promotion needs evaluation gates, approval, and artifact lineage—not a shared notebook.
- Shadow and canary traffic beat big-bang model swaps for critical paths.
- Define rollback as a first-class runbook with owners and time bounds.
FAQ
What metadata must travel with a model?
Training data version, code commit, metrics, approval record, and the serving config that was validated—enough to reproduce or reverse the change.
How do we avoid notebook-driven production?
Export training to pipelines, store models in a registry, and allow only registry artifacts into serving environments.