Serving ML Models in Production

Cover this or you fail the question

• Online inference means a request-response service under a latency budget, and the budget determines nearly every subsequent decision including model size and hardware.
• Batch scoring runs on a schedule over a large dataset and has no latency constraint, so it is a data pipeline rather than a service and is far simpler to operate.
• Streaming inference sits between them, scoring events as they arrive with a bounded delay.
• Ask which one is actually needed, since a great deal of unnecessary complexity comes from building online serving for a use case that a nightly batch job would satisfy. Precomputing predictions and serving them from a key-value store is often the right answer and is dramatically cheaper.

Cover this or you fail the question

• Models are versioned, stored in a registry with the data and code that produced them, and promoted through environments like any other build artifact. Reproducibility means being able to say which version served a given prediction.
• Preprocessing at inference must be exactly what was applied during training. Training-serving skew — a different tokenizer version, a different normalisation constant, a feature computed slightly differently — is the most common production failure in machine learning systems and it produces degraded quality rather than an error, so nothing alerts.
• The standard mitigation is to share the transformation code between training and serving rather than reimplementing it, and to use a feature store where features are genuinely shared across models.
• Rollback means serving the previous model version, so both should remain deployable and the switch should be as fast as any other configuration change.

Common trap: Treating the model file as configuration rather than as a build artifact. It needs the same versioning, promotion and rollback discipline as code, and more testing rather than less.

What separates a solid answer

• Batching is the main lever on accelerator utilisation: processing requests one at a time leaves the hardware idle, so requests are grouped within a short window, trading a little latency for a large throughput gain.
• Model optimisation — quantisation, distillation, compilation for the target hardware — reduces cost and latency at some accuracy cost, and the trade should be measured rather than assumed.
• Choose hardware from the workload: accelerators are worth their cost only at sufficient utilisation, and a low-traffic model on a dedicated accelerator is expensive idle capacity. Consolidating several models onto shared hardware is frequently the right answer.
• Cache aggressively where inputs repeat, which for many production workloads is far more often than expected. An exact-match cache on the request is trivial and removes real cost.
• Autoscaling is harder than for ordinary services because loading a large model takes time, so scale-up is slow. That argues for keeping warm capacity, or for a queue that absorbs bursts while capacity is added.

What separates a solid answer

• The service needs the ordinary signals: latency percentiles, error rate, saturation, and hardware utilisation. Necessary and not sufficient.
• The model needs its own: the distribution of incoming features compared with training data, so a shift in inputs is detected; the distribution of predictions, since a model suddenly predicting one class far more often is a strong signal; and confidence distributions where the model produces them.
• Accuracy itself usually arrives late, because ground truth does — whether a recommendation was clicked, whether a flagged transaction was actually fraudulent. That delay has to be designed for, with a pipeline joining predictions to outcomes when they arrive.
• This is the property that makes model serving distinctive and is worth stating plainly: a model can serve every request successfully, within latency, at a hundred percent availability, while being progressively more wrong, and no conventional monitoring notices.

The senior signal

• Shadow deployment runs the new model on real traffic without serving its results, so its behaviour and cost can be compared on production inputs with no user risk. It is the closest thing to a safe test available and is underused.
• A canary limits exposure while measuring, and the promotion criteria must include a quality metric rather than only technical ones — a model with excellent latency and worse predictions is a regression that technical thresholds will approve.
• Where the outcome is measurable, an A/B test against the incumbent is the real evaluation, and it needs enough traffic and enough time for the metric to be meaningful.
• Retraining should be a scheduled, automated pipeline with the same validation gates, because a model degrades as the world moves away from its training data, and manual retraining reliably slips.
• Two governance points matter in regulated contexts and are increasingly asked about: being able to explain a specific prediction, and being able to say which model version and which data produced it. Both are much cheaper designed in than retrofitted.