Does serving always happen online?

No. Batch prediction is also a serving mode.

Does inference mean the same thing?

Inference is the model computation; serving includes making that computation usable within an application, although terminology sometimes overlaps.

Can a model be served locally?

Yes. Serving can occur on a device, in an internal process, or through a remote system.

AI Glossary / MLOps and deployment / Intermediate

Serving

Definition

Serving is the process of making a trained machine-learning model available to produce predictions for an application or workflow. It can provide responses to individual requests or run predictions over collections of records. Serving connects the model to usable inputs and outputs, with the operational behavior needed for the surrounding product to rely on it.

What it means

A trained model file is not yet a complete prediction service. Something must load it, supply correctly prepared inputs, run inference, and deliver the result in a form the application understands. That surrounding process is part of serving.

Online serving responds when a request arrives, while offline or batch serving produces predictions for later consumption. Neither mode is inherently more intelligent or accurate. The choice depends on when results are needed, how quickly inputs change, and what the application can afford to wait for.

How it works

The serving system loads a particular model version with its compatible preprocessing and output interpretation. It receives input records, checks their format, obtains any required features, and performs inference. It then returns or stores predictions with enough context to identify the version and input assumptions used.

Operational design covers response time, capacity, failure handling, and observability. A batch process might need to finish before a daily planning deadline. An interactive service might need to respond consistently within a short interval even when demand rises.

Serving also needs a controlled path for model changes. A new model can be evaluated on shadow traffic or introduced gradually when appropriate, with a way to restore a known version. Reliable delivery and predictive quality remain separate: a service can return an incorrect prediction promptly, or fail to deliver an otherwise accurate model's output.

Three practical examples

1. Product search: a ranking model scores candidate items during a user's search request so that the application can order the results.

2. Daily planning: a batch process generates next-day demand forecasts for a planning system to read each morning.

3. On-device assistance: a local model converts an input into a prediction without requiring every request to be sent to a remote server.

Think of it this way

Think of a restaurant using a prepared recipe to fulfill orders. The recipe resembles the trained model, while the kitchen process includes ingredients, timing, delivery, and handling unavailable items. The analogy has limits: model inference may be deterministic or stochastic, and operationally smooth delivery does not establish that the model's answer is appropriate.

Worked example

Consider an invented interactive prediction path with five sequential stages: input validation takes 5 milliseconds, feature retrieval 25, model inference 35, output formatting 10, and network transit 20. With no overlap or queueing, the end-to-end time is 5 + 25 + 35 + 10 + 20 = 95 milliseconds.

If the target is 120 milliseconds, the simplified path leaves 25 milliseconds of margin. Reporting only the 35-millisecond model computation would hide 60 milliseconds of work needed to deliver the response. If a feature request adds 40 milliseconds, total time rises to 135 and misses the target even though the model itself is unchanged.

Now consider a separate batch workflow with 240,000 records. At a sustained measured rate of 800 records per second, scoring alone would take 300 seconds. If preparation and writing add 60 seconds sequentially, total duration is 360 seconds, or six minutes.

These calculations illustrate what to include in a service budget. They do not predict a real deployment: concurrency, queueing, data size, cache behavior, and startup costs can change observed timing. Percentiles also cannot generally be obtained by adding each stage's independently reported percentile. End-to-end performance should be measured directly under representative conditions.

Illustrative case study

Fictional case: Elm Recommendations has a model that performs well in an offline evaluation. During a limited release, the prediction endpoint is slow because it fetches several features serially before inference.

The team traces the full request path, identifies the feature dependency, and evaluates an appropriate change to data access. It retains model and transformation versions in its records and checks that the faster path produces compatible inputs. A fallback behavior is defined for unavailable features, then tested separately. The scenario describes service diagnosis; it does not claim a measured speedup or improved recommendation quality.

Common misconceptions and mistakes

Serving is not synonymous with training. A serving system usually consumes a trained artifact; updating the model from new data is a separate process unless explicitly designed otherwise. It is also not limited to a public web API or a cloud platform.

More replicas may increase capacity, but they do not eliminate every bottleneck. Shared feature services, network limits, or insufficient memory can still constrain performance. Likewise, a faster model may not noticeably improve user experience if most time is spent elsewhere.

An available endpoint does not prove that its feature meanings, class-label order, or thresholds match those used during evaluation.

Business use

Define serving requirements around the user's workflow: how fresh the prediction must be, when it is needed, and what happens if it is missing. Track successful response rates, latency or batch completion times, resource cost, and prediction-quality signals as distinct measures.

Store the model version and compatible transformation artifacts together. For delayed outcomes, plan how later labels will be joined to earlier predictions so that quality can be assessed without guessing which model produced them.

Check your understanding

If inference takes 30 milliseconds but the full request takes 150, which number reflects the user's waiting time?

Answer: The end-to-end 150 milliseconds. The inference time is one component and should not be presented as the complete service response time.

Related terms

Learn next

trace one prediction from raw input to delivered result, then compare the freshness and reliability requirements of interactive and batch serving.

Sources

Primary sources: Google's Machine Learning Glossary, serving: https://developers.google.com/machine-learning/glossary#serving ; TensorFlow's serving overview: https://www.tensorflow.org/tfx/guide/serving . These pages were read. The explanation is platform-neutral; the timing calculations and fictional case are original illustrations, not product benchmarks.

Previous: Service principal Next: Session (web analytics)