Does inference always happen in real time?

No. It can be scheduled, event-driven or requested interactively.

Does an inference request change model weights?

Usually not in standard serving, though a system may include a separate adaptation or learning process that should be identified explicitly.

Are latency and throughput the same?

No. Latency is time to complete a request or batch, while throughput is the amount processed per unit time. Improving one does not guarantee improvement in the other.

AI Glossary / Statistics and probability / Intermediate

Model Inference

Definition

Model inference is the use of a trained machine-learning model to produce outputs from input data. Depending on the model, those outputs may be predictions, scores, classifications, embeddings or generated content. In operational ML, inference usually applies learned parameters rather than updating them through training. The surrounding inference pipeline can include feature retrieval, preprocessing, model execution and postprocessing. This usage differs from statistical inference about unknown population parameters, although statistical reasoning may be used to develop and evaluate the deployed model.

What it means

Training establishes a model's learned behavior, while inference applies that behavior to a particular input. A model can be evaluated through inference on historical examples as well as used on new requests; newness alone is not the defining distinction. In production, the useful output depends on the complete pipeline and on how a person or service consumes it. A probability score, for example, may be converted into an action by a threshold, and that action rule is separate from the numerical model output.

How it works

Load the intended model version and obtain inputs in the expected schema. Apply the same required transformations and feature definitions used during development, including any learned preprocessing parameters. Execute the model in the appropriate inference mode, then transform or interpret the output for the application. Record relevant version and operational information, and monitor quality when outcomes become available. Inference may run in scheduled batches, request-response services, streams or on devices. Stateful and generative systems can maintain context during inference without necessarily changing the learned model weights.

Three practical examples

Batch prediction: generate a scheduled set of demand forecasts and store them for planning.
Online prediction: score an incoming request and return a response within an operational latency requirement.
Stream inference: process sensor events as they arrive and produce alerts or updated estimates. Each pattern has different freshness, throughput, state and failure-handling requirements, even when the underlying model is the same.

Think of it this way

Think of using a calibrated instrument after it has been built and adjusted. Each measurement applies the instrument's existing mechanism to a new input. The surrounding handling still matters: wrong units, stale inputs or an incorrect interpretation can make the result unsuitable. Using the instrument does not automatically recalibrate it or prove that its output is correct.

Worked example

Consider an illustrative logistic model with stored coefficients z = -1 + 0.5x1 - 0.25x2 and output p = 1/(1 + exp(-z)). For input x1 = 4 and x2 = 2, z = 0.5 and p is approximately 0.62246. A decision threshold of 0.6 produces a positive classification, while a threshold of 0.7 produces a negative one without changing the model score or coefficients. Suppose feature retrieval takes 12 milliseconds, preprocessing 8, model execution 20 and postprocessing 5 in a sequential request. End-to-end time is 45 milliseconds before any additional network or queue delay, not merely the 20 milliseconds spent executing the model. This original example distinguishes prediction, decision policy and service latency. The score is not claimed to be calibrated for a real application.

Illustrative case study

In a fictional case, Rowan Search deploys a relevance model whose laboratory execution time is fast. Users still experience slow responses because feature retrieval dominates the request path. Engineers measure the full inference pipeline and improve feature availability before changing the model. They also discover that a deployment used a different text-normalization rule from training and correct it. The resulting evaluation considers both ranking quality and user-visible latency rather than treating successful model execution as the whole service.

Common misconceptions and mistakes

Inference is not the same as training, and ordinary prediction does not automatically make a model learn from each request. Online inference does not necessarily mean online learning. A model score is not always a calibrated probability, and generated text is not automatically factual. Low execution latency does not imply low end-to-end latency under load. Batch inference is not inherently less accurate, though stale data can matter. Correct model weights cannot compensate for inconsistent preprocessing, missing features or an inappropriate downstream decision rule.

Business use

Businesses should define output quality, freshness, latency, throughput and cost requirements together. Measure the complete pipeline under realistic traffic and monitor failures, not only average model execution time. Keep model and preprocessing versions compatible and document how predictions become decisions. Plan for missing inputs, unavailable dependencies and rollback. For generative systems, evaluate content quality and the effects of decoding settings as well as infrastructure performance. A reliable inference service requires both an appropriate model and a dependable way to supply and use its inputs and outputs.

Check your understanding

Why can the same score yield two different classifications in the example?

Answer: The decision threshold is a separate application rule. The model produces approximately 0.62246 in both cases, but that value exceeds 0.6 and falls below 0.7. Changing the threshold changes the decision without retraining or altering the prediction calculation.

Related terms

Inference Pipeline
Batch inference
Online inference
Stream processing
Training
Latency
model serving
feature retrieval
decision threshold
throughput

Learn next

Learn how preprocessing and model artifacts are versioned together, then compare batch and online serving architectures. Study performance under load, prediction monitoring and the distinction between probabilistic outputs, calibrated probabilities and downstream actions.

Sources

Hopsworks MLOps Dictionary, Model Inference: https://www.hopsworks.ai/dictionary/model-inference . Google, Productionization: https://developers.google.com/machine-learning/managing-ml-projects/production . The logistic-score and latency calculations and fictional search-service case above are original illustrations.

Previous: Model-Independent Transformations Next: Model Interpretability