Does online inference require retraining on every request?

No. A fixed trained model can generate many on-demand predictions without changing its parameters.

Can it use precomputed information?

Yes. Request-time predictions can combine fresh inputs with stored features, provided their meaning and freshness are suitable.

Can online and offline inference coexist?

Yes. A system may precompute common results and use on-demand computation for other cases, with clear rules for each path.

AI Glossary / Graphs and symbolic AI / Beginner

Online inference

Definition

Online inference is the use of a trained model to generate a prediction when an application requests it. The model processes the request's inputs and returns a newly computed result, rather than relying entirely on predictions prepared earlier. It is also called dynamic inference. Online here describes when prediction happens; it does not mean that the model is learning during every request.

What it means

A request might ask for a delivery-time estimate using a newly entered address or for a classification of a message that has just arrived. The application cannot necessarily know those exact inputs in advance, so it runs the model after receiving them.

The main benefit is responsiveness to request-specific information. The main operational challenge is that the user or another system may be waiting. Data retrieval, preparation, model computation, and response handling all contribute to the delay. A fast model alone does not guarantee a fast prediction service.

How it works

The service receives and validates an input, obtains the features required by the deployed model, and applies compatible transformations. The model computes an output, which may then be converted into the form expected by the client.

Some features can be prepared earlier and retrieved at request time. Others may come directly from the request or require fresh calculation. Online inference therefore does not imply that every ingredient is calculated from scratch for every call.

The serving system needs an explicit response when inputs are missing, the model is unavailable, or a request exceeds the allowed time. It also needs records that distinguish the deployed model version from the request data. Updating that model is a separate process; online prediction can use the same fixed parameters for many requests.

Three practical examples

Customer support: classify a newly submitted message so it can be routed to an appropriate queue.
Delivery estimates: generate a prediction after a customer enters a destination and selects a service option.
Interactive recommendations: score candidate items using the current session context, while perhaps retrieving other features prepared earlier.

Think of it this way

Think of a calculator answering after you enter the numbers, rather than handing you a table of answers prepared yesterday. The answer can reflect the particular inputs you just supplied.

The analogy is limited because a learned prediction is not exact arithmetic. It may be uncertain, biased, or outside the model's reliable input range. Immediate computation does not establish that the resulting answer is accurate or that all its supporting data are fresh.

Worked example

Suppose a hypothetical prediction service has a target response budget of 150 milliseconds. For one request, validation takes 10 milliseconds, feature retrieval takes 35, transformation takes 15, model computation takes 40, and response processing takes 10.

If these stages run sequentially, their total is 10+35+15+40+10=110 milliseconds. That leaves 40 milliseconds for any additional delays within the stated measurement boundary. If queueing adds 60 milliseconds, the total becomes 170 milliseconds and misses the target even though the model itself still takes only 40.

Now suppose the model is replaced with one that takes 20 milliseconds. The queued request would take 150 milliseconds under these exact assumptions, but this single calculation does not establish that the service meets its target reliably. Different requests and traffic levels can produce different queueing and retrieval times.

The team therefore measures end-to-end latency distributions under representative load and defines timeout behavior. These numbers are original teaching assumptions, not benchmark results. The example illustrates why a serving decision depends on the complete request path, rather than only the model's computation time.

Illustrative case study

Fictional case: an online shop adds a request-time recommendation model. The model runs quickly in a local test, but customers experience delays because a feature lookup is repeatedly waiting on another service.

The team measures the stages separately, establishes a response-time limit, and defines a fallback recommendation list for requests that cannot finish in time. It checks the quality of both normal and fallback responses. This case illustrates diagnosing a serving bottleneck, not a reported improvement or a claim that fallback results are equivalent to model predictions.

Common misconceptions and mistakes

Online inference and online learning are different. Serving a thousand requests does not automatically train the model on those interactions. Training requires its own data, objective, and update procedure.

Online also does not mean instantaneous or hard real-time. A service may compute on demand and still take too long for a particular application. Its timing guarantees need to be stated separately.

Fresh requests can contain old features. A customer activity summary retrieved now may have been computed yesterday. Finally, the ability to accept a previously unseen input does not guarantee a reliable prediction for an unfamiliar situation outside the model's intended domain.

Business use

Online inference is useful when decisions depend on information available only at request time or when precomputing all relevant inputs is impractical. It can support interactive services, but introduces operational dependencies that should be included in the business case.

Teams should define acceptable delay, expected request volume, input coverage, feature freshness, and behavior during failure. They should monitor prediction quality as outcomes become available as well as the service's technical performance. A reliable endpoint that returns poor predictions and an accurate model that regularly times out are different problems requiring different responses.

Check your understanding

Why does a forty-millisecond model fail the example's 150-millisecond target?

Answer: The total includes other processing stages and queueing. With the assumed sixty-millisecond queue delay, end-to-end processing reaches 170 milliseconds.

Related terms

Offline inference
Online learning
Inference Pipeline
Latency
model serving
feature store
throughput
fallback behavior

Learn next

Trace a prediction request from arrival to response and label which stages read stored information and which compute new values. Then study online inference pipelines and evaluate both response-time distributions and feature freshness for the intended application.

Feature Freshness

Sources

Google Machine Learning Crash Course, Static versus dynamic inference: https://developers.google.com/machine-learning/crash-course/production-ml-systems/static-vs-dynamic-inference — request-time prediction and serving tradeoffs.

Hopsworks, Online Inference Pipeline: https://www.hopsworks.ai/dictionary/online-inference-pipeline — combining request inputs, features, transformations, and model execution. All timings and operational scenarios above are original illustrations.

Previous: Online gradient descent Next: Online Inference Pipeline