Is preprocessing part of the pipeline?

Yes. Transformations required to turn the request's information into valid model inputs belong to the inference workflow.

Must every feature be calculated during the request?

No. Pipelines often combine on-demand features with values prepared earlier and retrieved at serving time.

Is retraining part of online inference?

Usually it is a separate workflow. The inference pipeline uses a deployed model to answer requests, while training produces or updates that model.

AI Glossary / Graphs and symbolic AI / Intermediate

Online Inference Pipeline

Definition

An online inference pipeline is the sequence of processing steps that turns a prediction request into a response using a deployed machine-learning model. It can validate inputs, retrieve or compute features, apply transformations, run the model, and prepare the output. The pipeline is broader than the model: it includes the surrounding work needed to produce a usable request-time prediction.

What it means

A model usually expects a particular representation of its input. A client request may instead contain identifiers, raw measurements, or incomplete context. The pipeline bridges that gap while preserving the meaning the model learned during training.

For example, a delivery request might provide a destination identifier and parcel size. The model may also require a stored regional statistic and a transformed distance value. Obtaining and arranging those values correctly is part of the pipeline, even though it happens outside the model's numerical calculation.

How it works

The deployed service loads a compatible model and its required preprocessing artifacts. For each request, it checks the request format, determines the relevant entities, and obtains the required data. Some features may be computed from the request while others come from a feature store or another approved source.

The pipeline applies the correct transformations, builds the input in the expected order and format, invokes the model, and post-processes its output. The response may include a predicted value or class together with information the application needs to interpret it.

The stages must agree about units, feature definitions, missing values, and versions. An output can look plausible even when these agreements are broken. Observability should therefore cover intermediate processing and data quality, not just whether the final request returned successfully.

Three practical examples

Delivery prediction: combine a request's parcel details with stored route features before estimating arrival time.
Support routing: clean and encode a new message, invoke a classifier, and map its class identifier to a queue label.
Recommendation serving: retrieve account features, calculate session-dependent features, score candidate items, and format the resulting list for an application.

Think of it this way

Think of a laboratory workflow in which a sample is identified, prepared, measured, and reported. The measuring instrument corresponds to the model, while the complete handling process corresponds to the pipeline.

The analogy is limited because a machine-learning prediction is not necessarily a direct physical measurement. Still, it captures an important point: a correct instrument cannot compensate for a mislabeled sample or the wrong preparation procedure.

Worked example

Consider a toy model that predicts a delivery duration using p=5+2d+3s, where d is distance in kilometers and s is a stored congestion feature between 0 and 1. This deliberately simple formula illustrates the pipeline; it is not a trained production model.

A request supplies distance as 4,000 meters and an area identifier. The pipeline retrieves s=0.6 for that area, converts the distance to d=4 kilometers, and constructs the model input in the agreed order. The output is 5+2×4+3×0.6=14.8 minutes.

If the conversion is omitted, the model receives d=4,000 and produces 8,006.8 minutes. If the two features are accidentally swapped, it computes 5+2×0.6+3×4=18.2 minutes. The second mistake looks more plausible but is still wrong for the specified formula.

The pipeline should also decide what happens if the congestion value is missing or too old. Silently replacing it with zero changes the prediction to 13 minutes and implies a particular condition that may not be justified. A documented fallback, rejected request, or alternative model could be appropriate depending on the application. These are hand calculations demonstrating representation errors, not evidence from a deployed service.

Illustrative case study

Fictional case: a logistics team replaces its model but keeps an older preprocessing component. Requests still return valid-looking numbers, so basic uptime monitoring reports no problem. A review discovers that the new model expects a different distance normalization.

The team packages compatible model and transformation versions together and checks representative requests across the full serving path before release. It also records which feature definitions produced each response. The case illustrates compatibility management; it does not claim a particular deployment product automatically prevents this failure.

Common misconceptions and mistakes

An online inference pipeline is not just a network endpoint around a model. Input preparation and feature retrieval can determine whether the prediction means what the application thinks it means.

It does not have to use a feature store. That is one architecture for supplying precomputed features, not a requirement for every system. Similarly, a model registry can support version management without needing to be contacted for every request.

Do not confuse pipeline success with prediction correctness. A request can pass format checks and return on time while using stale features, mismatched units, or a model unsuitable for the case. Conversely, a deliberate rejection can be the correct response to invalid input.

Business use

The pipeline is a useful unit of responsibility for operational AI. It makes clear that model quality, data contracts, response time, and failure handling are connected parts of one service.

A business review can identify who owns each feature, how compatibility is checked, and what users receive when a dependency fails. Evaluation should include representative complete requests, missing-data cases, version changes, and end-to-end timing. The goal is a predictable service contract, not merely a model that performs well on a prepared dataset.

Check your understanding

Why is the swapped-feature output especially easy to miss in the worked example?

Answer: It remains numerically plausible at 18.2 minutes, although the model received values in the wrong positions. Range checks alone may not detect the semantic error.

Related terms

Learn next

Write down a model's required inputs, including units, order, transformation versions, and missing-value rules. Trace a single request through every stage and calculate the expected intermediate values before examining throughput or scaling.

Sources

Hopsworks, Online Inference Pipeline: https://www.hopsworks.ai/dictionary/online-inference-pipeline — the supplied term's deployment workflow and combination of on-demand and precomputed features.

Google Machine Learning Crash Course, Static versus dynamic inference: https://developers.google.com/machine-learning/crash-course/production-ml-systems/static-vs-dynamic-inference — request-time serving context. The formula, compatibility scenario, and review recommendations are original teaching material.

Previous: Online inference Next: Online learning