Is latency the same as throughput?

No. Latency is delay for an operation; throughput is the rate of completed work.

Does a low median describe the slowest users?

No. Tail measurements are needed to understand slower requests.

Does faster model computation always halve user waiting time?

No. Other stages and queueing can remain substantial.

AI Glossary / MLOps and deployment / Advanced

Latency

Definition

Latency is the elapsed time between a defined initiating event and a defined response or completion event. In computing, it can describe network delay, request response time, storage access or an AI system's response behavior. A useful latency measurement states its start and end points, units and measurement conditions. Without those details, two reported values may measure different experiences.

What it means

A system can perform many operations per second while an individual user still waits a long time. Latency describes that delay; throughput describes the rate of completed work. The two can interact but are not interchangeable. A user-facing request may include network travel, queueing, computation and rendering. Measuring only the model's calculation time can therefore miss much of the actual wait. For streamed AI output, time to the first visible result and time to complete the response also answer different questions.

How it works

Instrument a clearly defined interval and collect measurements across representative requests. Summaries such as the median and high percentiles describe different parts of the distribution. Averages can hide a minority of slow requests, while a high percentile is still not a maximum or a guarantee. Breakdowns and traces can help identify contributing stages, but overlapping work means stage durations cannot always be added naively. Load, payload size, cache state and errors affect interpretation. Measurements across different machines also need appropriate clock handling if timestamps are directly compared.

Three practical examples

An AI assistant measures both time to its first displayed output and time until the complete response is available.
A search service compares typical response time with the slowest portion of its request distribution during busy periods.
A storage benchmark specifies whether timing ends when a write is accepted, buffered or durably completed.

Think of it this way

Imagine ordering a meal. The kitchen may prepare many meals each hour, but your wait depends on when your order arrives, the queue and the work needed for your dish. Receiving a drink quickly is also different from receiving the entire meal. The analogy separates throughput from individual delay and illustrates why the start and end events must be clear before people compare response times.

Worked example

In an original sequential request illustration, network and transfer overhead takes 30 milliseconds, queueing takes 20, computation takes 80 and final delivery takes 10. With these stages explicitly nonoverlapping, total latency is 140 milliseconds. Reducing computation by half makes it 40 milliseconds and the total becomes 100, a reduction of 40/140, approximately 28.6%. It does not halve the total delay because other stages remain. The example uses stated stage boundaries and no parallel overlap; it is not a general rule for summing arbitrary traces or a prediction for a particular deployed service.

Illustrative case study

A fictional customer-support tool reports faster average model execution after an upgrade, but users still complain during peak hours. End-to-end measurements show longer queueing and a small group of very slow requests. The team reviews request-size differences and percentile distributions before changing capacity and scheduling. It also checks response quality so faster output does not conceal poorer answers. This scenario illustrates why one component's improvement and one average can be insufficient to explain the experience users actually encounter.

Common misconceptions and mistakes

Low average latency does not mean every request is fast. A 99th percentile is not a guaranteed upper bound, and its estimate depends on the data and calculation method. Another mistake is averaging percentiles from different groups as though that gives the combined percentile. Throughput and latency must also be distinguished: batching may improve processing efficiency while increasing waiting time for some requests. Finally, comparing benchmarks without matching request content, concurrency and timing boundaries can create an apparent speed advantage that does not carry over to real use.

Business use

Define latency targets around user tasks and the consequences of delay. Measure representative end-to-end behavior and report enough distributional information to expose slow experiences. Segment results when request types differ materially, and include failures or timeouts in an explicit reporting policy. Use stage measurements to locate bottlenecks, then verify that changes improve the intended experience without unacceptable cost or quality tradeoffs. A useful performance claim describes what was measured and under which load, rather than presenting one attractive number as universal system speed.

Check your understanding

What was the initial example latency?

Answer: One hundred forty milliseconds.

What happened after computation was halved?

Answer: Total latency fell to 100 milliseconds.

Why was the percentage improvement only about 28.6%?

Answer: The unchanged stages still contributed 60 milliseconds, so the full request did not scale with computation alone.

Related terms

Throughput
tail latency
response time
queueing delay
service-level objective
time to first token

Learn next

Draw the timing boundaries for a request you understand and identify sequential versus overlapping stages. Measure several requests rather than one, then compare a median with a high percentile. Explain which part of the user's experience each metric captures and which parts remain outside its scope.

Sources

Google, The Site Reliability Workbook, Monitoring: https://sre.google/workbook/monitoring/ The sequential-stage calculation, meal-order analogy and support-tool case are original educational illustrations. The arithmetic assumes nonoverlapping stages and does not treat percentile estimates as hard guarantees or substitute component timing for end-to-end measurement.

Previous: Late move reduction Next: Latent class model