Does compute mean a GPU?

No. GPUs are one source of compute; CPUs and other accelerators also perform computational work.

Is training compute the same as inference compute?

No. Training updates model parameters, while inference applies a model, and their resource patterns can differ substantially.

Can software reduce compute needs?

Yes. Algorithm changes, precision choices and efficient implementations can reduce work or improve utilization, subject to quality and compatibility checks.

AI Glossary / MLOps and deployment / Beginner

Compute

Definition

Compute is the processing capability or computational work used to run software. In AI, the term can refer to hardware resources, the amount of numerical computation required for training or inference, or the capacity available to perform that work.

What it means

An AI model needs processors to execute operations, but it also depends on memory, data movement, storage and software. Saying a project needs more compute can therefore mean several things: more processing devices, longer runtime, faster operations or a larger computational budget. The intended meaning should be specified. Training compute and inference compute serve different workloads, and peak hardware performance is not the same as achieved application performance. A processor can spend time waiting for data even when its theoretical arithmetic capacity is high.

How it works

A workload is mapped onto processors such as CPUs, GPUs or other accelerators. Software schedules operations and moves data through memory and across devices. Performance depends on numerical precision, operation types, parallelism, utilization and communication overhead.
Computational work may be estimated in floating-point operations, while hardware throughput may be expressed in operations per second. Device-hours describe resource use over time but do not make different devices equivalent. Latency measures how long one request takes; throughput measures work completed per unit time. These quantities answer different planning questions. A useful compute estimate connects the model and workload to measured system behavior.

Three practical examples

A research team budgets processing resources for several training experiments rather than only one final run.
A service operator measures how many inference requests a deployment can complete within its latency requirement.
An optimization project reduces unnecessary operations or data transfers to make better use of existing hardware.

Think of it this way

Think of compute as a workshop's ability to carry out tasks. More machines can help, but only if materials arrive, jobs can be divided and workers are not waiting on one shared station. The analogy explains bottlenecks without implying that every AI workload scales like an independent set of physical jobs.

Worked example

Suppose a hypothetical job requires 6 × 10¹⁴ floating-point operations. A system sustains 2 × 10¹² operations per second on that exact workload. Dividing work by sustained throughput gives an idealized arithmetic time of 300 seconds, or 5 minutes.
If the same machine advertises a much higher peak rate, using that peak would underestimate runtime when the workload cannot achieve it. Data loading, communication and setup may add time beyond the arithmetic estimate. Similarly, running two devices does not automatically halve elapsed time if synchronization or shared memory traffic becomes limiting. These invented figures illustrate the calculation, not a product benchmark.

Illustrative case study

In a fictional image-analysis service, a team adds an accelerator expecting lower response times. Profiling shows that image decoding and transfer dominate many small requests, leaving the accelerator underused. The team improves preprocessing and batching, then checks how batching affects individual-request delay. It evaluates the whole service rather than only the model kernel. This case illustrates why buying processing capacity and achieving useful application performance are different steps, without claiming a measured speedup.

Common misconceptions and mistakes

Compute is not identical to memory capacity, although both constrain AI systems. Peak operations per second do not directly predict real runtime. Device-hours are not universally comparable across hardware and precision modes. More compute also does not guarantee better model quality: data, architecture, optimization and evaluation matter. Another mistake is optimizing throughput while overlooking latency, or reporting average latency while rare slow requests violate the service's requirements.

Business use

For AI planning, define the workload, quality target, latency needs and expected volume before estimating resources. Include experimentation, preprocessing and operational overhead. Benchmark representative inputs using the intended precision and software stack. Track utilization and bottlenecks so optimization addresses the limiting factor. Compare resource cost with useful completed work rather than a hardware headline. Capacity planning should also consider demand variation and the time needed to make additional resources ready.

Check your understanding

A model runs faster in a large batch but each request waits longer before the batch starts. Has every performance measure improved?

Answer: No. Throughput may improve while end-to-end request latency worsens. The relevant service requirements determine whether the tradeoff is useful.

Related terms

Latency
Training
Inference
CPU
GPU
accelerator
FLOPs
throughput
utilization
memory bandwidth
capacity planning

Learn next

Learn latency, throughput and memory bandwidth next. Profile a small AI pipeline by stage, separating data preparation, transfer and model execution before deciding whether additional hardware or software changes would help.

Latency

Sources

NVIDIA, technical discussion of precision-dependent accelerator compute: https://www.nvidia.com/en-gb/data-center/tensorcore/ MLCommons, benchmark organization and workload-based evaluation: https://mlcommons.org/ Primary hardware and benchmarking-source excerpts consulted; full workload-specific and editorial review remain pending.

Previous: Computational statistics Next: Computer Assisted Survey Information Collection