What is a typical batch size?

Common values are powers of two between 16 and 512, but the right choice depends on the data, the model and the hardware.

Does batch size affect prediction speed?

Only for batch inference. A single live prediction is unaffected by the training batch size.

Is there one best batch size?

No. It is a trade-off tested by comparing trained models on held-out data.

AI Glossary / AI and machine learning / Beginner

Batch size

Definition

Batch size is the number of training examples a model works through before it makes one adjustment to itself, a setting that trades speed, memory use and stability during training.

What it means

Training means showing a model examples and nudging its parameters to reduce errors. The batch size decides how many examples are bundled into each nudge: 16, 64, 256 or whatever the engineer chooses.

A small batch gives frequent, noisy updates. Each bundle is a rough sample, so the model zigzags a little as it learns, which can actually help it settle into a better final state.

A large batch gives smooth, steady updates and lets modern hardware run efficiently, but it needs more memory, and very large batches can produce models that perform slightly worse on new data.

The batch size is not a property of your data; it is a dial. The same dataset can be trained with many settings, and the right one depends on the hardware, the model and how much time you have.

For a business, batch size mostly matters as a cost and feasibility question: it shapes how long training takes, how much computing power you rent, and whether the job fits on one machine.

How it works

The training data is shuffled and cut into batches of the chosen size. One full pass through all the batches is called an epoch, so 10,000 examples at a batch size of 100 means 100 updates per epoch.

For each batch, the model makes predictions, the errors are averaged across the batch, and the optimizer adjusts the parameters once to reduce that average.

Batch size interacts with the learning rate, the size of each adjustment. Larger batches usually pair with a larger learning rate; changing one without the other is a common cause of poor runs.

Hardware sets the ceiling. Graphics processors have fixed memory, and if the batch does not fit, training stops. Engineers shrink the batch or use smaller models to make the job fit.

The same word appears at prediction time: batch inference means scoring many records in one scheduled run, which uses the same hardware-efficiency idea.

Three practical examples

Fine-tuning a text classifier on 20,000 support tickets with batches of 32, giving 625 updates per epoch.
Training an image model on a single graphics card, where a batch of 128 images is the largest that fits in memory.
A nightly job that scores every customer for churn risk in one batch run, ready for the morning outreach list.

Think of it this way

Think of marking exam papers. Marking one at a time and adjusting your marking scheme after each is jumpy and slow. Marking in stacks of 30 and reviewing your scheme after each stack is steady and fast. The stack size is the batch size.

The limit: if the stack is the whole school, you only adjust your scheme once and miss what the early papers were telling you.

Worked example

tuning a ticket classifier on one machine

Suppose a small software firm fine-tunes a model to tag support tickets, with 8,000 labeled examples and one rented graphics card.

A batch size of 256 runs out of memory. The engineer drops to 64, and training completes in 40 minutes. A run at 16 also works, takes longer, and its held-out accuracy is a touch higher. The numbers are illustrative.

The engineer keeps 64 because the difference is small and the run is faster, and notes the choice in the project log so it can be revisited.

The lesson for the buyer: batch size is a practical tuning decision, usually made for hardware reasons, and checked by comparing results rather than by theory.

Illustrative case study

A fictional startup trains a demand model on cloud machines and picks the largest batch that fits to finish faster and cut the bill.

The model trains quickly but performs worse on new weeks than an earlier experiment. An engineer halves the batch size, lowers the learning rate to match, and quality recovers at a higher cost per run.

The invented lesson: the cheapest training run is not always the cheapest model, because a model that misses more costs you every day after training.

Common misconceptions and mistakes

A larger batch always trains a better model. It trains faster per example but can hurt final quality.
Batch size changes what the model is. It changes how the model is trained, not what kind of model it is.
Batch and epoch mean the same thing. A batch is one bundle; an epoch is one full pass through every bundle.

Business use

When a vendor quotes training speed or price, ask what batch size and hardware the figure assumes, because both change the number.

If you train models on your own machines, expect batch size to be set by memory limits, and judge settings by held-out results, not by how fast the run finishes.

If you run training yourself, log the batch size with every experiment. Two runs with different batch sizes are not directly comparable, and an unlogged change can look like an improvement that is not.

Check your understanding

Why is the whole dataset rarely used as one batch for large data?

Answer: It would need far too much memory, give only one update per pass, and lose the frequent, varied updates that help models learn well.

Related terms

Learn next

AI for Business Owners explains how to weigh the cost and quality trade-offs behind model training decisions: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]

Model training

Sources

[S1] Russell and Norvig, AIMA glossary; [S2] Google, Machine Learning Glossary; [S18] NVIDIA, AI Enterprise Glossary. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Batch processing Next: BatchBALD