One full pass through the training data, made up of many batches.

A batch smaller than the whole dataset, which is what most training uses.

What is batch inference?

Scoring many records in one scheduled run, rather than one at a time on demand.

Does batch size affect cost?

Indirectly. Larger batches use hardware more efficiently, so training can be cheaper per example, up to the memory limit of the machine.

AI Glossary / AI and machine learning / Beginner

Batch

Definition

In machine learning, a batch is a group of training examples processed together in one step before the model updates its parameters.

What it means

Models learn by looking at examples and adjusting themselves. Looking at one example at a time makes learning noisy, and looking at the whole dataset every time is slow and memory-hungry.

A batch is the compromise: a chunk of examples, say 32 or 256, passed through the model together. The model computes its errors over that chunk and then makes one update.

Going through all the training data once, batch by batch, is called an epoch. If there are 10,000 examples and a batch size of 100, one epoch has 100 updates.

The batch size is a setting chosen by the engineer. Small batches give noisier, more frequent updates; large batches give smoother updates but need more memory and can generalize differently.

The word also appears in business computing with a different meaning, such as a batch job that processes a pile of records overnight. In AI, context makes the meaning clear.

Batch size also interacts with the learning rate, the size of each adjustment. Larger batches usually pair with a larger learning rate, and changing one without the other is a common cause of poor training runs.

How it works

The training data is shuffled and split into batches of the chosen size. Shuffling matters so that each batch is a fair mix, not, say, all of one customer type.

For each batch, the model makes predictions, the loss measures the average error across the batch, and the optimizer adjusts the parameters to reduce it.

Using a batch also lets modern hardware work efficiently, as graphics processors are built to do the same calculation on many examples at once.

The same word appears at prediction time: batch inference means scoring many records in one go, for example nightly, as opposed to answering one request instantly.

Grouping requests into batches at prediction time raises hardware efficiency, which is why providers charge less for batch jobs that can wait in a queue.

Three practical examples

Training an image classifier on batches of 64 images at a time.
Scoring all customers for churn risk each night in a single batch run.
Sending a batch of documents to an AI service for overnight summarizing at a lower price.

Think of it this way

Think of a teacher marking homework. Marking one paper and changing the lesson plan each time would be chaotic; marking the whole year before changing anything would be slow. Marking a pile of 30 and then adjusting strikes a balance.

The limit: if the pile is unrepresentative, say all from one class, the adjustment is skewed.

Worked example

choosing between nightly batch scoring and live scoring

Suppose a business wants to flag customers at risk of leaving. It can score everyone nightly in a batch, or score each customer live when they visit the site.

Nightly batch scoring is cheap and simple, and the results are ready for the morning's outreach list. Live scoring needs more engineering and is worth it only if the decision must happen in the moment.

The team decides that outreach happens by email the next day, so a nightly batch is enough. The decision and costs here are illustrative.

They log the date of each score, so staff know how fresh the information is.

Illustrative case study

A fictional startup trains a model with a very large batch size to speed things up. Training is fast but the final model performs worse on new data than earlier runs.

An engineer reduces the batch size and adjusts the learning rate. Performance recovers at the cost of longer training.

The invented lesson: batch size is a trade-off between speed, memory and quality, and it should be tested.

Common misconceptions and mistakes

A larger batch always trains better. It can speed up training but may hurt generalization.
Batch means the whole dataset. That is full-batch training, which is rare for large data.
Batch always means training. It also describes processing many records at once at prediction time.

Business use

As a buyer, know that batch processing is often cheaper and simpler than real-time, and many AI services discount batch jobs.

Ask whether a decision truly needs to be instant. If not, a nightly batch may save money and complexity.

When a vendor quotes throughput, ask whether it refers to batch or real-time processing. Batch figures look faster per record but say nothing about how quickly one request is answered.

Check your understanding

A dataset has 10,000 examples and the batch size is 100. How many updates happen in one epoch?

Answer: 100: the data is split into 100 batches, and the model updates once per batch.

Related terms

Learn next

AI Agents and Automation for Small Business shows when scheduled batch jobs beat live automation: https://aiforbusinessandlife.com/academy/ai-agents-automation-small-business. [C4]

Sources

[S2] Google, Machine Learning Glossary; [S12] Aalto University, Dictionary of Machine Learning. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Basis function regression Next: Batch active learning