Does every mini-batch update improve the full training loss?

No. The gradient is based on a subset, and its direction may not reduce the full objective at that step.

Is a larger batch always better?

No. Memory, computation, sampling variability, and optimization behavior create tradeoffs that need evaluation.

Can one epoch contain many updates?

Yes. With fixed-size nonoverlapping batches, a pass through the data ordinarily contains one update per batch, subject to the chosen accumulation policy.

AI Glossary / Mathematics and optimization / Intermediate

Mini-batch stochastic gradient descent

Definition

Mini-batch stochastic gradient descent is an optimization method that updates model parameters using gradients calculated from a relatively small group of training examples. Each update uses that mini-batch to estimate the gradient of the full training objective. It balances frequent updates with efficient processing of several examples together.

What it means

Full-batch gradient descent uses the entire training dataset for each gradient update. In the narrow terminology used by the supplied glossary, ordinary stochastic gradient descent uses one example, while mini-batch SGD uses a group. In practice, people also use SGD more broadly to include mini-batch updates.

The batch size is the number of examples contributing to one update. An epoch usually means one pass through the training dataset, often after shuffling. These are different units: one epoch can contain many parameter updates. Sampling with replacement or streaming data can require different descriptions of training progress.

How it works

At the current parameter vector, calculate the gradient of the loss for each selected example and average those gradients. Subtract the learning rate times this average from the parameters. Repeat with another mini-batch.

For uniformly sampled examples from a fixed dataset, the average mini-batch gradient has a useful relationship to the full-data gradient. At a fixed parameter vector, independent uniform draws produce an unbiased estimate. Random reshuffling without replacement is common but has a different conditional sampling structure, so not every theoretical statement transfers unchanged.

Larger batches often reduce sampling variability and can use vectorized hardware efficiently, but consume more memory and change the number and character of updates. Batch size and learning rate interact. A training step need not lower the full objective, and a decreasing training loss alone does not establish good performance on unseen examples.

Three practical examples

1. Neural-network training: process groups of images or text examples together to calculate each update within available memory.

2. Large-scale regression: estimate updates from subsets when calculating a full-data gradient at every step would be expensive.

3. Experiment design: compare batch sizes using both elapsed training time and held-out performance, rather than comparing update counts alone.

Think of it this way

Think of revising a proposal after reading a small group of feedback forms. Each group gives a partial picture, and successive revisions reflect different groups.

The analogy does not imply that a small group is automatically representative or that every revision improves the full set of responses. It also misses the precise gradient calculation: mini-batch SGD follows a mathematical local direction, not a simple vote among examples.

Worked example

Fit one scalar w to four target values 1, 3, 5, and 7 using per-example loss (w−y)²/2. The full average-loss gradient is w−4, because the targets' mean is 4. Start at w=0 with learning rate 0.1.

Choose the first mini-batch as {1,3}. Its gradient is the average of w−1 and w−3, which at zero equals −2. The update gives w=0−0.1×(−2)=0.2.

Use {5,7} next. At w=0.2, the gradients are −4.8 and −6.8, with average −5.8. The next update gives w=0.2−0.1×(−5.8)=0.78.

A single full-batch update from zero would instead give w=0.4. The two procedures are not equivalent because the second mini-batch is evaluated after the first update changed w. Reversing the two mini-batches yields w=0.6 after the first and w=0.74 after the second, showing that order can matter.

If all four gradients were evaluated at w=0, accumulated without intervening parameter changes, and averaged correctly, one update would match the full-batch step in this simple example. That is a different schedule from updating after each mini-batch.

Illustrative case study

Fictional case: a team doubles batch size to use available hardware more efficiently. It keeps the same number of epochs and initially compares the runs as if they had performed the same number of updates.

A reviewer notes that the larger-batch run takes fewer updates per pass through the data. The team records examples processed, update count, elapsed time, and validation performance, and tunes the learning-rate schedule within a controlled comparison. The result is an interpretable experiment rather than a blanket claim that a bigger or smaller batch is always better.

Common misconceptions and mistakes

Do not confuse an epoch with an update. Do not assume that averaging gradients from successive parameter states is the same as evaluating one large batch at a single state.

Summing losses instead of averaging them changes gradient scale unless the learning-rate or normalization convention compensates. Batch construction can also matter when examples are correlated or badly imbalanced. Finally, naming the optimizer SGD does not guarantee a particular batch size, momentum setting, or learning-rate schedule; those details should be recorded explicitly.

Business use

Mini-batch SGD helps make training practical on large datasets, but its operational value depends on the complete training configuration. It affects memory use, hardware utilization, update noise, and the schedule over which the model learns.

For project reporting, describe the effective batch size and what counts as one update, especially when gradients are accumulated across devices or smaller batches. Compare final quality and resource use under clearly stated budgets. A faster individual step can still produce a slower or less effective overall training run.

Check your understanding

A dataset has 1,200 examples and uses nonoverlapping mini-batches of 100, with one update per batch. How many updates occur in one complete epoch?

Answer: Twelve. This assumes every example is used once and there is no gradient accumulation across batches.

Related terms

Learn next

compare full-batch, single-example, and mini-batch updates on the same small objective. Then examine learning-rate schedules and gradient accumulation, keeping track of where gradients are evaluated and how they are normalized.

Sources

Google Machine Learning Glossary, mini-batch and mini-batch stochastic gradient descent: https://developers.google.com/machine-learning/glossary#mini-batch_stochastic_gradient_descent . Relevant definitions were read. The two-batch ordering and accumulated-gradient comparisons are original hand calculations; no training execution is claimed.

Previous: Mini Batch K-Means Next: Minimal Cover