Must the whole dataset fit in memory at once?

No. Gradients can be accumulated across chunks without updating parameters between chunks, provided the computation matches the full objective.

Is batch gradient descent always deterministic?

It is deterministic for fixed initial conditions and a deterministic objective and implementation, but other randomness or numerical execution details may affect reproducibility.

Does a zero gradient prove the best possible model was found?

Not generally. In nonconvex settings it can indicate a stationary point that is not a global minimum, and optimization quality is separate from generalization.

AI Glossary / Mathematics and optimization / Intermediate

Batch gradient descent

Definition

Batch gradient descent updates a model's parameters using the gradient of the loss calculated across the entire training dataset at the current parameter values. Each update follows the negative full-data gradient, scaled by a learning rate. It is also called full-batch gradient descent to distinguish it from mini-batch training.

What it means

The batch here is the complete training set, not simply any group of examples. With an objective equal to the average of individual losses, the update uses their average gradient.

This produces a direction based on all current training examples rather than a randomly selected subset. It removes sampling noise from the gradient calculation for a fixed deterministic objective, but does not remove uncertainty in the dataset or guarantee good generalization. Other randomness in a model, such as stochastic transformations, can still make training nondeterministic unless controlled.

How it works

Initialize the parameter vector. Evaluate every training example using those same current parameters, accumulate the loss gradients, and divide by the number of examples when optimizing a mean loss. Include any regularization gradient consistently with the defined objective.

Update the parameters by subtracting the learning rate times the resulting gradient. Repeat until a suitable stopping rule is met, such as a small gradient or an improvement threshold under appropriate conditions.

The computation can be split into chunks for memory management while keeping parameters fixed until all gradients have been accumulated. That still represents a full-batch update if scaling is correct. Updating parameters after each chunk instead changes the method to mini-batch-style training.

Three practical examples

1. Small regression problem: optimize a smooth training objective when evaluating the full dataset at each step is computationally affordable.

2. Algorithm demonstration: compare a stable full-gradient trajectory with stochastic updates on the same simple objective to understand sampling effects.

3. Distributed optimization: aggregate gradients from data partitions before making a synchronized full-data update, accounting for partition sizes and communication costs.

Think of it this way

Imagine a committee waiting for every department's measured contribution before deciding one adjustment. The decision uses the complete current report rather than acting after each partial report arrives.

The analogy does not mean the result is automatically correct. The reports may reflect imperfect data, and an adjustment can be too large. Gradient descent still depends on the objective, learning rate, numerical precision, and the model's structure.

Worked example

Fit a constant prediction w to observations 1 and 5 using J(w)=[(w-1)^2+(w-5)^2]/4. The divisor includes both averaging over two examples and the conventional one-half factor in squared loss.

The full gradient is J'(w)=[(w-1)+(w-5)]/2=w-3. Starting at w=0 with learning rate 0.5 gives w1=0-0.5(-3)=1.5. The next gradient is -1.5, giving w2=2.25; then w3=2.625. The sequence approaches the optimum w=3.

The objective values start at J(0)=6.5, then J(1.5)=3.125 and J(2.25)=2.28125. The optimum has J(3)=2 because the two observations cannot both equal one constant prediction.

If the learning rate is 2, the update becomes w-next=6-w. Starting at zero, it alternates between 0 and 6 instead of converging. A full-data gradient does not prevent overshooting. These exact hand calculations use one fixed objective and do not claim a general learning-rate rule for other problems.

Illustrative case study

Fictional case: an analyst trains a small numerical model with full-batch updates. The loss decreases smoothly, but the same procedure becomes slow when the dataset expands.

The team compares full-batch and mini-batch training under matched evaluation conditions, recording total runtime rather than only the number of updates. It also checks whether changing the loss from a sum to a mean has inadvertently changed the effective step size.

The final choice reflects dataset size, hardware, convergence behavior, and held-out performance. The team does not assume that the least noisy gradient produces the fastest or best overall training process.

Common misconceptions and mistakes

One full-batch parameter update usually requires processing the complete dataset, whereas a mini-batch epoch contains several parameter updates. Comparing update counts alone can therefore be misleading.

Do not confuse batch gradient descent with batch learning as opposed to online learning. A batch-learning system can use mini-batch or stochastic optimization internally.

Summed and averaged gradients differ by dataset size. The learning rate must be interpreted with the objective's scaling.

Full-batch optimization does not guarantee a global optimum for a nonconvex model. Even on a simple convex objective, unsuitable step sizes can cause oscillation or divergence.

Business use

The method offers a clear reference point for training smooth objectives and can be effective when full gradient calculations are affordable. It may also exploit efficient matrix operations or synchronized data partitions.

Assess the cost per update, memory usage, communication overhead, and stopping behavior. Monitor validation performance separately from training loss. If gradients are accumulated across chunks or devices, check sample weighting and ensure all contributions correspond to the same parameter version before treating the result as the intended full-data gradient.

Check your understanding

Why does the learning-rate-2 example fail even though every gradient uses both observations?

Answer: The step is too large for convergence on that quadratic. The iteration jumps between equally distant points on opposite sides of the optimum, showing that complete data do not eliminate step-size issues.

Related terms

Learn next

Continue the scalar example with learning rates 0.25, 1, and 2, comparing updates and losses. Then compute individual-example gradients and compare them with the full average. Study mini-batch optimization to understand the trade-off between update cost and gradient variability.

Mini-batch

Sources

Primary author textbook discussion consulted: Dive into Deep Learning, Minibatch Stochastic Gradient Descent: https://d2l.ai/chapter_optimization/minibatch-sgd.html . Supplied AIMA glossary repository opened, but the retrieved page did not expose the term definition: https://github.com/aimacode/aima-glossary . The full-batch interpretation is explicit, and the quadratic example is original and hand-checked.

Previous: Batch active learning Next: Batch inference