Does SGD require a neural network?

No. Many linear and other differentiable models can be trained with it.

Is one epoch the same as one update?

Usually not. An epoch typically processes the training data once, requiring many mini-batch updates.

Are momentum and Adam identical to basic SGD?

No. They modify the update mechanism or maintain additional state, although they also use sampled gradients in common applications.

AI Glossary / Mathematics and optimization / Intermediate

Stochastic gradient descent

Definition

Stochastic gradient descent, or SGD, is an optimization method that updates parameters using a randomly sampled estimate of the objective's gradient. It avoids computing the full dataset gradient before every update, making many large training problems more manageable.

What it means

A gradient indicates how an objective changes with the model parameters. Ordinary full-batch gradient descent uses all training observations to compute that direction. SGD estimates it from one sampled observation; mini-batch SGD averages a small sampled group. The broader name SGD is often used for both.

The estimated direction is noisy: it can favor the sampled observations while temporarily worsening the full objective. Its usefulness comes from how repeated, appropriately controlled updates behave, not from a guarantee that every step improves every example.

SGD is an optimizer, not a particular model. It can train different models with different losses. A linear classifier fitted using SGD remains a linear classifier; a neural network does not become a linear model because it uses the same update principle.

How it works

Write the empirical objective as F(w) = (1/n) Σℓi(w). At step t, sample an index i and compute gi = ∇ℓi(wt). The basic update is w(t+1) = wt − ηt gi, where ηt is the learning rate.

With uniform independent sampling at the current parameters, the expected sampled gradient equals ∇F(wt). For a mini-batch, average its gradients using a normalization consistent with the objective. Nonuniform sampling may require weights if the goal is to preserve the same expected gradient.

A learning-rate schedule controls step sizes. Large steps can move quickly but overshoot; very small steps can waste computation. Constant steps can leave persistent variation near an optimum, while suitable decreasing schedules support convergence results under additional assumptions.

Shuffling data each epoch is common, but sampling without replacement is not identical to fresh independent sampling at every step. Feature scaling, regularization and stopping rules also influence practical behavior.

Three practical examples

A text classifier updates a linear model from small batches of sparse document features.
A neural-network experiment computes gradients from mini-batches that fit the available memory.
An incremental learning system processes arriving observations, while separately addressing whether the incoming distribution is changing.

Think of it this way

Think of estimating the slope of a large landscape from a few local measurements before taking a step. Each estimate is cheaper, but it can point somewhat differently from a more comprehensive measurement.

The analogy explains noisy guidance. Parameter spaces can have many dimensions, and mathematical convergence depends on assumptions that a physical hillside does not represent.

Worked example

Two examples produce two different steps

Fit a constant w to targets 0 and 4 using individual losses ℓ1(w) = (w − 0)²/2 and ℓ2(w) = (w − 4)²/2. The average objective is F(w) = [w² + (w − 4)²]/4, minimized at w = 2.

Start at w = 0 and use learning rate 0.5. The individual gradients are 0 and −4. Sampling the first observation gives no movement; sampling the second gives w_new = 2. Their average gradient is −2, so a full-batch step would give w_new = 1.

The expected sampled update is also 1, but no individual sampled step in this setup produces 1. Moreover, expected objective values differ: F(0) = 4 and F(2) = 2, so the expected objective after the sampled step is 3. The full-gradient step has F(1) = 2.5.

At the optimum w = 2, individual gradients are 2 and −2. A sampled step with the same learning rate moves to 1 or 3, increasing F from 2 to 2.5. This explains why noisy constant-step iterates need not settle exactly at the optimum.

Illustrative case study

A fictional demand-modeling team changes from full-dataset updates to mini-batches. Its training curve now fluctuates, and a reviewer initially interprets every upward movement as a software defect.

The team separates batch loss from loss measured on a fixed evaluation set. It reviews learning-rate settings, batch normalization conventions and feature scaling, then compares time to reach a defined validation target.

It also records multiple runs because sampling order affects the trajectory. The final assessment recognizes that some fluctuation is expected while sustained divergence still requires investigation. The lesson is to evaluate the optimizer's progress at an appropriate scale rather than demanding that every sampled update improve the full objective.

Common misconceptions and mistakes

SGD guarantees a lower objective at every step. The sampled direction can temporarily increase it.
An unbiased gradient gives an exact gradient each time. Unbiasedness concerns an average over sampling.
Mini-batch size only affects memory. It also changes gradient variability and update frequency.
SGD is a model architecture. It is a training method.
Convergence always means a global optimum. Guarantees depend on properties such as convexity and the step-size assumptions.

Business use

SGD is useful when full gradients are expensive or observations arrive incrementally. Compare computational cost and learning progress together. A cheaper update is valuable only if enough useful progress accumulates to meet the task's quality requirement.

Record loss normalization, batch size, sampling procedure and learning-rate schedule so that results are interpretable. Monitor validation performance separately from optimization loss. When comparing batch sizes, be explicit about whether the budget holds examples processed, updates, elapsed time or total compute constant; those comparisons answer different questions.

Check your understanding

Why can SGD move away from the optimum in the worked example?

Answer: The full gradient is zero there, but each individual observation has a nonzero gradient. Sampling one observation does not cancel the opposing contributions.

Related terms

Learn next

Read learning-rate schedules and gradient accumulation. Rework the example using both observations in one mini-batch, then explain how averaging restores the full gradient and what computational tradeoff that creates in a large dataset.

Sources

Aalto Dictionary of Machine Learning, stochastic gradient descent: https://aaltodictionaryofml.github.io/ADictML_Main.pdf. Sampling-based gradient approximation.

Scikit-learn, Stochastic Gradient Descent guide: https://scikit-learn.org/stable/modules/sgd.html. Optimizer distinction and implementation considerations. Calculations and the fictional demand-model case are original examples.

Previous: Stochastic games Next: Stochastic gradient Hamiltonian Monte Carlo