Does every iteration need to move a point back to the boundary?

No. A proposed point already inside C projects to itself.

Can the optimum have a nonzero gradient?

Yes. Constraints can block every improving feasible movement.

Is it related to proximal gradient?

Yes. Projection is the proximal operator of a convex-set indicator function, making projected gradient a special case.

AI Glossary / Mathematics and optimization / Advanced

Projected gradient descent

Definition

Projected gradient descent is an optimization method that combines a gradient step with a projection onto a feasible set. The gradient step seeks to reduce a differentiable objective; the projection returns the proposed point to the allowed region. It is useful when constraints can be expressed through a set whose nearest-point projection is tractable.

What it means

Ordinary gradient descent can propose values that violate a model's restrictions. Nonnegative coefficients, bounded settings, or probabilities summing to one are common examples. Projected gradient descent incorporates these restrictions into each update rather than treating them as an afterthought.

In its usual Euclidean form, “projection” means finding the closest feasible point to the proposed update. It is not any convenient repair that happens to restore validity. Different feasible sets have different projection formulas, and the correct projection may itself require an optimization calculation.

How it works

For an objective f, current point w, feasible set C, and positive step size η, compute v = w − η∇f(w). Then set the next iterate to projection_C(v).

A nonempty closed convex set has a unique Euclidean projection. For a convex differentiable objective with a Lipschitz-continuous gradient, suitable step sizes support convergence results when a solution exists. A common conservative choice uses η no greater than the reciprocal of the gradient's Lipschitz constant; practical methods may use a suitable line search when that constant is unknown.

At a constrained optimum the ordinary gradient need not vanish. The gradient may point toward improvements that are prohibited by the constraint, while the projected update stays fixed. A projected-gradient residual can therefore be more informative than the raw gradient norm for checking constrained stationarity.

Three practical examples

1. Probability weights: optimize weights on a simplex so that they remain nonnegative and sum to one.

2. Bounded control settings: adjust continuous settings while respecting lower and upper limits represented by a box constraint.

3. Norm-limited models: train parameters inside a specified norm ball when an explicit size constraint is part of the model definition.

Think of it this way

Think of taking a step toward a lower point on a landscape, then returning to the nearest point inside a fenced area if the step crossed the fence. The analogy explains the two operations. It does not mean walking along the fence's surface in every case, nor does an irregular nonconvex fence retain the usual unique-projection guarantees.

Worked example

Minimize f(a, b) = ((a − 2)² + (b − 0.5)²)/2 subject to a ≥ 0, b ≥ 0, and a + b = 1. Start at (0.5, 0.5) and use step size 0.5.

The gradient is (a − 2, b − 0.5), so the first proposed point is (0.5, 0.5) − 0.5(−1.5, 0) = (1.25, 0.5). It is outside the simplex because its entries sum to 1.75.

For this point, subtracting 0.375 from both entries gives the Euclidean projection (0.875, 0.125). Both coordinates remain nonnegative and now sum to one. Simply dividing by 1.75 would instead give (5/7, 2/7), which is feasible but is not the nearest-point projection.

The next gradient proposal from (0.875, 0.125) is (1.4375, 0.3125). Its projection is (1, 0): the nearest point occurs at the endpoint of the simplex segment. The objective values at the initial point, first iterate, and second iterate are 1.125, 0.703125, and 0.625.

At (1, 0), the gradient is (−1, −0.5), not zero. Another half-step proposes (1.5, 0.25), which projects back to (1, 0). This is the constrained optimum. Substituting b = 1 − a verifies that the unconstrained minimizer along the line would have a = 1.25, outside the permitted interval [0, 1]. The calculations are original and hand-checked.

Illustrative case study

Fictional case: Larch Forecasting combines three prediction sources using nonnegative weights that must sum to one. Its first implementation normalizes every proposed update by its sum and calls that operation projection.

The team tests the operation on simple known vectors and discovers that normalization differs from Euclidean simplex projection. It replaces the update with the intended projection and checks both feasibility and the projected residual. Forecast quality is assessed on later observations, separately from mathematical correctness of the optimizer. The scenario does not claim that the corrected method automatically improves business outcomes.

Common misconceptions and mistakes

Coordinate clipping is the Euclidean projection onto a box, but it is not a universal projection algorithm. It does not enforce a sum-to-one condition by itself. Likewise, normalization can restore a probability sum while failing to give the nearest feasible vector.

A feasible iterate need not be optimal. A small change between iterates can also result from an unsuitable step size or numerical limitations. Nonconvex feasible sets can have multiple nearest points and require more careful analysis.

Projection does not erase the importance of scaling. Changing coordinate units changes Euclidean distance and therefore can change the projected point unless the formulation accounts for those units.

Business use

Use the method when constraints are explicit and projection is affordable relative to gradient evaluation. Compare the cost of each projection with the full training budget; simple-looking constraints can still make the operation expensive.

Record the feasible set, metric, step-size policy, and stopping criterion. Check constraints independently after loading or exporting a model, especially if downstream software rounds or transforms its parameter values.

Check your understanding

Why is normalizing a vector to sum to one not always a valid substitute for simplex projection?

Answer: Feasibility and nearest-point optimality are different requirements. Normalization may achieve the first without the second.

Related terms

Learn next

derive projection onto a box and a simplex, then compare raw-gradient and projected-gradient stopping checks at a boundary optimum.

Sources

Primary sources: Aalto Dictionary, projected gradient descent: https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; Parikh and Boyd, Proximal Algorithms, Section 4.2: https://web.stanford.edu/~boyd/papers/pdf/prox_algs.pdf . Relevant update and convergence passages were read. The simplex calculation and fictional case are original.

Previous: Progression planning Next: Projection filters