Are bounds a form of constraint?

Yes. They are simple because their linear representation is exact.

Does linearized feasibility guarantee nonlinear feasibility?

No. The circle example demonstrates the difference.

Is this the same as generalized least squares?

No. Generalized least squares primarily concerns accounting for error covariance in statistical fitting, not this constrained nonlinear update procedure.

AI Glossary / Mathematics and optimization / Advanced

Generalized Gauss - Newton method

Definition

In the constrained least-squares sense used here, the generalized Gauss - Newton method extends Gauss - Newton fitting to problems with equality or inequality constraints. Each iteration solves a local least-squares approximation subject to linearized constraints, then uses an appropriate update strategy.

What it means

A good fit may also need to respect physical limits, balance equations or parameter bounds. Treating those requirements inside the optimization differs from finding an unconstrained fit and adjusting the answer afterward.

The name is not universal. Some machine-learning literature uses generalized Gauss - Newton for a curvature approximation involving a convex loss composed with a nonlinear model. The constrained least-squares version is also called constrained Gauss - Newton. Always identify the mathematical problem and update rule instead of relying on the acronym alone.

How it works

Let r(x) be the residual vector and minimize one-half its squared length, subject to c(x) = 0 and h(x) ≥ 0. At the current estimate, approximate the residual by r(x) + J_r p. Approximate the constraints by c(x) + J_c p = 0 and h(x) + J_h p ≥ 0.

The resulting subproblem minimizes a quadratic least-squares objective with linear constraints. Its solution proposes a parameter step p. Bounds that are already linear need no further approximation.

Nonlinear constraints are only locally represented. An update can satisfy the linearized equations while violating the original equations. Practical implementations therefore need suitable step acceptance, feasibility handling and stopping checks. Convergence depends on assumptions about derivatives, constraint regularity and the local problem; global optimality is not automatic.

Three practical examples

1. Calibration with bounds: fit a nonlinear response model while keeping a gain within its permitted range. The bound participates in calculating the step rather than being applied only after fitting.

2. Conservation-based estimation: estimate parameters while enforcing a balance relationship. Linearizing that relationship gives a local constraint, which must still be checked in its original form after an update.

3. Dynamic-system fitting: estimate model parameters from observations subject to equations linking successive states. A structured constrained formulation can represent both the measurement discrepancies and the model requirements.

Think of it this way

Imagine adjusting a recipe to match a target taste while respecting a fixed ingredient budget. A local approximation estimates how changes affect taste, while separate equations describe permitted changes. The suggested adjustment is useful only if it also behaves acceptably when tested against the real recipe and its actual restrictions.

Worked example

Consider a fitting problem with residuals r_1(x) = x squared − 4 and r_2(x) = x − 2. Minimize phi(x) = [r_1(x) squared + r_2(x) squared] / 2, subject to 0 ≤ x ≤ 1.5.

Start at x = 1. The residual vector is (−3, −1), and its Jacobian is (2, 1). The local objective for the step is [(-3 + 2p) squared + (-1 + p) squared] / 2. The bounds require −1 ≤ p ≤ 0.5.

Without the bounds, differentiating the local objective gives 5p − 7 = 0, so p = 1.4. That step would produce x = 2.4 and violate the upper bound. Over the permitted step interval, the local objective is still decreasing at p = 0.5, so the constrained subproblem chooses p = 0.5.

The new estimate is x = 1.5. Its residuals are −1.75 and −0.5, giving actual objective (3.0625 + 0.25)/2 = 1.65625. This improves on the starting objective of 5 while satisfying both bounds.

At x = 1.5, the unconstrained objective derivative is 2x(x squared − 4) + (x − 2) = −5.75. It is not zero. Nevertheless, moving in the direction that decreases the objective would require increasing x beyond its bound. The active upper constraint therefore explains why an unconstrained stationarity test would be inappropriate.

For a nonlinear-constraint check, consider x squared + y squared = 1 at (1, 0). Its linearized equation requires 2p_x = 0. The step (0, 0.2) satisfies that equation, but the updated point gives 1 squared + 0.2 squared = 1.04. Linearized feasibility has not preserved the original equality.

Illustrative case study

In a fictional equipment-modeling project, a team fits parameters that must obey an operating bound and a nonlinear balance equation. An early prototype reports success whenever the residual sum decreases and the local subproblem is solved.

Testing reveals that some accepted updates violate the original balance equation despite satisfying its linearization. The team adds explicit checks of the original constraints, a suitable step-control strategy and stopping criteria that account for active bounds.

The revised report separates fitting error from constraint violation. This distinction helps reviewers judge whether the estimated model is both useful and consistent with the requirements it was intended to represent.

Common misconceptions and mistakes

A constrained optimum need not have a zero unconstrained gradient. Active constraints can prevent further descent along otherwise attractive directions.

Solving the local constrained subproblem does not solve the original nonlinear problem in one iteration. The residual and constraint approximations must be updated.

Clipping an unconstrained answer to bounds is not a general substitute for constrained optimization. With several coupled parameters or constraints, the correct adjustment can change more than the offending coordinate.

Do not assume all papers using generalized Gauss - Newton describe the same algorithm. Distinguish the constrained least-squares formulation from generalized curvature methods for other losses.

Business use

This approach is relevant when a fitted model must respect operational or scientific requirements. The constraints should express genuine requirements, with units, tolerances and feasible ranges documented clearly.

Evaluation should report original constraint violations, residual quality, sensitivity to starting values and the reliability of derivative calculations. A low fitting error does not compensate for breaking a required balance, and an exactly feasible estimate can still fit observations poorly.

Check your understanding

Why is the unconstrained step rejected in the worked example?

Answer: It would move x to 2.4, exceeding 1.5.

Answer: What step does the constrained subproblem choose? p = 0.5, reaching the upper bound.

What must be checked after a step involving nonlinear constraints?

Answer: The original constraint values, not only their linear approximations.

Related terms

Gauss - Newton algorithm
Constrained least squares
Jacobian
Levenberg - Marquardt algorithm
sequential quadratic programming
active constraint
feasibility

Learn next

Derive the scalar constrained step and compare its stationarity conditions with the unconstrained problem. Then examine a two-parameter example with a nonlinear equality and test how step control affects constraint violation.

Sources

Florian Messerer, Katrin Baumgärtner and Moritz Diehl, Survey of Sequential Convex Programming and Generalized Gauss-Newton Methods, 2021, naming discussion and constrained formulations: https://publications.syscop.de/Messerer2021a.pdf

The bounded fitting calculation, circle check and equipment case are original illustrations.

Previous: Generalized Few-shot Learning Next: Generalized Hebbian algorithm