Why not fit first and fix invalid values afterward?

Post-processing can break the optimal balance among coupled variables and may violate other constraints. Solving the constrained problem handles those relationships together.

Are all constrained least-squares problems convex?

No. Linear models with convex constraints have useful convex structure, but nonlinear or nonconvex restrictions can change the problem substantially.

What if no point satisfies all constraints?

The problem is infeasible and the requirements or model need review; a valid solution should not be invented.

AI Glossary / Computer vision and multimodal AI / Advanced

Constrained least squares

Definition

Constrained least squares finds model parameters that minimize a sum of squared residuals while obeying specified restrictions. The constraints may impose bounds, equalities or other admissible relationships, allowing a fitted result to respect requirements that ordinary unconstrained least squares could violate.

What it means

Least squares chooses values that make predictions close to observations according to squared error. But the numerically closest fit may be unusable: an estimated physical intensity could be negative, mixture proportions might not add to one, or a parameter may exceed an allowed range. Constraints define which solutions are acceptable before the best fit is selected. This is different from simply clipping an unconstrained answer afterward, because constrained optimization considers how the variables interact within the feasible set. It also differs from regularization, which usually adds a preference or penalty rather than imposing an absolute requirement.

How it works

In a linear model, the objective often takes the form minimize ||Ax−b||², where A maps parameters x to predicted measurements and b contains observations. Constraints might require x≥0, lower and upper bounds, or Cx=d. With linear equality and inequality constraints, the problem remains a convex quadratic optimization problem. That structure supports reliable global optimization when the problem is feasible, though uniqueness depends on additional conditions.
The solver balances residuals while remaining within the permitted region. Some constraints may be active at the solution, meaning their boundary is reached. If the requirements conflict, no feasible solution exists. Numerical tolerances, scaling and data quality affect how the computed result should be interpreted.

Three practical examples

An image-reconstruction method estimates nonnegative intensities instead of allowing negative values that have no meaning in its physical model.
A material-mixture analysis fits component proportions subject to nonnegativity and a sum-to-one condition, provided that the mixture model itself is appropriate.
A calibration routine fits parameters within known operating bounds and reports when the data push the optimum against those bounds.

Think of it this way

Think of fitting furniture into a room while keeping doors clear. The preferred arrangement minimizes inconvenience, but it must still obey the room's physical restrictions. The analogy is limited because least squares uses a specific numerical error measure, and constraints can interact in ways that are not obvious from considering each variable separately.

Worked example

Suppose a single parameter x is fitted to observations 2 and 4 by minimizing (x−2)²+(x−4)². The unconstrained optimum is their mean, x=3, with total squared error 2. Now impose x≤2.5. The constrained optimum is x=2.5, with error 0.5²+(−1.5)²=2.5. The best permitted fit is worse numerically than the unconstrained fit but satisfies the requirement.
This one-variable example can be understood as clipping because the geometry is simple. In a coupled multivariable problem, independently clipping an unconstrained solution generally does not produce the constrained optimum. The optimization must account for how changing one parameter affects all predictions and constraints.

Illustrative case study

Fictional example: Cedar Spectral fits three material contributions to sensor readings. An unconstrained fit assigns one material a negative share. The team checks whether the model and calibration are credible, then uses nonnegative, sum-to-one constraints where those assumptions are justified. It examines the remaining residuals rather than treating the constrained answer as automatically correct. A poor fit can still signal a missing material or an unsuitable model. The case demonstrates physically motivated restrictions, not proof that every mixture can be explained by the chosen components.

Common misconceptions and mistakes

Constraints do not make an incorrect model true. Nonnegative coefficients are meaningful only when the variables and model justify them. A feasible solution is not necessarily unique. A solver reporting success does not remove the need to inspect residuals and numerical tolerances. Regularization and hard constraints are related tools but are not interchangeable in every formulation. Finally, setting arbitrary bounds merely to produce plausible-looking values can conceal a data or modeling problem rather than solve it.

Business use

Businesses using fitted models should document why each constraint exists and what happens when it becomes active. Separate physical requirements from business preferences and numerical conveniences. Check feasibility before relying on the output, and retain diagnostics for residuals and bound violations within tolerance. Compare the constrained model with suitable reference data. When parameters repeatedly sit at limits, investigate whether the limits are correct or the model is compensating for missing effects. Explain to decision-makers that a constrained estimate is the best fit within a declared set of assumptions.

Check your understanding

A parameter reaches its upper bound in the fitted solution. Does this prove the bound is the true physical value?

Answer: No. It shows that the constraint is limiting the fitted result. The data, model and bound should be examined before drawing a physical conclusion.

Related terms

Least squares
Residual
Convex optimization
Regularization
nonnegative least squares
feasible set
active constraint

Learn next

Learn ordinary least squares and residual plots, then solve a one-variable bounded example. Study convex feasible sets and compare hard constraints with penalties before fitting coupled physical or business models.

Sources

CVXPY, introductory constrained least-squares example: https://www.cvxpy.org/ CVXPY, Least-squares example: https://www.cvxpy.org/examples/basic/least_squares.html Official optimization documentation excerpts consulted; full solver-specific and editorial review remain pending. Worked arithmetic and the fictional case are original.

Previous: Constrained Hidden Markov Model Next: Constrained minimization