Must residuals be linear?

No. Specialized nonlinear least-squares solvers handle nonlinear residual functions.

Does a success flag prove a global optimum?

No. It usually describes satisfaction of a termination condition within the solver's local process.

Can parameter bounds change the best fit?

Yes. They restrict the feasible set and can exclude the unconstrained minimizer.

AI Glossary / Mathematics and optimization / Intermediate

Least-squares minimization

Definition

Least-squares minimization is the process of finding parameters that make a collection of residuals as small as possible under a squared-error objective. In numerical optimization, it often means solving a nonlinear residual-fitting problem, possibly with parameter bounds. The objective, constraints, starting point, and stopping conditions all affect what a solver returns.

What it means

The least-squares principle defines what is being minimized; minimization describes the computational attempt to find suitable parameters. A residual function reports the discrepancy for each observation. A specialized solver can exploit this collection of residuals instead of treating the problem as an arbitrary scalar objective.
In the SciPy context associated with this term, least_squares accepts residuals and supports bounded nonlinear problems. Ordinary least squares corresponds to the standard squared-residual loss; optional robust losses change how large residuals influence the objective.
For nonlinear models, a successful local solve is not a certificate of the best possible global fit. A stopping message also does not establish that the resulting parameters are physically meaningful, identifiable from the available data, or useful for future predictions.

How it works

Define residuals r₁(θ),…,rₘ(θ). A common ordinary objective is F(θ)=½Σrᵢ(θ)². The one-half factor simplifies derivatives without changing the minimizers. For differentiable residuals, the gradient is Jᵀr, where J contains the derivatives of residuals with respect to parameters.
Iterative methods use local information to propose parameter changes. A Gauss-Newton approximation models how the residual vector changes through its Jacobian; trust-region or damping strategies can limit unreliable steps. Bounds restrict the permitted parameter region.
Choose residual scaling deliberately. Dividing a residual by a meaningful scale changes its relative contribution. Also distinguish scaling the parameters, which can help numerical progress, from changing the residual weights, which changes the fitting objective. After termination, inspect the residuals and constraints as well as the reported status.

Three practical examples

A calibration model can estimate nonlinear response parameters from discrepancies between predicted and reference measurements.
A curve-fitting task can estimate a decay rate and amplitude while restricting parameters to a meaningful domain.
A reconstruction problem can combine many measurement residuals into one fit, using sparse derivatives when each measurement depends on only a few unknowns.

Think of it this way

Think of adjusting several controls until a collection of mismatches is reduced. A local guide indicates promising adjustments, but it may lead to one favorable setting among several. The analogy is limited: numerical algorithms have explicit objectives and stopping rules, and an apparently quiet set of controls can still be at a poor stationary point.

Worked example

Consider an original one-parameter nonlinear example with residuals r₁(θ)=θ²−1 and r₂(θ)=θ²−3. The ordinary half-squared objective is F(θ)=½[(θ²−1)²+(θ²−3)²]=θ⁴−4θ²+5.
Rewrite it as F(θ)=(θ²−2)²+1. Without bounds, the global minimum is 1, achieved at both θ=√2 and θ=−√2. The two parameter values give identical predictions θ²=2. The data therefore cannot distinguish their signs in this model.
The derivative is F′(θ)=4θ(θ²−2). At θ=0 it is zero, but F(0)=5. For a small nonzero θ, the objective decreases because the negative quadratic term dominates the quartic term. Thus θ=0 is a stationary local maximum, illustrating why a small gradient alone does not prove a minimum.
Now impose 0≤θ≤1. On the interior of this interval the derivative is negative, so the constrained optimum is θ=1. Its residuals are 0 and −2, giving F=½(0+4)=2. The unconstrained minimizers are infeasible. A solver that reports the bounded answer should therefore not be judged against the impossible target cost of 1 without considering the constraints.
These conclusions follow algebraically; they are not a report of a particular software run. The example separates objective value, stationarity, parameter ambiguity, and feasibility - four issues that can otherwise be hidden behind one success flag.

Illustrative case study

In a fictional sensor-development team, an engineer fits a nonlinear response curve to reference readings. Several starting points produce similarly small residual costs but noticeably different parameter values. The team investigates whether the measurements cover enough of the response range to distinguish those parameters.
It reviews residual plots and repeats the comparison on additional reference conditions before relying on the curve. The hypothetical workflow treats numerical minimization as one part of calibration evidence, rather than presenting a solver's completion message as proof of a uniquely determined model.

Common misconceptions and mistakes

Do not pass already squared residuals to an interface that expects raw residuals unless a fourth-power penalty is actually intended. Do not confuse the solver's half-sum cost with an unscaled sum of squares or a mean squared error.
Bounds can place a valid optimum where the unconstrained gradient is nonzero. Conversely, an unconstrained zero gradient may identify a maximum or saddle. Robust-loss options are not merely faster ways to obtain the same ordinary least-squares answer: they change the objective. Poor derivative information and badly scaled variables can also make a useful mathematical formulation difficult to solve numerically.

Business use

A reviewable fitting workflow records the residual definition, units, parameter bounds, starting values, loss choice, and termination reason. Report enough information to distinguish a low residual cost from a credible parameter estimate. If several solutions fit equally well, discuss whether their predictions differ in the operating range.
For consequential forecasts or calibration decisions, evaluate the fitted model on suitable observations outside the fitting set. Tightening numerical tolerances can improve optimization precision while leaving model mismatch and measurement uncertainty unchanged. The business question is whether the resulting fit supports the intended use, not simply whether another decimal place can be obtained.

Check your understanding

A solver expects residuals r₁ and r₂, but receives r₁² and r₂². What ordinary objective does it minimize?

Answer: One-half of r₁⁴+r₂⁴, rather than one-half of r₁²+r₂².

Related terms

Least squares
Residual
Jacobian
Trust region
Gauss-Newton method
Levenberg-Marquardt algorithm
parameter bounds
robust loss
identifiability

Learn next

Next, work through one nonlinear residual function and its Jacobian by hand. Then compare unconstrained and bounded fits, inspecting residuals and feasible parameters before interpreting the solver's stopping message.

Sources

Primary software reference: SciPy, scipy.optimize.least_squares, official documentation for residual inputs, bounds, loss functions, methods, and termination: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html . The polynomial objective and constrained solution in this article are original algebraic examples.

Previous: Least squares estimate Next: Least squares regression