Is a large condition number a software error?

No. It can reflect an inherently sensitive mathematical problem even when the solver is stable.

Can a small residual hide a large solution error?

Yes. An ill-conditioned matrix can map a substantial solution error to a small residual.

Does multiplying the whole matrix by a nonzero constant improve conditioning?

Not for the standard norm-based matrix condition number; the scale factors cancel.

AI Glossary / Mathematics and optimization / Advanced

Condition number

Definition

A condition number measures how sensitive a mathematical problem's solution can be to changes in its input. For an invertible matrix A, a standard matrix condition number is ‖A‖‖A⁻¹‖ in a specified norm. In the Euclidean norm, it is the ratio of the largest to the smallest singular value.

What it means

The supplied glossary emphasizes a symmetric positive-definite matrix, where the Euclidean condition number equals its largest eigenvalue divided by its smallest eigenvalue. This entry includes that optimization setting while distinguishing the more general singular-value definition.

A condition number near one indicates comparatively balanced directions. A large number warns that some perturbations can be amplified much more strongly than others. It describes sensitivity inherent in the mathematical problem and representation, not simply the quality of the software used to solve it. A stable algorithm cannot remove uncertainty already present in sensitive input data.

How it works

For Ax=b with fixed invertible A, changing b changes the solution by A⁻¹ times that change. The matrix norm gives a bound on amplification. Relative solution sensitivity can be bounded using the condition number, with the relevant assumptions and norms stated.

For a positive-definite quadratic objective, eigenvalues describe curvature in different directions. A large ratio means a narrow, elongated landscape. Basic gradient descent must respect the steep direction when choosing a stable step size, which can make progress slow in a shallow direction.

Conditioning depends on the problem and its scaling. Preconditioning or suitable coordinate changes can make numerical solution easier. Regularization can also improve matrix conditioning, but it changes the estimation problem. None of these facts makes a large condition number an automatic verdict that every computed answer is unusable: the actual error depends on perturbation size, direction, and the algorithm.

Three practical examples

1. Linear regression: diagnose nearly dependent feature directions that can make fitted coefficients sensitive to small data changes.

2. Optimization: understand why a simple gradient method makes uneven progress across directions of a quadratic loss.

3. Numerical analysis: distinguish a small residual in an equation from a reliable solution when the underlying matrix is ill-conditioned.

Think of it this way

Think of a control panel with one dial that barely affects the reading and another that changes it dramatically. Recovering dial settings from small changes in the readings can be difficult because the sensitivities differ so much.

The analogy concerns mathematical sensitivity. It does not say the equipment is faulty, nor that every small input change causes the worst possible output change. A condition number describes a potential amplification pattern under a particular formulation.

Worked example

Let A be the diagonal matrix with diagonal entries 1 and 0.001. Its Euclidean condition number is 1/0.001=1,000. For b=(1,0), the exact solution is x=(1,0).

Now perturb b to b_new=(1,0.001). The change in b has length 0.001 relative to the original length 1, a relative change of 0.1%. The new solution is x_new=(1,1), so the solution change has length 1 relative to the original solution length 1, a relative change of 100%. This example attains an amplification factor of 1,000.

A residual can also look reassuring while solution error is large. For the original b, take the approximate solution (1,1). Its residual A(1,1)−b=(0,0.001) has length 0.001, although its error from the true solution has length 1.

For the separate quadratic f(w)=(w_1²+0.001w_2²)/2, gradient descent with step size 1 changes w_1 to zero immediately but multiplies w_2 by 0.999 each step. The same eigenvalue imbalance explains slow progress along the second direction. These are exact hand calculations for an idealized diagonal example.

Illustrative case study

Fictional case: an analytics team finds that tiny revisions to two highly similar input features produce large changes in regression coefficients. The predictions on familiar observations change less dramatically, leaving the team unsure which result to trust.

The team examines singular values and feature redundancy, checks scaling, and evaluates alternative formulations. It distinguishes coefficient interpretation from predictive performance. If regularization is introduced, the team documents its modeling effect and tests predictions separately. The investigation treats sensitivity as a property to understand rather than dismissing every coefficient change as a software bug.

Common misconceptions and mistakes

Do not compute a general matrix's condition number as a ratio of eigenvalue magnitudes without checking the required matrix properties. Singular values provide the Euclidean matrix measure in the general invertible case.

A small determinant alone is not a reliable measure of poor conditioning; multiplying every entry by a nonzero scalar leaves the standard condition number unchanged. Likewise, a small residual does not guarantee a small forward error. Finally, conditioning and algorithmic stability are distinct: one describes the problem, the other the numerical method's behavior.

Business use

Conditioning matters when teams need dependable coefficients, reproducible numerical solutions, or predictable optimization behavior. It helps explain why more decimal places in the output may not represent more trustworthy information.

When reporting sensitivity, specify the matrix, norm, scaling, and source of perturbations. Relate the potential amplification to realistic input uncertainty and to the decision being made. A large abstract number is less useful than a clear explanation of which outputs can change and whether those changes alter the business conclusion.

Check your understanding

A symmetric positive-definite matrix has smallest eigenvalue 2 and largest eigenvalue 50. What is its Euclidean condition number?

Answer: It is 50/2=25. The eigenvalue ratio applies because of the stated positive-definite symmetric setting.

Related terms

Learn next

compare forward error, residual, and backward error on a small linear system. Then investigate feature scaling and preconditioning, keeping numerical improvements distinct from changes to the statistical objective through regularization.

Sources

Aalto Dictionary of Machine Learning, condition number entry: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . NumPy, matrix condition-number definitions and norm options: https://numpy.org/doc/stable/reference/generated/numpy.linalg.cond.html . Relevant passages were read. Positive-definite and general-matrix cases are distinguished; perturbation and gradient calculations are original.

Previous: Condition-action rule Next: Conditional average treatment effect