Are iterative methods always faster?

No. Performance depends on matrix structure, accuracy requirements, hardware, and setup costs.

Can an iterative method give a highly accurate answer?

Yes, when its convergence and numerical behavior support the required tolerance.

Does a small residual prove a small solution error?

Not by itself; conditioning and the relevant error measure matter.

AI Glossary / Mathematics and optimization / Intermediate

Iterative method

Definition

An iterative method is a numerical procedure that constructs a sequence of approximate answers by repeatedly applying an update rule. It stops when a suitable accuracy criterion or computation limit is reached. Iterative methods appear in equation solving, optimization, eigenvalue calculations, and many other numerical tasks. Repetition alone does not ensure convergence or a correct answer.

What it means

Instead of completing one predetermined sequence of elimination operations, an iterative solver starts with an estimate and uses the problem's structure to revise it. A useful method balances the cost of each revision against the number of revisions needed.
The word iterative describes a broad approach, not a particular level of accuracy. Some iterations converge quickly, some slowly, and some diverge. Even a mathematically convergent method can struggle because of finite precision, poor scaling, an unsuitable initial estimate, or an ineffective stopping rule.

How it works

Specify the problem, an initial estimate, and an update rule. For a fixed-point formulation, the rule may be written x_next=G(x_current). In linear-system solvers, updates are often constructed from matrix operations and a residual, the difference between the requested right-hand side and the one produced by the current estimate.
After each update, assess progress using a criterion appropriate to the task. Include iteration or time limits and report why the method stopped. Large sparse problems often benefit from methods that use matrix-vector products without explicitly forming dense factorizations. Preconditioning can improve convergence, but its setup and application also cost resources.

Three practical examples

An engineering simulation can solve a large sparse linear system arising from a discretized physical model.
A fitting routine can iteratively update parameters until an optimization stopping criterion is met.
A scientific application can approximate an eigenvector through repeated matrix operations, while checking the associated eigenvalue residual and the method's assumptions.

Think of it this way

Think of adjusting a measurement estimate after each new consistency check. Each revision aims to reduce the mismatch, but a faulty adjustment rule can overshoot or move away from the target. Unlike casual trial and error, a numerical method requires a specified update and evidence about when it should work.

Worked example

Consider an original two-equation system: 4x+y=9 and x+3y=7. The exact solution is x=20/11 and y=19/11, approximately (1.818182,1.727273). Use a Jacobi iteration, where both new coordinates are calculated from the old coordinates: x_next=(9-y)/4 and y_next=(7-x)/3.
Start at (0,0). The first estimate is (2.25,2.333333). Using those old values together gives the second estimate (1.666667,1.583333). The third is (1.854167,1.777778), and the fourth is approximately (1.805556,1.715278).
For the fourth estimate, compute the residual r=(9-4x-y,7-x-3y). Its components are approximately (0.0625,0.048611). The largest absolute residual has fallen from 9 initially to 0.0625. The largest coordinate error is approximately 0.012626, which is different from the residual magnitude.
The convergence can be checked directly here. If the current errors are ex and ey, the next errors are -ey/4 and -ex/3. After two updates, each original error is multiplied by 1/12. Consequently both errors approach zero from any starting estimate in exact arithmetic. This is a property of this specific iteration and matrix, not a promise for every Jacobi system.
Now compare a separate example that exposes a stopping-rule trap. Let A be the diagonal matrix with entries 1 and 0.000001, and let b=(1,0.000001). The exact solution is (1,1), but the estimate (1,0) has residual (0,0.000001) and coordinate error (0,-1). A small residual can coexist with a large error when the system is sensitive in a particular direction.
Finally, the scalar equation x=2 can be represented by the update x_next=2x-2. Its fixed point is indeed 2, but starting at 0 produces -2, then -6, then -14. A correct fixed-point equation is not enough: the chosen update must have suitable convergence behavior.

Illustrative case study

In a fictional heat-transfer project, Meadow Systems compares iterative solvers on the same discretized model. The team records setup time, iteration time, memory use, residual histories, and whether a preconditioner was applied. It checks small instances against an independently computed solution.
One run reaches its iteration limit with an acceptable-looking plot but a residual above the specified tolerance. The analyst marks it as incomplete and investigates scaling before using its outputs. The case illustrates verification practice and makes no claim of measured commercial performance.

Common misconceptions and mistakes

Do not equate iteration count with accuracy across different methods. One expensive update may accomplish more than many cheap ones. A tiny change between successive estimates can mean convergence, but it can also mean numerical stagnation.
Residual and solution error are related through the problem, not interchangeable. Likewise, a decreasing residual in a few early steps does not establish eventual convergence. Direct methods also use loops in their implementations; the numerical distinction concerns how an answer is obtained, not whether computer code contains repetition.

Business use

Business-facing evaluations should state the accuracy required for the downstream decision, rather than demanding an arbitrary number of decimal places. Compare methods on representative problem sizes, include preconditioner costs, and retain termination reasons. If a simulation feeds an optimization or forecast, determine whether solver error is small enough relative to modeling uncertainty and decision thresholds. Faster approximate computation is useful only when its limitations remain visible.

Check your understanding

Why does the diagonal-matrix example challenge a residual-only interpretation?

Answer: Its residual is only 0.000001 in the second coordinate, yet that coordinate's solution error has magnitude 1 because the matrix scales it by 0.000001.

Related terms

Convergence
Residual
Conditioning
Fixed-point iteration
Jacobi method
Gauss-Seidel method
Krylov subspace
Preconditioner
Stopping criterion
Numerical stability

Learn next

Compute two more Jacobi updates for the two-equation example and compare the errors with those from two steps earlier. Then change the initial estimate to (10,-10) and explain why the same two-step error factor still applies in exact arithmetic.

Sources

Primary technical source: Barrett and coauthors, Templates for the Solution of Linear Systems, iterative-methods chapter: https://www.netlib.org/linalg/old_html_templates/chapter2.6.html . The linear systems, convergence derivation, residual counterexample, and divergent scalar update are original.

Previous: Iterative expansion Next: Iterative rational Krylov algorithm