AI Glossary / AI and machine learning / Advanced
Iteratively Reweighted Least Squares
Definition
Iteratively Reweighted Least Squares is a fitting method that solves certain regression problems by repeatedly solving a weighted least-squares problem, updating the weights each round until the solution settles.
What it means
Some fitting problems resist a direct solution: logistic regression has no tidy formula for its best weights the way ordinary linear regression does. IRLS is the classic workaround, reaching the answer through a sequence of solvable approximations.
Each round pretends the problem is an ordinary weighted regression, which has a closed-form answer, solves it, then recomputes the weights from the current solution's errors. Rounds continue until the answer stops moving.
The method is the standard engine inside generalized linear models: when your statistics package fits a logistic regression in milliseconds, IRLS or a close relative is usually doing the work.
Its behavior is mostly well-mannered, fast and reliable for the problems it was designed for, but it can wobble when data is sparse in the wrong places, such as classes that separate perfectly, where the weights it computes can explode.
For a business, this is plumbing: you will never operate it directly, but knowing the name helps when a model report mentions convergence failures, which are usually this loop failing to settle and always worth a question.
The reweighting is where the name earns itself: observations are not all treated alike, and the weight given to each is revised as the current answer improves, so the approximation shadows the true problem a little more closely with every round.
How it works
Start with an initial guess at the model's coefficients, often zeros or a cheap estimate.
Compute, from the current guess, the working weights and adjusted targets of an approximate least-squares problem.
Solve that weighted regression exactly, producing a new guess.
Repeat until successive guesses stop changing within tolerance, declaring convergence.
If the guesses diverge or oscillate, diagnose separability or collinearity in the data rather than blindly running longer.
Three practical examples
- A statistics package fitting a bank's logistic scorecard via IRLS in a fraction of a second.
- A convergence warning in a risk-model fit, traced to one field that perfectly separated defaulters from survivors.
- A textbook generalized linear model, logit, probit, Poisson, whose coefficients all came out of the same IRLS loop.
Think of it this way
Think of weighing yourself on a scale that reads slightly wrong, then adjusting your stance based on the reading, weighing again, adjusting again. Each weighing uses the last result, and after a few rounds you stop moving: the reading and the stance agree. IRLS is that loop, with matrices.
The limit: if the scale is broken in a systematic way, the loop settles confidently on nonsense. Convergence proves the equations agree, not that the problem was well-posed, which is why the diagnostics still matter.
Worked example
the scorecard that would not settle
Suppose a lender fits a new scorecard and the software reports failure to converge. The figures are illustrative.
The cause turns out to be one variable, an internal flag, that alone almost perfectly separates good loans from bad, sending its coefficient climbing without bound across iterations.
Removing the leakage-flagged variable lets the loop settle in eight iterations, with sensible, stable weights.
The incident earns a standing rule: a convergence failure is a data question first, not a compute problem.
Illustrative case study
A fictional analyst hit a convergence warning, raised the iteration limit until the warning disappeared, and shipped the resulting model with a coefficient of several thousand on one field.
The model scored absurdly on a handful of live cases before anyone checked, and the rollback cost more than the diagnosis would have.
The invented lesson: silence is not convergence. Forcing the loop to stop complaining is not the same as the loop finding a stable answer.
Common misconceptions and mistakes
- It is an alternative to regression. It is a method for fitting certain regressions; the model is unchanged, only the engine that finds its coefficients.
- Convergence means correctness. It means the loop stabilized; the stability of a badly posed problem is still a bad answer.
- Iteration limits fix convergence problems. Raising the limit helps only when the loop was merely slow; systematic problems need data fixes.
Business use
Treat convergence warnings in model fits as review triggers, not noise: they usually announce a data problem.
Ask which fields could separate the outcome almost perfectly, since those are the classic cause of IRLS blow-ups and often leakage.
Keep the fitting diagnostics with the model record: iteration counts and warnings are cheap context for future reviewers.
People also ask
What does IRLS stand for?
Iteratively Reweighted Least Squares: repeatedly solving a weighted least-squares problem with weights updated from the current solution.
Which models use it?
Generalized linear models, most famously logistic regression, where no closed-form solution exists.
What causes it to fail?
Perfect or near-perfect separation, extreme collinearity or sparse data in key regions, all data problems rather than engine problems.
Is IRLS the same as gradient descent?
No; both iterate, but IRLS jumps to the exact solution of each weighted approximation, while gradient descent takes small steps along the slope.
Check your understanding
Why does Iteratively Reweighted Least Squares use a sequence of approximations instead of solving directly?
Answer: Because models like logistic regression have no closed-form solution, so the method converts the problem into a series of weighted regressions, each of which does have one, and repeats until the answer settles.
Related terms
Learn next
AI for Business Owners explains how to read model diagnostics and know which warnings deserve a question: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]
More in AI and machine learning
- Inverse-Wishart distribution - The inverse-Wishart distribution is a probability distribution over real symmetric positive-definite matrices.
- Inverted Dirichlet distribution - The inverted Dirichlet distribution is a multivariate probability distribution over positive quantities.
- Isotropic position - Isotropic position describes a normalization in which a vector distribution has equal second-moment spread in every direction.
- Jackknife variance estimates for random forest - Jackknife variance estimates for random forests are methods for estimating how much a forest's prediction would vary across repeated training samples.
- Jarvis&Patrick clustering - Jarvis-Patrick clustering is a clustering method that groups items by shared near neighbors: two items belong together when they appear high on each other's...
- Jensen bound integration - Jensen bound integration is a technique for approximating an average that is hard to compute exactly: it uses Jensen's inequality, which relates the average...
Sources
[S13] Minka, Statistical Learning/Pattern Recognition Glossary. Full URLs are in the source ledger. Verified 11 October 2026.
Previous: Iterative soft thresholding algorithm Next: Jaccard similarity coefficient score