Must the input-output relationship be a straight line?

It must be linear in the estimated coefficients, but features can include predetermined nonlinear transformations such as squared inputs.

Does RLS need all previous records for each update?

The standard recursion uses the current summary, although records may still be retained for validation and audit.

Is a forgetting factor always required?

No. Equal-weight estimation uses no forgetting; discounting is a separate choice for changing conditions.

AI Glossary / Probabilistic machine learning / Advanced

Recursive least squares

Definition

Recursive least squares is an algorithm for updating a least-squares parameter estimate as new observations arrive. It retains a compact summary of earlier measurements and adjusts the current estimate using each new prediction error, avoiding a complete refit over the full historical dataset at every step.

What it means

Suppose a model predicts y from a feature vector phi using y_hat = phi^T theta. Recursive least squares, often abbreviated RLS, updates the coefficient vector theta whenever another feature-and-target pair becomes available.
The word recursive refers to reusing the previous estimate and associated matrix, not to a recurrent neural network. For a fixed least-squares objective, a correctly initialized recursion can reproduce the corresponding batch solution in exact arithmetic. Variants with a forgetting factor deliberately change the objective to emphasize recent observations, which can help track changing relationships.

How it works

For the ordinary unit-weight update, let P be the inverse accumulated feature-information matrix and theta the current coefficient estimate. Given a new feature vector phi and target y, compute gain K = P phi/(1 + phi^T P phi), prediction error e = y - phi^T theta, and updated coefficients theta_new = theta + K e. Update the matrix as P_new = P - K phi^T P.
A positive-definite initial P can represent a regularizing quadratic penalty. With theta_0 = 0 and P_0 = I, the recursion minimizes the sum of squared residuals plus theta^T theta, rather than unregularized least squares with no initial information.
A forgetting-factor version modifies these equations and discounts earlier contributions. The matrix is not automatically a calibrated parameter covariance unless a compatible probabilistic model and noise scaling justify that interpretation. Stable factorized implementations may be preferable when features are poorly conditioned.

Three practical examples

Adaptive calibration: a device updates a linear correction model whenever a new reference measurement is available, rather than waiting for a large scheduled refit.
System identification: an engineer estimates coefficients relating recent control inputs to measured outputs while new experiments are recorded.
Streaming prediction: a service updates a small linear predictor from incoming labeled records, tracking both prediction errors and changes in the estimated coefficients.

Think of it this way

Think of maintaining a running financial total together with the supporting subtotals needed to revise an estimate. Each new receipt updates the summary; the accountant does not reread every old receipt for the same calculation. The analogy is about efficient bookkeeping. RLS still depends on the chosen model and does not discover a suitable feature set automatically.

Worked example

Use an original one-coefficient model y_hat = phi theta with theta_0 = 0 and P_0 = 1. This initialization corresponds to minimizing theta^2 plus the accumulated squared errors.
The first observation is phi_1 = 1, y_1 = 2. Its gain is K_1 = 1/(1 + 1) = 1/2. The prediction error is 2, so theta_1 = 1 and P_1 = 1 - (1/2)(1)(1) = 1/2.
The second observation is phi_2 = 2, y_2 = 5. Its gain is K_2 = [(1/2)2]/[1 + 2(1/2)2] = 1/3. The old prediction is 2 and the error is 3, giving theta_2 = 1 + (1/3)3 = 2. The updated matrix is P_2 = 1/2 - (1/3)2(1/2) = 1/6.
Check the result in batch form: minimize J(theta) = theta^2 + (2 - theta)^2 + (5 - 2 theta)^2. Its derivative is 12 theta - 24, yielding theta = 2. Without the initial penalty, the batch estimate would be 12/5 = 2.4. Initialization therefore has a concrete effect, even though the sequential calculation is exact for its stated objective.

Illustrative case study

In a fictional project, Willow Sensors updates a calibration slope whenever a technician supplies a trusted reference reading. The initial implementation uses a large confidence matrix without documenting what that initialization means. Different engineers obtain different early estimates despite processing identical observations.
The team writes down the implied initial penalty, verifies small sequences against batch fits, and records predictions before each update for evaluation. It also separates suspicious reference readings from ordinary calibration data. This case illustrates reproducible online estimation; it does not claim that recursion alone protects a system from bad measurements or drifting relationships.

Common misconceptions and mistakes

RLS is not the same as taking a fixed-size gradient step on the newest example. Its gain depends on accumulated feature information. It also does not guarantee a unique unregularized solution before the observed features span the required coefficient directions.
A small prediction error can coexist with poorly determined coefficients when features are strongly correlated. Squared loss remains sensitive to large erroneous observations. Forgetting older data changes the estimation problem; it should not be described as exactly equivalent to fitting all historical observations with equal weight. Reordering fixed-weight data changes intermediate estimates but not the final exact batch objective.

Business use

Use RLS when a linear-in-parameters model is appropriate and updates must be fast. Specify feature transformations, target timing, initialization, and any forgetting factor. Keep evaluation chronological: measure a prediction before using its target to update the coefficients.
Monitor conditioning, residual patterns, and coefficient stability. Compare against periodic batch fits to detect implementation problems. Retain sufficient records for auditing even when the algorithm does not require the entire history in memory. If the process changes, test whether a forgetting scheme improves future predictions rather than assuming that faster adaptation is always beneficial.

Check your understanding

Why does the worked example end at theta = 2 rather than the unregularized estimate 2.4?

Answer: P_0 = 1 with theta_0 = 0 introduces the penalty theta^2. The recursive and batch calculations agree once they are compared using the same objective.

Related terms

Least squares
Online learning
Regularization
Feature vector
matrix inversion lemma
adaptive filter
forgetting factor
system identification
parameter covariance

Learn next

Reproduce the two observations in the opposite order and verify the same final regularized estimate. Then compare fixed-weight and discounted estimates on a simulated sequence whose true coefficient changes halfway through.

Sources

Boyd, Stanford EE263 Lecture6, growing measurements and fast recursive least-squares updates: https://see.stanford.edu/materials/lsoeldsee263/06-ls-app.pdf Sarkka, Bayesian Filtering and Smoothing, Chapter3, recursive linear regression and its probabilistic interpretation: https://users.aalto.fi/~ssarkka/pub/cup_book_online_20131111.pdf Boyd Lecture 6, pages 20–22, independently consulted for growing-data least squares and inverse rank-one updates.

Previous: Recursive Language Model Next: Recursive Neural Network