Is it the same as LOESS?

LOESS belongs to the locally weighted polynomial-regression family, with particular neighborhood and weighting conventions. The labels are related but do not guarantee identical implementations.

Must weights sum to one?

No. A common positive scaling of all least-squares weights leaves the fitted coefficients unchanged.

Does a smaller bandwidth always help?

No. It can reduce smoothing bias while increasing variance and numerical instability.

AI Glossary / Statistics and probability / Intermediate

Locally weighted regression

Definition

Locally weighted regression predicts a response by fitting a regression that gives greater influence to observations near the query point. The weights change when the query changes, so different locations can have different fitted coefficients. A common version uses a local straight line with distance-based weights. It combines simple fitting calculations with a flexible overall response curve, but its success depends on meaningful distances, adequate local data, and a suitable bandwidth.

What it means

The key distinction is between one global weighted fit and a query-specific fit. In ordinary weighted regression, weights may represent different measurement precisions and remain fixed across predictions. In locally weighted regression, proximity to the current query usually determines the weights.

The resulting predictor can be nonlinear overall even when every local fit is a straight line. The method is often described as nonparametric because the representation and predictions depend on the training observations rather than on one fixed global parameter vector alone.

How it works

At query x0, calculate weights such as wi = exp[-(xi - x0) squared/(2h squared)], where h is a positive bandwidth. Fit coefficients a and b to minimize the sum of wi times [yi - a - b(xi - x0)] squared. The prediction at x0 is a.

Repeat this calculation for each query. Nearby observations have greater influence, but a Gaussian weighting rule gives every finite-distance observation positive weight. Other kernels can assign zero weight beyond a boundary.

Choose bandwidth and feature scaling using appropriate validation. Very small bandwidths may leave too little effective information for a stable fit. Very large bandwidths make the procedure resemble a global fit. Robust response weights or regularization are additional choices, not implied by proximity weighting alone.

Three practical examples

A manufacturing analyst estimates output near a particular process setting using comparable nearby settings more strongly than distant ones.
A robotics team fits a local relationship around the current operating state when a single global linear dynamics model is inadequate.
A service analyst predicts handling time from similar case characteristics, validating that the chosen distance represents genuine similarity rather than arbitrary differences in measurement units.

Think of it this way

Think of asking nearby witnesses for an estimate of local conditions. Their reports may be more relevant than reports from far away. But you still fit a relationship among those reports rather than simply choosing the nearest witness. If all witnesses stand on one side or report nearly identical inputs, the local calculation can become unreliable.

Worked example

Take fictional observations (x,y) = (0,0), (1,2), and (2,3). Predict at x0 = 0.5 with Gaussian bandwidth h = 1.

Distances are 0.5, 0.5, and 1.5, giving weights exp(-0.125), exp(-0.125), and exp(-1.125), approximately 0.882497, 0.882497, and 0.324652.

Fit the centered local line y approximately a + b(x - 0.5). Weighted least squares gives a approximately 0.935219 and b approximately 1.611312. Therefore the local prediction at 0.5 is approximately 0.9352.

By contrast, the distance-weighted response average is approximately 1.3107. That is a local constant fit, not the local linear fit: it ignores the slope across nearby inputs. The ordinary unweighted global line is 1/6 + 1.5x, giving approximately 0.9167 at the same query.

These differences result from explicitly different fitting rules. Multiplying all three local weights by the same positive constant would not change the local least-squares minimizer. The example demonstrates one query calculation and offers no evidence that one rule will predict better on unseen data.

Illustrative case study

Fictional case: Stonefield Controls predicts a machine's response near its current settings. The system uses temperature and pressure as distance features, but the first implementation measures one in units hundreds of times larger than the other.

Engineers discover that numerical scale dominates neighborhood selection. They introduce a documented scaling rule based on training data and process knowledge, then compare bandwidth choices on separate operating runs.

They also flag queries far from historical conditions for review. The local model remains useful within supported regions, while the team avoids treating a mathematically available prediction as proof that the machine has been observed in a comparable state.

Common misconceptions and mistakes

Do not assume locally weighted regression is simply a weighted average; a local line or higher-degree polynomial can produce a different estimate. Do not interpret distance weights as probabilities that observations are correct.

Using Gaussian-shaped weights does not require normally distributed predictors. A local fit also does not guarantee resistance to response outliers or stable extrapolation. In high dimensions, neighborhoods may contain few genuinely comparable cases. Highly correlated predictors or too few effective neighbors can make the weighted design matrix nearly singular.

Business use

This method can provide useful low-dimensional predictions and diagnostic baselines for AI applications where relationships change gradually across operating conditions. It also helps explain why a locally flexible method may outperform a single rigid line.

Account for prediction-time computation and retention of training information. Document the bandwidth, kernel, scaling, local polynomial form, and any regularization. Evaluate accuracy and latency together. If records must be removed or updated, consider how the stored local information and future predictions will change.

Check your understanding

Why is the worked local-linear prediction about 0.935 rather than the weighted mean 1.311?

Answer: The local-linear fit estimates both an intercept at the query and a slope. The weighted mean estimates only a constant. They optimize different model classes even though they use the same proximity weights.

Related terms

Local regression
Weighted least squares
Nearest neighbor
bandwidth
Gaussian kernel
nearest neighbors
design matrix
nonparametric regression

Learn next

Next, compare local constant and local linear prediction near boundaries. Then examine bandwidth validation, feature scaling, and stable numerical solvers for weighted least-squares systems.

Sources

Sources: Ma and Ng, Stanford CS229 Lecture Notes, Locally Weighted Linear Regression, primary weighting and prediction construction: https://cs229.stanford.edu/main_notes.pdf . NIST/SEMATECH, LOESS, for the related local polynomial family: https://www.itl.nist.gov/div898/handbook/pmd/section1/pmd144.htm . The supplied AIMA repository opened but did not expose its glossary body. The Gaussian-weight example and fictional case are original; weights and three contrasting estimates were independently checked.

Previous: Locally weighted learning Next: Location-based recommendation