Is k-nearest neighbors the only instance-based method?

No. It is a familiar example within a broader family that can select, weight or adapt stored instances in different ways.

Can the method predict numbers?

Yes. Nearest-neighbor regression combines numerical targets rather than class votes.

Does adding every new example always help?

No. Noisy, redundant or unrepresentative cases can increase cost or distort local predictions.

AI Glossary / Graphs and symbolic AI / Intermediate

Instance-based learning

Definition

Instance-based learning predicts a new case by comparing it with stored examples. Instead of relying only on a compact global rule learned in advance, it uses particular instances and a defined similarity or distance measure to make a local prediction.

What it means

An instance is an example with its relevant features and, in supervised learning, its known target. A familiar instance-based method is k-nearest neighbors, which finds nearby stored examples and combines their labels or numerical values.

The approach depends on what nearby means. Two records may look similar under one representation and quite different under another. Feature scaling, irrelevant attributes and missing values can therefore change the prediction even when the stored labels stay the same.

Instance-based learning is often described as lazy learning because much of the decision work occurs when a query arrives. That does not mean there is no preparation: selecting examples, constructing representations and building a search index can all require substantial work.

How it works

Store suitable training examples and choose the representation and comparison measure. When a new case arrives, identify relevant neighbors and combine their targets using a specified rule.

For classification, a simple rule takes the majority class among k neighbors. Distance weighting can give nearer examples more influence. For regression, the system may average their numerical targets, with or without weights.

Choose settings using appropriate validation, and keep preprocessing fitted within the training procedure. Some instance-based methods retain only selected examples to reduce storage or limit the influence of noise. Prediction quality still depends on the retained cases covering the intended use. A nearest neighbor can be far away, and retrieving one does not by itself establish that the query resembles anything the system has learned.

Three practical examples

A teaching classifier labels a new measurement using nearby labeled measurements.
A service analysis tool retrieves similar historical cases to support a suggested category, while showing the examples used.
A regression prototype estimates a continuous response from nearby examples and compares its performance with a simple global model.

Think of it this way

Think of consulting a file of previous jobs to find those most similar to a new request. Their outcomes can inform your expectation for the new job.

The analogy explains learning from particular cases. It does not guarantee that superficial similarity implies the same outcome, or that a previous case provides a causal explanation for what will happen now.

Worked example

Changing the neighbor-combination rule

Suppose three stored examples use two already normalized numerical features. Example A has coordinates (1, 2) and label Blue; B has (2, 4) and label Red; C has (4, 3) and label Red. The new query is (2, 2).

Euclidean distances are 1 to A, 2 to B and square root of 5, approximately 2.236, to C. A one-nearest-neighbor classifier predicts Blue because A is closest.

An unweighted three-nearest-neighbor classifier predicts Red because two of the three neighbors are Red. If the same three neighbors vote with weight 1/distance, Blue receives weight 1. Red receives 1/2 + 1/square root of 5, approximately 0.947. This weighted rule predicts Blue.

The examples have not changed, but the prediction rule has. No result is declared correct without the true label or evaluation evidence. This calculation also assumes the features were meaningfully normalized; applying Euclidean distance directly to incompatible raw scales could select different neighbors for reasons unrelated to the task. A practical inverse-distance implementation must additionally define its behavior for an exact zero-distance match.

Illustrative case study

The fictional Willow Cases team builds a support-request classifier from past labeled cases. Its initial representation includes a numerical customer identifier alongside meaningful request features.

The identifier's large scale dominates distance, causing neighbors to be selected for accidental numerical proximity. Reviewers remove that identifier from the similarity calculation and inspect the representation of the actual task features. They compare several neighbor counts and weighting rules on held-out cases.

Willow also checks whether duplicate requests from the same incident have crossed the evaluation split. The case illustrates representation and evaluation issues rather than a measured accuracy gain or a claim that nearest-neighbor methods are always the best choice.

Common misconceptions and mistakes

Instance-based learning merely memorizes and cannot generalize. It generalizes through a similarity assumption and local prediction rule.
The nearest record must be a good match. Every stored record may be far from an unusual query.
Increasing k always improves accuracy. Larger neighborhoods can reduce sensitivity to noise but mix genuinely different regions.
The algorithm has no settings to learn or choose. Representation, metric, example selection and aggregation choices strongly affect behavior.

Business use

The approach can provide a useful baseline and make supporting examples visible. It is attractive when local similarity has a credible connection to the target and suitable historical cases are available.

Review storage, query latency and the handling of obsolete or mislabeled examples. In high-dimensional representations, distance can become less useful, and indexing methods may lose their expected efficiency. Showing neighbors can help users inspect a recommendation, but it does not prove that those cases justify the outcome. Define what happens for poor matches and validate performance on genuinely new cases from the intended population.

Check your understanding

Why do unweighted and inverse-distance three-neighbor voting disagree in the worked example?

Answer: Unweighted voting counts two Red cases against one Blue case. Inverse-distance voting gives the closest Blue case weight 1, slightly exceeding the Red total of approximately 0.947.

Related terms

Nearest neighbor
Lazy learning
Case-based reasoning
Classification
Local regression
Nearest neighbors
distance metric
feature scaling
prototype selection

Learn next

Move query point (2, 2) to (3, 3) and recompute all three distances. Compare one-neighbor, unweighted three-neighbor and inverse-distance predictions, explicitly handling any ties that arise.

Sources

Aha, Kibler and Albert's original paper describes instance-based learning and selective storage of examples: https://doi.org/10.1007/BF00153759

The scikit-learn developers document nearest-neighbor classification, regression, weighting and search considerations: https://scikit-learn.org/stable/modules/neighbors.html

The coordinate calculation and Willow Cases scenario are original examples. The cited paper's abstract and relevant implementation documentation were reviewed; no historical experiment is presented as a current performance guarantee.

Previous: Instance Next: Instance selection