Does k-NN work only for classification?

No. It also supports regression and neighbor retrieval.

Must k equal the number of classes?

No. It is the number of neighbors used.

Is there no training at all?

Basic k-NN mainly stores data, but preprocessing, distance design, indexing, and parameter selection still require work.

AI Glossary / Data science and analytics / Intermediate

k-nearest neighbors

Definition

K-nearest neighbors, often written k-NN, is a method that uses the k closest reference examples to make a prediction or retrieve similar cases. In supervised classification, neighboring labels vote for a class. In regression, neighboring target values are combined, often by averaging. Closeness is defined by a chosen distance measure and feature representation, so the method's usefulness depends strongly on what similar means for the task.

What it means

Instead of learning one global prediction equation, a basic k-NN model keeps reference examples and consults nearby ones when a new case arrives. The value k determines how many neighbors contribute. A small k responds closely to local details but can be sensitive to noise. A larger k smooths over more examples and may blur useful boundaries. Neither choice is universally best, and nearby records are only informative if the representation captures relevant differences.

How it works

Prepare the features, select a distance measure, and store the reference data with their labels or target values. For a query, find the k nearest examples, then combine their outcomes using equal weights or a documented distance-based weighting rule. Numerical features may need scaling so a large measurement unit does not dominate distance. Choose k and preprocessing using appropriate validation data, keeping evaluation examples out of the reference set. Search can use a direct comparison or an indexing method, with efficiency depending on dataset size and geometry.

Three practical examples

Classification: a system assigns a simple document category using the labels of nearby examples in an appropriate numerical representation.
Regression: a team estimates a property of an item from measured values of similar labeled items.
Retrieval: an analyst finds reference cases closest to a new observation for inspection, without automatically treating similarity as a final decision.

Think of it this way

Think of asking the most relevant nearby examples for advice. Their collective answer guides the prediction. The analogy depends on a defensible definition of nearby: sharing an irrelevant characteristic does not make two cases useful comparisons. The algorithm does not understand why a neighbor is relevant unless that relevance is reflected in its inputs and distance rule.

Worked example

Suppose a one-dimensional training set contains positions 1 and 3 labeled A, and positions 5, 7, and 9 labeled B. A new query is at position 4.2, and distance is absolute difference. The distances are 3.2, 1.2, 0.8, 2.8, and 4.8, respectively. For k = 3, the neighbors are position 5 labeled B, position 3 labeled A, and position 7 labeled B. An equal-weight majority vote predicts B by two votes to one.
If k = 1, only position 5 contributes, again producing B in this case. That agreement does not establish that either k is generally better. A query closer to a class boundary or a mislabeled record can behave differently. For a regression variant using these same neighbor positions but target values 50, 30, and 70, the equal-weight prediction would be (50 + 30 + 70)/3 = 50. A distance-weighted prediction would use a different calculation. Classification votes and regression averages should not be confused.

Illustrative case study

In a fictional equipment dataset, a team measures one feature in millimeters and another in thousands of operating hours. Raw Euclidean distance is dominated by the numerically larger feature, even though the team intended both to matter. It evaluates a scaling procedure fitted only on training data and reviews retrieved neighbors with engineers. It also checks performance on equipment not represented during training. The case illustrates that preprocessing defines the neighborhood; it does not prove that standardizing every feature is always the correct choice.

Common misconceptions and mistakes

K-nearest neighbors is not k-means clustering: k-NN selects nearby reference examples, whereas k-means learns cluster centers. The k in k-NN is not the number of classes. A large dataset does not automatically provide useful neighbors in a high-dimensional space. Class imbalance and repeated near-duplicate records can influence votes. Training-set evaluation can be misleading if an example retrieves itself. Ties in distance or votes need a documented policy, and a predicted class does not establish that the associated vote fraction is a well-calibrated probability.

Business use

Use k-NN as an interpretable baseline or retrieval method when meaningful comparisons can be constructed. Measure prediction quality, response time, memory requirements, and the availability of reference data at deployment. Review whether stored examples contain sensitive information and whether retained records remain representative. If approximate neighbor search is used for scale, evaluate whether missed neighbors change outcomes. Keep preprocessing and distance conventions consistent between reference records and new queries, and define what happens when a query is unlike anything in the reference collection.

Check your understanding

In the worked example, why does k = 3 predict B?

Answer: The three closest reference points have labels B, A, and B. Under the specified equal-weight rule, B receives the majority. A different representation or weighting scheme could change another case's result.

Related terms

Learn next

Calculate neighbors manually for a small two-feature dataset before and after changing one feature's units. Then compare several k values with a validation split appropriate to the task. Learn how exact and approximate search differ from the prediction rule that uses the returned neighbors.

Sources

Scikit-learn, nearest-neighbor classification, regression, and search: https://scikit-learn.org/stable/modules/neighbors.html Scikit-learn, KNeighborsClassifier weighting and distance parameters: https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html Scikit-learn, feature scaling and distance-sensitive methods: https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html

Previous: K-nearest Neighbor Classification Algorithm Next: k-nearest neighbors regression