K-Nearest Neighbors

Predicting from the most similar known cases. Euclidean, Manhattan and Minkowski distances, majority votes and averages worked on eight points, why the inputs must be put on a common scale (proline and the wine data), k as the control of flexibility on a problem with a known Bayes error, Cover and Hart's bound of twice the Bayes error checked by simulation, the curse of dimensionality computed four ways, distance-weighted votes, the cost of prediction, and the method in scikit-learn.

Eight points lie in the plane, each labeled A or B, and a new point at $(3, 3)$ needs a label. Its nearest neighbor, one unit away at $(3, 4)$, is an A. The next three, at distances between 2.1 and 2.4, are all B. Asked to copy the label of the closest point, the answer is A; asked to take a vote among the three closest, the answer is B. The k-nearest-neighbor method (k-NN) predicts in exactly this way: it finds the $k$ training instances most similar to the new one and lets their known outputs decide, by a vote for classification or an average for regression. It has no equations to fit and no coefficients to estimate. Training consists of storing the data.

The method rests on one assumption: instances that are close in their inputs tend to have similar outputs. This article works out what "close" means, why the inputs must be put on a common scale, how $k$ controls the balance between following the data and smoothing it, how good the method can be in principle, and why it degrades when there are many inputs. Introduction to Supervised Learning introduced it as one way of approximating the Bayes classifier, and the Evaluating Model Performance articles already use it: nearest-neighbor regression on the diabetes study shows the bias-variance tradeoff, is tuned by Cross-Validation, and is the model chosen on a validation set in Training, Validation, and Test Sets. Those experiments are not repeated here. The course wiki's Vector Space Classification applies the method to text, where documents are compared by cosine similarity.

History

Cover and Hart (1967) credit the first nearest-neighbor rule and the first analysis of its properties to Fix and Hodges, in a 1951 report for the US Air Force School of Aviation Medicine. Fix and Hodges studied the $k$-nearest-neighbor rule and showed that its error approaches the best possible as the sample grows, provided $k$ grows too but more slowly than the sample. Cover and Hart's own result, that even a single neighbor achieves at most twice the best possible error rate in large samples, is the subject of a section below.

Measuring distance

Each instance is a vector of $p$ numerical inputs, a point in $p$-dimensional space. The usual measure of closeness between instances $a = (a_1, \dots, a_p)$ and $b = (b_1, \dots, b_p)$ is the Euclidean distance, the length of the straight line between them:

$$d_2(a, b) = \sqrt{\sum_{j=1}^{p} (a_j - b_j)^2}.$$

The Manhattan distance adds the absolute differences instead, the distance traveled along a grid of streets:

$$d_1(a, b) = \sum_{j=1}^{p} |a_j - b_j|.$$

Both are members of the Minkowski family $d_q(a, b) = \bigl(\sum_j |a_j - b_j|^q\bigr)^{1/q}$, with $q = 2$ and $q = 1$ (Hechenbichler and Schliep, 2004, §2). Other distances suit other data: great-circle distance for latitude and longitude, cosine similarity for documents, and distances for categorical and mixed inputs, which Hastie, Tibshirani and Friedman (2009, §14.3) discuss.

Eight points worked by hand

The query is $q = (3, 3)$. Sorted by Euclidean distance, the eight training points are:

Point Inputs Class Output Euclidean Manhattan
1 (3, 4) A 14 1.000 1.0
2 (5.1, 3) B 9 2.100 2.1
3 (1.5, 4.5) B 8 2.121 3.0
4 (3, 0.6) B 6 2.400 2.4
5 (4.5, 5) A 15 2.500 3.5
6 (0.5, 1) A 12 3.202 4.5
8 (6, 1) B 5 3.606 5.0
7 (1, 6.2) A 13 3.774 5.2

For point 3, the Euclidean distance is $\sqrt{(1.5 - 3)^2 + (4.5 - 3)^2} = \sqrt{2.25 + 2.25} = 2.121$ and the Manhattan distance is $1.5 + 1.5 = 3$. The two measures rank the points differently. By Euclidean distance point 3 is third, by Manhattan distance fourth, behind point 4, which differs from the query in one coordinate only. Manhattan distance penalizes a difference spread over several coordinates more than Euclidean distance does, relative to the same total in one coordinate. With $k = 3$ the two distances choose different neighbor sets, although here both sets vote B.

The eight training points and the query (star). The dashed circle encloses its three nearest neighbors by Euclidean distance (one A, two B), the dotted circle its five nearest (two A, three B). The vote changes with k even though the data do not.
The eight training points and the query (star). The dashed circle encloses its three nearest neighbors by Euclidean distance (one A, two B), the dotted circle its five nearest (two A, three B). The vote changes with k even though the data do not.

Classification and regression

The algorithm is the same for both tasks (Hastie et al., 2009, §13.3):

  1. Choose $k$ and a distance.
  2. Compute the distance from the new instance to every training instance.
  3. Take the $k$ training instances with the smallest distances, the neighborhood $N_k(x)$.
  4. For classification, predict the most common class in $N_k(x)$; for regression, predict the average output, $\hat{f}(x) = \frac{1}{k} \sum_{i \in N_k(x)} y_i$.

The vote is an estimate of the conditional class probabilities: the fraction of the neighbors in class $j$ estimates $\Pr(Y = j \mid X = x)$, and the vote picks the largest estimate, as the Bayes classifier would with the true probabilities (James, Witten, Hastie and Tibshirani, 2021, eq. 2.12). The average likewise estimates the regression function $E(Y \mid X = x)$ by the mean output of nearby instances, since there are usually no training instances at exactly $x$ (Hastie et al., 2009, eq. 2.14).

For the query $(3, 3)$, the votes are:

$k$ Neighbors Votes A : B Prediction
1 1 1 : 0 A
3 1, 2, 3 1 : 2 B
5 1 to 5 2 : 3 B
7 1 to 6, 8 3 : 4 B

As a regression with $k = 3$, the prediction is the mean output of points 1, 2 and 3: $(14 + 9 + 8)/3 = 10.333$.

With two classes and an odd $k$ the vote cannot tie. With an even $k$, or more than two classes, it can, and a rule is needed: Hastie et al. (2009, §13.3) break ties at random. scikit-learn's KNeighborsClassifier takes the first of the tied classes in its sorted list of class labels (classes_), as its source code shows (version 1.6). A different tie, between the $k$th and $(k+1)$th neighbors at the same distance, happens with rounded or repeated values; which of them is counted then depends on the order of the training data, as the scikit-learn documentation warns (scikit-learn developers, 2024).

Scaling the inputs

Distance treats every input alike, so an input measured in large units dominates it. The wine data (Aeberhard and Forina, 1992) record 13 chemical measurements on 178 wines from three cultivars. Their standard deviations range from 0.124 for nonflavanoid phenols to 14.3 for magnesium and 314.9 for proline, which runs from 278 to 1,680. Summed over all pairs of wines, proline contributes 99.77% of the squared Euclidean distance, magnesium 0.21%, and the other eleven measurements together 0.02%. Nearest neighbors on the raw measurements are, in effect, nearest neighbors on proline alone.

Ten-fold cross-validation (stratified folds, random seed 0) measures the cost:

Inputs to 5-nearest neighbors Cross-validated accuracy
raw measurements, all 13 0.675
proline alone 0.670
standardized (mean 0, standard deviation 1) 0.961
rescaled to the range 0 to 1 0.955

On raw inputs the classifier is barely better than proline alone, and both are far worse than the same classifier after standardizing each input, subtracting its mean and dividing by its standard deviation, after which each input contributes exactly 1/13 of the total squared distance over all pairs. Rescaling each input to run from 0 to 1 (min-max scaling) works nearly as well. Hastie et al. (2009, §13.3) recommend standardizing before computing distances, since the inputs may be measured in different units.

Standardizing is not a neutral choice either. It gives every input equal weight, which is right if each matters equally and wrong if some are irrelevant, a problem the section on dimensionality returns to. And the means and standard deviations must be computed on the training data only and then applied to new data, or the scaling leaks information from the test set into the model (Training, Validation, and Test Sets).

Choosing k

The number of neighbors controls how flexible the method is. With $k = 1$ each prediction copies a single training instance, so the boundary between classes follows every training point, and a single mislabeled or unusual point claims a region of its own. With $k$ equal to the size of the training set every prediction is the same, the most common class or the overall mean, and the inputs are ignored. Between the two extremes, increasing $k$ trades variance for bias, as The Bias-Variance Tradeoff derives for regression: the average of $k$ outputs varies less as $k$ grows, but the neighbors reach farther from the query, where the output may differ.

The simulated problem of Introduction to Supervised Learning shows the effect on a classifier, against a known floor. Each of two equally likely classes is a mixture of three clusters, the Bayes decision boundary bends around a cluster of class 0 that lies among the clusters of class 1, and the Bayes error rate is 0.1816. On one training set of 200 instances (seed 742), tested on 20,000 new ones:

$k$ Training error Test error
1 0.000 0.2505
3 0.135 0.2303
7 0.195 0.2117
15 0.190 0.2006
31 0.190 0.1911
51 0.235 0.1971
101 0.300 0.2787
151 0.320 0.3068
Decision regions of the 1-nearest-neighbor (left) and 101-nearest-neighbor (right) classifiers on the same 200 training instances, with the Bayes boundary dashed. With one neighbor, every training point claims its own island; with 101, the boundary is smooth but has lost the cluster of class 0 in the middle.
Decision regions of the 1-nearest-neighbor (left) and 101-nearest-neighbor (right) classifiers on the same 200 training instances, with the Bayes boundary dashed. With one neighbor, every training point claims its own island; with 101, the boundary is smooth but has lost the cluster of class 0 in the middle.

With one neighbor the training error is zero, since every training instance is its own nearest neighbor, and the test error is 0.25. With 101 neighbors, half the training set, each neighborhood stretches across clusters and the boundary loses the curve that the Bayes boundary follows. Averaged over 100 training sets (seeds 1000 to 1099), the test error is lowest at $k = 21$, 0.1987, within two points of the Bayes error rate; it is 0.2555 at $k = 1$ and 0.2837 at $k = 151$. The test error curve is flat near its minimum, between about 0.199 and 0.200 for $k$ from 15 to 25, so the choice within that range matters little.

Training and test error of k-nearest neighbors against k (log scale), averaged over 100 training sets of the simulated problem. Training error rises from zero as k grows; test error falls to a minimum near k = 21 and rises again. The dotted line is the Bayes error rate, which no classifier can beat. Distance-weighted votes (dashed) are worse for small k and better for large k.
Training and test error of k-nearest neighbors against k (log scale), averaged over 100 training sets of the simulated problem. Training error rises from zero as k grows; test error falls to a minimum near k = 21 and rises again. The dotted line is the Bayes error rate, which no classifier can beat. Distance-weighted votes (dashed) are worse for small k and better for large k.

In practice the truth is unknown and $k$ is chosen by Cross-Validation, as the diabetes example in that article does; Model Selection adds the one-standard-error rule, which here would favor the larger, smoother $k$ within the flat stretch.

How good can one neighbor be?

The 1-nearest-neighbor rule ignores everything except the single closest training instance, which sounds wasteful. Cover and Hart (1967) proved a surprising bound on how much it loses. Suppose the training set grows without limit, so that the nearest neighbor of any point $x$ lies essentially at $x$ and has the same conditional class probabilities. With two classes, write $\eta(x) = \Pr(Y = 1 \mid X = x)$. The new instance is in class 1 with probability $\eta(x)$, and so, independently, is its neighbor; the rule errs when they differ, with probability

$$\eta(x)\bigl(1 - \eta(x)\bigr) + \bigl(1 - \eta(x)\bigr)\eta(x) = 2\eta(x)\bigl(1 - \eta(x)\bigr).$$

The Bayes classifier errs at $x$ with probability $r^*(x) = \min\bigl(\eta(x), 1 - \eta(x)\bigr)$, and $2\eta(1 - \eta) = 2r^*(1 - r^*)$ whichever of the two is smaller. Averaging over $x$, the large-sample error rate of the 1-nearest-neighbor rule is

$$R = E\bigl[2r^*(X)\bigl(1 - r^*(X)\bigr)\bigr] = 2R^*(1 - R^*) - 2\operatorname{Var}\bigl(r^*(X)\bigr) \le 2R^*(1 - R^*),$$

where $R^* = E[r^*(X)]$ is the Bayes error rate (Cover and Hart, 1967, eqs. 24 to 26). The second form follows from $E[(r^*)^2] = (R^*)^2 + \operatorname{Var}(r^*)$. Since $2R^*(1 - R^*) \le 2R^*$, the rule's error is at most twice the Bayes rate. It is also at least the Bayes rate itself, because $r^* \le \frac{1}{2}$ makes $2r^*(1 - r^*) \ge r^*$ at every $x$. In Cover and Hart's words, "half the classification information in an infinite sample set is contained in the nearest neighbor". They prove the result for any number of classes, under smoothness conditions on the distributions (Hastie et al., 2009, eq. 13.5, gives the general bound).

On the simulated problem all three quantities can be computed. The Bayes error rate is $R^* = 0.1816$; integrating $2\eta(1 - \eta)$ over the input distribution gives the large-sample 1-nearest-neighbor error, 0.2505; and the bound $2R^*(1 - R^*)$ is 0.2972. Simulation, with ten training sets of each size, shows how quickly the limit is reached:

Training instances per class 1-NN test error SD over 10 training sets
100 0.2643 0.0125
1,000 0.2497 0.0042
10,000 0.2514 0.0026
100,000 0.2516 0.0027

By 1,000 instances per class the error has reached its large-sample value within the noise of the test set. As Hastie et al. (2009, §13.3) point out, the bound also works backward: if a 1-nearest-neighbor classifier has an error rate of 10% on a large sample, the Bayes error rate is at least about 5%, so no method can have an error rate much below half that of the nearest neighbor. The catch is the word "asymptotically". The argument assumes the neighbor is at essentially the same point as the query, which needs a training set dense enough to fill the input space, and in many dimensions no training set is.

The curse of dimensionality

Nearest-neighbor methods rely on there being training instances close to every query. In a few dimensions that is easy; in many it is impossible. Bellman named the difficulty the curse of dimensionality, and Hastie et al. (2009, §2.5) give several ways to see it. Each is computed here.

Neighborhoods stop being local. Suppose the inputs are spread uniformly over the unit cube in $p$ dimensions, and a neighborhood is a smaller cube that must hold a fraction $r$ of the data. Its volume is $r$, so its edge length is $e_p(r) = r^{1/p}$.

Dimension $p$ Edge to hold 1% of the data Edge to hold 10%
1 0.010 0.100
2 0.100 0.316
3 0.215 0.464
10 0.631 0.794
100 0.955 0.977

In ten dimensions, a neighborhood holding 1% of the data spans 63% of the range of every input. An average over such a neighborhood is no longer an average over similar instances.

Every point is near the edge. For $N$ points spread uniformly in the unit ball in $p$ dimensions, the median distance from the center to the nearest point is $d(p, N) = \bigl(1 - \tfrac{1}{2}^{1/N}\bigr)^{1/p}$ (Hastie et al., 2009, eq. 2.24). With $N = 500$ it is 0.037 in two dimensions, 0.518 in ten, more than halfway to the boundary, and 0.936 in a hundred; simulations of 400 samples each give 0.037, 0.516 and 0.937. A prediction at the center of the data then has to extrapolate from points near the edge rather than interpolate between nearby ones.

Distances become alike. Beyer, Goldstein, Ramakrishnan and Shaft (1999) proved that under broad conditions the distance from a query to its nearest data point approaches the distance to its farthest as the dimension grows, and found the effect in practice with as few as 10 to 15 dimensions. For 1,000 points uniform in the unit cube and 100 random queries, the median ratio of farthest to nearest distance is:

Dimension $p$ Median nearest distance Median farthest distance Farthest / nearest
2 0.014 1.070 79.0
10 0.506 1.873 3.66
100 3.334 4.764 1.43
1,000 12.176 13.607 1.12

In two dimensions the farthest point is dozens of times farther away than the nearest (79 times in this run; in two dimensions the figure varies noticeably from run to run, because the nearest distance is small and variable). In a thousand dimensions it is only 12% closer, and "nearest" has lost most of its meaning.

Two views of the curse of dimensionality. Left: the edge length of a sub-cube that holds a fraction r of uniformly spread data, for p = 1, 2, 3 and 10 inputs; in ten dimensions, 10% of the data needs 79% of each input's range. Right: the median ratio of the farthest to the nearest of 1,000 uniform points from a random query, falling toward 1 as the dimension grows (both axes logarithmic).
Two views of the curse of dimensionality. Left: the edge length of a sub-cube that holds a fraction r of uniformly spread data, for p = 1, 2, 3 and 10 inputs; in ten dimensions, 10% of the data needs 79% of each input's range. Right: the median ratio of the farthest to the nearest of 1,000 uniform points from a random query, falling toward 1 as the dimension grows (both axes logarithmic).

Irrelevant inputs dilute the relevant ones. Real data are not uniform, and their inputs are usually related, so the effective dimension can be much lower than the number of inputs. The more common practical problem is inputs that carry no information. Standardizing gives each one the same weight in the distance as a useful input, so each added irrelevant input adds noise to every distance. Adding columns of pure noise, drawn independently from a standard normal distribution, to the 13 standardized wine measurements (20 draws for each number of added columns, seeds 0 to 19):

Noise inputs added 5-nearest neighbors Logistic regression
0 0.961 0.983
2 0.959 0.983
5 0.955 0.982
10 0.949 0.979
20 0.930 0.967
50 0.890 0.962
100 0.813 0.943
Ten-fold cross-validated accuracy on the wine data as columns of pure noise are added to the 13 standardized measurements, averaged over 20 draws of the noise. Nearest neighbors loses accuracy steadily, from 0.961 to 0.813 with 100 noise inputs; logistic regression, which can give the noise small coefficients, falls from 0.983 to 0.943.
Ten-fold cross-validated accuracy on the wine data as columns of pure noise are added to the 13 standardized measurements, averaged over 20 draws of the noise. Nearest neighbors loses accuracy steadily, from 0.961 to 0.813 with 100 noise inputs; logistic regression, which can give the noise small coefficients, falls from 0.983 to 0.943.

With 100 noise columns, nearest neighbors drops from 0.961 to 0.813 while logistic regression, which fits a coefficient to each input and can give the useless ones small values, falls only to 0.943.

This is the sense in which Hastie et al. (2009, §2.5) contrast local methods with structured ones: imposing a form such as linearity lets a method ignore what does not matter, at the price of bias when the form is wrong. Choosing inputs with care (feature selection, in the course wiki's text-classification setting) or reducing their number first, for example with principal component analysis, helps nearest-neighbor methods more than most.

Weighting the votes

A plain vote counts every neighbor equally, however far away. A natural refinement gives closer neighbors more say. With inverse-distance weights, neighbor $i$ counts $1/d_i$, and the class with the largest total weight wins; for regression, the prediction is the weighted mean $\sum_i y_i/d_i \big/ \sum_i 1/d_i$. Hechenbichler and Schliep (2004, §3) describe a general family of weighting schemes, which turn distances into weights through a kernel function that decreases with distance.

For the query $(3, 3)$, the weighted votes are:

$k$ Weight of A Weight of B Weighted prediction Plain vote
1 1.000 0 A A
3 1.000 0.948 A B
5 1.400 1.364 A B
7 1.712 1.642 A B

Point 1, at distance 1, has weight 1, while the three B points at 2.1 to 2.4 have weights of 0.42 to 0.48 each, so the close A outweighs the two nearest B points together. The weighted prediction is A for every $k$, while the plain vote changes from A to B at $k = 3$. Weighting makes the prediction less sensitive to the choice of $k$. As a regression with $k = 3$, the weighted mean of outputs 14, 9 and 8 is 11.325, closer to the nearest point's 14 than the plain mean of 10.333.

Whether weighting improves accuracy depends on $k$ and on the data. On the simulated problem, averaged over the same 100 training sets, inverse-distance weighting is worse for small $k$ (0.2240 against 0.2142 at $k = 5$; 0.2060 against 0.1997 at $k = 15$), and better for large $k$ (0.2056 against 0.2204 at $k = 51$; 0.2267 against 0.2657 at $k = 101$). Its best, 0.2020 at $k = 29$, is close to the best plain vote, 0.1987 at $k = 21$. With large $k$ the weights let the nearby neighbors dominate the distant ones the large neighborhood takes in; with small $k$ they give the single closest neighbor even more influence than it already has, which adds variance. On the wine and breast cancer data, with standardized inputs and ten-fold cross-validation:

$k$ Wine, equal Wine, weighted Breast cancer, equal Breast cancer, weighted
1 0.955 0.955 0.953 0.953
5 0.961 0.961 0.965 0.965
15 0.978 0.972 0.960 0.961
25 0.977 0.977 0.953 0.954
51 0.956 0.972 0.949 0.949
101 0.860 0.972 0.933 0.940

The pattern is the same. Near the best $k$ the two kinds of vote are within a point of each other; for large $k$, weighting protects against neighborhoods that are too wide. With 101 neighbors, more than half of the 160 or so wines in each training fold, the plain vote on the wine data falls to 0.860, because the smallest cultivar, 48 wines in all, is outnumbered in its own neighborhoods: 20 of its 48 wines are assigned to cultivar 2. The weighted vote stays at 0.972. Weighting is therefore a cheap insurance against choosing $k$ too large rather than a way to beat a well-chosen $k$.

Cost and practice

A nearest-neighbor model costs nothing to fit and a great deal to use. A naive prediction computes the distance to all $n$ training instances, $p$ subtractions each, for every query, and the whole training set must be kept. Spatial data structures such as k-d trees and ball trees organize the training points so that most of them can be ruled out without computing their distances; scikit-learn chooses among them and brute-force search automatically. They work well in a few dimensions and lose their advantage as the dimension grows, for the reason Beyer et al. (1999) gave: when all distances are similar, little can be ruled out. Beyer et al. inferred from published experiments that, at 10 to 15 dimensions, a simple linear scan through all the points would often outperform the high-dimensional indexing methods being proposed. Approximate search methods trade a small chance of missing the true nearest neighbor for large gains in speed.

The method's strengths are the reverse of its costs. It assumes nothing about the form of the relationship, so it can follow boundaries of any shape, including classes made of several separate clusters. It handles any number of classes without modification. And each prediction comes with its own explanation: the training instances that produced it, which can be shown to a person. What it cannot offer is a global summary, such as a coefficient saying how much an input matters, since it has fitted none.

In Python

from sklearn.datasets import load_wine
from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
folds = StratifiedKFold(10, shuffle=True, random_state=0)

raw = KNeighborsClassifier(n_neighbors=5)
scaled = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
print(cross_val_score(raw, X, y, cv=folds).mean())      # 0.675
print(cross_val_score(scaled, X, y, cv=folds).mean())   # 0.961

search = GridSearchCV(scaled, {'kneighborsclassifier__n_neighbors': range(1, 52, 2),
                               'kneighborsclassifier__weights': ['uniform', 'distance']}, cv=folds)
search.fit(X, y)
print(search.best_params_)

model = scaled.fit(X, y)
dist, idx = model[-1].kneighbors(model[0].transform(X[:1]), n_neighbors=5)
print(idx, dist)                    # the five training wines nearest the first wine

n_neighbors sets $k$, weights='distance' gives inverse-distance votes, and metric and p choose the distance (the default is Minkowski with p=2, the Euclidean distance; p=1 gives Manhattan). KNeighborsRegressor averages instead of voting. The kneighbors method returns the neighbors themselves, which is how a single prediction is explained. Putting the scaler in a pipeline makes cross-validation fit it to the training folds only. The numbers in this article are computed by tools/experiments/data-science/k-nearest-neighbors.py in the site's repository.

References

  • Aeberhard, S., and Forina, M. (1992). Wine [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5PC7J record · 8 pages
  • Beyer, K., Goldstein, J., Ramakrishnan, R., and Shaft, U. (1999). When is "nearest neighbor" meaningful? In C. Beeri and P. Buneman (Eds.), Database Theory: ICDT'99 (Lecture Notes in Computer Science, Vol. 1540, pp. 217–235). Springer. record
  • Cover, T. M., and Hart, P. E. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27. record
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. record · 23 pages
  • Hechenbichler, K., and Schliep, K. (2004). Weighted k-nearest-neighbor techniques and ordinal classification (Discussion Paper 399, SFB 386). Ludwig-Maximilians-Universität München. record
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. record · 30 pages
  • scikit-learn developers (2024). sklearn.neighbors.KNeighborsClassifier. scikit-learn 1.6 documentation. https://scikit-learn.org/1.6/modules/generated/sklearn.neighbors.KNeighborsClassifier.html record
  • Wolberg, W., Mangasarian, O., Street, N., and Street, W. (1993). Breast Cancer Wisconsin (Diagnostic) [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5DW2B record · 14 pages