Detecting Outliers

Finding observations that do not fit, and deciding what they are. Errors against genuine extremes; z-scores and the masking that lets two recording errors hide each other, with the bound (n - 1)/sqrt(n) on any z-score proved; modified z-scores built on the median and MAD; Tukey's fences on skewed NHANES columns; Mahalanobis distance with chi-squared cutoffs for waist and BMI, where an adult ordinary on each measurement is unusual in combination, and its robust MCD version; isolation forests and the local outlier factor briefly; and what to do with an outlier once found, including one created by a processing rule.

One NHANES adult has a mean diastolic blood pressure of 8 mm Hg, far below any other value in the survey's 5,232 diastolic means. Another weighs 242.6 kg. Both are outliers by any rule, and they call for opposite responses. The weight is very rare but physiologically possible, and deleting it would remove a real person from the data. The diastolic value is not a measurement at all: the adult's three readings were 0, 0 and 8 mm Hg, the survey's protocol allows a reading of 0, and the table this article uses treated zeros as missing, leaving the mean of the one remaining reading. The outlier was made by a processing rule.

Detecting outliers is the easy half of the problem; deciding what each one is takes judgment. This article covers both halves: the rules that flag unusual values in one variable, and why the most familiar of them can fail exactly when it is needed; distances that flag unusual combinations of several variables; two machine-learning detectors; and what to do with an outlier once it is found. The data are again the NHANES 2017 to 2018 adults described in Performing Exploratory Data Analysis. Density-based detection, in which an outlier is a point in a sparse region, is covered in Detecting Outliers Using DBSCAN; outliers and influential points in regression, measured by residuals and leverage, in Linear Regression Assumptions.

What an outlier is

Hawkins (1980) gave the definition most often quoted: an outlier is "an observation that deviates so much from other observations as to arouse suspicion that it was generated by a different mechanism" (as quoted by Breunig, Kriegel, Ng and Sander, 2000). The definition is about suspicion, not proof. A detection rule can only say that a value is far from the others by some measure; whether it came from a different mechanism, a slipped decimal point, a faulty sensor, a different population or a processing step, is a question about how the data were produced.

Iglewicz and Hoaglin (1993) separate three activities, a distinction the NIST/SEMATECH handbook (§1.3.5.17) adopts: labeling potential outliers for further investigation, accommodating them by using methods that are not unduly affected by them, and identifying them by formal tests. Exploratory analysis is mostly about labeling, and the rules below are labeling rules. A label is a reason to look at a record, not a verdict on it.

The causes fall into two broad groups. Errors include data entry slips (a decimal point in the wrong place, digits transposed), measurement faults, unit confusions, codes for missing values stored as numbers, and artifacts of processing such as the diastolic value above. Genuine extremes are correct values from the tail of the distribution, and also correct values from a different population that has been mixed in, such as a child's record in a table of adults. The response differs: errors are corrected, or set to missing when the correct value cannot be recovered; genuine extremes are kept, and the analysis is made robust to them or reported with and without them.

One variable: z-scores and masking

The most common rule standardizes each value by the mean $\bar{x}$ and the standard deviation $s$ of the sample, giving its z-score $z_i = (x_i - \bar{x})/s$ (Measures of Center and Spread), and flags values with $|z_i| > 3$, or sometimes $2.5$. For a normal distribution, 0.27% of values lie more than three standard deviations from the mean, so in a large normal sample the rule flags few values, and those few are genuinely extreme.

A bound on every z-score

The rule has a weakness that small samples make obvious. In a sample of $n$ values, no z-score can exceed $(n-1)/\sqrt{n}$ in absolute value (Shiffler, 1988; NIST/SEMATECH, §1.3.5.17). The proof is short. Write $d_j = x_j - \bar{x}$ for the deviations; they sum to zero, so for any one observation $i$, $\sum_{j \ne i} d_j = -d_i$. By the Cauchy–Schwarz inequality, $\left(\sum_{j \ne i} d_j\right)^2 \le (n-1)\sum_{j \ne i} d_j^2$, so $\sum_{j \ne i} d_j^2 \ge d_i^2/(n-1)$. Then

$$(n-1)s^2 = \sum_{j=1}^n d_j^2 = d_i^2 + \sum_{j \ne i} d_j^2 \ge d_i^2\left(1 + \frac{1}{n-1}\right) = d_i^2 \,\frac{n}{n-1},$$

and dividing by $(n-1)s^2 \cdot n/(n-1)$ gives $z_i^2 = d_i^2/s^2 \le (n-1)^2/n$. Equality holds when all the other observations are equal to each other. For $n = 10$ the bound is $9/\sqrt{10} = 2.85$, so a rule of $|z| > 3$ can never flag anything in a sample of ten, however wild one value is. For $n = 20$ it is 4.25.

The reason is that the extreme value is part of the mean and the standard deviation that measure it. A single wild value pulls the mean toward itself and inflates the standard deviation, which shrinks its own z-score. With more than one wild value, the effect compounds.

Masking

Take twenty systolic blood pressures drawn at random (seed 645) from the NHANES adults: values from 99.3 to 188.0 mm Hg with a mean of 127.6, a standard deviation of 20.7, a median of 125.3 and a median absolute deviation (MAD) of 13.7. The largest, 188.0, has $z = 2.91$, just under 3. Now suppose the value 129.3 is entered as 1293.0, a slipped decimal point. The mean jumps to 185.8 and the standard deviation to 261.4, and the error has $z = 4.24$, close to the largest possible value of 4.25, so the rule flags it. Suppose instead that two values are entered with the same slip, 129.3 as 1293.0 and 131.3 as 1313.0. The mean is now 244.9 and the standard deviation 362.5, and the two errors have z-scores of 2.89 and 2.95. Neither is flagged at 3. Two absurd values, ten times any real blood pressure, hide each other. This is masking: outliers inflating the scale estimate enough to conceal themselves. Its counterpart, swamping, is the labeling of ordinary values as outliers, for instance because they lie close to real ones or because a test was told to expect more outliers than there are (NIST/SEMATECH, §1.3.5.17; Liu, Ting and Zhou, 2008, §3).

Twenty systolic pressures, clean and with one or two recording errors (crosses: a value multiplied by ten). Left: z-scores. One error stands out at 4.24, near the largest possible z-score for n = 20 (dotted line); with two errors, both have z-scores below 3 (dashed line) and the rule flags nothing. Right: modified z-scores based on the median and MAD, on a scale that is linear up to 10 and logarithmic beyond; both errors score about 46, far beyond the cutoff of 3.5, whether there are one or two.
Twenty systolic pressures, clean and with one or two recording errors (crosses: a value multiplied by ten). Left: z-scores. One error stands out at 4.24, near the largest possible z-score for n = 20 (dotted line); with two errors, both have z-scores below 3 (dashed line) and the rule flags nothing. Right: modified z-scores based on the median and MAD, on a scale that is linear up to 10 and logarithmic beyond; both errors score about 46, far beyond the cutoff of 3.5, whether there are one or two.

Modified z-scores

The remedy is to measure distance from a center and in a scale that a few wild values cannot move. The median is such a center and the median absolute deviation, $\mathrm{MAD} = \operatorname{median}_i |x_i - \tilde{x}|$ where $\tilde{x}$ is the median, such a scale: both stay within the range of the good data as long as fewer than half the values are bad (Measures of Center and Spread calls such summaries resistant). Iglewicz and Hoaglin (1993) define the modified z-score

$$M_i = \frac{0.6745\,(x_i - \tilde{x})}{\mathrm{MAD}}$$

and recommend labeling values with $|M_i| > 3.5$ as potential outliers (NIST/SEMATECH, §1.3.5.17). The constant 0.6745 is the 75th percentile of the standard normal distribution: for normal data the MAD is about $0.6745\sigma$, so $\mathrm{MAD}/0.6745$ estimates the standard deviation and $M_i$ is on the same scale as an ordinary z-score. On the twenty pressures, the median and MAD barely move when errors are added (median 125.3, 126.3 and 128.0; MAD 13.7, 17.0 and 17.0), and the errors have modified z-scores of 46.3 with one error and 46.2 and 47.0 with two. Masking has no grip on them. The genuine 188.0 has $M = 3.09$ in the clean sample and is not labeled, which is reasonable for a high but real blood pressure.

Tukey's fences

The box plot's rule labels values below $Q_1 - 1.5\,\mathrm{IQR}$ or above $Q_3 + 1.5\,\mathrm{IQR}$, where $Q_1$ and $Q_3$ are the quartiles and IQR their difference; values beyond $3\,\mathrm{IQR}$ from the quartiles are sometimes called "far out". The construction and its behavior on NHANES blood pressures are in Tables and Graphs for Data. Quartiles, like the median, resist a few wild values, so the fences do not suffer from masking in small numbers: in the twenty pressures they label 188.0 in the clean sample (the upper fence is 175.9) and both errors in the corrupted one. The two errors also move the quartiles slightly, since two middle values became the two largest, and the upper fence rises to 190.7, so that 188.0 is no longer labeled.

On whole columns, and the problem of skew

All three rules assume, explicitly or not, a roughly symmetric distribution with normal-looking tails. On real columns they disagree, and the disagreement tracks skewness:

Column n Skewness $\lvert z \rvert > 3$ $\lvert M \rvert > 3.5$ 1.5 IQR fences 3 IQR fences
Systolic pressure 5,247 1.04 60 66 149 11
BMI 5,434 1.26 72 78 156 21
HDL cholesterol 5,176 1.31 44 36 128 19
C-reactive protein 5,146 7.89 104 510 422 181
log₁₀ C-reactive protein 5,146 0.35 23 2 32 0
Normal data, n = 5,000 (expected) 0 13.5 2.3 34.9 0.0

For a normal sample of 5,000, the expected numbers labeled are 13.5 by $|z| > 3$, 2.3 by $|M| > 3.5$ and 34.9 by the 1.5 IQR fences (whose share for a normal distribution is 0.70%). The moderately skewed columns exceed all three expectations several times over, almost entirely in their upper tails: of the 156 BMI values outside the fences, all 156 are above. CRP, with skewness 7.9, has 510 values labeled by the modified z-score and 422 by the fences, about one adult in ten. Those adults are not errors. They are the right tail of a distribution that is roughly normal on a log scale, and on that scale the counts fall to 23, 2 and 32, close to what normal data would give.

The lesson is that a univariate rule labels values that are unusual relative to a symmetric model, so applying it to a skewed variable on its original scale mostly labels the tail. Look at the distribution first (Visualizing Data with One Feature), transform a variable whose variation is multiplicative, and then apply a rule. With thousands of observations, some values are always labeled; the question is whether more are labeled than the model would predict, and whether the labeled values look like errors when their records are examined.

Several variables: unusual combinations

A value can be ordinary on each variable and unusual in combination. Waist circumference and BMI are strongly related among the 5,175 NHANES adults with both measured ($r = 0.905$): the least-squares line predicts waist $= 2.170 \times \mathrm{BMI} + 35.99$ cm, with a typical scatter of 7.3 cm around the line. One adult has a BMI of 43.7 and a waist of 95.7 cm. Her BMI has $z = 1.97$ and her waist $z = -0.26$, neither remarkable. But the line predicts a waist of 130.8 cm at her BMI, and her measurement is 35 cm, nearly five times the typical scatter, below it. No univariate rule can find her; a measure of distance that accounts for the correlation can.

Mahalanobis distance

For observations $\mathbf{x}_i$ with $p$ variables, sample mean vector $\bar{\mathbf{x}}$ and sample covariance matrix $S$, the squared Mahalanobis distance (Mahalanobis, 1936) is

$$d_i^2 = (\mathbf{x}_i - \bar{\mathbf{x}})^\top S^{-1} (\mathbf{x}_i - \bar{\mathbf{x}}).$$

For one variable it is the square of the z-score. In general it measures distance in units of the data's own spread in each direction: the inverse covariance matrix shrinks distances along directions in which the data vary a lot (along the waist-BMI line) and stretches them across directions in which they vary little (across the line). Points with the same $d^2$ lie on an ellipse centered at the mean and aligned with the data, as in the figure below. A useful identity, checked on the data, is that the average of $d_i^2$ over the sample is always $p(n-1)/n$, here $2 \times 5174/5175 = 1.9996$.

If the data come from a multivariate normal distribution with known mean $\boldsymbol{\mu}$ and covariance $\Sigma$, then $(\mathbf{x} - \boldsymbol{\mu})^\top \Sigma^{-1}(\mathbf{x} - \boldsymbol{\mu})$ has a chi-squared distribution with $p$ degrees of freedom. The reason is that $\Sigma^{-1/2}(\mathbf{x} - \boldsymbol{\mu})$ is a vector of $p$ independent standard normal variables, and $d^2$ is the sum of their squares. With the mean and covariance estimated from a large sample, the same distribution holds approximately, and quantiles of $\chi^2_p$ serve as cutoffs. For $p = 2$, the 0.975 and 0.999 quantiles are 7.378 and 13.816.

Waist circumference against BMI for 5,175 NHANES adults. The solid and dashed ellipses are the classical Mahalanobis contours at the chi-squared 0.999 and 0.975 cutoffs; diamonds are the 47 adults beyond the 0.999 contour. The dash-dot ellipse is the 0.999 contour from the robust MCD estimate, which is tighter and points slightly differently because it is fitted to the central 4,700 adults. The arrow marks an adult with a BMI of 43.7 and a waist of 95.7 cm, unremarkable on either axis but far from the band.
Waist circumference against BMI for 5,175 NHANES adults. The solid and dashed ellipses are the classical Mahalanobis contours at the chi-squared 0.999 and 0.975 cutoffs; diamonds are the 47 adults beyond the 0.999 contour. The dash-dot ellipse is the 0.999 contour from the robust MCD estimate, which is tighter and points slightly differently because it is fitted to the central 4,700 adults. The arrow marks an adult with a BMI of 43.7 and a waist of 95.7 cm, unremarkable on either axis but far from the band.

On the waist and BMI data, 184 adults (3.6%) are beyond the 0.975 contour and 47 (0.91%) beyond the 0.999 contour, against expected shares of 2.5% and 0.1%. The excess comes from the right skew of both variables: the ten largest distances all belong to women with high BMI, most of them with BMI between 58 and 68 and waists that, while large, are smaller than the line predicts for their BMI. The adult with BMI 43.7 has $d^2 = 26.9$, the eighth largest of 5,175, and she is the only adult beyond the 0.999 contour whose z-scores on both variables are below 2. On the logarithms of the two variables, which are close to symmetric (skewness 0.29 and 0.02), the 0.999 contour labels 10 adults (0.19%), and she has the second largest distance.

Masking again, and a robust version

The mean vector and covariance matrix are as easily distorted by outliers as the mean and standard deviation, and in several dimensions the distortion is harder to see. A cluster of outliers can inflate the covariance in their direction and pull the mean toward themselves until their own distances look ordinary. The multivariate counterpart of the median and MAD is a robust estimate of location and scatter. The minimum covariance determinant (MCD) estimator (Rousseeuw, 1984; computed by the algorithm of Rousseeuw and Van Driessen, 1999) finds the subset of $h$ observations, by default a little over half, whose covariance matrix has the smallest determinant, the most tightly concentrated subset, and estimates the mean and covariance from it, with corrections for consistency. Distances computed from these estimates are not inflated by the observations outside the core.

On waist and BMI, the MCD estimate is centered at a BMI of 28.1 and a waist of 97.5 cm, with standard deviations of 5.3 and 14.8, against 7.2 and 17.2 for the classical estimate; after scikit-learn's reweighting step it rests on 4,700 adults. Its 0.999 contour labels 207 adults, including all 47 labeled by the classical contour. Here the robust estimate does not uncover hidden outliers so much as describe the bulk of the data more tightly, and labels much of the long right tail. On the log scale it labels 31. Robust estimation protects against masking, but it does not remove the need for a model that fits the shape of the data: the chi-squared cutoff assumes a roughly normal core, and skewed data should be transformed first.

Machine-learning detectors

Distances to a center describe outliers relative to one elliptical cloud. Data with several clusters, curved structure or many dimensions need detectors that make fewer assumptions. Two widely used ones take opposite approaches.

Isolation forests

Liu et al. (2008) observed that anomalies are "few and different", and that this makes them easy to isolate. An isolation tree splits a sample recursively: pick a variable at random, pick a split value uniformly between that variable's minimum and maximum in the current node, and send points left or right, until every point is alone or a height limit is reached. An unusual point tends to be separated after few splits, because a random cut is likely to fall between it and everything else; a point in a dense region needs many. The path length $h(x)$, the number of splits from the root to the point's leaf, is short for anomalies. An isolation forest averages $h(x)$ over many trees, each built on a small random subsample, and converts it to a score

$$s(x, n) = 2^{-E[h(x)]/c(n)}, \qquad c(n) = 2H(n-1) - \frac{2(n-1)}{n},$$

where $E[h(x)]$ is the average path length, $n$ the subsample size, $H(i)$ the $i$th harmonic number (about $\ln i + 0.5772$) and $c(n)$ the average path length of an unsuccessful search in a binary search tree of $n$ points, used to normalize. A score near 1 marks a clear anomaly, scores well below 0.5 mark normal points, and if all scores are near 0.5 the sample has no distinct anomalies (Liu et al., 2008, §2). The authors found that small subsamples suffice, larger ones adding time and memory without improving detection, and used 256 points and 100 trees as defaults (§4); subsampling also reduces masking and swamping, because each tree sees only a few of the anomalies (§3). For $n = 256$, with the approximation to $H$ that Liu et al. use, $c(n) = 10.245$.

Local outlier factor

Breunig, Kriegel, Ng and Sander (2000) defined outlyingness relative to a point's neighborhood. Let $k$-distance$(o)$ be the distance from $o$ to its $k$th nearest neighbor. The reachability distance of $p$ from $o$ is $\max\{k\text{-distance}(o), d(p, o)\}$, the actual distance, but no less than $o$'s own $k$-distance, which smooths out fluctuations among close points. The local reachability density of $p$ is the inverse of the average reachability distance from $p$ to its $k$ nearest neighbors, and the local outlier factor is the average, over those neighbors, of the ratio of their local reachability density to $p$'s:

$$\mathrm{LOF}_k(p) = \frac{1}{|N_k(p)|}\sum_{o \in N_k(p)} \frac{\mathrm{lrd}_k(o)}{\mathrm{lrd}_k(p)}.$$

A point inside a cluster has a density similar to its neighbors' and an LOF near 1; a point in a sparser region than its neighbors has an LOF well above 1. Because the comparison is local, LOF can label a point just outside a dense cluster even when points in a diffuse cluster elsewhere are farther from their neighbors, a case that no single global density threshold handles (Detecting Outliers Using DBSCAN shows the same problem for DBSCAN). The authors suggest a neighborhood of at least 10 points to keep the estimate stable (Breunig et al., 2000, §6.2).

On the waist and BMI data

The same standardized waist and BMI data scored three ways; color shows each adult's rank on the score, yellow most unusual, and the ten highest are circled. Mahalanobis distance and the isolation forest (100 trees, 256-point subsamples, seed 645) both rank the far end of the band highest, where adults with very high BMI are few. The local outlier factor (20 neighbors) instead picks points that are sparse relative to their neighborhoods, several of them at the low-BMI end of the band and beside it.
The same standardized waist and BMI data scored three ways; color shows each adult's rank on the score, yellow most unusual, and the ten highest are circled. Mahalanobis distance and the isolation forest (100 trees, 256-point subsamples, seed 645) both rank the far end of the band highest, where adults with very high BMI are few. The local outlier factor (20 neighbors) instead picks points that are sparse relative to their neighborhoods, several of them at the low-BMI end of the band and beside it.

On the two standardized variables, the isolation forest's scores have a median of 0.446 and a maximum of 0.770, and the local outlier factor a median of 1.013 and a maximum of 3.42 (with 20 neighbors, scikit-learn's default). The isolation forest agrees broadly with the Mahalanobis distance (a Spearman rank correlation of 0.923 between the two scores, and three of their top ten in common; forests grown from other seeds give rank correlations of 0.92 to 0.93 and two or three shared cases), because in a single elongated cloud "few and different" and "far from the center" mostly coincide. LOF ranks the adults quite differently (rank correlations of 0.32 to 0.37 with the other scores) and shares none of its top ten with the isolation forest. Its top ten include adults at the low end of BMI, such as a BMI of 14.9 with a waist of 80 cm, whose neighborhoods are sparser than those of their neighbors, and the adult with BMI 43.7 and waist 95.7, whom it ranks second.

The methods answer different questions, and none is the correct one. Distance from a center finds the edges of one cloud; isolation finds what is quickly separated; LOF finds what is sparse relative to its surroundings. On real data they should be compared, and their top-ranked cases examined, as the next section describes. All three are also sensitive to scaling, since they work with distances or with splits whose randomness depends on each variable's range, and the variables were standardized first.

What to do with an outlier

The detection rules produce a list. Each entry needs a decision, and the decision depends on the cause, not on the score.

  1. Look at the whole record. The diastolic value of 8 mm Hg is explained by its raw readings (0, 0 and 8). Of the 5,533 adults, 53 have at least one diastolic reading of 0, 83 readings in all; for 15 of them every recorded diastolic reading is 0, which is why they have a systolic but no diastolic mean, and 38 have zeros mixed with nonzero readings, whose means are pulled low or rest on one reading. The adult with BMI 43.7 and a waist of 95.7 cm should be checked against her height and weight (165.1 cm, so a weight of about 119 kg) and against any comment codes recorded with the waist measurement.
  2. Correct errors where the correct value is known, such as a unit or a decimal slip confirmed by another field. Where it is not known, set the value to missing and handle it as a missing value (Cleaning Data), recording the reason.
  3. Fix the process when the outlier came from processing. The diastolic artifact is a property of the rule "treat 0 as missing", not of the person (Cleaning Data discusses these zeros). A better rule might keep zeros, or average only nonzero readings and flag means based on one reading; whichever is chosen should be applied to every adult, not to the one that happened to be noticed.
  4. Keep genuine extremes, and protect the analysis from them instead. Resistant summaries (medians, quartiles), transformations (logarithms for CRP), robust loss functions in regression (Loss Functions for Regression) and robust estimates such as the MCD all reduce the pull of extreme values without discarding them. How much a summary depends on its extremes varies: removing the heaviest adult changes the mean weight of 5,444 adults from 82.469 to 82.440 kg, while removing the largest 1% of CRP values (51 adults) changes its mean from 4.170 to 3.554 mg/L, a drop of 15%. The median moves from 1.940 to 1.910 and the geometric mean from 2.050 to 1.982.
  5. Report the sensitivity. When an analysis's conclusions change if a few observations are removed, that fact is part of the result. Fitting with and without the labeled cases and reporting both is often more informative than choosing one.
  6. Never delete because of the score alone. A z-score of 4 says a value is far from the mean; it does not say the value is wrong, and deleting real extremes biases every later estimate toward the center and understates the variability that a model must handle.

Corrections to the course notes

The unit's notes recommend $|z| \ge 3$ as the z-score rule and mention that extreme values inflate the mean and standard deviation, which can make an extreme value less likely to be flagged. Two statements can be made more precise. First, the effect is not only probabilistic: in a sample of $n$, no z-score can exceed $(n-1)/\sqrt{n}$, so the rule $|z| \ge 3$ cannot flag anything when $n \le 10$. Second, the standard robust alternative to the z-score is the modified z-score based on the median and MAD (Iglewicz and Hoaglin, 1993), which the notes do not mention; Tukey's fences are another, but they share the z-score's assumption of a roughly symmetric distribution and label much of the tail of a skewed variable, as the CRP column shows. The notes' discussion of leverage and influence belongs with regression and is covered in Linear Regression Assumptions.

In Python

import numpy as np
from scipy import stats
from sklearn.covariance import MinCovDet
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor

x = adults["sbp"].dropna().to_numpy()
z = (x - x.mean()) / x.std(ddof=1)
med = np.median(x)
mad = np.median(np.abs(x - med))
m = 0.6745 * (x - med) / mad                      # modified z-score; label |m| > 3.5
q1, q3 = np.percentile(x, [25, 75])
outside = (x < q1 - 1.5 * (q3 - q1)) | (x > q3 + 1.5 * (q3 - q1))

X = adults.loc[:, ["bmi", "waist"]].dropna().to_numpy()
diff = X - X.mean(axis=0)
d2 = np.einsum("ij,jk,ik->i", diff, np.linalg.inv(np.cov(X, rowvar=False)), diff)
cut = stats.chi2.ppf(0.999, df=X.shape[1])        # 13.816 for two variables
d2_robust = MinCovDet(random_state=645).fit(X).mahalanobis(X)

Z = (X - X.mean(axis=0)) / X.std(axis=0, ddof=1)
iso = IsolationForest(n_estimators=100, max_samples=256, random_state=645).fit(Z)
s = -iso.score_samples(Z)                        # the score s(x, n) of Liu et al.
lof = LocalOutlierFactor(n_neighbors=20).fit(Z)
lof_score = -lof.negative_outlier_factor_

scikit-learn's IsolationForest.score_samples returns the negative of Liu et al.'s score, so it is negated here; decision_function and predict additionally subtract a threshold set by the contamination parameter, which presumes a known share of outliers. Every number in this article is computed by tools/experiments/data-science/detecting-outliers.py in the site's repository, from the NHANES table built by tools/experiments/data-science/exploredata.py; the figures are drawn by tools/figures/detecting-outliers.py.

References

  • Breunig, M. M., Kriegel, H.-P., Ng, R. T., and Sander, J. (2000). LOF: Identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (pp. 93–104). https://doi.org/10.1145/342009.335388 record
  • Hawkins, D. M. (1980). Identification of Outliers. Chapman and Hall. record
  • Iglewicz, B., and Hoaglin, D. C. (1993). How to Detect and Handle Outliers. ASQC Quality Press. record
  • Liu, F. T., Ting, K. M., and Zhou, Z.-H. (2008). Isolation forest. In Proceedings of the Eighth IEEE International Conference on Data Mining (ICDM '08) (pp. 413–422). https://doi.org/10.1109/ICDM.2008.17 record
  • Mahalanobis, P. C. (1936). On the generalised distance in statistics. Proceedings of the National Institute of Sciences of India, 2(1), 49–55. record
  • National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographics (DEMO_J), Blood Pressure (BPX_J), Body Measures (BMX_J) and High-Sensitivity C-Reactive Protein (HSCRP_J) data files and documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
  • NIST/SEMATECH. e-Handbook of Statistical Methods, §1.3.5.17 (Detection of outliers). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm record · 30 pages
  • Rousseeuw, P. J. (1984). Least median of squares regression. Journal of the American Statistical Association, 79(388), 871–880. record
  • Rousseeuw, P. J., and Van Driessen, K. (1999). A fast algorithm for the minimum covariance determinant estimator. Technometrics, 41(3), 212–223. record
  • Shiffler, R. E. (1988). Maximum Z scores and outliers. The American Statistician, 42(1), 79–80. record