Among the 5,444 adults measured in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, men average 173.5 cm tall and women 159.7 cm. A difference of 14 cm between two groups of roughly equal size sounds as if it should give the combined heights two peaks, and textbooks have long used height as the standard example of a bimodal distribution. A smooth estimate of the distribution drawn with the usual default settings has one peak. Drawn with half the default smoothing, it has two, the second a slight bump at 171 cm. Which picture is right is a question about the data, and also a question about the plot: how much a one-variable display smooths, how its axis is scaled and how it summarizes are choices, and each choice can show or hide a feature of the data.
This article is about those choices. The standard displays of one variable, frequency tables, bar charts, histograms with their bin rules, the empirical distribution function and box plots with Tukey's fences, are defined in Tables and Graphs for Data on the blood pressures of the same NHANES adults, and the summaries they show are in Measures of Center and Spread. What follows takes those as known and adds the displays a data scientist reaches for next: kernel density estimates, logarithmic axes, violin and strip plots, and bar charts for a categorical feature with hundreds of levels. The data are the NHANES adults again, joined from several survey files as described in Performing Exploratory Data Analysis, and the Old Faithful geyser eruptions used in Detecting Outliers Using DBSCAN.
What a one-feature plot is for
Every display of one variable answers some of the same questions: where the values are centered, how widely they spread, whether the distribution is symmetric, whether it has one peak or several, whether there are gaps, and whether a few values sit far from the rest. The kind of variable decides which displays can answer them (Types of Numerical Data). A categorical feature has counts per category and nothing between categories, so its display is a bar chart. A numerical feature has a distribution along a line, and its displays either estimate the density of that distribution (histograms and their smooth relatives), summarize it (box plots), or show every value (strip plots and the empirical distribution function).
In an exploratory analysis these plots are made quickly and in bulk, often one per column of a table, and their defaults are rarely examined. The examples below are cases where the defaults matter.
Kernel density estimates
A histogram estimates a density by counting values in bins, and Tables and Graphs for Data shows how the picture changes with the bin width. The bins also have edges, and a value just left of an edge counts in a different bar from one just right of it. A kernel density estimate (KDE) removes the edges. Instead of dropping each observation into a bin, it places a small smooth bump centered on the observation and adds the bumps together.
The estimate
Let $x_1, \dots, x_n$ be the observations and $K$ a kernel, a symmetric probability density, usually the standard normal density $\phi(u) = e^{-u^2/2}/\sqrt{2\pi}$. The kernel density estimate at a point $t$ is
$$\hat{f}_h(t) = \frac{1}{nh} \sum_{i=1}^{n} K\!\left(\frac{t - x_i}{h}\right),$$
where $h > 0$ is the bandwidth. Each term $\frac{1}{h}K((t - x_i)/h)$ is a normal density with mean $x_i$ and standard deviation $h$, a bump of area 1 centered on the $i$th observation. Dividing the sum by $n$ makes the total area 1, so $\hat{f}_h$ is itself a probability density: it is never negative and integrates to 1. Where observations are crowded, many bumps overlap and the estimate is high; where they are sparse, it is low. Silverman (1986, ch. 3) gives the definition and its properties; the estimator is due to Rosenblatt (1956) and Parzen (1962).
The kernel's shape matters little. The bandwidth matters a great deal, in the same way as the bin width of a histogram: a small $h$ gives each observation a narrow spike and the estimate follows the noise of the sample, while a large $h$ spreads each observation widely and can flatten real features. The SciPy documentation for its Gaussian KDE states the point directly: the bandwidth affects the estimate more than the kernel does (SciPy developers, gaussian_kde).
Choosing the bandwidth
Rules of thumb set $h$ from the spread of the data and the sample size, shrinking it like $n^{-1/5}$, the rate that balances bias and variance for a smooth density (Silverman, 1986, §3.3). Two are in common use. With $s$ the sample standard deviation and IQR the interquartile range:
- Scott's factor, as SciPy implements it and seaborn uses by default: $h = s\,n^{-1/5}$.
- Silverman's rule of thumb: $h = 0.9 \min(s, \mathrm{IQR}/1.34)\, n^{-1/5}$ (Silverman, 1986, eq. 3.31). The IQR term protects against a standard deviation inflated by a long tail or outliers, and the factor 0.9 makes the rule smaller, which suits densities that are not normal.
For the 5,444 heights, $s = 10.067$ cm and $\mathrm{IQR} = 14.70$ cm, so $n^{-1/5} = 0.179$ and Scott's factor gives $h = 1.802$ cm, Silverman's rule $0.9 \times \min(10.067, 10.97) \times 0.179 = 1.622$ cm. Both rules are derived for data close to a normal distribution, and the SciPy documentation warns that bimodal or multimodal data tend to be oversmoothed by them. They are starting points, like the histogram bin rules.
At Scott's bandwidth the estimate of the heights has a single peak at 163.8 cm. At twice that bandwidth it still has one, lower and broader. At half, 0.90 cm, a second local maximum appears: the curve falls to 0.0315 at 169.8 cm and rises again to 0.0321 at 171.2 cm, a dip of 2%. At a quarter, 0.45 cm, the curve has 21 local maxima, almost all of them noise. A reader who changes only the bandwidth can therefore find one, two or twenty peaks in the same data. Features that persist over a range of reasonable bandwidths deserve attention; a feature that appears only in a narrow range, as the bump at 171 cm does, needs other evidence before it is believed.
Why height is not bimodal
The other evidence is the composition of the sample. Schilling, Watkins and Watkins (2002) work out when a mixture of two normal distributions has two peaks. For two components in equal proportions with a common standard deviation $\sigma$, the mixture is bimodal if and only if the means differ by more than $2\sigma$. For unequal standard deviations $\sigma_1 \le \sigma_2$ they prove a condition that holds whatever the proportions: with $r = \sigma_1^2/\sigma_2^2$ and a separation factor $S(r)$, which equals 1 when $r = 1$ and stays close to 1 when the variances are similar, the mixture is unimodal for every mixing proportion if $|\mu_1 - \mu_2| < S(r)(\sigma_1 + \sigma_2)$. Roughly, the means must differ by more than the sum of the two standard deviations before two peaks are possible. In NHANES the women's heights have standard deviation 7.03 cm and the men's 7.68 cm, so $r = 0.838$, $S(r) = 0.996$ and the threshold is $0.996 \times 14.70 = 14.64$ cm. The difference in means, 13.77 cm, is below it, so a mixture of two normal curves with these means and standard deviations has one peak whatever the share of men (in the sample it is 48.3%). Measured in the pooled standard deviation of 7.35 cm, the difference is 1.87 standard deviations, short of the 2 that equal spreads would need. The right panel of the figure shows the same thing without the normal assumption: the two sexes' estimates overlap so much that their sum is one hump with a shoulder on its right.
Schilling et al. traced the textbook example to a photograph of students lined up by height, a "living histogram" with two peaks. Normal curves with that class's own means, standard deviations and proportions are still unimodal; the authors point instead to students heaping on popular heights such as six feet, to the irregularity of histograms of small samples, and to selection: they had found only two photographs that purport to show two peaks, and wonder how many classes lined up and found nothing worth photographing. The lesson for exploration is the one the bandwidth already taught. A second peak in a plot is a hypothesis about subgroups; it should be checked by splitting the data by the suspected grouping variable, which is a multi-feature display (Visualizing Data with Multiple Features), and by asking whether the peak survives a change of smoothing.
Where a KDE misleads
Two failures of the KDE are worth knowing, both visible on the C-reactive protein (CRP) measurements described below. First, the estimate places mass where no data can be. A normal bump centered on a CRP of 0.5 mg/L extends below zero, and a concentration cannot be negative; for the 5,146 CRP values, with Scott's bandwidth of 1.485 mg/L, 14.5% of the area of the estimate lies below zero. Seaborn's documentation notes this behavior for bounded data and that data concentrated near a natural boundary may be better served by another display (Waskom, kdeplot). Second, plotting software extends the curve past the extreme observations: seaborn draws it to three bandwidths beyond the data by default (its cut parameter). On the Old Faithful eruption times, 10.2% of the area of the estimate lies outside the observed range of 1.6 to 5.1 minutes. A smooth curve also suggests a continuous variable even when the data are discrete or heaped on round values, a point the documentation also makes. A histogram shows such structure; a KDE smooths it away.
Logarithmic scales
C-reactive protein is a marker of inflammation measured in NHANES blood samples. Its 5,146 adult values run from 0.11 to 182.82 mg/L, with a median of 1.94 and a mean of 4.17 mg/L. The quartiles are 0.88 and 4.48, the 95th percentile 13.17 and the 99th 39.33. The skewness is 7.9. On a linear axis almost everything falls in the first one or two bars of a histogram, and the 8.1% of adults above 10 mg/L stretch the axis out to 180 with bars too short to see.
On a logarithmic axis, equal distances stand for equal ratios: the step from 0.1 to 1 is the same length as the step from 1 to 10. The CRP values span $\log_{10}(182.82/0.11) = 3.22$ factors of ten, and on that scale their distribution is nearly symmetric: the logarithms have skewness 0.35, mean 0.312 and median 0.288 (in units of $\log_{10}$ mg/L). The histogram on the right uses bins of equal width on the log scale, that is, of equal ratio between their edges, which is what makes the bars comparable. A log axis is appropriate when the variable is positive and its variation is multiplicative, when ratios such as "twice as high" are the natural comparison, and when values span several orders of magnitude. Concentrations, incomes, populations and reaction times are common cases.
The scale changes what the eye reads, so it must be labeled as such; the ticks on the right show the actual values 0.1, 0.3, 1, 3 and so on, not their logarithms, which keeps the axis readable. Three cautions follow. Zero and negative values have no logarithm, so a variable that can be zero needs a decision: a small offset such as $\log(x + 1)$ changes the shape near zero and its result depends on the units, and a symmetric log scale that is linear near zero is another option. Bars on a log axis should not be read as lengths from zero, since the axis has no zero; a log axis suits points, lines and histograms of counts, not bar charts of the logged quantity. And values that were not measured but filled in appear as artifacts. Twelve of the CRP values were below the laboratory's detection limit of 0.15 mg/L and were recorded as $0.15/\sqrt{2} = 0.106$ mg/L, which rounds to 0.11 in the file (National Center for Health Statistics, 2020b); on a log scale they form a small isolated bar at the left edge. Blood cadmium, examined in Performing Exploratory Data Analysis, has 357 such values and a much more conspicuous spike.
Violin and strip plots
When the box plot hides the shape
The 272 eruptions of the Old Faithful geyser last from 1.6 to 5.1 minutes. Their quartiles are 2.16 and 4.45 minutes and their median is 4.00 minutes (NumPy's default quartile method), so the box of a box plot runs from 2.16 to 4.45 and no eruption is beyond Tukey's fences at $-1.28$ and $7.89$. The box plot describes a broad, slightly left-skewed distribution. In fact the eruptions fall into two clusters: 92 lasted 2.5 minutes or less, 168 lasted 3.5 minutes or more, and only 12 lie between. The box spans the gap without any sign of it, and its median line sits in the lower part of the long cluster.
A box plot reduces a distribution to five numbers and a list of points beyond the fences, and two distributions with the same quartiles look the same however different their shapes (Tables and Graphs for Data notes that a box plot cannot show two peaks). That economy is its strength when many groups are compared side by side, and its weakness when a single distribution is being explored.
Violin plots
A violin plot (Hintze and Nelson, 1998) draws a kernel density estimate on both sides of a center line, so that the width of the shape at each value is proportional to the estimated density, and usually adds the box plot's quartiles and median inside it. It shows the two Old Faithful clusters as two bulges joined by a thin neck. With the KDE come its settings: the shape depends on the bandwidth, here Scott's factor of 0.372 minutes, which finds peaks at 1.99 and 4.36 minutes, and seaborn's default drawing range for violins, two bandwidths beyond the data, extends the violin from 0.86 to 5.84 minutes, well past the shortest and longest eruptions. Setting the drawing range to the data (cut=0 in seaborn) avoids implying eruptions that never happened. Violins of groups of very different sizes look equally substantial, since each density has area 1; a group of five observations gets as full a violin as a group of five thousand, so the group sizes should be printed or the violins scaled by count.
Strip plots and swarm plots
A strip plot draws every observation as a point along the axis. With 272 eruptions all the points are visible once each is moved a small random distance perpendicular to the axis, which is jitter; the vertical position in the bottom-right panel of the figure carries no information. A swarm plot moves the points systematically instead, so that none overlap, which makes the outline of the swarm a rough density. Both show the gap between the clusters, the exact values and the number of observations at once, and they make no smoothing choices. Their limit is size: with thousands of observations the points merge into a band, and a histogram, a density estimate or a box plot becomes more readable. A useful combination for moderate samples is a box plot or violin with the points drawn over it, so that the summary and the data behind it can be compared.
A categorical feature with many levels
Bar charts for a categorical feature are straightforward when it has a handful of categories (Tables and Graphs for Data shows race and self-rated health). Many real categorical features have hundreds or thousands: product codes, diagnoses, zip codes, words. NHANES asks every participant which prescription medicines they took in the past month. Of the 5,533 examined adults, 3,235 (58.5%) reported at least one, for 12,699 reports in all, and the reports name 635 different generic drugs or combinations, plus three codes: 7 reports of an unidentifiable drug, 24 refusals and 70 "don't know" (National Center for Health Statistics, 2020a). A bar chart with 635 bars cannot be read.
The figure shows three standard remedies:
- Sort by frequency. Alphabetical order is the default of many tools and is almost never informative; the first names alphabetically are abacavir combinations and abatacept, reported between one and three times each. Sorted, the chart shows at once that atorvastatin (563 reports, 4.4% of all reports), metformin (547) and lisinopril (483) lead, and that cholesterol, diabetes and blood pressure medicines dominate the top of the list. Wickham (2010, §7) lists alphabetical ordering among common graphical mistakes, citing Wainer's name for it, "Alabama first!".
- Show the top $k$ and say what the rest are. The 20 bars in the figure cover 42.9% of reports. A chart of the top 20 that does not say so invites the reader to assume it covers most of the data. A final bar labeled "all other 615 names" with its total, or a caption as here, keeps the denominator visible.
- Describe the tail separately. The cumulative share on the right answers questions a bar chart cannot: how many categories it takes to cover a given share (29 for half of all reports), and how long the tail is. Of the 635 names, 165 (26%) were reported exactly once. A plot of counts against rank on logarithmic axes is the usual way to look at such a tail, and the term statistics of the course wiki show the same long-tailed pattern for words in text.
Three practical points complete the picture. Horizontal bars leave room for long labels such as "hydrochlorothiazide" without rotating text. Levels that mean the same thing must be merged before counting: the NHANES file lists multi-ingredient products as separate entries (112 of the 635 contain a semicolon, such as "enalapril; hydrochlorothiazide"), and whether those belong with their components depends on the question. And special codes, here 55555, 77777 and 99999, are not categories of the variable; they are counted separately above, and the treatment of such codes is covered in Cleaning Data.
Choosing a one-feature display
| Question | Display | Watch for |
|---|---|---|
| What is the shape: peaks, skew, gaps? | histogram at two or three bin widths; KDE at two or three bandwidths | features that appear at only one setting; KDE mass outside the possible range |
| Do values span orders of magnitude? | histogram or KDE on a log axis, labeled in original units | zeros and negative values; filled-in values at a detection limit |
| What are the median, quartiles and extreme values? | box plot | two clusters inside one box; the fences assume rough symmetry |
| Shape and summary together, for one or several groups | violin plot, or box plot with points drawn over it | bandwidth; violins drawn past the data; unequal group sizes |
| Every value, for a small or moderate sample | strip plot with jitter, swarm plot | overplotting beyond a few hundred points |
| What share is below a value? | empirical distribution function | shape is harder to see than in a histogram |
| How common is each category? | bar chart sorted by frequency | alphabetical order; a top-$k$ chart without the total; codes counted as categories |
In Python
import numpy as np
import seaborn as sns
from scipy import stats
# adults: the NHANES table; eruptions: the 272 eruption times; drugs: one row per reported medicine
h = adults["height"].dropna()
kde = stats.gaussian_kde(h) # Scott's factor: bandwidth = sd * n**(-1/5)
kde_half = stats.gaussian_kde(h, bw_method=kde.factor / 2)
sns.kdeplot(data=adults, x="height", bw_adjust=0.5) # half the default bandwidth
sns.kdeplot(data=adults, x="height", hue="sex", common_norm=True) # each sex scaled by its share
sns.histplot(data=adults, x="crp", log_scale=True) # log axis, equal-ratio bins
sns.violinplot(x=eruptions, cut=0, inner="quart") # violin limited to the data range
sns.stripplot(x=eruptions, jitter=0.2)
top = drugs["drug"].value_counts() # sorted by frequency
top.head(20).plot.barh()
cum = top.cumsum() / top.sum() # cumulative share by rank
seaborn's kdeplot with hue scales each group's curve by its share of the data by default (common_norm=True), so the curves' areas show group sizes, as in the right panel of the first figure; with common_norm=False each curve has area 1 and group sizes are hidden. Every number in this article is computed by tools/experiments/data-science/visualizing-one-feature.py in the site's repository, which joins the NHANES files through tools/experiments/data-science/exploredata.py and loads Old Faithful through dsdata.py; the figures are drawn by tools/figures/visualizing-one-feature.py.
References
- Hintze, J. L., and Nelson, R. D. (1998). Violin plots: A box plot-density trace synergism. The American Statistician, 52(2), 181–184. record
- National Center for Health Statistics (2020a). National Health and Nutrition Examination Survey 2017–2018 Data Documentation, Codebook, and Frequencies: Prescription Medications (RXQ_RX_J). Centers for Disease Control and Prevention. https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/DataFiles/RXQ_RX_J.htm record · 8 pages
- National Center for Health Statistics (2020b). National Health and Nutrition Examination Survey 2017–2018: Demographics (DEMO_J), Body Measures (BMX_J), High-Sensitivity C-Reactive Protein (HSCRP_J) and Prescription Medications (RXQ_RX_J) data files and documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- Parzen, E. (1962). On estimation of a probability density function and mode. The Annals of Mathematical Statistics, 33(3), 1065–1076. record
- Rosenblatt, M. (1956). Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statistics, 27(3), 832–837. record
- Schilling, M. F., Watkins, A. E., and Watkins, W. (2002). Is human height bimodal? The American Statistician, 56(3), 223–229. record
- SciPy developers.
scipy.stats.gaussian_kde, documentation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html record - Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. Chapman and Hall. record
- Waskom, M. L.
seaborn.kdeplot, documentation. https://seaborn.pydata.org/generated/seaborn.kdeplot.html record - Wickham, H. (2010). A layered grammar of graphics. Journal of Computational and Graphical Statistics, 19(1), 3–28. record · 2 pages