A list of 5,247 blood pressures is complete and unreadable. Before anything is estimated or tested, the list has to be turned into something a person can take in. How do we compress thousands of values into one table or picture that shows what is there, without hiding real features or inventing false ones?
The answer is a small set of standard displays. A frequency table counts how many values fall in each range, and a graph draws the same counts as lengths, areas or positions. Done well, a table or graph shows at a glance where the values are centered, how widely they spread, whether they are symmetric, whether a few are far from the rest, and how groups differ. Done carelessly, it can hide all of that or invent features that are not there.
The dataset is the one used throughout this unit: the mean systolic blood pressures of the 5,247 adults examined in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, described in Types of Numerical Data. That article's types decide which displays each variable can use. First we build frequency tables, the source of most graphs. Then come the graphs of one variable, from bar charts to box plots, and of two, scatter plots and line graphs. Finally we look at how graphs mislead.
Frequency tables
A frequency table lists the possible values or classes of a variable and how many observations fall in each.
- Dividing each frequency $f_i$ by the total $n$ gives the relative frequency $f_i/n$, the share of observations in the class. Relative frequencies are between 0 and 1 and add to 1.
- For ordered values, adding the relative frequencies of a class and all classes below it gives the cumulative relative frequency, the share at or below the top of the class.
For a nominal or ordinal variable, the classes are its categories, and the table is just a count per category. The tables of race and Hispanic origin and of self-rated health in the previous article are examples. For a continuous variable, the classes are intervals chosen by the analyst. Here are the systolic pressures in intervals of 10 mm Hg:
| Systolic (mm Hg) | Frequency | Relative frequency | Cumulative relative frequency |
|---|---|---|---|
| 70 to under 80 | 3 | 0.0006 | 0.0006 |
| 80 to under 90 | 15 | 0.0029 | 0.0034 |
| 90 to under 100 | 227 | 0.0433 | 0.0467 |
| 100 to under 110 | 809 | 0.1542 | 0.2009 |
| 110 to under 120 | 1,155 | 0.2201 | 0.4210 |
| 120 to under 130 | 1,138 | 0.2169 | 0.6379 |
| 130 to under 140 | 787 | 0.1500 | 0.7879 |
| 140 to under 150 | 447 | 0.0852 | 0.8731 |
| 150 to under 160 | 317 | 0.0604 | 0.9335 |
| 160 to under 170 | 156 | 0.0297 | 0.9632 |
| 170 to under 180 | 98 | 0.0187 | 0.9819 |
| 180 to under 190 | 46 | 0.0088 | 0.9907 |
| 190 to under 200 | 24 | 0.0046 | 0.9952 |
| 200 to under 210 | 14 | 0.0027 | 0.9979 |
| 210 to under 220 | 6 | 0.0011 | 0.9990 |
| 220 to under 230 | 3 | 0.0006 | 0.9996 |
| 230 to under 240 | 2 | 0.0004 | 1.0000 |
| Total | 5,247 | 1.0000 |
Three rules make such a table unambiguous:
- the classes must not overlap, so every value belongs to exactly one;
- they must cover every value;
- the boundary convention must be stated. Here each class includes its lower limit and excludes its upper one.
The convention matters more than it may seem. Blood pressures are recorded in even numbers and averaged, so values pile up on round numbers: 316 of the 5,247 adults have a mean of exactly a multiple of 10. Classes written as "110 to 120" and "120 to 130" without a convention would leave those people with two homes.
The table already describes the distribution. Most adults are between 100 and 140 mm Hg (78.8% minus 4.7%, about 74%). The highest frequencies are in the classes from 110 to 130. Below 100 the frequencies fall off within two classes; above 140 they decline slowly over ten classes. The distribution is skewed to the right, with a long tail of high pressures.
Check your understanding. Using only the cumulative column, which class contains the middle adult, the one with half the others below?
The class from 120 to under 130. The cumulative share is 0.4210 at the top of the class below it and 0.6379 at the top of this one, so the 50% point is crossed inside it. The median, computed from the individual values, is 123.33 mm Hg.
Relative frequencies are also what make groups of different sizes comparable. A group of 1,923 older adults and one of 1,745 younger adults cannot be compared by their counts in a class, but they can by their shares, as the frequency polygons below do. First, though, the table has to become a picture, and the kind of variable decides which picture.
Bar charts for categories
A bar chart draws one bar per category, with length proportional to its frequency or relative frequency. The bars are separated by gaps, a reminder that the categories are distinct and nothing lies between them.
For a nominal variable the order of the bars is free, and sorting them by size makes the chart easier to read. For an ordinal variable the order is part of the data, and the categories must stay in it. Sorting the self-rated health categories by size would destroy the information that "poor" is the end of a scale. Labeled values on short bars help when exact shares matter.
Why bars rather than a pie? A pie chart shows the same shares as angles of a circle. Take two nearby shares, such as the 13.5% of Mexican American adults and the 14.2% of non-Hispanic Asian adults. Their slices differ by about 2.5 degrees of arc and sit in different parts of the circle. Two bars drawn from one baseline can be compared directly by where they end.
Check your understanding. A bar chart of household size has bars for 1, 2, 3, 4, 5, 6 and "7 or more" people. Should the bars be sorted by height, as the race chart is?
No. Household size is ordered, so, like self-rated health, its bars keep their natural order, which shows the shape. The last bar also needs its label, because it is top-coded.
Histograms
Bar charts handle categories. A measurement such as blood pressure has no categories until we make them. A histogram is the graph of a frequency table for a continuous variable: a bar over each class interval, with no gaps between bars, because the classes together cover a continuous range.
A histogram shows the shape of the distribution (NIST/SEMATECH, §1.3.3.14):
- where its center is, and how spread out it is;
- whether it is symmetric or skewed;
- whether it has one peak or several;
- whether there are values far from the rest.
The left panel of the figure of three levels of detail in the previous article is the histogram of the table above.
How many bins
The picture a histogram gives depends on the width of its classes, or bins, and no width is right for every purpose. The figure shows the same 5,247 pressures with four choices.
With too few bins, the histogram is a few blocks, and the asymmetry of the distribution is barely visible. With too many, each bin holds few observations and the bar heights are dominated by chance, or, as here, by the way the data were recorded. The 200 bins expose the lattice of even readings and their averages described in Types of Numerical Data. That is a real feature of the data, worth knowing. But it is a feature of the measurement process, not of blood pressure.
Several rules choose the number of bins from the data. Each gives a starting point, not an answer.
- Sturges's rule: $k = 1 + \log_2 n$ bins, equivalently $1 + 3.322\log_{10} n$. For $n = 5247$, $1 + \log_2 5247 = 13.36$, rounded up to 14 bins. Each is $165.3/14 = 11.81$ mm Hg wide over the range from 72.7 to 238.0. The rule depends only on $n$. It is the default of R's
histfunction. The NumPy documentation notes that it assumes roughly normal data and gives too few bins for large datasets that are not normal (NumPy,histogram_bin_edges). - Scott's rule: bin width $h = \sigma\,(24\sqrt{\pi}/n)^{1/3} \approx 3.49\,\sigma n^{-1/3}$, using the standard deviation $\sigma$. Here $h = 4.04$ mm Hg, or 41 bins.
- Freedman–Diaconis rule: $h = 2\,\mathrm{IQR}\;n^{-1/3}$, using the interquartile range (IQR), the width of the middle half of the data, which an extreme value cannot inflate. Here the IQR is 24.67 mm Hg and $h = 2.84$, or 59 bins.
The last two widths shrink like $n^{-1/3}$. So more data allow narrower bins, but more slowly than the sample grows. The NumPy documentation notes that a number of bins proportional to $n^{1/3}$ is asymptotically optimal, which is why the rate appears in most such rules. It describes the rules as plug-in methods that give good starting points (NumPy, histogram_bin_edges).
Check your understanding. If the survey had examined twice as many adults, with the same distribution, how many bins would Sturges's rule give?
One more. Doubling $n$ adds exactly 1 to $\log_2 n$, so the rule moves from 13.36 to 14.36 and rounds up to 15.
For these data, the moderate choices, 14 to 41 bins, all tell the same story: a single peak between 110 and 125 mm Hg and a long right tail. A useful habit is to draw a histogram at two or three widths and believe only the features that persist.
Unequal classes and the density scale
So far every bin has had the same width. Classes need not have equal widths, though. Sparse tails are often pooled into one wide class, as in this table:
| Systolic (mm Hg) | Width | Frequency | Relative frequency | Relative frequency per 10 mm Hg |
|---|---|---|---|---|
| 70 to under 100 | 30 | 245 | 0.0467 | 0.0156 |
| 100 to under 110 | 10 | 809 | 0.1542 | 0.1542 |
| 110 to under 120 | 10 | 1,155 | 0.2201 | 0.2201 |
| 120 to under 130 | 10 | 1,138 | 0.2169 | 0.2169 |
| 130 to under 140 | 10 | 787 | 0.1500 | 0.1500 |
| 140 to under 150 | 10 | 447 | 0.0852 | 0.0852 |
| 150 to under 160 | 10 | 317 | 0.0604 | 0.0604 |
| 160 to under 240 | 80 | 349 | 0.0665 | 0.0083 |
Suppose the bars are drawn with heights equal to the relative frequencies. Then the class from 160 to 240 becomes a broad slab as tall as the bar for 150 to 160. The eye, which reads area, sees a large mass of very high pressures.
The right-hand column fixes this. Dividing each relative frequency by the class width (here, per 10 mm Hg) gives a density. On the density scale, each bar's area, height times width, equals the share of observations in the class, and the total area is 1.
The NIST/SEMATECH handbook (§1.3.3.14) describes both normalizations. Dividing counts by $n$ makes heights shares; dividing by $n$ times the class width makes the total area 1. It recommends the second whenever a histogram is to be compared with a probability density. With equal widths the two differ only by a constant, and the choice is cosmetic. With unequal widths only the density scale is honest.
Check your understanding. On the density scale, the bar for 70 to under 100 has height 0.0156 per 10 mm Hg. Is 1.56% the share of adults in that class?
No. The height is a share per 10 mm Hg, and the class is 30 mm Hg wide. The share is the area, height times width: three times the height, or 0.0467 before the height is rounded, which is the 4.67% in the relative frequency column. On a density scale, only areas are shares.
Frequency polygons: comparing groups
Each graph so far has shown one group. How do the distributions of younger and older adults differ? Overlaid histograms become cluttered.
A frequency polygon joins the tops of a histogram's bars at the class midpoints with straight lines. It adds a point of zero height one class beyond each end, so that the polygon closes on the axis. Polygons for several groups can be drawn on one set of axes.
The groups have 1,745, 1,579 and 1,923 adults, so their polygons must use relative frequencies. Counts would make the largest group look as if it had more of everything.
The figure shows more than a shift of the center. Among adults aged 18 to 39, 59.8% are in the two classes from 100 to 120 mm Hg. Among those 60 and over, the distribution is wider and more skewed, with 24.8% at 150 mm Hg or above, against 1.8% of the youngest group.
The empirical distribution function
Every graph so far has required a choice of classes. Is there a display that needs none? Cumulative relative frequencies can be graphed as well. Taken to the limit of one class per observed value, they become the empirical distribution function (ECDF), $\hat{F}(x)$, the share of observations less than or equal to $x$. It rises in steps from 0 to 1, jumping by $1/n$ at each observation, or by a multiple of $1/n$ at tied values.
The ECDF needs no choice of bins and loses no information: every observation is a step. It answers questions of the form "what share are below this value?" directly. For instance, $\hat{F}(120) = 0.4324$, so 43.24% of the adults have a mean systolic pressure of 120 or less. Likewise $\hat{F}(130) = 0.6503$ and $\hat{F}(140) = 0.7951$.
Read the other way, it gives percentiles: the value below which a given share lies. The 25th, 50th and 75th percentiles, the quartiles, are 112.0, 123.3 and 136.7 mm Hg.
Shape is harder to see in an ECDF than in a histogram. Skewness shows only as a difference in how quickly the curve approaches 0 and 1, so the two graphs complement each other. The ECDF is also the basis of percentile-based summaries and of the bootstrap (The Bootstrap).
Box plots
The quartiles just read from the ECDF lead to the most compact display of all. A box plot draws five numbers:
- a box runs from the first quartile $Q_1$ to the third quartile $Q_3$, so it contains the middle half of the data;
- a line across the box marks the median;
- lines called whiskers extend from the box toward the extremes.
In the simplest version the whiskers reach the minimum and the maximum, and the plot shows the five-number summary: minimum, $Q_1$, median, $Q_3$, maximum.
The version in most use, due to Tukey, flags values far from the rest. With the interquartile range $\mathrm{IQR} = Q_3 - Q_1$, the fences are
$$Q_1 - 1.5\,\mathrm{IQR} \quad\text{and}\quad Q_3 + 1.5\,\mathrm{IQR}.$$
Each whisker is drawn from the box to the most extreme observation that is still inside its fence, and observations beyond a fence are drawn individually (NIST/SEMATECH, §1.3.3.7). The whisker therefore ends at a data value, not at the fence itself. The fences are used only in the calculation.
| Age | $n$ | $Q_1$ | Median | $Q_3$ | IQR | Upper fence | Upper whisker ends at | Beyond the fences |
|---|---|---|---|---|---|---|---|---|
| 18 to 39 | 1,745 | 106.00 | 113.33 | 122.67 | 16.67 | 147.67 | 146.7 | 37 above |
| 40 to 59 | 1,579 | 114.00 | 124.00 | 135.33 | 21.33 | 167.33 | 167.3 | 51 above, 4 below |
| 60 and over | 1,923 | 122.67 | 134.00 | 149.33 | 26.67 | 189.33 | 189.3 | 38 above |
(The quartiles here use NumPy's default method. Measures of Center and Spread explains why different software can give slightly different quartiles.)
Box plots are compact enough to compare many groups side by side, and they show differences in center and spread at once (NIST/SEMATECH, §1.3.3.7). Their weaknesses are the other side of their compactness. A box plot cannot show two peaks, and for a few observations it gives a false impression of precision.
Are the flagged points errors?
The points beyond the fences need careful reading. With these data, 130 adults in the three groups are flagged. For all adults together, 147 are above the upper fence of 173.67 mm Hg and 2 below the lower one.
Check your understanding. For a normal distribution, 0.70% of values fall beyond Tukey's fences. If the 5,247 pressures were normal, about how many would be flagged, and would the real count be higher or lower?
About 37 (the script gives 36.6). The real count is 149, four times as many, and almost all of them are above the upper fence. A long right tail sends far more values past the upper fence than a normal curve would.
Few of the flagged adults are errors. They are the tail of a skewed distribution, and the rule of 1.5 IQR was designed with roughly symmetric data in mind. For a normal distribution, the fences lie 2.698 standard deviations from the center and 0.70% of values fall beyond them. A right-skewed distribution sends many more past the upper one. A flagged point is a value to check, by looking for a recording error or a reason for it, not a value to delete (NIST/SEMATECH, §1.3.5.17).
Every display so far has described one variable at a time. Many questions are about how two variables move together.
Scatter plots
Two measurements on the same people are shown together in a scatter plot (or two-way scatter plot), one point per person.
The plot shows things no table of either variable alone can. Systolic pressure tends to increase with age. The mean rises from 110.79 mm Hg among adults in their late teens (ages 18 and 19) to 116.99 in their thirties, 130.10 in their fifties and 137.17 in their seventies. The spread increases as well, which is the box plots' widening boxes seen another way.
The association is clear but loose: at every age the pressures are widely scattered around the mean. The correlation coefficient, which measures how closely the points follow a straight line, is $r = 0.487$. Its definition, and the least-squares line through a scatter plot, are developed in Simple Linear Regression.
Check your understanding. Does $r = 0.487$ mean that systolic pressure rises by about 0.487 mm Hg per year of age?
No. The correlation coefficient has no units and does not change if age is measured in months or pressure in kilopascals. It measures how tightly the points cluster around a line, not how steep the line is. The steepness, in mm Hg per year, is the slope of the regression line.
The plot also shows a recording artifact. The dense column at 80 years is the 350 adults aged 80 and over, all recorded as 80. A regression of pressure on age that ignored this would treat a 95-year-old as 80.
Line graphs
A line graph joins points in the order of the horizontal variable. It suits data where that order is meaningful, most often time: monthly case counts, yearly death rates, a patient's temperature through the day. The line in the scatter plot above is a line graph of mean pressure against year of age.
Joining the points emphasizes trend. It also invites the reader to interpolate between them, which is reasonable for time or age and meaningless for categories.
The mean at each year of age is based on between 40 and 350 adults, so the line wobbles from year to year by a few mm Hg. At age 18 the mean is 111.65 (128 adults), and at age 79 it is 140.28 (41 adults). Those wobbles are mostly sampling variation. A line graph shows every fluctuation with the same confidence. Before reading meaning into a single jump, a reader should ask how many observations stand behind each point.
Graphs that mislead
A graph is a set of encoding rules: the length of a bar stands for a frequency, the area of a histogram bar for a share, the position of a point for a value. A graph misleads when it breaks its own rule. The unequal-width histogram above is one example. The most common is a bar chart whose axis does not start at zero.
Among the nonpregnant NHANES adults whose hypertension status is defined in The Normal Approximation to the Binomial, 58.56% of men and 51.08% of women have hypertension. The men's proportion is $0.5856/0.5108 = 1.15$ times the women's.
With the axis starting at 0.50, the visible parts of the bars are $0.5856 - 0.50 = 0.0856$ and $0.5108 - 0.50 = 0.0108$, in a ratio of 7.93. A reader comparing the bars, which is what bars invite, sees an eightfold difference where the real one is 15%. The numbers printed on the bars are correct, and the graph still misleads, because the length of the bar is its message.
Check your understanding. Suppose the axis started at 0.40, or at 0.25. Would the distortion disappear?
It would shrink but not disappear. From 0.40 the visible bars are 0.1856 and 0.1108, a ratio of 1.68. From 0.25 they are 0.3356 and 0.2608, a ratio of 1.29. Only a baseline at zero makes the ratio of the bars equal the true ratio of 1.15.
The rule is specific to encodings by length or area. A bar's length means "how much", so it must be measured from zero. A point's position does not. A dot plot or line graph of the same two proportions may use a narrow axis, since the reader compares positions. Starting a line graph of blood pressure by age at zero would only flatten the trend into the bottom of the frame.
Other common distortions follow the same logic:
- a pictogram that scales an icon in both height and width turns a doubling of the value into a fourfold area;
- bars and pies drawn in three dimensions add depth and perspective that stand for no value, so the drawn sizes no longer follow the encoding rule;
- two vertical scales on one chart can each be stretched or shifted independently, so any two lines can be made to appear to move together.
Choosing a display
| Question | Variable | Display |
|---|---|---|
| How many in each category? | nominal or ordinal | frequency table, bar chart |
| What is the shape of the distribution? | discrete or continuous | histogram (density scale for unequal classes) |
| What share is below a value, or what value cuts off a share? | ordinal, discrete or continuous | cumulative table, ECDF |
| How do several groups compare? | continuous, by group | box plots side by side, relative frequency polygons |
| Are two measurements related? | two continuous | scatter plot |
| How does a quantity change over time? | ordered by time | line graph |
Corrections to the course notes
The unit's notes give the box plot whiskers as reaching $\max(\text{minimum}, Q_1 - 1.5\,\mathrm{IQR})$ and $\min(\text{maximum}, Q_3 + 1.5\,\mathrm{IQR})$, that is, to the fence when there are values beyond it. In Tukey's construction, and in the NIST/SEMATECH handbook (§1.3.3.7), the whisker ends at the most extreme observation inside the fence. That is short of the fence unless a value lies exactly on it.
The upper whiskers in the table above show both cases. For the youngest group the whisker ends at 146.7, short of the fence at 147.67. In the two older groups a recorded mean pressure happens to lie exactly on the upper fence, so there the two rules agree.
The notes also state that a bar chart's axis must start at zero because the bars represent quantities through area. The reason is that they represent quantities through length (and, for bars of equal width, area). The same requirement does not apply to graphs that encode values by position.
What we have learned
A frequency table, with relative and cumulative columns, is the source of most graphs, and the kind of variable decides which graph to draw from it: bars in a meaningful order for categories, a histogram for a measurement. A histogram's message depends on its bins, so it is worth drawing at more than one width, and unequal classes need the density scale. The ECDF and the box plot avoid bins altogether, and the box plot's flagged points in skewed data, such as these blood pressures, are mostly tail rather than error. Every graph is a set of encoding rules, and a truncated bar axis breaks the most important one.
Graphs show the whole distribution, but a report often has room for two numbers, one for center and one for spread. Which two, and what each is sensitive to, is the subject of the next article, Measures of Center and Spread.
In Python
import numpy as np
import matplotlib.pyplot as plt
counts, edges = np.histogram(sbp, bins=np.arange(70, 250, 10)) # frequency table, [a, a + 10)
rel = counts / counts.sum() # relative frequencies
cum = np.cumsum(rel) # cumulative
np.histogram_bin_edges(sbp, bins='sturges') # or 'fd', 'scott', 'auto'
plt.hist(sbp, bins=edges, density=True) # density scale: areas are shares
plt.boxplot([young, middle, old], vert=False, whis=1.5) # Tukey's fences at 1.5 IQR
plt.ecdf(sbp) # matplotlib 3.8 or later
NumPy's histogram uses half-open bins $[a, b)$ except for the last, which includes its upper edge. Every number in this article is computed by tools/experiments/applied-statistics/tables-and-graphs-for-data.py in the site's repository, and every figure is drawn by tools/figures/tables-and-graphs-for-data.py from that script's output.
References
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographic Variables and Sample Weights (DEMO_J), Blood Pressure (BPX_J) and Blood Pressure and Cholesterol (BPQ_J) data files and documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- NIST/SEMATECH. e-Handbook of Statistical Methods, §1.3.3.7 (Box plot), §1.3.3.14 (Histogram) and §1.3.5.17 (Detection of outliers). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
- NumPy Developers.
numpy.histogram_bin_edges, documentation for NumPy 2.2. https://numpy.org/doc/stable/reference/generated/numpy.histogram_bin_edges.html record - Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages