Standard Scores, Chebyshev's Inequality and Grouped Data

The standard deviation used as a ruler. z-scores, which have mean 0 and standard deviation 1; the empirical rule checked against NHANES 2017-2018 heights, blood pressures, body mass index and household size; Chebyshev's inequality proved, for population and sample standard deviations; and the mean, standard deviation, median and quartiles recovered from a frequency table, with Sheppard's correction and the effect of a recording lattice.

A systolic blood pressure of 180 mm Hg and a height of 190 cm are both large values. Which is the more unusual? The pressure is. Among the 5,247 adults of the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 it lies 2.665 standard deviations above the mean, and the height lies 2.143 standard deviations above the mean of the men. Counting in standard deviations puts values from different distributions on one scale.

This article uses the standard deviation as a ruler. It continues Measures of Center and Spread, which defines the mean, the median, the variance and the standard deviation, proves the rules for changes of units that are used below, and defines the skewness coefficient quoted here. The data are the same NHANES adults. "The eleven adults" are the sample of eleven systolic pressures worked by hand in that article, with mean 126.55 and standard deviation 23.52 mm Hg. Their largest value is 170.7.

First we define z-scores. Then we test the empirical rule on real data, and see why it is a weak test of normality. Next we prove Chebyshev's inequality, which holds for data of any shape. It is numbered Proposition 3, after the two propositions of the previous article. Finally we recover a mean, a standard deviation and a median from a frequency table, when the individual values are not available.

Standard deviations as a ruler

z-scores

Shifting by the mean and dividing by the standard deviation expresses a value as a number of standard deviations from the mean, its z-score:

$$z = \frac{x - \bar{x}}{s} \quad\text{(or } (x - \mu)/\sigma \text{ for a population).}$$

By the rules for changes of units proved in Measures of Center and Spread, with $a = -\bar{x}/s$ and $b = 1/s$, z-scores have mean $-\bar{x}/s + \bar{x}/s = 0$ and standard deviation $s/s = 1$. A systolic pressure of 180 mm Hg has $z = (180 - 126.36)/20.12 = 2.665$ among the 5,247 adults. A man 190 cm tall has $z = 2.143$ among the men. The z-score makes values from different distributions comparable in the sense of "how unusual". It is the standardization that The Normal Distribution uses to reduce every normal distribution to one table.

Example (comparing a height with a pressure). The man on the first line of Types of Numerical Data is 185.0 cm tall, and that is less unusual among men than a systolic pressure of 180 is among all adults. His height has $z = 1.492$, about one and a half standard deviations above the men's mean, while the pressure is 2.665 standard deviations above the adults' mean.

The empirical rule

For data whose histogram is close to a normal curve, the empirical rule says that about 68% of the values lie within one standard deviation of the mean, about 95% within two, and about 99.7% within three. The exact normal shares are 0.6827, 0.9545 and 0.9973; The Normal Distribution computes them. The rule is a property of the normal distribution. It describes other data only as well as the normal curve describes them.

Example (a skewed distribution within one standard deviation). The blood pressures are skewed to the right, with a peak between 110 and 125 mm Hg and a long upper tail, and more than 68% of them lie within one standard deviation of the mean: 70.92%. The few values far out in the tail add large squared deviations, so they inflate the standard deviation. The one-standard-deviation band is then wide compared with the bulk of the data, and it holds more of the bulk than the normal curve's 68%. This is a tendency of long-tailed data rather than a rule, and BMI in the table below shows it too.

Within $k$ standard deviations of the mean $k = 1$ $k = 2$ $k = 3$
Normal distribution 0.6827 0.9545 0.9973
Height of men 0.6869 0.9503 0.9984
Systolic pressure 0.7092 0.9537 0.9886
BMI 0.7307 0.9585 0.9869
Household size 0.6484 0.9552 1.0000
Chebyshev's guarantee, any data at least 0 at least 0.75 at least 0.8889

Men's heights follow the rule closely. The skewed variables have too many values within one standard deviation, about the right share within two, and too many beyond three. For example, 1.14% of the blood pressures are more than three standard deviations from the mean, 4.24 times the normal share of 0.27%. Household size, with only seven possible values, matches at two standard deviations by coincidence.

Example (a common misstatement). "The empirical rule is a quick check for normality" asks more of the rule than it can give. The table shows data that match the rule at two standard deviations while being clearly skewed: the blood pressures have 95.37% within two standard deviations, against the normal 95.45%, and a skewness of 1.038. The rule is a useful description of data already known to be roughly normal and a weak test of normality. Agreement at two standard deviations says little, and the shape is better judged from a histogram or a normal probability plot.

Chebyshev's inequality

The last row of the table holds for data of any shape. The idea in one sentence: values far from the mean contribute large squared deviations, and the variance is a fixed total, so not many values can be far away.

Proposition 3 (Chebyshev's inequality). In any population of $N$ values with mean $\mu$ and standard deviation $\sigma > 0$, and for any $k > 0$, the share of values at least $k\sigma$ from the mean is at most $1/k^2$.

Proof. Let $m$ be the number of values with $|x_i - \mu| \ge k\sigma$. Every term of the variance is nonnegative, so dropping the terms of the other values can only decrease the sum. Each remaining term is at least $k^2\sigma^2$:

$$\sigma^2 = \frac{1}{N}\sum_{i=1}^{N}(x_i - \mu)^2 \ \ge\ \frac{1}{N}\sum_{|x_i - \mu| \ge k\sigma}(x_i - \mu)^2 \ \ge\ \frac{m}{N}\,k^2\sigma^2.$$

Dividing by $k^2\sigma^2$ gives $m/N \le 1/k^2$. $\square$

So at least $1 - 1/k^2$ of any data lie within $k$ standard deviations of the mean: at least 75% within two, and at least 88.9% within three. The same holds for a sample with its standard deviation computed either way. With divisor $n$, the sample is a population of $n$ values and the proposition applies as it stands. The standard deviation $s$ with divisor $n - 1$ is larger than that one, so every value at least $ks$ from the mean is also at least $k$ of the smaller standard deviations from it. The share of such values is therefore again at most $1/k^2$. For the eleven adults and $k = 1.5$, the bound allows at most $11/2.25 = 4.89$, so four, values at least $1.5 \times 23.52 = 35.28$ mm Hg from the mean. There is one, 170.7. Applied directly to $(n-1)s^2 = \sum (x_i - \bar{x})^2 \ge m k^2 s^2$, the same argument gives the slightly sharper bound $m \le (n-1)/k^2$ on the number $m$ of such values: here $10/2.25 = 4.44$, still four.

On the NHANES data the comparison is easy: every variable in the table clears Chebyshev's bound by a wide margin, the blood pressures with 95.37% within two standard deviations against the guaranteed 75%. The bound is far below what roughly normal data achieve. That is the price of assuming nothing about shape, and for $k \le 1$ it says nothing at all. It is what makes the standard deviation meaningful for every distribution, not only normal ones.

Summaries from a frequency table

Everything so far has used the individual values. Many published reports give only a frequency table. Can the mean and standard deviation be recovered from it?

Approximately. Treat every value in a class as if it were at the class midpoint $m_j$. With class frequencies $f_j$ and $n = \sum f_j$,

$$\bar{x} \approx \frac{\sum_j f_j m_j}{n}, \qquad s^2 \approx \frac{\sum_j f_j (m_j - \bar{x})^2}{n - 1}.$$

The first is a weighted mean, with the frequencies as weights.

Example (the grouped mean of the blood pressures). The blood pressure distribution declines to the right of its peak, and the grouped mean, computed from the 10 mm Hg frequency table, comes out above the exact mean. In a declining tail, more of each class's values sit near its lower end, so the midpoint overstates them. Applied to the frequency table of Tables and Graphs for Data, the formulas give a mean of 126.68 mm Hg and a standard deviation of 20.35, against the exact 126.36 and 20.13 from the individual values. The values in a class sit, on average, 0.32 mm Hg below its midpoint, which is exactly the gap between the two means. Two causes add up. One is the declining tail: in the classes from 140 up the values sit 0.83 mm Hg below their midpoints. The other is the recording lattice. Mean pressures are averages of up to three even readings, so in a class from $a$ to under $a + 10$ they can take only twenty values, $a,\ a + \tfrac23,\ a + 1,\ \ldots,\ a + 9\tfrac13$: the multiples of $\tfrac23$ and the odd whole numbers. The largest is $a + 9\tfrac13$, not $a + 10$, and the twenty values average $a + 4.75$, a quarter of a mm Hg below the midpoint, whatever the shape of the distribution. Below 140 the offset is 0.18 mm Hg, smaller than that quarter, so there the lattice is enough to account for it.

Grouping also inflates the variance slightly. A classical adjustment, Sheppard's correction, subtracts $h^2/12$ from the grouped variance for classes of equal width $h$. Sheppard (1898) derived it for a smooth distribution that tapers off at both ends, with classes narrow compared with its spread. The idea behind it: a midpoint is the value plus a rounding error. For such a distribution the error is spread nearly evenly over an interval of width $h$, which gives it variance $h^2/12$, and it is nearly uncorrelated with the value, so the two variances add. The derivation is cited here, not proved. Here $h^2/12 = 100/12 = 8.33$, and the corrected standard deviation is 20.15, close to the exact 20.13.

A median and quartiles can be read from the cumulative column of the table by interpolating within a class. If the class containing the middle observation starts at $L$, has width $h$ and frequency $f$, and $F$ observations lie below it, the grouped median is

$$L + h\,\frac{n/2 - F}{f},$$

which treats the class's values as evenly spread across it. For the blood pressures the class is 120 to under 130, with 2,209 adults below it and 1,138 in it, so the grouped median is $120 + 10 \times (2{,}623.5 - 2{,}209)/1{,}138 = 123.64$, against the exact 123.33. Quartiles use $n/4$ and $3n/4$ in place of $n/2$.

Worked problems

The problem is an exercise from the course text's chapter on descriptive statistics (Pagano, Gauvreau and Mattie, 2022), restated in our own words with its data. The script named under "In Python" computes every number and asserts the main ones.

Problem 1 (serum zinc, from a frequency table). The serum zinc levels of 462 males aged 15 to 17, in micrograms per deciliter, have this frequency distribution:

Serum zinc (µg/dl) Males Relative frequency
50–59 6 0.0130
60–69 35 0.0758
70–79 110 0.2381
80–89 116 0.2511
90–99 91 0.1970
100–109 63 0.1364
110–119 30 0.0649
120–129 5 0.0108
130–139 2 0.0043
140–149 2 0.0043
150–159 2 0.0043
Total 462 1.0000

(a) What type of data is serum zinc level? (b) Compute the relative frequencies and describe the distribution. (c) Find the mean, median, range, interquartile range and standard deviation. (d) Which measure of center, and which measure of spread, should be reported?

The exercise in the text computes the summaries from the 462 individual measurements, which are in a dataset distributed with the book. This solution uses only the table, so its summaries are the grouped approximations of the section on frequency tables.

Solution. (a) Continuous: a concentration, recorded here to the whole microgram per deciliter and then grouped.

(b) The relative frequencies are in the third column; for example $6/462 = 0.0130$. The distribution has one peak, in the classes from 70 to 89, which together hold 48.9% of the males. It is skewed to the right: below the peak the frequencies reach zero within two classes, and above it they decline over seven classes, with six males (1.3%) at 130 or above. A histogram on the eleven classes, which all have width 10, shows this shape directly; the exercise asks for one, and it is not drawn here.

(c) The classes hold whole numbers, so the class "50–59" covers the interval from 49.5 to 59.5, with midpoint 54.5 and width 10. The midpoints are 54.5, 64.5, and so on up to 154.5.

  • Mean. $\sum_j f_j m_j = 6 \times 54.5 + 35 \times 64.5 + \cdots + 2 \times 154.5 = 40{,}689.0$, so $\bar{x} \approx 40{,}689.0/462 = 88.07$ µg/dl.
  • Median. Half of 462 is 231. The cumulative frequencies are 6, 41, 151 and 267, so the middle observation is in the class 80–89, which starts at 79.5, with 151 observations below it and 116 in it. The grouped median is $79.5 + 10 \times (231 - 151)/116 = 86.40$ µg/dl.
  • Standard deviation. $\sum_j f_j (m_j - \bar{x})^2 = 115{,}807.1$, so $s^2 \approx 115{,}807.1/461 = 251.21$ and $s \approx 15.85$ µg/dl. Sheppard's correction subtracts $100/12 = 8.33$ from the variance and gives 15.58.

  • Interquartile range. The same interpolation gives quartiles of 76.27 and 98.24, an IQR of 21.96 µg/dl.

  • Range. The table only bounds it. The smallest value is between 50 and 59 and the largest between 150 and 159, so the range is between $150 - 59 = 91$ and $159 - 50 = 109$ µg/dl. The exact range needs the individual measurements.

(d) The mean, 88.07, is above the median, 86.40, as the right skew suggests, but the gap is about a tenth of a standard deviation. Either center describes these data, and the median with the IQR is the safer pair because the few values above 130 pull on the mean and inflate the standard deviation. If the purpose is inference about the mean zinc level, the mean and standard deviation are the ones to report, as the section on what to report in Measures of Center and Spread explains.

What we have learned

A z-score counts standard deviations from the mean, and z-scores have mean 0 and standard deviation 1. The empirical rule describes roughly normal data, such as men's heights, and fits skewed data well at two standard deviations and poorly at one and three, so it is a weak test of normality. Chebyshev's inequality needs no assumption about shape: at least $1 - 1/k^2$ of any data lie within $k$ standard deviations of the mean. From a frequency table, class midpoints recover the mean and the standard deviation approximately, and interpolation in the cumulative column recovers the median and the quartiles.

This completes the description of one population. The next unit compares populations of different sizes and age structures, beginning with Rates in Public Health.

In Python

import numpy as np

z = (x - x.mean()) / x.std(ddof=1)                    # z-scores: mean 0, standard deviation 1
np.mean(np.abs(x - x.mean()) <= 2 * x.std())          # share within two standard deviations
f, edges = np.histogram(x, bins=np.arange(70, 250, 10))
mid = edges[:-1] + 5                                  # class midpoints
gm = (f * mid).sum() / f.sum()                        # grouped mean
gs = np.sqrt((f * (mid - gm) ** 2).sum() / (f.sum() - 1))   # grouped standard deviation

Every number in this article is computed by tools/experiments/applied-statistics/measures-of-center-and-spread.py in the site's repository, the script it shares with Measures of Center and Spread.

References