A health survey hands us 5,247 adults. Each has a sex code, an age, a race code, a household size, up to three blood pressure readings, a height and a rating of their own health. Every entry is a number, and software will average any column on request. Which of those averages mean anything, what does the whole column look like, and which one or two numbers summarize it honestly? This first unit of the course answers those questions.
Descriptive statistics is the work of saying what a set of data contains, through classification, tables, graphs and numerical summaries, before anything is estimated or tested. It is less routine than it looks. A column of codes may be names that cannot be averaged, or ratings whose average depends on an unstated assumption. A histogram can show a long tail or hide it, depending on the width of its bars. A mean and a standard deviation describe a symmetric distribution well and a skewed one poorly. A single misplaced decimal point can move them almost anywhere, while the median does not notice.
The unit covers the subject in three steps:
- first, what kind of measurement each variable is, and what goes wrong when a summary assumes more than the data contain;
- then, the standard tables and graphs, and the rules that keep them honest;
- finally, the numbers that summarize center and spread, what makes each one the right answer to a particular question, and how the familiar rules of thumb fare on real data.
One population runs through all three articles. It is the 5,247 adults examined in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 who had a systolic blood pressure reading. Later units of the course sample from the same adults to check confidence intervals and tests. Their records include every kind of variable. Race and Hispanic origin, and sex, are nominal. Self-rated health is ordinal, and household size is discrete. Blood pressure, height and body mass index (BMI) are continuous.
The data also carry the marks that recording rules leave: blood pressures recorded only in even numbers, and every age of 80 and over recorded as 80. Every number comes from a script in the site's repository.
Articles
- Types of Numerical Data: which averages of a coded column mean anything? Nominal, ordinal, discrete and continuous data, on one participant's record and on the whole survey. Why the mean of race codes is meaningless and the mean of a 0/1 variable is a proportion. When can two groups' mean ratings be compared? A proof that their order survives every order-preserving coding exactly when one group's cumulative distribution lies on or above the other's, shown on self-rated health in two pairs of age groups. Categories cut from measurements; ranks; top-coding and even-numbered blood pressures; Stevens's (1946) scales of measurement and Velleman and Wilkinson's (1993) criticism.
- Tables and Graphs for Data: how do we see 5,247 values at once, without inventing or hiding features? Frequency, relative and cumulative tables, and bar charts for nominal and ordinal data. Histograms, the bin rules of Sturges, Scott and Freedman and Diaconis, and the density scale for unequal classes. Relative frequency polygons for three age groups, and the empirical distribution function. Box plots with Tukey's fences, and why most flagged points in skewed data are not errors. Scatter plots and line graphs, and a truncated axis that turns a 15% difference in hypertension prevalence into an eightfold one.
- Measures of Center and Spread: if a report has room for two numbers, which two? The mean, median and mode worked on eleven blood pressures, with proofs that the mean minimizes squared deviations and the median absolute ones. Skewness, the mean-median rule and its exceptions, and resistance to a recording error. Range, quartiles (and the five methods software uses for them), variance with the divisor $n - 1$, standard deviation and coefficient of variation. z-scores, the empirical rule checked against heights and blood pressures, Chebyshev's inequality proved, and summaries from grouped data.
How this unit relates to the Data Science collection
The Library's Data Science collection has its own articles on summarizing data with pandas. Its Data Exploration topic covers visualizing data with one feature, exploratory data analysis and detecting outliers. The two treatments complement each other. This unit defines the summaries and graphs, proves their properties and ties them to the kinds of measurement found in health data. The Data Science articles focus on the working routine of exploring a new dataset with pandas and plotting libraries, many variables at a time. They link back here for the definitions.
Related topics
The ideas of this unit are used throughout the rest of the course, and later units link back here rather than define them again.
- Proportions, introduced here as means of 0/1 variables, are the subject of Inference on Proportions.
- The divisor $n - 1$ is derived in Confidence Intervals for a Mean.
- The sum of squared deviations reappears as the decomposition of variation in One-Way Analysis of Variance.
- Ranks, the ordinal transformation of any ordered data, are the basis of Nonparametric Methods.
- Scatter plots and the correlation coefficient lead to Simple Linear Regression, and the empirical distribution function to the bootstrap.
The next unit, Rates and Standardization, turns from describing one population to comparing populations of different sizes and age structures.
References
- NIST/SEMATECH. e-Handbook of Statistical Methods, §1.3.3.7 (Box plot), §1.3.3.14 (Histogram), §1.3.5.1 (Measures of location) and §1.3.5.6 (Measures of scale). National Institute of Standards and Technology. https://www.itl.nist.gov/div898/handbook/ record · 30 pages
- Pagano, M., Gauvreau, K., and Mattie, H. (2022). Principles of Biostatistics (3rd ed.). CRC Press. The course text. record · 35 pages
- Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680. record · 2 pages
- Velleman, P. F., and Wilkinson, L. (1993). Nominal, ordinal, interval, and ratio typologies are misleading. The American Statistician, 47(1), 65–72. record · 2 pages