A table of 5,533 adults and 20 columns, joined from nine public files of the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018, arrives ready for analysis: blood pressure, body measurements, five blood tests, smoking and prescription medicines for every adult the survey examined. A first call to describe() reports, among other things, that the mean of the race and ethnicity column is 3.504. That number means nothing, because the column holds codes, and the fact that it was computed at all is the first finding of the exploration. Six steps later the same routine has found lab results missing in blocks, a waist measurement that fails twice as often for the oldest adults, hundreds of values that were never measured but filled in, a diastolic blood pressure of 8 mm Hg created by a processing rule, and one strong relationship that no one asked about: blood cadmium is almost four times as high in current smokers as in people who have never smoked.
This article presents a systematic routine for that first look at a dataset with many features, and follows it through the NHANES table step by step. The individual plots it uses are covered in Visualizing Data with One Feature and Visualizing Data with Multiple Features, the work of repairing what it finds in the Data Wrangling topic (Introduction to Data Wrangling, Cleaning Data), and the methods for unusual values in Detecting Outliers.
What exploratory analysis is
John Tukey gave the activity its name and its program. In Exploratory Data Analysis (1977) he described it as detective work: looking at data to see what they seem to say, with simple arithmetic and many pictures, before, and as a guide to, the formal work of estimating and testing. He contrasted it with confirmatory analysis, which tests a hypothesis stated in advance and quantifies the evidence, and he argued that both are needed (Tukey, 1980): exploration without confirmation finds patterns that may be chance, and confirmation without exploration tests the wrong hypotheses on data it has not understood.
In data science practice, exploration has three overlapping aims:
- Data quality. Find what is wrong or strange in the data: coding errors, impossible values, missing values and their pattern, artifacts of collection and processing. These are cheap to fix early and expensive to discover after a model is built on them.
- Structure. Learn the shapes of the distributions, the relationships among the features and the groups in the data, which decide what transformations, models and summaries are appropriate.
- Questions. Produce hypotheses worth testing and questions worth asking of the people who collected the data.
The steps below are a routine, not a recipe. Each step can send the analyst back to an earlier one, and the most useful findings often come from following something odd that a step turned up.
The data
The table has one row per adult aged 18 or over who was interviewed and examined at an NHANES mobile examination center, 5,533 adults in all. It is built by joining, on the respondent number SEQN, the demographics file to eight others: blood pressure (each adult's mean of up to three readings), body measures, glycohemoglobin (HbA1c), total and HDL cholesterol, high-sensitivity C-reactive protein (CRP), blood metals (lead and cadmium), the smoking questionnaire, and the prescription medicines file, reduced to one count per person (National Center for Health Statistics, 2020). NHANES is a complex survey whose sampling weights would be needed to estimate quantities for the US population; this exploration describes the examined sample itself and ignores the weights, as the other articles on the site do.
Step 0: the question and the provenance
Before computing anything, two things should be written down. The first is the question, even a vague one ("how do blood pressure, body size and blood markers relate in adults?"), because it decides which columns matter and which findings are interesting. The second is the provenance of every column: who measured it, how, on whom, in what units, and with what codes for refusals, unknown values and values below a laboratory's detection limit. For NHANES this is the codebook of each file, and several findings below are explained, or only explainable, by it. The blood pressure documentation states that a diastolic reading can be zero; the laboratory files state their detection limits and how values below them were filled in; the prescription file warns that it has one row per medicine, not per person, and must be reduced before joining. Exploration without the codebook rediscovers its contents slowly and sometimes wrongly.
Step 1: structure
Size and keys. The table has 5,533 rows and 20 columns. SEQN is unique, and every join was checked to be one-to-one (pandas' merge(..., validate="one_to_one") raises an error otherwise), so no adult was duplicated or silently dropped by a join. Enriching Data shows what an unchecked join on a non-unique key can do.
Types. pandas reads 13 columns as floating point, 4 as integers, 2 as true or false (the detection-limit flags) and 1 as text (smoking status). The integers are the problem. Sex is stored as 1 and 2, and race and Hispanic origin as six codes from 1 to 7 (5 is unused). describe() treats them as numbers and reports a mean race code of 3.504 with standard deviation 1.655, numbers with no meaning, since the codes are labels whose order is arbitrary (Types of Numerical Data). Converting such columns to a categorical type, with their labels, before anything else prevents nominal codes from being averaged, correlated or fed to a model as quantities. The race counts are then the right summary: 1,898 non-Hispanic White, 1,283 non-Hispanic Black, 793 non-Hispanic Asian, 752 Mexican American, 519 other Hispanic and 288 other or multiracial adults.
Units and ranges. Each numerical column's minimum and maximum, compared with its documented units and a plausible range, catches unit errors and codes for missing values stored as numbers (the "9999" of many files). In this table the special codes were removed during preparation; the ranges are examined in step 3.
Step 2: missing values
How much. Age is complete; CRP is missing for 387 adults (7.0%), cholesterol and HDL for 357, waist circumference for 349, diastolic pressure for 301, systolic for 286 and body measurements for 89 to 99. Only 4,669 adults, 84.4%, have all fourteen numerical columns. An analysis that silently drops incomplete rows, as many model-fitting functions do, would lose one adult in six.
Which together. Missing values in different columns are rarely independent, and their joint pattern often reveals how the data were collected:
- Cholesterol and HDL are missing for exactly the same 357 adults: they come from the same blood specimen and the same laboratory run.
- 262 adults are missing all six laboratory columns, and 396 are missing at least one, so most missing lab values belong to adults who gave no blood sample at all.
- Every adult missing systolic pressure is also missing diastolic, as the protocol requires ("if there is no systolic BP, there can be no diastolic BP"); 15 more have a systolic but no usable diastolic reading, which step 3 explains.
- 266 adults have a height but no waist circumference.
Who is missing. Comparing missingness across groups shows whether the complete cases are representative. For systolic pressure, cholesterol and cadmium the share missing varies only between about 3% and 8% across age groups, with no steady trend. Waist circumference is the exception: it is missing for between 4.5% and 5.8% of adults in each age group up to 69, and for 11.3% of those aged 70 and over. Whatever the reason, which the examination documentation should explain, an analysis of waist circumference that drops missing values will under-represent the oldest adults. The mechanisms behind missing values and the remedies, from complete-case analysis to imputation, are the subject of Cleaning Data; exploration's job is to measure the pattern and ask the question.
Step 3: one column at a time
The next step looks at every column's distribution, with a summary table and a small plot of each.
| Column | n | Distinct | Mean | Median | SD | Skewness | Min | Max |
|---|---|---|---|---|---|---|---|---|
| age (years) | 5,533 | 63 | 49.8 | 51.0 | 18.6 | −0.06 | 18 | 80 |
| systolic (mm Hg) | 5,247 | 237 | 126.4 | 123.3 | 20.1 | 1.04 | 72.7 | 238.0 |
| diastolic (mm Hg) | 5,232 | 163 | 72.5 | 72.7 | 12.0 | 0.18 | 8.0 | 135.3 |
| height (cm) | 5,444 | 505 | 166.4 | 165.9 | 10.1 | 0.14 | 138.3 | 197.7 |
| weight (kg) | 5,444 | 1,022 | 82.5 | 78.9 | 23.0 | 1.12 | 32.6 | 242.6 |
| BMI (kg/m²) | 5,434 | 409 | 29.7 | 28.5 | 7.4 | 1.26 | 14.2 | 86.2 |
| waist (cm) | 5,184 | 810 | 100.2 | 98.9 | 17.2 | 0.48 | 56.4 | 169.5 |
| HbA1c (%) | 5,261 | 94 | 5.84 | 5.60 | 1.09 | 3.19 | 3.8 | 16.2 |
| cholesterol (mg/dL) | 5,176 | 250 | 186.9 | 183.0 | 41.2 | 0.67 | 76 | 446 |
| HDL (mg/dL) | 5,176 | 107 | 53.2 | 51.0 | 15.6 | 1.31 | 10 | 189 |
| CRP (mg/L) | 5,146 | 1,191 | 4.17 | 1.94 | 8.20 | 7.89 | 0.11 | 182.82 |
| cadmium (µg/L) | 5,264 | 287 | 0.479 | 0.310 | 0.566 | 5.30 | 0.07 | 13.03 |
| lead (µg/dL) | 5,264 | 491 | 1.25 | 0.92 | 1.41 | 9.26 | 0.05 | 42.48 |
| prescription medicines | 5,524 | 22 | 2.30 | 1 | 3.08 | 1.79 | 0 | 22 |
Several things stand out, each a question about how the values came to be.
Skew and scale. CRP, lead and cadmium have skewness of 5 to 9 and means far above their medians; on a log scale their skewness falls to 0.35, 0.27 and 0.34. Any analysis of them, from a correlation to a regression, is better done on the logarithms (Visualizing Data with One Feature discusses log scales). HbA1c, HDL, BMI, weight and systolic pressure are moderately right-skewed.
Recording rules. Age has 63 distinct values: the whole years 18 to 79 and 80, which stands for 80 and over. The histogram's last bar holds 382 adults, against 43 at 79 and 43 at 78. Systolic pressure has only 237 distinct values in 5,247 adults, because the readings are even numbers and the mean of up to three of them takes few values; 35.1% of the means are whole even numbers. HbA1c is reported to one decimal place, so it has 94 values. These are properties of the measurement process, described in Types of Numerical Data, and they matter for plots (overplotting) and for any method that assumes continuous data.
Values that were not measured. On a log axis, cadmium has a separate bar at 0.07 µg/L. The laboratory's detection limit for blood cadmium is 0.100 µg/L, and results below it are recorded as the limit divided by $\sqrt{2}$, 0.0707, which the file stores as 0.07 (National Center for Health Statistics, 2020). 357 adults, 6.8% of those measured, have this filled-in value. They are not 357 people with the same cadmium level; they are people whose level is somewhere below 0.100. Treating the filled value as a measurement understates the uncertainty and can distort a distribution or a model fitted to it. CRP has the same rule, with 12 adults at its filled value, and so does lead, with one. A flag column recording which values are below the limit, which the files provide, should travel with the data.
Extreme values. The maxima include a weight of 242.6 kg, a BMI of 86.2, an HbA1c of 16.2%, an HDL of 189 mg/dL, a blood lead of 42.48 µg/dL and a CRP of 182.82 mg/L. All are possible, and the next step for each would be a look at the rest of the person's record. The minimum diastolic pressure is a different case. One adult has a mean diastolic pressure of 8 mm Hg, and 21 have means below 40. The raw readings of the adult at 8 are 0, 0 and 8 mm Hg. The blood pressure documentation says only that a diastolic reading can be zero (Cleaning Data discusses these zeros and how a file reader can hide them); in preparing the table, readings of 0 were treated as missing, following the convention of the site's other blood pressure articles, and the mean of the one remaining reading is 8. The value is an artifact of a processing decision interacting with an unusual measurement, and it would not exist under a different rule. Detecting Outliers takes up this case.
Counts. The number of prescription medicines is zero for 41.4% of adults, and 199 adults report ten or more, up to 22. A mean of 2.30 describes this distribution poorly; the share with none and the distribution among the others are better summaries.
Step 4: pairs
With the columns understood one at a time, the next step looks at how they relate, using a correlation heatmap to survey all pairs and scatter plots for the pairs it points to, as described in Visualizing Data with Multiple Features. The strongest correlations in this table are expected: weight, BMI and waist circumference correlate at about 0.9 with one another, systolic pressure with age (0.487) and with diastolic pressure (0.453). Two are informative about the data: the number of prescription medicines correlates with age at 0.506, the strongest relation of age with anything in the table, and with HbA1c at 0.336, presumably through diabetes treatment.
The pairs worth a second look are those where Pearson's and Spearman's coefficients disagree, which flags skew and extreme values. Blood cadmium and blood lead correlate at 0.156 by Pearson's coefficient and 0.358 by Spearman's, a sign that their association is hidden by the long tails of both. The visualization article also found in this table a near-zero correlation hiding a curve (diastolic pressure and age) and a correlation produced entirely by a difference between the sexes (HDL and height). Those findings are the reason a correlation matrix is a map of where to look, not a result.
Step 5: groups, and a finding
Breaking the distributions down by categorical features (sex, age group, race and ethnicity, smoking) is the last routine step, done with grouped summaries (Exploring Data with Groups and Pivot Tables) and grouped plots. For most columns it confirms what is known: blood pressure rises with age and the gap between men and women reverses after 50. For blood cadmium, it turns the isolated bar and the long tail found in step 3 into a finding.
| Smoking status | Adults | Median (µg/L) | Geometric mean | Interquartile range | Below detection limit |
|---|---|---|---|---|---|
| never | 3,128 | 0.240 | 0.245 | 0.160 to 0.390 | 9.3% |
| former | 1,218 | 0.315 | 0.307 | 0.200 to 0.480 | 4.9% |
| current | 918 | 0.930 | 0.903 | 0.570 to 1.440 | 0.7% |
The median blood cadmium of current smokers is 3.88 times that of never smokers, and their geometric mean (the exponential of the mean logarithm, the natural center for a quantity analyzed on a log scale) is 3.68 times as high. Former smokers are only slightly above never smokers. The tail belongs to the smokers: of the 20 highest cadmium values in the table, 19 are current smokers, and above the 95th percentile of 1.45 µg/L, 85.8% are current smokers, against 17.4% of all adults with a cadmium measurement. The filled-in values below the detection limit, conversely, belong almost entirely to non-smokers.
The NHANES laboratory documentation explains the finding: inhaled cigarette smoke is a predominant source of cadmium for smokers, and food is the largest source for non-smokers without occupational exposure (National Center for Health Statistics, 2020). The documentation states that smokers' blood cadmium has been observed to be about twice that of non-smokers; in this sample, comparing current with never smokers, the ratio is closer to four, a difference worth a note but not a surprise, since "smokers" and "non-smokers" can be defined in several ways.
Following the finding one step further shows how exploration generates questions. Among never smokers, cadmium still rises with age (Spearman correlation 0.306), consistent with an element that accumulates in the body over time, and women who never smoked have a higher median than men who never smoked (0.290 against 0.190 µg/L). Neither pattern was looked for, and neither is established by a plot: each is a hypothesis that would need a model with the other variables, and ideally data not used to find it, before it was reported as a result.
Recording what was found
An exploration ends with a written list, kept with the code that produced it. For this table:
- Sex and race are coded categories; convert to categorical types before any summary.
- 15.6% of adults are missing at least one numerical column; lab values are missing in a block of 262 adults with no blood sample; waist is missing for 11.3% of adults 70 and over.
- CRP, cadmium and lead are log-normal in shape; analyze on log scales.
- Age is top-coded at 80 (382 adults); blood pressures and HbA1c are recorded on coarse grids.
- 357 cadmium, 12 CRP values and one lead value are filled in below detection limits; keep the flags.
- Diastolic means below 40 mm Hg arise from readings of 0 treated as missing; decide on a rule for zeros.
- Cadmium is strongly related to smoking, and among never smokers to age and sex; a candidate question for a confirmatory analysis.
The list is the bridge to the next stages: items 1 to 6 are cleaning and structuring decisions, item 7 a modeling question.
Exploration and confirmation
Exploration searches many views of the data for anything interesting, and some of what it finds will be chance. With 14 numerical columns there are 91 pairs, and several group comparisons for each; a few will look striking by luck alone (Multiple Comparisons and the Bonferroni Correction). A pattern found by searching cannot be tested fairly on the same data, because the search has already selected it for looking unusual. The remedies follow Tukey's division of labor: treat exploratory findings as hypotheses, test them on data not used to find them, or set aside a test set at the start and explore only the rest (Training, Validation, and Test Sets), and report how many things were looked at. A finding as strong and as well explained as the cadmium difference survives such scrutiny easily; the age and sex patterns among never smokers are exactly the kind that need it.
Corrections to the course notes
The unit's notes present exploratory analysis as four steps (understand the data, identify relationships, describe shapes, detect outliers and missing data). The routine here keeps those steps and adds two that the notes leave implicit: recording the question and the provenance of every column before starting, since several findings above can only be interpreted from the codebook, and looking at groups as well as pairs. The notes also give, as an example of a bimodal distribution, body weight with peaks for two subpopulations such as the sexes; a mixture of two groups is bimodal only if their means are far enough apart relative to their spreads, and for human height, the standard example, they are not (Visualizing Data with One Feature works this out on NHANES heights). The classification of missing data as MCAR, MAR and MNAR that the notes include here is covered in Cleaning Data.
In Python
import pandas as pd
adults.shape # (5533, 20)
adults.dtypes.value_counts()
adults["race"] = adults["race"].map(RACE).astype("category") # RACE: code -> label, from the codebook
adults.describe(include="all").T # one row per column
adults.isna().sum().sort_values() # missing per column
adults.isna().value_counts().head() # most common missingness patterns
adults.groupby(adults.age // 10 * 10)["waist"].apply(lambda s: s.isna().mean())
adults.select_dtypes("number").skew()
adults.hist(bins=40, figsize=(10, 8)) # one histogram per numerical column
adults["sbp"].nunique(), (adults["age"] == 80).sum()
adults.select_dtypes("number").corr(method="spearman")
adults.groupby("smoking")["cadmium"].describe()
Every number in this article is computed by tools/experiments/data-science/exploratory-data-analysis.py in the site's repository, from the table built by tools/experiments/data-science/exploredata.py, which downloads the nine NHANES files, checks each against a SHA-256 hash and reads them with the Applied Statistics helper statdata.read_xpt(); the figures are drawn by tools/figures/exploratory-data-analysis.py.
References
- National Center for Health Statistics (2020). National Health and Nutrition Examination Survey 2017–2018: Demographics (DEMO_J), Blood Pressure (BPX_J), Body Measures (BMX_J), Glycohemoglobin (GHB_J), Total Cholesterol (TCHOL_J), HDL Cholesterol (HDL_J), High-Sensitivity C-Reactive Protein (HSCRP_J), Lead, Cadmium, Total Mercury, Selenium and Manganese in Blood (PBCD_J), Smoking: Cigarette Use (SMQ_J) and Prescription Medications (RXQ_RX_J) data files and documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?BeginYear=2017 record · 45 pages
- Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley. record · 2 pages
- Tukey, J. W. (1980). We need both exploratory and confirmatory. The American Statistician, 34(1), 23–25. record