Why Data Science

What data can and cannot tell us, as an orientation to the collection. Datasets, instances and features; the three kinds of question data answer, description, prediction and counterfactual prediction, following Hernán, Hsu and Healy, each with worked results from this collection's articles; a seeded simulation in which a model that predicts the outcome gives the wrong sign for the effect of an exposure, and adjustment recovers it only because the confounder is known; what the field adds beyond its methods; and the limits that no amount of data removes.

Among 1,975 adults examined in the National Health and Nutrition Examination Survey (NHANES) of 2017 to 2018 who had fasted overnight and had never been told they had diabetes, 115 had a fasting blood glucose in the diabetic range: 5.8%. That number answers one kind of question, how common something is. A second question is whether a blood test that needs no fasting, HbA1c, can pick out those 115 people; at the American Diabetes Association's cut-off of 6.5% it finds 54 of them, and 54 of the 79 people it flags have a diabetic fasting glucose. A third question is different again: would lowering a person's weight lower their glucose? The first two can be answered from these data. The third cannot, at least not without assumptions that the data do not supply.

Data science is the practice of answering questions like these from data, and the reason to study it as a discipline is that the three questions look alike, are often computed with the same software, and require different things to be answered correctly. This article sets out what kinds of question data can answer, shows with a small simulation how a good answer to one can be a wrong answer to another, and points to the articles in this collection that work each kind through on real data. The history of the field is in A Historical Overview of Data Science, and the organization of a project from question to deployment in The Data Science Lifecycle.

Datasets, instances and features

Most of the data in this collection are tables. Each row is an instance, also called an observation, case or record: one adult in NHANES, one tumor in the breast cancer biopsies, one shuttle flight before the Challenger accident. Each column is a feature, also called a variable or attribute: a measurement or property recorded for every instance, such as age, systolic blood pressure or launch temperature. The vocabulary differs between communities. Statisticians speak of variables and observations, and machine learning of features and examples, and the inputs and output of a model have further names of their own, listed in Introduction to Regression.

The distinction between instances and features is not always given by the file. A table of blood pressure readings with three columns, one per reading, has one row per person; the same readings stacked into one column have three rows per person, and which layout is right depends on the question. Structuring Data treats this under the name of tidy data, where each variable is a column, each observation a row and each kind of observational unit a table. The kind of feature matters as well: a code for race and ethnicity stored as a number can be averaged by software, but the average means nothing, which is the first finding of Performing Exploratory Data Analysis. Types of Numerical Data sets out the distinctions between nominal, ordinal, discrete and continuous data that decide which summaries are meaningful.

The rise of data science is often explained by the size of modern datasets, an explanation Donoho (2017) calls the "Big Data" meme and is skeptical of: census data have long been as large as much of today's "big data", and the skills needed for very large data are, in his phrase, "coping skills" for the constraints of cluster computing, not new ways of learning from data. Size changes the engineering, which Data Engineering describes. It does not change which questions the data can answer.

Three kinds of question

Hernán, Hsu and Healy (2019) organize "the scientific contributions of data science" into three classes of task, and their classification is a useful starting point for a newcomer to the field.

  • Description is "using data to provide a quantitative summary of certain features of the world". The share of examined adults with a diabetic fasting glucose is a description, and so is a map of a social network.
  • Prediction is "using data to map some features of the world (the inputs) to other features of the world (the outputs)". Using HbA1c to flag adults whose fasting glucose is diabetic is a prediction, and so is a model that scores the risk of death in intensive care from measurements taken at admission.
  • Counterfactual prediction is "using data to predict certain features of the world if the world had been different", which is what causal inference requires. Their example is the mortality rate that would have been observed if everyone in a population had been screened for colorectal cancer, compared with the rate if no one had been.

The methods overlap. A mean, a correlation or a regression can serve any of the three, and statistical inference, with its confidence intervals and tests, is often needed for all three when the data are a sample. What differs is what must be true for the answer to be right. A description needs the data to represent the population described. A prediction needs the future cases to resemble the ones the model learned from. A counterfactual prediction needs, in addition, knowledge of the causal structure: which variables affect which, and which common causes have to be accounted for. Hernán and colleagues argue that "causal analyses typically require not only good data and algorithms, but also domain expert knowledge", and that failing to say so is "a source of widespread misunderstandings about data science". They also note that the boundaries between the categories "are not always sharp".

Description in this collection

Most of the early topics are description, and they show how much care a summary takes. Exploring Data with Groups and Pivot Tables compares the blood pressure of NHANES adults who have no usual place of health care with those who have one: the first group's mean systolic pressure is 4.2 mm Hg lower overall, yet higher in every age group, by 1.8, 1.1 and 6.1 mm Hg. The reversal, an instance of Simpson's paradox, comes from the groups' different age make-up, and direct standardization removes it. The course pages on probability work the same phenomenon through formally in Conditional Probability and the Law of Total Probability, and Direct and Indirect Standardization gives the standard remedy for rates.

Other descriptive results include a blood cadmium level almost four times as high in current smokers as in people who never smoked, found by a systematic exploration rather than by a question anyone asked in advance (Performing Exploratory Data Analysis), and state death rates built by joining weekly death counts to Census populations and checked against the published national rate (Enriching Data). Unsupervised learning is also description: Introduction to Unsupervised Learning asks the wine data where they are dense, with the cultivar labels hidden, and finds that the answer depends on how the measurements are scaled.

Prediction in this collection

Prediction is the subject of most of the later topics. In the diabetes study used throughout the regression topic, body mass index alone explains a share $R^2 = 0.344$ of the variation in disease progression after a year (Introduction to Regression), and all ten baseline measurements together explain 0.518 (Multiple Linear Regression). On two interleaved crescents of simulated points, a network with one hidden layer classifies a median of 93.6% of new points correctly against 86.1% for logistic regression, while on the breast cancer biopsies it does no better than logistic regression (The Multilayer Perceptron). Introduction to Supervised Learning derives the best possible prediction, the Bayes classifier, and computes its error rate for a simulated problem, which tells the other articles how far from the ideal each method is.

Every one of these numbers is measured on data the model did not see while it was fitted, or by cross-validation. That is the defining discipline of prediction, explained in Training, Validation, and Test Sets and Cross-Validation: a prediction is judged by how well it predicts, and the only honest evidence is new cases.

Counterfactual prediction in this collection

The collection's methods are mainly descriptive and predictive, and the causal questions that run through it are flagged as such. Introduction to Regression points out that calling BMI the "independent variable" does not make it a cause of progression, and Multiple Linear Regression works out exactly what "holding the other inputs constant" means for a coefficient, which is less than the phrase suggests. The clearest causal analysis on the site is historical: John Snow's comparison of the two London water companies, retold in A Historical Overview of Data Science, in which the assignment of houses to companies did the work of a randomized experiment. The course pages treat the study designs on which such questions depend: paired against independent comparisons in Comparison of Two Means, where the design decides the analysis, and cohort, cross-sectional and case-control studies in Relative Risk and the Odds Ratio.

A good predictor and the wrong answer

A simulation makes the difference between prediction and counterfactual prediction concrete, because in a simulation the true effect is known. Each simulated person has a background variable $Z$, standard normal, which could stand for age. An exposure $X$, yes or no, is more common when $Z$ is high: $P(X = 1) = 1/(1 + e^{-1.5 Z})$. The outcome is

$$Y = 2Z - 0.5X + \varepsilon, \qquad \varepsilon \sim N(0, 1),$$

so the exposure lowers the outcome by 0.5 for everyone, while $Z$ raises it. $Z$ is a confounder: a common cause of the exposure and the outcome. The script tools/experiments/data-science/why-data-science.py draws 5,000 people, fits two least-squares models, and repeats this for 30 seeds; the values below are medians, with the interquartile range in parentheses.

Model Coefficient of $X$ $R^2$
$Y$ on $X$ alone +1.631 (1.596 to 1.665) 0.147
$Y$ on $X$ and $Z$ −0.492 (−0.518 to −0.473) 0.780

The first model is a legitimate predictor. Knowing whether someone is exposed explains 15% of the variation in their outcome, because exposure tells us something about $Z$: exposed people tend to have high $Z$ and therefore high $Y$. As a description, the coefficient is also correct: the exposed have outcomes 1.63 higher on average. But as an answer to the counterfactual question, what would happen to a person's outcome if they were exposed, it has the wrong sign. A decision maker who used it would withhold an exposure that helps.

The second model recovers the true effect, within sampling error, because $Z$ is in it. The reason this works is not anything in the data. It works because the simulation was written so that $Z$ is the only common cause, and the model includes it. With real data, no table of numbers says which variables are confounders or whether all of them were measured; that knowledge comes from outside the data, from understanding how the exposure came to be assigned. This is the point Hernán and colleagues make about domain expertise, and the reason a higher $R^2$ is not evidence of a better causal estimate: a model can predict well because it includes a consequence of the outcome, or a variable that should not have been adjusted for.

What data science adds

If its methods come from statistics and computer science, what does data science add? The articles of this collection suggest three answers, none of them a new algorithm.

The first is the work around the model. Donoho's six divisions of a "greater data science" put data exploration and preparation, data representation, computing, and visualization beside modeling as equal parts of the discipline (Donoho, 2017), and the order of this collection follows that view: Python, SQL and data wrangling come before regression. Many of the most consequential errors in the collection happen before any model is fitted: a zero that a file reader turns into $5.4 \times 10^{-79}$ (Cleaning Data), a join on a non-unique key that turns 2,808 rows into 12 million (Enriching Data), or a query whose NOT IN returns nothing at all because of one missing value (Special Operators and Clauses).

The second is measurement of predictive performance as a routine, the habit Donoho calls the "common task framework": a fixed dataset, a held-out test set, and a score that settles which method predicts better. Model Selection and The Bias-Variance Tradeoff explain why the score on the training data cannot settle it.

The third is reproducibility. An analysis written as a script can be rerun on new data, checked by someone else, and corrected. Donoho describes this as the change that made a science of data analysis possible: a prose description of an analysis "becomes instead tangible and useful, as one can download and execute code immediately". Programming with Python and Jupyter shows that the tools alone do not guarantee it, with notebooks that give 16.85, 2.25 or 126.36 depending on the order their cells are run.

What data cannot do

Some limits do not shrink as data grow.

  • The data may not measure what the question is about. Graunt had to decide which causes of death the parish Searchers could recognize by sight (A Historical Overview of Data Science), and the screening articles show that "diabetes" in a survey is a definition based on a single measurement, not a diagnosis (Sensitivity and Specificity).
  • The data may not represent the population of interest. The NHANES examples in this collection ignore the survey's sampling weights, so their figures describe the adults examined rather than the U.S. population, as each unit states (Screening and Diagnostic Tests).
  • A model says nothing about cases unlike those it learned from. The Challenger launch forecast of 31 °F was 22 degrees colder than any earlier flight in the record (Logistic Regression), and a model deployed to a population with a different prevalence can change the meaning of its own outputs, as The Data Science Lifecycle measures.
  • Looking at data long enough finds patterns that are not there. Exploration generates hypotheses; testing them needs data that were not used to find them, and a report of how many things were tried (Training, Validation, and Test Sets, Multiple Comparisons and the Bonferroni Correction).
  • Data about people carry obligations. Records stripped of names can often be matched back to individuals, and a risk score can be fair by one reasonable definition and unfair by another, sometimes unavoidably, as Ethics in Data Science shows.

None of these is a reason not to analyze data. They are the reasons the field needs people trained to ask, before computing anything, which kind of question is being asked and what the answer would require.

A path through the collection

Question Kind Where to start
How common is it, and how does it differ between groups? Description Descriptive Statistics, Exploring Data with Groups and Pivot Tables, Performing Exploratory Data Analysis
What structure is in the data without labels? Description Introduction to Unsupervised Learning
How does an outcome change with an input, on average? Description or prediction Introduction to Regression, Multiple Linear Regression
Which category will a new case fall in? Prediction Introduction to Supervised Learning, Logistic Regression, Introduction to Decision Trees
How good is this model on new data? Prediction Binary Classification Metrics, Cross-Validation, Model Selection
What would happen if we intervened? Counterfactual prediction the study designs in Comparison of Two Means and Relative Risk and the Odds Ratio, and the cautions in Multiple Linear Regression

References