The Group Insurance Commission of Massachusetts, which bought health insurance for state employees, held patient records, with nearly a hundred fields per hospital encounter, for about 135,000 employees and their families. Names and addresses had been removed, and "because the data were believed to be anonymous", the commission gave a copy to researchers and sold one to industry. Latanya Sweeney bought the voter registration list for Cambridge, Massachusetts, for twenty dollars. It gave each voter's name, address, ZIP code, birth date and sex, three fields that the health records also contained. William Weld, then the governor, lived in Cambridge and his records were in the commission's data. "According to the Cambridge Voter list, six people had his particular birth date; only three of them were men; and, he was the only one in his 5-digit ZIP code" (Sweeney, 2002).
Nobody had published the governor's name beside his diagnoses, and nobody had needed to. The example contains most of the subject of this article: data collected for one purpose and used for another, a protection that seemed adequate and was not, and a harm that came from combining sources rather than from any one of them. Data science works with records about people at every phase of a project, from the collection described in The Data Science Lifecycle to the deployed model, and each phase carries its own obligations. Risks specific to artificial intelligence systems are taken up in Risks and Ethics in AI.
Principles from research ethics
In the United States, a standard statement of principles for research on people is the Belmont Report of the U.S. National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, published for comment in the Federal Register in April 1979. It distinguishes research, "an activity designed to test an hypothesis, permit conclusions to be drawn, and thereby to develop or contribute to generalizable knowledge", from practice, and names three basic principles (National Commission, 1979).
- Respect for persons: "individuals should be treated as autonomous agents", and "persons with diminished autonomy are entitled to protection".
- Beneficence, expressed as two rules: "(1) do not harm and (2) maximize possible benefits and minimize possible harms".
- Justice, in the sense of "fairness in distribution": who receives the benefits of research and who bears its burdens.
The report applies them through informed consent, the assessment of risks and benefits, and the selection of subjects. Consent, it says, "can be analyzed as containing three elements: information, comprehension and voluntariness": people must be told the procedure, purposes, risks and benefits and that they may withdraw; the information must be presented so they can understand it; and the agreement must be free of "coercion and undue influence".
Much of data science is not research in the report's sense, and much of it uses data collected by someone else, long after any consent was given. The principles still organize the questions well. Respect for persons asks whether the people in the data agreed to this use, or would have. Beneficence asks what could go wrong for them if the data or the model's outputs were misused. Justice asks who bears the errors of a model and who gets its benefits, which is the question behind the fairness criteria later in this article. History offers a contrast. When John Snow compared the customers of two London water companies in 1854, he described more than 300,000 people "divided into two groups without their choice, and, in most cases, without their knowledge" (Snow, 1855). The division was not his doing, and his study is a landmark of epidemiology (A Historical Overview of Data Science); but his sentence describes exactly the conditions the Belmont Report was written to keep researchers from creating.
Privacy and re-identification
Why removing names is not enough
Sweeney's linkage worked because a few ordinary fields, each harmless alone, combine into something close to a unique label. Using 1990 census summaries, she estimated that 87% of the U.S. population, 216 million of 248 million people, "had reported characteristics that likely made them unique based only on {5-digit ZIP, gender, date of birth}"; with the city or town in place of the ZIP code the figure was 53%, and with the county, 18% (Sweeney, 2000). Fields like these, which are not identifiers themselves but can be linked to a source that has names, are called quasi-identifiers.
The arithmetic behind such figures is short. Suppose a population of $N$ people is spread evenly and independently over $K$ possible combinations of the quasi-identifiers. A given person is unique if none of the other $N - 1$ falls in that person's combination, which has probability
$$\left(1 - \frac{1}{K}\right)^{N-1} \approx e^{-(N-1)/K}.$$
Uniqueness depends on the ratio of combinations to people. Sweeney's Cambridge voter list had 54,805 voters, five ZIP codes, two sexes and birth dates spread over about 100 years (Sweeney, 2000). A simulated town with those numbers, every field drawn independently and uniformly, gives the following shares of people who are unique; the script tools/experiments/data-science/ethics-in-data-science.py repeats the draw for 30 seeds, and the medians agree with the formula.
| Fields known | Combinations $K$ | Unique (median of 30 towns) | Formula |
|---|---|---|---|
| ZIP, sex, date of birth | 365,000 | 86.1% | 86.1% |
| ZIP, date of birth | 182,500 | 74.1% | 74.1% |
| Sex, date of birth | 73,000 | 47.2% | 47.2% |
| ZIP, sex, year of birth | 1,000 | 0.0% | 0.0% |
Real towns are not uniform: births cluster by year, and some ZIP codes are much larger than others, which leaves people in small or unusual groups more exposed than the average suggests. The simulation shows the mechanism, not the true rate for Cambridge. The last row shows the remedy that the mechanism suggests. Replacing the date of birth with the year leaves 1,000 combinations for 54,805 people, about 55 per combination, and nobody is unique.
Rules built on this arithmetic
The U.S. health privacy rule under HIPAA allows health information to be treated as de-identified by an expert's determination, or by the "Safe Harbor" method of removing eighteen kinds of identifier. Two of them are exactly Sweeney's quasi-identifiers. All "elements of dates (except year)" directly related to a person, including birth date, must be removed, as must ages over 89; and all geographic subdivisions smaller than a state must go, except the first three digits of a ZIP code when the area they define contains more than 20,000 people (45 CFR § 164.514(b)(2)).
Sweeney (2002) turned the idea into a formal guarantee. A release has k-anonymity "if the information for each person contained in the release cannot be distinguished from at least k-1 individuals whose information also appears in the release". In the simulated towns, the release with full dates of birth is not even 2-anonymous, since 86% of its records are unique; with years of birth instead, each combination holds about 55 people on average, and the median town has no unique record.
The public NHANES files used throughout this collection apply the same ideas. Age is reported in years up to 79, and "all responses of participants aged 80 years and older are coded as '80'", because "the reporting of age in single years for adults 80 years and older was determined to be a disclosure risk"; family income relative to poverty is capped at 5; household sizes are capped at 7; pregnancy status is released only for women aged 20 to 44 (National Center for Health Statistics, 2020). When an article in this collection reports an age of 80, it means 80 or over, for that reason.
Sparse data: the Netflix Prize
Fixed lists of quasi-identifiers protect against linkage on those fields. They do not protect records that contain many fields, each rare. Netflix released the movie ratings of about 500,000 subscribers for its prize competition, with names removed. Narayanan and Shmatikov (2008) showed that very little outside knowledge identifies a record. With 8 movie ratings, of which 2 may be completely wrong, and dates known to within 14 days, 99% of records can be uniquely identified in the dataset; for 68%, two ratings with dates known to within three days are enough. Matching against public ratings on the Internet Movie Database, for a small sample of users, they identified the Netflix records of two people, and from one person's full viewing history inferred his apparent political orientation and made "strong guesses" about his religious views, information his public ratings did not reveal.
The reason is the sparsity of such data: in the space defined by thousands of movies, "for the vast majority of records, there isn't a single similar record in the entire 500,000-record dataset". Every record is an outlier, and an outlier is an identifier. Removing a list of fields cannot fix that, because there is no short list.
The legal definitions
The European Union's General Data Protection Regulation, which applies from 25 May 2018, defines personal data as "any information relating to an identified or identifiable natural person" (Article 4(1)). Its Recital 26 draws the line that the examples above suggest: data that have been pseudonymized, with identifiers replaced but re-identifiable with additional information, remain personal data, while the regulation does not apply to information that has been rendered genuinely anonymous. Article 22 gives a person the right "not to be subject to a decision based solely on automated processing", that "produces legal effects concerning him or her or similarly significantly affects him or her" (Regulation (EU) 2016/679).
Fairness of risk scores
Three reasonable requirements
A risk score assigns each person a number meant to be the probability of some outcome, and a decision follows from the score: flag for review, offer a loan, recommend a test. Kleinberg, Mullainathan and Raghavan (2017) formalized three conditions that people have proposed as requirements of fairness between two groups:
- Calibration within groups: among people of either group given a score of, say, 0.3, a fraction 0.3 have the outcome. In the authors' words, the condition "asks that the scores mean what they claim to mean", even within each group separately.
- Balance for the negative class: people who do not have the outcome receive the same average score in both groups.
- Balance for the positive class: people who do have the outcome receive the same average score in both groups.
Each is natural. Their theorem is that, apart from two special cases, no score satisfies all three: if a score satisfies them, then either it predicts perfectly or the two groups have the same base rate, the same share of people with the outcome. Even approximate versions of the three conditions require the data to be close to one of those cases.
When the score is turned into a yes-or-no decision at a threshold, Chouldechova (2017) states the conflict in terms of the rates of Binary Classification Metrics: predictive parity asks for the same positive predictive value (PPV) in both groups, error rate balance for the same false positive rate (FPR) and false negative rate (FNR), and statistical parity for the same share flagged.
The identity behind the conflict
Write $p$ for a group's base rate and $N$ for its size, and count true positives TP, false positives FP, false negatives FN and true negatives TN. By definition $\text{PPV} = \text{TP}/(\text{TP} + \text{FP})$, so $\text{FP} = \text{TP}\,(1 - \text{PPV})/\text{PPV}$. The people with the outcome number $pN = \text{TP} + \text{FN}$, so $\text{TP} = (1 - \text{FNR})\,pN$. The people without it number $(1 - p)N = \text{FP} + \text{TN}$. Then
$$\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}} = \frac{(1 - \text{FNR})\,pN\,(1 - \text{PPV})/\text{PPV}}{(1-p)N} = \frac{p}{1 - p}\cdot\frac{1 - \text{PPV}}{\text{PPV}}\cdot(1 - \text{FNR}),$$
which is Chouldechova's equation (2.6). If two groups have the same PPV and the same FNR, the right-hand side differs only through $p/(1-p)$, so their false positive rates must differ whenever their base rates do. Something has to give: equal PPV, equal FPR or equal FNR.
A synthetic example
Real cases in this literature involve comparisons between social groups that are best studied with care and full context; the papers above discuss a risk tool used in criminal courts. A synthetic example shows the arithmetic without them. Two groups, A and B, of 50,000 people each, have true individual risks drawn from different distributions: Beta(2, 5) in group A, with mean $2/7 = 0.286$, and Beta(4, 4) in group B, with mean 0.5. Each person's outcome is drawn with their own risk as its probability, and the score handed to the decision maker is that risk, so the score is calibrated within each group by construction; no bias has been put into it. Everyone with a score of 0.5 or more is flagged. The script repeats this for 30 seeds and reports medians; the interquartile ranges are at most about 0.01 wide.
| Group A | Group B | |
|---|---|---|
| Base rate | 0.286 | 0.500 |
| Share flagged | 0.109 | 0.500 |
| PPV | 0.594 | 0.637 |
| False positive rate | 0.062 | 0.363 |
| False negative rate | 0.774 | 0.364 |
| Mean score, people without the outcome | 0.250 | 0.444 |
| Mean score, people with the outcome | 0.375 | 0.556 |
Calibrated scores and a common threshold give very different error rates. A person in group B who will not have the outcome is flagged with probability 0.363, almost six times the 0.062 of a person in group A. A person in group A who will have the outcome is missed with probability 0.774, against 0.364 in group B. The two balance conditions fail as the theorem says they must: people without the outcome average a score of 0.250 in group A and 0.444 in group B. The identity above holds for each group's counts to rounding error, and the exact values for the two Beta distributions, computed by numerical integration, agree with the medians to within 0.003 (group A: PPV 0.592, FPR 0.0625, FNR 0.773).
Equalizing one criterion moves the others. Raising group B's threshold until its false positive rate equals group A's, at a score of 0.694, gives group B a PPV of 0.767 against 0.594 for group A, and a false negative rate of 0.794. Group B is now flagged at a rate of 13.5%, while half its members have the outcome.
None of this says which criterion should win. Chouldechova (2017) notes that statistical parity suits contexts like employment or admissions, where equal selection rates may be desired or legally required, and that it is hard to motivate for predicting reoffending, the setting of her paper. Kleinberg and colleagues describe calibration as the property without which a decision maker "has a clear incentive" to treat people with the same score differently by group. The choice depends on what the decision does to people and on which errors are worse, a judgment the analyst cannot make alone. What the mathematics settles is that the choice exists, and that a report claiming a model is "fair" without saying in which sense is incomplete.
Aggregation
A comparison between groups can reverse when a third variable is taken into account. In Exploring Data with Groups and Pivot Tables, NHANES adults with no usual place of health care have the lower mean blood pressure overall and the higher mean in every age group, because they are younger on average. This is Simpson's paradox, developed with its explanation in Conditional Probability and the Law of Total Probability, and the usual remedy for rates is standardization. Hernán, Hsu and Healy (2019) trace the paradox to "a failure to recognize that the choice of data analysis depends on the causal structure of the problem": whether the aggregated or the stratified comparison answers the question depends on what the third variable is and how it relates to the others.
The ethical point is that the same table can support opposite statements about a group, each true of the numbers. Reporting a comparison between groups therefore includes reporting what it was adjusted for and why, and resisting the summary that happens to support a preferred conclusion. The simulation in Why Data Science shows a related case, in which an exposure that lowers an outcome appears to raise it until a common cause is accounted for.
Documenting data
Many of these problems are invisible to someone who receives a dataset without its history. Gebru et al. (2021) proposed that "every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on", by analogy with the electronics industry, where every component comes with a datasheet "describing its operating characteristics, test results, recommended usage, and other information". Their questions fall into seven groups: motivation; composition; collection process; preprocessing, cleaning and labeling; uses; distribution; and maintenance. They name two audiences: creators, who are prompted to reflect on how the data were made, and consumers, who need to know whether the data suit their task.
The NHANES documentation that this collection relies on answers many of those questions. It records how each variable was collected, how values were edited, which variables were coarsened for disclosure and why, and how the sampling weights should be used; ignoring the weights, as the examples here do for teaching, means the results describe the examined adults rather than the U.S. population, which each unit says. Performing Exploratory Data Analysis makes reading the documentation the first step of every exploration.
Energy
Computation has a physical cost. The International Energy Agency estimates that "data centres accounted for around 1.5% of the world's electricity consumption in 2024, or 415 terawatt-hours", and projects that their consumption will "more than double to around 945 TWh by 2030" (IEA, 2025). Most analyses in this collection run in seconds on a laptop, and the cost becomes a real consideration at the scale of training large models, where Risks and Ethics in AI takes it up. At any scale, it is a reason to prefer the simplest model that answers the question, which Model Selection supports on statistical grounds as well.
How this collection handles data
The examples in this collection use public data published for reuse, such as the NHANES files, Census tables and the datasets distributed with scikit-learn, or synthetic data generated with a fixed seed, as in the fairness example above. Every number comes from a script kept with the site's source, so that each analysis can be rerun and checked. The public NHANES files have already been coarsened by their publisher for disclosure control.
A checklist by phase
| Phase | Questions to ask |
|---|---|
| Business understanding | Who is affected by the decision this model supports? Which errors harm whom? Which fairness criterion matters here, and who decides? |
| Data understanding | Who collected the data, for what purpose, under what consent or legal basis? Is there documentation, and what does it say about collection and coarsening? |
| Data preparation | Which fields are quasi-identifiers? Are joins creating combinations that identify people? Are any groups too small to report? |
| Modeling | Do base rates differ between groups? Which proxies for group membership are in the inputs? |
| Evaluation | Are error rates, PPV and calibration reported for each group, with the trade-off stated? |
| Deployment | Can a person contest an automated decision? Is the model monitored for changes in the population (The Data Science Lifecycle)? Who is responsible for it? |
References
- Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163. https://doi.org/10.1089/big.2016.0047 record · 2 pages
- Code of Federal Regulations, Title 45, § 164.514. Other requirements relating to uses and disclosures of protected health information. https://www.law.cornell.edu/cfr/text/45/164.514 record
- Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., and Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92. https://doi.org/10.1145/3458723 record
- Hernán, M. A., Hsu, J., and Healy, B. (2019). A second chance to get causal inference right: A classification of data science tasks. CHANCE, 32(1), 42–49. https://doi.org/10.1080/09332480.2019.1579578 record · 4 pages
- International Energy Agency (2025). Energy and AI. IEA. https://www.iea.org/reports/energy-and-ai record
- Kleinberg, J., Mullainathan, S., and Raghavan, M. (2017). Inherent trade-offs in the fair determination of risk scores. In 8th Innovations in Theoretical Computer Science Conference (ITCS 2017). arXiv:1609.05807. record · 2 pages
- Narayanan, A., and Shmatikov, V. (2008). Robust de-anonymization of large sparse datasets. In 2008 IEEE Symposium on Security and Privacy (pp. 111–125). https://doi.org/10.1109/SP.2008.33 record
- National Center for Health Statistics (2020). NHANES 2017–2018 Demographic Variables and Sample Weights (DEMO_J), data documentation. Centers for Disease Control and Prevention. https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/DataFiles/DEMO_J.htm record
- National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1979). The Belmont Report: Ethical principles and guidelines for the protection of human subjects of research. Federal Register, 44(76), 23192–23197. record
- Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation). Official Journal of the European Union, L 119. https://eur-lex.europa.eu/eli/reg/2016/679/oj record
- Snow, J. (1855). On the Mode of Communication of Cholera (2nd ed.). John Churchill. record · 2 pages
- Sweeney, L. (2000). Simple Demographics Often Identify People Uniquely (Data Privacy Working Paper 3). Carnegie Mellon University. record
- Sweeney, L. (2002). k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10(5), 557–570. https://doi.org/10.1142/S0218488502001648 record