Data Science

A collection on the working methods of data science, from Python, SQL and data wrangling through probability, regression, model evaluation, supervised and unsupervised learning, trees, ensembles and neural networks, with worked examples on public data computed by scripts kept with the site's source.

Data science is the practice of answering questions with data: collecting it, getting it into a usable shape, looking at it, building models that describe or predict it, and checking whether those models deserve to be believed. Most of its methods are older than the name. Least squares dates from the early nineteenth century, and logistic regression and the perceptron were both in use by the 1960s. What changed is the scale of the data and the cheapness of computation, which turned a statistician's toolkit into an everyday engineering activity and added the software and database skills needed to apply it.

This collection follows a standard introductory sequence in fourteen topics. The early topics are tools: Python and its numerical libraries, SQL, and the work of cleaning and restructuring data. The middle topics are the statistical core: probability, inference, exploratory analysis and regression. The later topics are machine learning: how to evaluate a model honestly, then the main families of supervised and unsupervised methods, decision trees, ensembles and neural networks, and a closing look at artificial intelligence more broadly.

How these articles are written

Each article teaches one method from first principles, as a textbook chapter would, and derives the results a student needs, with every symbol defined. Examples use public data that anyone can obtain, such as the datasets shipped with scikit-learn or the UCI Machine Learning Repository, or synthetic data generated with a fixed random seed. Every number in an article is produced by a script in the site's repository (tools/experiments/data-science/), and the figures are drawn from the same scripts' output, so each example can be rerun and checked. Definitions and results are cited to standard textbooks, chiefly James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning (2nd ed., 2021), and to the original papers where a method has one.

Some concepts are already covered in depth by the course wiki on this site, written for information retrieval. Where that is so, the articles here link to the existing article instead of repeating it: for example The k-means Algorithm, Hierarchical Clustering, Naive Bayes Classification, evaluating classifiers and significance tests.

Articles not yet written appear below as plain text.

Introduction to Data Science

What the field is, where it came from and how a project runs, from the first question to a deployed model, along with the people who do the work and the obligations that come with handling data about people. These articles orient a reader to the rest of the collection and link into it rather than teaching methods. The history goes back to sources that can be checked: Graunt's bills of mortality, Snow's cholera table, whose printed rate for the rest of London does not follow from its own counts, Nightingale's army returns, Tukey's 1962 paper, Breiman's two cultures and Donoho's six divisions of the field. A seeded simulation shows a model that predicts an outcome while giving the wrong sign for the effect of an exposure. The careers article reports official Bureau of Labor Statistics figures with their dates. The lifecycle follows CRISP-DM through the site's own NHANES screening project, from 9,254 survey participants to a deployed test whose positive rate signals a change in the population. The ethics article recomputes how quickly ordinary fields make people unique, covers the Netflix de-anonymization, and derives why calibrated risk scores give unequal error rates when base rates differ.

Python for Data Science

Python is the common language of the field because of its libraries rather than its speed. This topic covers the language as analysis code uses it, the tools for installing and versioning its libraries, and the three packages almost every later article relies on: NumPy for arrays, pandas for tables and matplotlib for plots. The examples run on NHANES adults and concentrate on behavior that gives wrong answers without an error: three notebook cells that print 16.85, 2.25 or 126.36 depending on the order they are run in, a float that makes round(2.675, 2) equal 2.67, a default argument created once, respondent numbers whose squares wrap around in 32-bit integers to a sum a ninth of the true one, a sorted table on which loc[0:2] returns 1,022 rows, the same pandas code giving different results under pandas 2.2 and 3.1 because of copy-on-write, and a plot title that lands on the wrong panel. The matplotlib article takes one figure apart into its 144 objects.

Probability and Statistics

The language for reasoning about uncertainty. How samples are drawn and what they can say about a population, how to summarize data, the rules of probability and the common distributions, and how to turn a sample into an estimate or a test with a stated margin of error. The Applied Statistics course pages cover the same ground at more depth, starting with Descriptive Statistics (types of data, tables and graphs, measures of center and spread), and the articles here link to them for definitions and proofs.

SQL for Data Science

Much of the world's structured data lives in relational databases, and SQL is how it is retrieved. The topic moves from the relational model through filtering, aggregation, joins and subqueries to running queries from Python, on one database built from eleven NHANES 2017-2018 tables in SQLite. It starts from Codd's 1970 paper and normalizes the survey's prescription file, which stores one diagnosis 2,483 times, into first, second and third normal form. The later articles keep returning to NULL and SQL's three-valued logic, which makes a condition and its negation together miss 1,231 people and a NOT IN query return nothing at all, and to the number of rows a join produces, which decides whether a mean blood pressure is computed per person or per pill. Along the way the queries reproduce the grouped means computed in pandas in Exploring Data with Groups and Pivot Tables to the hundredth, demonstrate SQL injection on a copy of the database, and note where SQLite, PostgreSQL, MySQL and SQL Server differ.

Data Wrangling

Raw data is rarely ready for analysis. Wrangling is the work of summarizing it by groups, reshaping and rescaling it, repairing missing and inconsistent values, and combining it with other sources, together with the storage systems that hold data at scale. The examples are real public files with real problems: NHANES health survey records with refusal codes and a file format whose zeros a reader can lose, Census population tables that store totals beside their parts, and weekly death counts with suppressed cells, joined to give state death rates checked against the published national rate.

Data Exploration

Looking at data before modeling it. The topic covers plots for one feature and for several, the principles that make a chart honest and readable, the tools used to make them, a systematic routine for exploratory analysis, and the detection of outliers. The definitions of the individual graphs (histograms, box plots, the empirical distribution function) and the rules that keep them honest are in Tables and Graphs for Data; this topic is about the workflow of exploring a dataset with many features. The running example is a table of 5,533 NHANES adults joined from nine public survey files, on which the articles work through kernel density bandwidths, log scales, overplotting and correlations that mislead, Cleveland and McGill's ranking of perceptual tasks, color scales measured for lightness and color-vision deficiency, the grammar of graphics, a six-step exploration that ends in a finding about blood cadmium and smoking, and outlier rules from z-scores, whose masking is shown and bounded, to Mahalanobis distance, isolation forests and the local outlier factor.

Regression

Regression models how an outcome depends on one or more inputs. The topic starts with the vocabulary of inputs, outputs and residuals, derives the least-squares line and its connection to correlation, examines what must hold for its standard errors and intervals to be trusted, extends the model to several inputs, polynomial terms and interactions, and ends with logistic regression for yes-or-no outcomes. The running examples are a diabetes study of 442 patients and the 23 space shuttle flights before the Challenger accident.

Evaluating Model Performance

A model is only as good as the evidence that it works on data it has not seen. This topic covers the bias-variance tradeoff, the metrics and loss functions used for classification and regression, the division of data into training, validation and test sets, cross-validation and the bootstrap, and how to choose among candidate models. The course wiki's Evaluating Classifiers treats several of the same ideas for text classification.

Supervised Learning

Methods that learn from labeled examples to predict a label or a number for new ones. The overview derives the best possible predictions, the regression function and the Bayes classifier, and computes the Bayes error rate of a simulated problem that the later articles use as a benchmark. Three classifiers built on different ideas follow: nearness, probability and a separating margin. k-nearest neighbors is worked with distances, scaling, weighted votes and the curse of dimensionality computed; support vector machines are derived from the maximal margin to soft margins and kernels. Naive Bayes is covered by the course wiki's article, written for text, and the overview works its Gaussian version for measurements by hand on the wine data.

Unsupervised Learning

Methods that find structure in data without labels: groups of similar instances, dense regions and the points that fall outside them, and a smaller number of underlying dimensions. The overview puts all three questions to the wine data with the cultivar hidden, shows how much the answers depend on scaling, works the silhouette by hand, and checks the clusters by resampling, where a stable answer can still be the wrong one. K-means and hierarchical clustering are covered by the course wiki's articles, written for text, along with measures for comparing clusterings. DBSCAN is worked from its original definitions on twelve points and on the eruptions of the Old Faithful geyser; principal components are derived as eigenvectors of the covariance matrix and through the singular value decomposition, which the course wiki's Latent Semantic Indexing derives in full; and factor analysis is fitted to nine mental ability tests from 1939 and compared with principal components on the same correlations.

Decision Trees

Trees predict by asking a sequence of yes-or-no questions about the inputs. They are easy to read, handle numbers and categories alike, and are the building block of the strongest ensemble methods. The topic derives how a tree chooses its splits, by residual sum of squares for numeric outcomes and by the Gini index or entropy for categories, how a large tree is pruned back by cost-complexity pruning and cross-validation, and why averaging many decorrelated trees in a random forest reduces their variance. The examples include breast cancer biopsies, the diabetes study, the Challenger flights and a three-class wine dataset.

Artificial Neural Networks

From the single perceptron, a linear classifier much like logistic regression, to networks with hidden layers and nonlinear activations that can approximate a wide range of functions. The introduction shows that one neuron fitted to two breast cancer measurements is logistic regression, and builds XOR from three threshold units. The perceptron article proves the convergence theorem in the form that follows Novikoff's argument and checks its bound on iris flowers, then finds breast cancer data that are separable yet take the rule 234 epochs. The activation functions are derived with their gradients, and a ten-layer network with sigmoid units, measured layer by layer, learns nothing in 50 epochs while the same network with ReLU units fits its data. The multilayer perceptron article derives backpropagation, works it by hand on one tumor, checks it by finite differences, and trains a network written in NumPy against logistic regression and scikit-learn in 30 seeded splits: hidden units win on two interleaved crescents and add nothing on the nearly linear breast cancer and wine data, and a network with 74,000 parameters overfits in its probabilities while its accuracy hardly changes. Deeper networks are taken further in Language Models from Scratch.

Ensemble Techniques

Combining many models often beats the best single one, provided the members are better than chance and their errors differ. The topic works out those conditions for votes and averages, then the three standard methods: bagging averages models fitted to bootstrap samples and reduces their variance, boosting fits simple models in sequence, each aimed at the errors of the ensemble so far, and stacking learns how to combine different kinds of model from their cross-validated predictions. AdaBoost is derived from the exponential loss and gradient boosting from gradient descent, both worked by hand. The examples include breast cancer biopsies, the diabetes study and Friedman's simulated regression problem.

Artificial Intelligence

A closing topic on the wider field that data science feeds into, written as an overview that routes into the material elsewhere on the site. The introduction traces the field from Turing's imitation game and the Dartmouth proposal through two AI winters to deep learning. The machine learning overview reruns Sutton and Barto's 10-armed bandit, where an agent that never explores finds the best arm in about a third of problems. The vision article convolves a handwritten 7 by hand and shifts test digits by one pixel, which costs logistic regression 54 points of accuracy and a small convolutional network 37. The language article computes word vectors from the Brown corpus, whose nearest neighbors find the days of the week and whose analogies fail. The risks article fits a census income classifier that is about equally precise for women and men while its false positive rates are 0.02 and 0.10, and explains with Chouldechova's identity why both can hold. The generative article samples new digits from a Gaussian model and works Kaplan's scaling law, and the prompting article works temperature and nucleus sampling on one next-token distribution. The library's Anthropic Research collection covers current research on language models in depth, and Language Models from Scratch builds a small one.

References

  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. record · 23 pages
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. record · 30 pages
  • Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. record · 6 pages