Statistics is the study of what data can and cannot tell us: how to summarize observations, how to estimate a quantity that was not observed directly, and how sure that estimate deserves to be. This collection gathers the site's statistics in one map. Its base is the applied course in biostatistics that I took as MTH 645, whose twelve units are complete. Around that base the collection adds the mathematics the course took as given, the chapters of its textbook that the course did not reach, and the parts of statistics that the rest of this site uses most: likelihood, information theory and the evaluation of models.
The units are written as reference material. Each result is proved on the page or cited to a source that proves it, and the page says which. Examples use public data, and every number is produced by a script kept with the site's source (tools/experiments/), so it can be rerun and checked. Each article ends with worked problems solved in full.
Units and articles that are not yet written appear below as plain text.
The applied core
These twelve units follow the course text, Pagano, Gauvreau and Mattie, Principles of Biostatistics (3rd ed., 2022), and are listed on the course page. They are the starting point for everything else in the collection.
- Descriptive Statistics: types of data, tables and graphs, measures of center and spread.
- Rates and Standardization: rates in public health, and comparing populations with different age structures.
- Probability: the rules of probability, conditional probability, relative risk and the odds ratio.
- Screening and Diagnostic Tests: sensitivity, specificity, predictive values, likelihood ratios and ROC curves.
- Theoretical Probability Distributions: random variables and the binomial, Poisson and normal distributions.
- Sampling Distributions: the sampling distribution of the mean and the central limit theorem.
- Confidence Intervals: intervals for a mean, the chi-squared and t distributions, one-sided bounds and sample size.
- Hypothesis Testing: tests for a mean, the two kinds of error, power and sample size.
- Comparison of Two Means: paired and two-sample t tests, and Welch's test.
- Analysis of Variance: one-way and two-way analysis of variance and multiple comparisons.
- Nonparametric Methods: the sign, signed-rank, rank-sum and Kruskal–Wallis tests.
- Inference on Proportions: intervals and tests for one proportion and comparisons of two.
Mathematical statistics
The theory under the applied core. These units prove what the course states, beginning with the probability that every later unit needs.
- Probability Theory: distribution functions, expectation and conditional expectation, inequalities, the laws of large numbers, the central limit theorem with a full proof, the delta method, and the sampling distributions that come from the normal.
- Likelihood and Estimation: the likelihood function, maximum likelihood, sufficiency, Fisher information and the Cramér–Rao bound, the large-sample behavior of the estimator, the Neyman–Pearson lemma, the likelihood ratio, Wald and score tests, and the EM algorithm.
- Generalized Linear Models: exponential families, link functions, fitting by iteratively reweighted least squares, deviance, Poisson regression and softmax regression.
- Bayesian Inference: conjugate families, credible intervals, priors, Bayes factors, Markov chain Monte Carlo, hierarchical models and variational inference.
- Multivariate Statistics: the multivariate normal distribution, covariance matrices and quadratic forms, discriminant analysis, mixture models and the singular value decomposition.
Completing the applied core
Subjects of the course text that the course did not reach.
- Categorical Data Analysis: chi-square tests of fit and of independence, McNemar's test, the Mantel–Haenszel test and measures of agreement.
- Linear Models: correlation, least squares as a projection, the Gauss–Markov theorem, tests for coefficients, diagnostics, and ridge and lasso regression.
- Survival Analysis: life tables, censoring, the Kaplan–Meier estimator, the log-rank test and the Cox model.
- Study Design, Sampling and Causal Inference: randomized experiments, potential outcomes, confounding, propensity scores and survey sampling.
Computation and model choice
Methods that replace a formula with computation, and the theory of choosing among models.
- Resampling Methods: why the bootstrap works, bootstrap intervals, permutation tests and the jackknife.
- Multiple Testing: the false discovery rate, closed testing, q-values and selective inference.
- Prediction Error and Model Choice: optimism, the Akaike and Bayesian information criteria derived, and what cross-validation estimates.
Statistics for machine learning
Where the rest of the site draws on statistics: the language model pages, the Data Science collection and the information retrieval course.
- Information Theory: entropy, cross-entropy and Kullback–Leibler divergence, mutual information, perplexity and source coding, and maximum entropy distributions.
- Statistics of Model Evaluation: standard errors for benchmark scores, clustered questions, paired comparisons of models, power, calibration and proper scoring rules, and rare events.
- Stochastic Processes and Time Series: Markov chains, the Poisson process, random walks, autocorrelation and hidden Markov models.
- Statistical Learning Theory: empirical risk minimization, concentration inequalities, PAC learning, the VC dimension and Rademacher complexity.
Related collections
The Data Science collection treats regression, logistic regression, the bootstrap, cross-validation and model selection as working methods with code. The units here are the theory beneath those articles and link to them where the two meet.