The applied statistics course on this site uses probability constantly and proves only part of it. Its page on random variables treats discrete variables on a finite sample space. Its central limit theorem is proved, in Moment Generating Functions and Slutsky's Theorem, only for populations that have a moment generating function. Its derivation of the chi-squared and t distributions cites the properties of joint densities and the convolution formula that it uses. This unit supplies the missing mathematics. It is the first unit of the theory tier of the Statistics collection, and the units on likelihood, information theory, linear models and resampling all build on it.
The unit keeps the course's running data. The systolic blood pressures and C-reactive protein levels of the adults in the National Health and Nutrition Examination Survey of 2017 to 2018 appear in every article, treated as finite populations, and each theorem is checked on them or on a small exact case. Every number comes from a script in tools/experiments/statistics/ in the site's repository.
What the unit assumes
The first three units of probability in the course: The Rules of Probability, Conditional Probability and the Law of Total Probability, and the distributions of Theoretical Probability Distributions. Beyond that, one-variable calculus, double integrals, and for the last article some linear algebra. No measure theory is assumed. The few measure-theoretic results that the proofs need are stated and cited.
Conventions
- Probability is $P(\cdot)$, expectation $E[\cdot]$, and $\operatorname{Var}$, $\operatorname{Cov}$, $\operatorname{Corr}$ are written out. Random variables are capitals and their values lower case.
- The distribution function is $F_X(x) = P(X \le x)$, the mass function $p_X$, the density $f_X$, and the quantile function $Q_X(u) = \inf\{x : F_X(x) \ge u\}$.
- The indicator of an event is $\mathbf{1}_A$ or $\mathbf{1}\{\cdot\}$.
- The normal distribution with mean $\mu$ and standard deviation $\sigma$ is written $N(\mu, \sigma)$, with the standard deviation second, as on the course pages. Each article says in words which is meant. Its standard density is $\phi$ and its distribution function $\Phi$; the characteristic function of $X$ is $\varphi_X$.
- A sample is $X_1, \ldots, X_n$ with sum $S_n$, mean $\bar{X}_n$ and sample variance $s_n^2$. Convergence in probability, in distribution and almost surely are written $\xrightarrow{p}$, $\xrightarrow{d}$ and $\xrightarrow{\text{a.s.}}$.
- "Finite mean" means $E\lvert X\rvert < \infty$ and "finite variance" means $E[X^2] < \infty$. Each theorem says which it needs.
- The kurtosis is $\kappa = \mu_4/\sigma^4$, with $\mu_4 = E[(X - \mu)^4]$, as on the course page One-Sided Confidence Bounds and Sample Size. It is 3 for the normal distribution, and the excess kurtosis is written $\kappa - 3$.
What is proved and what is cited
Nearly everything is proved on the page. The exceptions are stated where they occur and are of two kinds.
The first kind is the measure theory that the unit avoids. Random variables are handled through distribution functions, mass functions and densities, with integrals in the sense of Riemann. Four kinds of thing are then taken as given. The sets in question have probabilities. Random variables with prescribed distributions exist. A distribution function determines the whole distribution, the product formula for intervals extends to all sets, and independent variables collected in disjoint groups form independent vectors. And a joint density gives the probability of a region as well as of a rectangle, with multiple integrals and sums computed in any order (Fubini's theorem). Random Variables and Distribution Functions is the home of that statement and lists exactly what is lost, and each article lists what its own proofs take as given. Within those limits the expected value is defined for every random variable and its rules are proved in general, and conditional expectation is treated in four settings, stated exactly in Conditional Expectation and the Laws of Total Expectation and Variance, that cover a discrete conditioning variable and the commonest cases of one with a density.
The second kind is a short list of theorems whose proofs are beyond the unit: the uniqueness and continuity theorems for characteristic functions, and the strong law of large numbers under a finite mean alone. They are stated precisely and cited, and the articles that use them say which step depends on them.
Articles
In reading order. Each article lists what it proves, what it cites and what its experiment shows, and ends with worked problems solved in full.
- Random Variables and Distribution Functions: one description for discrete, continuous and mixed variables. The properties of a distribution function proved from countable additivity, the quantile function and the equivalence that makes it an inverse, the probability integral transform in both directions, densities of transformed variables, independence, and the distribution of sums and extremes.
- Expectation, Variance and Covariance: the expected value defined by one formula for every random variable, with linearity and the product rule for independent variables proved in that generality. Variance, covariance and its bilinearity, the covariance matrix, the bounds on a correlation and on the correlation of two indicators, and the best linear predictor.
- Conditional Expectation and the Laws of Total Expectation and Variance: the conditional expectation as a function of the conditioning variable and as the best predictor, the laws of total expectation, variance and covariance, and the tower property. The course's design effect and overdispersion formulas, and the variance of a random sum, as corollaries.
- Probability Inequalities: Markov, Chebyshev, Jensen and Cauchy–Schwarz: what a mean, a variance or a convex function says about probabilities, with the distributions that attain each bound and the places the site uses them.
- Moment Generating and Characteristic Functions: the two transforms that turn sums of independent variables into products, derived for the standard families, with moments from derivatives and the expansion the central limit theorem needs.
- The Laws of Large Numbers: when and in what sense a sample mean settles at the population mean. The weak law by Chebyshev's inequality and by truncation, the Borel–Cantelli lemmas, and the strong law proved under a finite fourth moment.
- Convergence in Distribution and the Proof of the Central Limit Theorem: the modes of convergence and the implications among them, the continuous mapping theorem and Slutsky's theorem, and the Lindeberg–Lévy theorem proved by characteristic functions for every population with a finite variance.
- The Delta Method: the limit distribution of a function of an estimate, in first and second order and for several means, with the log relative risk, the log odds ratio and variance-stabilizing transformations.
- Sampling Distributions from the Normal: Chi-Squared, t and F: the exact theory of normal samples as one theorem about projections, with Cochran's theorem, the F density and the noncentral chi-squared distribution.
Where the unit leads
Likelihood and Estimation uses the limit theorems to find the large-sample behavior of maximum likelihood estimates and the conditional expectation for the Rao–Blackwell theorem. Information Theory uses the expected value and Jensen's inequality throughout. The units on linear models, multivariate statistics and resampling start from the covariance matrix, the normal sampling theory and the delta method. Several results that the course pages state and cite are proved in this unit, and each article names the course pages it completes.
References
- Billingsley, P. (1995). Probability and Measure (3rd ed.). Wiley. The source for the measure-theoretic results the unit cites. record · 8 pages
- Casella, G., and Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury. Chapters 1 to 4 cover the ground of the first three articles. record · 11 pages
- Grinstead, C. M., and Snell, J. L. (1997). Introduction to Probability (2nd rev. ed.). American Mathematical Society. The probability text behind the course pages. record · 20 pages