A linear model describes the mean of a response as a linear combination of known predictors, $E[\mathbf{Y}] = X\boldsymbol\beta$, and fits the unknown coefficients by least squares. The library's Data Science collection uses regression as a working method, in Simple Linear Regression, Multiple Linear Regression and Linear Regression Assumptions. This unit is the theory beneath those pages. It proves why least squares is the natural fit, what its estimates are worth, and exactly which tests and intervals follow from which assumptions.
The unit is organized around one idea. The responses form a vector in $n$-dimensional space, the possible fitted values form the subspace spanned by the columns of $X$, and least squares is orthogonal projection onto that subspace. The projection gives the fit and its algebra with no probability at all. Adding uncorrelated errors with a common variance gives the Gauss–Markov theorem. Adding normality gives the exact distributions, through the projection theorem of Sampling Distributions from the Normal: independent projections of a normal vector have independent chi-squared lengths. Every exact $t$ test, $F$ test and interval under normal errors in the unit is a case of that theorem; the unit's large-sample and permutation methods, such as Fisher's $z$ interval and the rank tests, rest on other arguments.
What it assumes. The unit on Probability Theory, especially Expectation, Variance and Covariance and Sampling Distributions from the Normal, and matrix algebra: products, transposes, inverses, rank and trace. The singular value decomposition and the multivariate normal distribution belong to Multivariate Statistics and are used only where an article says so.
What is proved and what is cited. Proved across the unit: the bounds of the three correlation coefficients, the exact null distribution of Pearson's under normal errors, the permutation means and variances of all three, and the limiting variance of Pearson's; the projection theorem and every property of the hat matrix used later; the Gauss–Markov and Aitken theorems; the joint distribution of the coefficients and the residual sum of squares under normal errors; the $F$ test of any linear hypothesis and its equivalence to the likelihood ratio test; what the three types of sums of squares test in unbalanced designs; the leave-one-out identities behind the diagnostics; the ridge and lasso solutions in the cases where they have closed forms. Cited rather than proved: a few results of linear algebra, named where they are used, and the theory of the lasso beyond its closed-form cases. Taken as a hypothesis rather than proved: that the mean of an unbiased estimator may be differentiated under the integral, in the Cramér–Rao argument of the Gauss–Markov article. The large-sample theory of least squares without normal errors, including robust standard errors, is not developed here; Linear Regression Assumptions measures by simulation what it changes in practice.
Notation. The unit writes $\mathbf{y} = X\boldsymbol\beta + \boldsymbol\varepsilon$, with $X$ of size $n \times p$ and the intercept column $\mathbf{1}$ counted in $p$ (the Data Science pages count $p$ inputs and $p + 1$ columns). The least squares estimate is $\hat{\boldsymbol\beta} = (X^{\mathsf T}X)^{-1}X^{\mathsf T}\mathbf{y}$, the hat matrix $H = X(X^{\mathsf T}X)^{-1}X^{\mathsf T}$ with leverages $h_{ii}$, the residuals $\mathbf{e} = (I - H)\mathbf{y}$, $\mathrm{RSS} = \lVert\mathbf{e}\rVert^2$ and $s^2 = \mathrm{RSS}/(n - p)$. A linear hypothesis is $C\boldsymbol\beta = \mathbf{d}$ with $C$ of size $q \times p$. Correlations are $r$ and $\rho$ (Pearson), $r_S$ and $\rho_S$ (Spearman), $\hat\tau$ and $\tau$ (Kendall).
Data. The running example is the 5,176 adults of the National Health and Nutrition Examination Survey 2017–2018 with a systolic blood pressure and a body mass index, treated as a finite population without the survey's weights; samples are drawn from it with stated seeds, and the smallest, twelve adults, is worked by hand. Pressure rises with age, differs slightly by sex and rises with body mass index; across the 5,176 adults these three predictors account for about a quarter of the variation in pressure. The ridge and lasso article uses the diabetes study of the Data Science pages, whose ten correlated inputs show shrinkage better. Every number is computed by a script in tools/experiments/statistics/.
Articles
Association and the geometry of the fit
- Correlation: Pearson, Spearman and Kendall: the three coefficients as cosines, with their bounds and cases of equality; their population values and the arcsine formulas for the bivariate normal; the exact $t$ test and permutation tests of zero correlation; the limiting variance of Pearson's coefficient for any population and Fisher's $z$ interval, with its coverage measured on the survey.
- The Geometry of Least Squares: least squares as orthogonal projection; the normal equations from orthogonality; the hat matrix and its properties; $R^2$ as a squared cosine; nested models and extra sums of squares; the Frisch–Waugh–Lovell theorem; rank-deficient designs and estimable functions; QR against the normal equations.
- The Gauss–Markov Theorem: least squares as the best linear unbiased estimator, proved by projection; unbiasedness of $s^2$ without normality; the stronger result under normal errors; Aitken's generalized least squares and the condition under which ordinary least squares is still best; an experiment on what fails without the theorem's conditions.
Inference under normal errors
- The Normal Linear Model: t and F Tests for Coefficients: the distribution of the coefficients and of the residual sum of squares, the $t$ test and interval for any linear combination of coefficients, intervals for a mean response and prediction intervals, the overall $F$ test, the joint confidence ellipsoid and Scheffé's simultaneous intervals.
- The General Linear Hypothesis: restricted least squares and one $F$ test for any set of linear restrictions, with its noncentral distribution and its equivalence to the likelihood ratio test; the two-sample $t$ test and one-way analysis of variance as cases.
- Unbalanced Designs and the Types of Sums of Squares: what the sequential, Type II and Type III sums of squares test in a two-way analysis of variance with unequal cells, each as a general linear hypothesis about the cell means, with the $F$ distributions that the course's two-way analysis of variance states without proof.
Checking and stabilizing the fit
- Leverage, Influence and Residual Diagnostics: leverage and its Mahalanobis form, studentized residuals and the exact distribution of the externally studentized one, the leave-one-out identities from the Sherman–Morrison formula, Cook's distance.
- Ridge and Lasso Regression: shrinkage and the bias it trades for variance; the ridge solution and its singular value form, with the theorem that some amount of shrinkage always lowers the mean squared error; the lasso's optimality conditions, soft thresholding in the orthogonal case, and coordinate descent.
Where the unit leads
Generalized Linear Models replaces the normal errors and the identity link with exponential families and fits by iteratively reweighted least squares, which is a sequence of the weighted projections of this unit. Prediction Error and Model Choice asks how many columns a model should have, using the trace of the hat matrix as the number of parameters. Study Design, Sampling and Causal Inference asks when a regression coefficient measures an effect rather than a comparison.
References
Each article lists its own references. A textbook that treats the unit's subject as a whole, with the estimation and distribution theory in its Chapter 3 (pp. 35–95): record
- Seber, G. A. F., and Lee, A. J. (2003). Linear Regression Analysis (2nd ed.). Wiley Series in Probability and Statistics. Wiley. https://doi.org/10.1002/9780471722199 record