Survival Analysis

The statistics of times to an event when many of those times are only partly observed, with every result proved or cited. Life tables built from a year's death rates; the survival function, density and hazard and their relations; censoring and truncation, what censored data can and cannot identify, and the likelihood of censored data; the Kaplan–Meier and Nelson–Aalen estimators with Greenwood's variance; the log-rank test and its weighted relatives; the Cox proportional hazards model and its partial likelihood; and exponential and Weibull models fitted with censoring. Worked on U.S. deaths of 2019 against the published NCHS life table and on NHANES adults linked to death records through 2019.

A survival time is the time from a defined start to an event: from an examination to death, from diagnosis to relapse, from the start of a trial to the failure of a part. What makes such times a subject of their own is that many of them are not seen in full. A study ends while most of its subjects are alive; people move away; a competing event intervenes. Such a time is censored: it is known only to exceed some value. Discarding censored times, or counting them as events, gives answers that are wrong in a predictable direction. Survival analysis is the set of methods that use the partial information correctly, and the theory of what that partial information can and cannot reveal.

The course pages meet the subject at its edges. Rates in Public Health counts person-time and turns a constant rate into a risk, and Direct Standardization compares death rates by age; a life table is the other classical use of the same rates. The Poisson Distribution and The Poisson Process study waiting times with a constant hazard, which is the simplest survival model. The library uses the ideas elsewhere: The EM Algorithm fits censored exponential lifetimes, Poisson Regression models deaths per person-time, and The Mantel–Haenszel Test supplies the test that, applied at each event time, becomes the log-rank test. This unit supplies the theory beneath them.

What it assumes. Probability Theory, in particular Expectation, Variance and Covariance and The Delta Method; Likelihood and Estimation, for likelihoods, their maximization and the score, Wald and likelihood ratio tests; and for the Cox model the large-sample theory of Asymptotics of the Maximum Likelihood Estimator.

What is proved and what is cited. The unit derives its estimators and proves their finite-sample properties, and cites the martingale theory behind their limits. Proved: the conversion of death rates to probabilities of dying under uniform deaths and a constant hazard, life expectancy as a mean remaining lifetime, the closing of the open age interval, and the delta-method variance of life expectancy with the open interval; the relations among survival function, density and hazard for continuous, discrete and mixed lifetimes, and the mean residual life as a description of the whole distribution; identification of the hazard under independent censoring, Tsiatis's theorem that independence cannot be checked from the data, and the sharp bounds that hold without it; the likelihood of censored data under independent noninformative censoring, and the limits of the naive analyses; the Kaplan–Meier estimator as a product of conditional probabilities and as the nonparametric maximum likelihood estimate, the Nelson–Aalen estimator with its exact discrete-time mean and variance, and Greenwood's formula by the delta method; the log-rank test as a sum of Mantel–Haenszel terms whose deviations are uncorrelated martingale increments, with its weighted versions and the optimal weights; Cox's partial likelihood, both as a product of conditional probabilities and as a profile likelihood, Breslow's estimator, the likelihoods for tied times, the score test at zero as the log-rank test, and the test of proportional hazards; and the likelihoods, estimates and checks of the exponential and Weibull models. Cited: the normal limits of the Kaplan–Meier, Nelson–Aalen and log-rank statistics and of the Cox estimates, which rest on counting processes and martingale central limit theorems; the product integral for general distributions; and the published methods and values of the NCHS life tables.

Notation. The survival time is $T$ and the censoring time $C$; what is observed is $Y = \min(T, C)$ and $\delta = \mathbf{1}\{T \le C\}$, with $\delta = 1$ for an event seen. The survival function is $S(t) = P(T > t)$, the distribution function $F = 1 - S$, the density $f$, the hazard $h(t) = f(t)/S(t)$ for continuous lifetimes and the cumulative hazard $H(t) = \int_0^t h$, so that $S = e^{-H}$. The censoring survival function is $G(t) = P(C > t)$. At the distinct event times $t_{(1)} < t_{(2)} < \cdots$ there are $d_j$ events among the $n_j$ subjects at risk, the risk set $R(t_{(j)})$; a subject censored at an event time counts as at risk there. Independent censoring means $T$ and $C$ are independent (given covariates, in a regression), and noninformative censoring means the distribution of $C$ shares no parameters with that of $T$. Life tables use the demographers' notation: ${}_nm_x$, ${}_nq_x$, $l_x$, ${}_nd_x$, ${}_nL_x$, $T_x$ and $e_x$, where the column $T_x$ of person-years is not the survival time $T$. The Cox model is $h(t \mid \mathbf{x}) = h_0(t)\exp(\mathbf{x}^{\mathsf T}\boldsymbol\beta)$, and the Weibull survival function is $S(t) = \exp(-(\lambda t)^k)$.

Data. Two kinds. For life tables, the deaths of U.S. residents in 2019 and 2020 by age from CDC WONDER, the data of the rates unit, against the NCHS life table for 2019. For follow-up studies, the NCHS public-use linked mortality files, which link NHANES participants to the National Death Index through December 31, 2019: the 5,219 adults of the course pages' 2017–2018 cycle, with 116 deaths in at most 37 months, and adults of the 1999–2000 cycle, with about twenty years of follow-up and well over a thousand deaths, on which the Kaplan–Meier curves, the log-rank tests and the Cox and Weibull models are fitted. The public-use files replace the follow-up time or cause of death of some records with synthetic values to protect confidentiality, but not the vital status; the articles say where this matters. NHANES sampling weights are not used: the cohorts serve as data to analyze, not as estimates for the U.S. population. Every number is computed by a script in tools/experiments/statistics/.

Articles

Lifetimes and what censoring hides

  • Life Tables: how a year's death rates by age become probabilities of dying, survivors, person-years and a life expectancy. The probability of dying derived from the death rate in general, under uniform deaths and under a constant hazard, with the gap between the two bounded; life expectancy proved to be the mean remaining lifetime; the open last interval, exact for a stationary population and never too short for a younger one when the hazard rises with age; period against cohort tables; and the standard error of life expectancy, Chiang's formula with the last interval added. The 2019 U.S. table rebuilt from CDC WONDER gives a life expectancy at birth of 79.32 years against the NCHS 78.85, and 0.39 of the difference comes from closing the table at 85 with one rate for ages 85 and over, whose census-based population is 1.109 times the one that the registered deaths and the NCHS rates together imply.
  • Censoring, Survival Functions and Hazards: the survival function, density and hazard, related for continuous, discrete and mixed lifetimes, and the mean residual life as a description of the distribution; right, left and interval censoring and truncation; identification of the hazard under independent censoring, Tsiatis's theorem that every dependent censoring has an independent twin with the same data, and the sharp bounds without assumptions; the likelihood of censored data, with the person-time rate as the exponential's estimate. On NHANES 2017–2018 one naive analysis turns out to be the upper bound and another falls below the lower one, and artificial censoring of the 1999–2000 cohort shows the product-limit estimate on target under independent censoring and too high when censoring depends on age.

Estimating and comparing survival curves

  • The Kaplan–Meier Estimator: estimating a survival function when many lives are still being watched at the end of a study. The estimator derived as a product of conditional probabilities and proved to be the nonparametric maximum likelihood estimate, equal to the empirical survival function without censoring; the Nelson–Aalen estimator of the cumulative hazard, with the Kaplan–Meier curve never above its exponential and the exact discrete-time mean and variance proved by a martingale argument; Greenwood's formula by the delta method, and the plain, log and log-log intervals; the tail of the curve, the median and its Brookmeyer–Crowley interval, and what the estimator converges to under informative censoring. Worked on the 6-MP leukemia trial and on twenty years of NHANES follow-up by diagnosed diabetes, with simulations of coverage and of bias.
  • The Log-Rank Test: testing whether two survival curves differ under censoring. The test built as a Mantel–Haenszel test over the two-by-two tables at each event time, with the hypergeometric mean and variance, and the tables' deviations proved to be uncorrelated martingale increments, so that their variances add; the observed-over-expected chi-square shown never to exceed it; Gehan's generalized Wilcoxon test as a weighted log-rank test, and the most powerful weights; stratified tests and the number of events a trial needs, by Schoenfeld's and Freedman's formulas. Worked on the 6-MP trial and on NHANES mortality by diabetes and smoking, where stratifying by age changes both comparisons, with simulations of level and power when hazards are proportional, early, late or crossing.

Regression for survival times

  • The Cox Proportional Hazards Model: regression for survival times that leaves the baseline hazard unspecified. The partial likelihood derived as a product of conditional probabilities of who fails at each event time and as a profile likelihood, which also gives Breslow's estimator of the baseline cumulative hazard; the score, the information and the concavity of the log partial likelihood; the exact, Breslow and Efron likelihoods for tied times; the score test at zero proved to be the log-rank test; the test of proportional hazards from a time-varying coefficient, with Schoenfeld residuals; and the proof that omitting an independent covariate pulls the marginal hazard ratio away from the conditional one (below it when the conditional ratio exceeds 1), even in a randomized comparison. Worked on 4,114 adults of NHANES 1999–2000 by smoking, diabetes, age and sex.
  • Parametric Survival Models: survival models that name the shape of the hazard, fitted by maximum likelihood with censoring and delayed entry. The exact chi-squared law of the exponential rate under type II censoring, with four intervals compared; a single maximum of the Weibull's profile likelihood in the shape; the Weibull proved to be both a proportional hazards and an accelerated failure time model, and the only such family when the covariate ranges over values; the log-logistic as a proportional odds model, Cox–Snell residuals, and the piecewise exponential model as a Poisson regression whose limit is Cox's model. Exponential, Weibull, Gompertz, log-logistic and log-normal models are fitted to 4,114 NHANES adults followed for twenty years and compared by AIC and with Kaplan–Meier, the Gompertz law is fitted to the U.S. life table of 2019, and simulations measure what a correct model gains and a wrong one costs.

Where the unit leads

The methods here assume one kind of event and censoring that is independent of the lifetime, given what is measured. Both assumptions have natural extensions that the unit names without developing: competing risks, where several kinds of event remove a person from risk and Tsiatis's theorem forbids recovering a "net" survival for each without further assumptions; and multistate and recurrent-event models, which follow people through several states and are where the counting-process theory cited here does its full work. The large-sample results cited throughout rest on martingale central limit theorems for counting processes; Random Walks and Martingales proves optional stopping and the other discrete-time martingale facts on which those limits build. The next unit of the collection, Statistical Learning Theory, leaves time-to-event data for the question of how well a learned predictor generalizes.