Censoring, Survival Functions and Hazards

The survival function, the density and the hazard of a lifetime, with their relations derived for continuous, discrete and mixed distributions, and the mean residual life shown to determine the distribution. Right, left and interval censoring, the types of censoring designs, and truncation; a proof that independent censoring makes the hazard identifiable from what is observed, Tsiatis's theorem that without it any dependent censoring is indistinguishable from an independent one, and the sharp bounds that hold with no assumption at all; the likelihood of censored data derived under noninformative censoring, with the exponential and piecewise-constant hazards as examples. On NHANES adults linked to death records, the naive analyses are shown to choose an extreme assumption without saying so, and artificial censoring of a cohort with complete follow-up measures what each analysis does. Worked problems on discrete hazards, a linear hazard, a small censored sample and two exponential clocks.

The National Center for Health Statistics (NCHS) links the adults examined in the National Health and Nutrition Examination Survey (NHANES) to the National Death Index. Of the 5,219 adults examined in 2017 and 2018 whose blood pressure the course pages use, and who could be linked, 116 had died by December 31, 2019. The rest are recorded as alive on that date, which for some was 11 months after their examination and for others 37. What share of adults like these die within two years of an examination? The 116 deaths are 0.0222 of the cohort, but half of the cohort was not followed for two years. Counting as dead everyone not seen alive beyond two years gives a survival of 0.4692. Neither is an answer. What do incomplete follow-up times reveal about a lifetime, and under what assumption?

A censored observation is one in which the event has not been seen: the time is known to exceed some value, and no more. Survival analysis is the statistics of times to an event when some of those times are censored. This article sets up its three basic functions (the survival function, the density and the hazard), proves how each determines the others, and then asks what censored data can reveal about them. It is the unit's home for these relations, for the identification results and for the likelihood of censored data, on which The Kaplan–Meier Estimator, The Log-Rank Test, The Cox Proportional Hazards Model and Parametric Survival Models build.

Here is the route. First the three functions and their relations, for continuous, discrete and mixed lifetimes, and the mean residual life. Then the kinds of censoring and truncation, on the NHANES follow-up. Then identification: what the observed data determine under independent censoring, why nothing replaces that assumption (Tsiatis's theorem), and what bounds hold without it. Then the likelihood of censored data, derived. Finally what the usual naive analyses estimate, proved and then measured by censoring a cohort whose deaths are all known. The article assumes Random Variables, Expectation and Variance and The Likelihood Function; Life Tables is the same subject seen through a year's death rates.

Notation and the scope of the proofs

A survival time $T \ge 0$ is the time from a defined origin (here an NHANES examination) to an event (here death). Its distribution function is $F(t) = P(T \le t)$ and its survival function is

$$S(t) = P(T > t) = 1 - F(t),$$

which is nonincreasing, right-continuous, equal to 1 at $t = 0$ when $P(T = 0) = 0$, and tends to 0 as $t \to \infty$ when every lifetime ends. Random variables are capitals and their values lower case. The censoring time is $C$, with survival function $G(t) = P(C > t)$; what is observed is

$$Y = \min(T, C), \qquad \delta = \mathbf{1}\{T \le C\},$$

with $\delta = 1$ when the event was seen and $\delta = 0$ when the observation is censored. A sample is $(Y_i, \delta_i)$, $i = 1, \ldots, n$. When an event and a censoring happen at the same recorded time the event is taken to come first, so an observation censored at $t$ counts as still at risk at $t$; this is the convention of the whole unit.

The proofs treat lifetimes with a density, lifetimes with countably many values, and mixtures of the two, with Riemann integrals and sums, in the manner of Probability Theory. Statements for general distributions, which need the product integral and the theory of counting processes, are mentioned without proof.

The survival function, the density and the hazard

The hazard is the rate at which events occur among those still at risk:

$$h(t) = \lim_{\Delta \downarrow 0}\frac{P(t \le T < t + \Delta \mid T \ge t)}{\Delta}.$$

For a person who has survived to $t$, $h(t)\Delta$ is approximately the probability of the event in the next short interval of length $\Delta$. It is a rate, not a probability, and can exceed 1. The cumulative hazard is $H(t) = \int_0^t h(u)\,du$. Demographers call the hazard of death the force of mortality; Life Tables writes it $\mu(x)$.

The three functions carry the same information. Here is the idea: the hazard is the density divided by the share still alive, which is the negative derivative of $\ln S$, and integrating that derivative recovers $S$.

Theorem 1 (continuous lifetimes). Let $T$ have a density $f$ that is continuous on $[0, \infty)$, and let $S(t) > 0$. Then

$$h(t) = \frac{f(t)}{S(t)} = -\frac{d}{dt}\ln S(t), \qquad S(t) = e^{-H(t)}, \qquad f(t) = h(t)\,e^{-H(t)}.$$

Conversely, any function $h \ge 0$ that is continuous with $\int_0^\infty h(u)\,du = \infty$ is the hazard of exactly one such distribution, the one with $S = e^{-H}$. If $\int_0^\infty h < \infty$, then $e^{-H}$ does not tend to 0 and describes a lifetime that is infinite with probability $e^{-H(\infty)}$.

Proof. For $\Delta > 0$, $P(t \le T < t + \Delta \mid T \ge t) = (F(t + \Delta) - F(t))/S(t)$, because $P(T \ge t) = P(T > t) = S(t)$ when $T$ has a density. Dividing by $\Delta$ and letting $\Delta \downarrow 0$ gives $F'(t)/S(t) = f(t)/S(t)$, by the fundamental theorem of calculus. Since $S = 1 - F$, $S' = -f$, and $-\frac{d}{dt}\ln S(t) = -S'(t)/S(t) = f(t)/S(t)$. Integrating from 0 to $t$ with $\ln S(0) = 0$ gives $-\ln S(t) = H(t)$, so $S = e^{-H}$, and $f = -S' = h\,e^{-H}$. Conversely, given $h$, the function $S = e^{-H}$ is continuous, nonincreasing, equal to 1 at 0, and has the continuous derivative $-h\,e^{-H}$, so $f = h e^{-H}$ is a density of the distribution with survival function $S$ when $S(t) \to 0$, that is, when $H(t) \to \infty$. Its hazard is $f/S = h$. Uniqueness: two distributions with the same continuous hazard have the same $-\ln S$, by the first part. $\square$

Example (the exponential). A constant hazard $h(t) = \lambda$ gives $H(t) = \lambda t$ and $S(t) = e^{-\lambda t}$: the exponential distribution. The Poisson Process (Theorem 2) proves it is the only memoryless waiting time; in the language of hazards, memorylessness is a hazard that does not change with time. Human mortality is the opposite: in the NCHS life table for 2019 the death rate at 84 is 2.59 times that at 75 (Life Tables). A hazard that rises with time is the common case for death, and Parametric Survival Models fits families such as the Weibull, whose hazard is a power of $t$.

Follow-up in the linked mortality files is recorded in whole months, and life tables group age by years. Grouped times are discrete, and the hazard becomes a conditional probability.

Theorem 2 (discrete lifetimes). Let $T$ take the values $t_1 < t_2 < \cdots$ with $P(T = t_j) > 0$, and define the discrete hazard $h_j = P(T = t_j \mid T \ge t_j)$. Then

$$S(t) = \prod_{j:\,t_j \le t}(1 - h_j), \qquad P(T = t_j) = h_j\prod_{i < j}(1 - h_i),$$

and $S(t) \le \exp\big(-\sum_{t_j \le t} h_j\big)$, with equality only when $t < t_1$.

Proof. The event $T > t_j$ is the event $T \ge t_j$ intersected with $T \ne t_j$, and $T \ge t_j$ is the event $T > t_{j-1}$ (with $T \ge t_1$ certain), since no value lies strictly between. So $P(T > t_j) = P(T > t_{j-1})(1 - h_j)$. Repeating from $j$ down to 1 gives the product for $S(t_j)$, and $S$ is constant between consecutive values. Then $P(T = t_j) = P(T \ge t_j)\,h_j = S(t_{j-1})\,h_j$. Finally $1 - x \le e^{-x}$ for every $x$, with equality only at $x = 0$ (the function $e^{-x} - 1 + x$ is zero at 0, with derivative $1 - e^{-x}$ negative before 0 and positive after), so each factor is at most $e^{-h_j}$. $\square$

The inequality matters when the hazards are not small. The estimators of The Kaplan–Meier Estimator are built on the product form, and the Nelson–Aalen estimator of the cumulative hazard on the sum; the two do not give the same survival curve, and Theorem 2 says which is lower. The life table is Theorem 2 with yearly steps: $l_x/l_0 = \prod_{y < x}(1 - q_y)$.

Many lifetimes are neither purely continuous nor purely discrete: a death can be recorded at an exact day for most people and at a fixed audit date for a few. The two theorems combine.

Theorem 3 (mixed lifetimes). Suppose $P(T = t_j) = p_j > 0$ at points $t_1 < t_2 < \cdots$ (finitely many in each bounded interval), and that between consecutive points $S$ is differentiable with continuous derivative. Let $h_j = p_j/S(t_j^-)$, where $S(t^-) = P(T \ge t)$, and let $h_c(t) = -S'(t)/S(t)$ at points other than the $t_j$, with $H_c(t) = \int_0^t h_c(u)\,du$. Then, while $S(t) > 0$,

$$S(t) = e^{-H_c(t)}\prod_{j:\,t_j \le t}(1 - h_j).$$

Proof. On an interval $[t_{j}, t_{j+1})$ with no point inside, the argument of Theorem 1 gives $S(t) = S(t_j)\exp\big(-\int_{t_j}^t h_c\big)$. At $t_j$ the function jumps: $S(t_j) = S(t_j^-) - p_j = S(t_j^-)(1 - h_j)$, and $S(t_j^-) = S(t_{j-1})\exp\big(-\int_{t_{j-1}}^{t_j} h_c\big)$. Multiplying these factors from 0 to $t$ gives the formula. $\square$

For a general distribution the same statement holds with a product integral, $S(t) = \prod_{u \le t}(1 - dH(u))$, which is beyond the scope here.

The mean and the mean residual life

For a lifetime, which is never negative, the mean is the area under the survival function, $E[T] = \int_0^\infty S(t)\,dt$: Expectation, Variance and Covariance takes this as the definition and proves it agrees with $\int t f(t)\,dt$. Applied to $T^2$, whose survival function is $P(T^2 > u) = S(\sqrt u)$, and with the substitution $u = t^2$, it gives $E[T^2] = \int_0^\infty 2t\,S(t)\,dt$.

The mean residual life at $t$ is the expected remaining lifetime of someone who has survived to $t$:

$$r(t) = E[T - t \mid T > t] = \frac{\int_t^\infty S(u)\,du}{S(t)},$$

the second form by the tail formula applied to the conditional survival function $S(t + s)/S(t)$. Life expectancy at age $x$ in a life table is $r(x)$ (Life Tables, Theorem 5). In the NCHS table for 2019 the life expectancy at birth is 78.85 years and at 65 it is 19.59 years: those who reach 65 live to 84.6 on average, beyond the newborns' average, because surviving to 65 already rules out the deaths before it. The mean residual life is not merely a summary.

Theorem 4 (the mean residual life determines the distribution). Let $T$ have a finite mean, let $\tau = \sup\{t : S(t) > 0\}$ (possibly $\infty$), and let $T$ have a density that is continuous on $[0, \tau)$. Then for $0 \le t < \tau$, $r$ is differentiable with $r'(t) = h(t)\,r(t) - 1$, and

$$S(t) = \frac{r(0)}{r(t)}\exp\Big(-\int_0^t \frac{du}{r(u)}\Big).$$

Proof. Work at $t < \tau$, where $S(t) > 0$ and every division below is by a positive number. Let $R(t) = \int_t^\infty S(u)\,du$, finite because $E[T] = R(0)$ is, and positive because $S > 0$ just after $t$. Then $R'(t) = -S(t)$, and $r = R/S$. By the quotient rule and $S' = -hS$, $r' = (R'S - RS')/S^2 = -1 + hR/S = h r - 1$. For the inversion, $R'/R = -S/R = -1/r$, so $\ln R(t) - \ln R(0) = -\int_0^t du/r(u)$, and $S(t) = R(t)/r(t) = \frac{R(0)}{r(t)}\exp\big(-\int_0^t du/r(u)\big)$, with $R(0) = E[T] = r(0)$. $\square$

Example (two mean residual lives). A constant $r(t) = 1/\lambda$ gives $S(t) = e^{-\lambda t}$: the exponential's remaining life does not depend on age. For $T$ uniform on $(0, \theta)$, $\tau = \theta$, $r(t) = (\theta - t)/2$ for $t < \theta$, and the inversion gives back, for $t < \theta$, $S(t) = \frac{\theta/2}{(\theta - t)/2}\exp\big(-\int_0^t \frac{2\,du}{\theta - u}\big) = \frac{\theta}{\theta - t}\Big(\frac{\theta - t}{\theta}\Big)^2 = 1 - t/\theta$.

Censoring and truncation

The NHANES follow-up shows the commonest kind of censoring. Examinations ran across 2017 and 2018, and the linked mortality files follow participants "from the date of survey participation through December 31, 2019" (NCHS, 2022). An adult examined early in 2017 can be followed for up to 37 months; one examined late in 2018, for as little as 11. For each adult the potential follow-up, from the examination to the closing date, is the censoring time $C$: the death is seen if it happens before $C$. Among the 5,103 adults still alive at the end, the censoring times range from 11 to 37 months, with a median of 24.

Right censoring means that for some subjects only $T > c$ is known. Its designs differ in what fixes $C$:

  • Type I: one fixed censoring time for everyone, as in a study that follows every subject for exactly two years.
  • Type II: the study stops at the $r$th event, so $C$ is the $r$th smallest lifetime, which is random and depends on the lifetimes; it is used in life testing of components.
  • Random censoring: $C$ is a random variable, such as the time to moving away or to withdrawal.
  • Administrative censoring with staggered entry: the closing date is fixed but entry dates differ, as in NHANES. A feature of this design is that $C$ is known for every subject, including those who died, because it is a calendar fact.

Left censoring means only $T \le c$ is known: the event had already happened when the subject was first checked. Interval censoring means $L < T \le R$ is known: periodic examinations find that a condition began between two visits. The monthly recording of the linked files is a mild interval censoring of every death, by at most a month.

Truncation is different. Censoring is incomplete information about a subject who is in the sample; truncation decides who is in the sample at all. Under left truncation, or delayed entry, a subject is seen only if $T > V$ for an entry time $V$. If NHANES adults are followed on the scale of age rather than time since examination, each enters at the age of examination, and anyone who died younger could never have been examined: their lifetimes are left truncated at the examination age. Under right truncation only subjects with $T \le c$ are seen, as in a registry that records cases only once they are diagnosed.

One more feature of the NHANES files matters for individual records. The public-use files "were subjected to data perturbation techniques to reduce participant disclosure risk. Synthetic data were substituted for follow-up time or underlying cause of death for select records. Information regarding vital status was not perturbed" (NCHS, 2022). The event indicator $\delta$ is therefore the recorded one, while a given person's follow-up time may be synthetic. The analyses here use the times as published.

What the observed data identify

Everything that can be learned from right-censored data is a property of the distribution of $(Y, \delta)$. That distribution is described by two sub-distribution functions,

$$F_1(t) = P(Y \le t,\ \delta = 1), \qquad F_0(t) = P(Y \le t,\ \delta = 0),$$

and $P(Y \ge t) = 1 - F_1(t^-) - F_0(t^-)$. The question is whether they determine $S$.

Under independent censoring

Here is the idea. Among those still under observation at $t$, independence of $T$ and $C$ means that the censoring has not selected anyone by their lifetime, so the rate of observed events among them is the hazard of the whole population.

Theorem 5 (identification under independent censoring). Let $T$ and $C$ be independent.

  1. If $T$ has a density $f$, then $F_1(t) = \int_0^t f(u)\,P(C \ge u)\,du$ and $P(Y \ge u) = S(u)\,P(C \ge u)$, so that for every $t$ with $P(Y \ge t) > 0$, $$H(t) = \int_0^t \frac{dF_1(u)}{P(Y \ge u)}, \qquad S(t) = \exp\Big(-\int_0^t \frac{dF_1(u)}{P(Y \ge u)}\Big),$$ where $dF_1(u) = f(u)P(C \ge u)\,du$ is the density of $F_1$.
  2. If $T$ is discrete with values $t_j$, then for every $t_j$ with $P(Y \ge t_j) > 0$, $$h_j = \frac{P(Y = t_j,\ \delta = 1)}{P(Y \ge t_j)}.$$

In both cases $S(t)$ is determined by the distribution of $(Y, \delta)$ for every $t$ with $P(Y \ge t) > 0$, and in general not beyond.

Proof. (1) The event $\{Y \le t, \delta = 1\}$ is $\{T \le t, T \le C\}$. By independence, conditioning on $T = u$ leaves $C$ with its own distribution, so $F_1(t) = \int_0^t f(u)P(C \ge u)\,du$. Likewise $\{Y \ge u\} = \{T \ge u\} \cap \{C \ge u\}$, whose probability is $S(u)P(C \ge u)$ by independence (and $P(T \ge u) = S(u)$ for a density). For $u \le t$, $P(Y \ge u) \ge P(Y \ge t) > 0$, so $P(C \ge u) > 0$, and the integrand $f(u)P(C \ge u)/(S(u)P(C \ge u)) = f(u)/S(u) = h(u)$. Theorem 1 gives $S = e^{-H}$. (2) $P(Y = t_j, \delta = 1) = P(T = t_j, C \ge t_j) = P(T = t_j)P(C \ge t_j)$ under the convention that an event at the censoring time is seen, and $P(Y \ge t_j) = P(T \ge t_j)P(C \ge t_j)$. The ratio is $P(T = t_j)/P(T \ge t_j) = h_j$, and Theorem 2 gives $S$. For the last claim: if $P(Y \ge t) = 0$ while $P(T \ge t) > 0$, then $P(C \ge t) = 0$ by independence. Every subject with $T \ge t$ is then censored, at $Y = C < t$, so the distribution of $(Y, \delta)$ depends on $T$ only through its distribution on $[0, t)$ and the probability $P(T \ge t)$. Two lifetime distributions that agree there but differ after $t$ give the same observations. $\square$

The ratio in part 2 is the population version of $d_j/n_j$, the deaths at a time over the number still at risk; The Kaplan–Meier Estimator multiplies those estimates as Theorem 2 multiplies the $1 - h_j$. In NHANES 2017–2018 no one was followed past 37 months, so the data say nothing about survival beyond 37 months, whatever the method.

Without independence: Tsiatis's theorem

Could the data themselves show whether censoring is independent? Tsiatis (1975) proved that they cannot. He posed it for competing risks, where an individual has a potential time to death from each of several causes and only the first is seen with its cause; with two "causes", the event and the censoring, it is our problem.

Theorem 6 (Tsiatis's nonidentifiability). Let $(T, C)$ have any joint distribution for which $Y$ has a continuous distribution with $P(Y \ge t) > 0$ for all $t$, and $F_1$ and $F_0$ have continuous densities $F_1'$ and $F_0'$. Define

$$h_1^*(t) = \frac{F_1'(t)}{P(Y \ge t)}, \qquad h_0^*(t) = \frac{F_0'(t)}{P(Y \ge t)},$$

and let $T^*$ and $C^*$ be independent with hazards $h_1^*$ and $h_0^*$ (lifetimes that may be infinite, as Theorem 1 allows). Then $(\min(T^*, C^*), \mathbf{1}\{T^* \le C^*\})$ has the same distribution as $(Y, \delta)$.

So every dependent model of censoring has an independent twin that produces exactly the same data. The twin's survival function, $\exp(-\int_0^t h_1^*)$, is what an analysis that assumes independence reports, and it need not resemble the true $S$.

Proof. This is Tsiatis's Theorem 2 for two risks, in our notation. First, $P(Y \ge t)$ has derivative $-(F_1'(t) + F_0'(t)) = -(h_1^*(t) + h_0^*(t))P(Y \ge t)$, since $Y$ is continuous and $P(Y \ge t) = 1 - F_1(t) - F_0(t)$. By the argument of Theorem 1, $P(Y \ge t) = \exp\big(-\int_0^t (h_1^* + h_0^*)\big)$. For the twin, independence gives $P(Y^* \ge t) = P(T^* \ge t)P(C^* \ge t) = \exp\big(-\int_0^t h_1^*\big)\exp\big(-\int_0^t h_0^*\big)$, the same function. The density of the twin's $F_1^*$ is, by part 1 of Theorem 5, $f^*(t)P(C^* \ge t) = h_1^*(t)P(T^* \ge t)P(C^* \ge t) = h_1^*(t)P(Y \ge t) = F_1'(t)$, and in the same way the density of $F_0^*$ is $F_0'$. Two distributions of $(Y, \delta)$ with the same sub-densities are the same. $\square$

Example (Tsiatis's example). Tsiatis took the joint survival function

$$P(T > t_1,\ C > t_2) = \exp(-\lambda t_1 - \mu t_2 - \theta t_1 t_2), \qquad \lambda, \mu > 0,\ \theta \ge 0.$$

Its marginal for $T$ (put $t_2 = 0$) is exponential with rate $\lambda$, whatever $\theta$. For it to be a distribution, its mixed second derivative, the joint density, must be nonnegative: that derivative is $[(\lambda + \theta t_2)(\mu + \theta t_1) - \theta]\,e^{-\lambda t_1 - \mu t_2 - \theta t_1 t_2}$, which is nonnegative for all $t_1, t_2 \ge 0$ exactly when $\theta \le \lambda\mu$ (at the origin it is $\lambda\mu - \theta$, and elsewhere the bracket is larger). Tsiatis's text uses $\lambda = 0.1$ and $\mu = 0.2$ with the two values $\theta = 0.1$ and $\theta = 0.02$. With $\theta = 0.1$ the would-be density is negative near the origin. Of the two values, only $\theta = 0.02$, which equals $\lambda\mu$ and so lies on the boundary of the allowed range $0 \le \theta \le \lambda\mu$, gives a proper joint distribution, and we use it. (The paper's Figure 1 labels the second curve $\vartheta = 0.2$; the text gives 0.02, and with 0.2 the drawn curve would be near 0 well before $t = 8$, so the label appears to be a misprint.) Here $P(Y \ge t) = P(T > t, C > t) = e^{-(\lambda + \mu)t - \theta t^2}$, and the sub-density of observed events is the negative partial derivative in $t_1$ at $t_1 = t_2 = t$ (Tsiatis's Theorem 1): $F_1'(t) = (\lambda + \theta t)\,e^{-(\lambda + \mu)t - \theta t^2}$. So $h_1^*(t) = \lambda + \theta t$, and an analysis assuming independence reports

$$S^*(t) = e^{-\lambda t - \theta t^2/2}, \qquad\text{against the true}\qquad S(t) = e^{-\lambda t}.$$

At $t = 10$ these are 0.135 and 0.368; at $t = 20$, 0.002 and 0.135. The median lifetime is 6.93 in truth and 4.71 under independence. Those still uncensored at $t$ have a higher hazard than the population ($\lambda + \theta t$ against $\lambda$), and the independent analysis takes their hazard for everyone's.

Tsiatis's example with \lambda = 0.1, \mu = 0.2 and \theta = 0.02. The solid curve is the true survival function of T; the dashed curve is the survival function of the independent twin of Theorem 6, which produces exactly the same observed data. Notice that the two agree near 0 and are far apart by t = 10: no amount of data of the form (Y, \delta) can tell them apart.
Tsiatis's example with $\lambda = 0.1$, $\mu = 0.2$ and $\theta = 0.02$. The solid curve is the true survival function of $T$; the dashed curve is the survival function of the independent twin of Theorem 6, which produces exactly the same observed data. Notice that the two agree near 0 and are far apart by $t = 10$: no amount of data of the form $(Y, \delta)$ can tell them apart.

Bounds with no assumption

If independence cannot be checked, what can be said without it? The answer is an interval for each $S(t)$, and it is often wide.

Theorem 7 (bounds without assumptions). For any joint distribution of $(T, C)$ and every $t$,

$$P(Y > t) + P(Y = t,\ \delta = 0) \le S(t) \le 1 - P(Y \le t,\ \delta = 1).$$

When $C$ has no atom at $t$, the second term of the lower bound, $P(Y = t, \delta = 0)$, is zero and the lower bound is $P(Y > t)$. Both bounds are sharp: for each, there are joint distributions with the given distribution of $(Y, \delta)$ whose $S(t)$ equals the bound, or comes within any $\varepsilon > 0$ of it.

Proof. If $Y > t$ then $T \ge Y > t$. If $Y = t$ and $\delta = 0$ then $T > C = t$. So $\{Y > t\}$ and $\{Y = t, \delta = 0\}$ are disjoint subsets of $\{T > t\}$, which is the lower bound. If $Y \le t$ and $\delta = 1$ then $T = Y \le t$, so $\{Y \le t, \delta = 1\} \subseteq \{T \le t\}$, which is the upper bound. For sharpness, keep the observed pairs and choose $T$ for the censored subjects, who are only known to have $T > C$. Setting $T = C + M$ for the censored, with $M > t$, makes every censored subject survive past $t$, and $S(t)$ equals the upper bound. For the lower bound, set $T = C + \varepsilon$ for the subjects censored strictly before $t$ and $T = C + M$ for the others. Every subject censored before $t - \varepsilon$ then dies before $t$, and $S(t)$ equals the lower bound plus $P(t - \varepsilon < Y < t, \delta = 0)$, which tends to 0 as $\varepsilon \downarrow 0$ because the intervals $(t - \varepsilon, t)$ shrink to the empty set. Both constructions leave $(Y, \delta)$ unchanged, since the censored still have $T > C$. $\square$

The width of the interval is $P(Y < t, \delta = 0)$, the share censored strictly before $t$. The NHANES follow-up is recorded in whole months, so censoring has atoms: 180 adults are censored at exactly 24 months, and under the unit's convention they were alive at 24 months. At 24 months the bounds are therefore 0.5037 and 0.9805, a width of 0.4767 (the 2,488 of the 5,219 adults censored before 24 months; the rounded bounds differ by 0.4768). Two naive answers at 24 months are related to these bounds. "Deaths so far over everyone", $1 - 102/5{,}219 = 0.9805$, is the upper bound: it assumes that everyone censored before 24 months survived to 24 months. "Everyone not seen beyond 24 months counts as dead" gives $P(Y > 24) = 0.4692$, a valid lower bound but not the sharp one, since it also counts as dead the 180 adults known to be alive at 24 months. (The share of all deaths in the whole follow-up, 0.0222 of the cohort, is neither: it mixes follow-up times up to 37 months.) Under independent censoring the product-limit estimate is 0.9791, with 2,629 adults still at risk at 24 months; at 36 months it is 0.9695 with 83 at risk.

NHANES 2017–2018 adults linked to death records through December 31, 2019, by month since examination. The shaded band holds every value of the survival function consistent with the observed data at each month (Theorem 7); its upper edge is the share not yet seen to die, and its lower edge counts as alive only those seen alive at that month. The dashed curve, just below the band, treats every censoring as a death at once, which also discards those censored at exactly that month. Notice that the band widens from month 11, when the first adults reach the closing date, and that the product-limit estimate, which assumes independent censoring, runs just below the upper edge because deaths are rare.
NHANES 2017–2018 adults linked to death records through December 31, 2019, by month since examination. The shaded band holds every value of the survival function consistent with the observed data at each month (Theorem 7); its upper edge is the share not yet seen to die, and its lower edge counts as alive only those seen alive at that month. The dashed curve, just below the band, treats every censoring as a death at once, which also discards those censored at exactly that month. Notice that the band widens from month 11, when the first adults reach the closing date, and that the product-limit estimate, which assumes independent censoring, runs just below the upper edge because deaths are rare.

The NHANES censoring is administrative: it depends on the examination date, not on health. That is the kind of censoring for which independence is most plausible, though not guaranteed (for example, if the people examined in late 2018 differed in health from those examined early in 2017). Censoring by withdrawal, by moving away or by a competing event is where the assumption is most often in doubt, and where Theorem 7 is the honest summary.

The likelihood of censored data

To fit a model, such as an exponential or a Weibull distribution, we need the probability of the observed data as a function of the model's parameters. Here is the idea: an observed death at $y$ contributes the density at $y$ times the probability that censoring had not yet happened; a censoring at $y$ contributes the density of censoring at $y$ times the probability that the death had not yet happened. When the censoring distribution has nothing to do with the parameters, its factors drop out.

Theorem 8 (the likelihood under independent, noninformative censoring). Let $(T_1, C_1), \ldots, (T_n, C_n)$ be independent pairs, with $T_i$ and $C_i$ independent; let $T_i$ have density $f(t; \theta)$ and survival function $S(t; \theta)$, and $C_i$ have density $g$ and survival function $G$ that do not involve $\theta$ (noninformative censoring). Then the joint density of the observations at $(y_i, \delta_i)$ is

$$\prod_{i=1}^n \big[f(y_i; \theta)\,G(y_i)\big]^{\delta_i}\big[g(y_i)\,S(y_i; \theta)\big]^{1 - \delta_i},$$

and the likelihood for $\theta$ is, up to a factor not involving $\theta$,

$$L(\theta) = \prod_{i=1}^n f(y_i; \theta)^{\delta_i}\,S(y_i; \theta)^{1 - \delta_i} = \prod_{i=1}^n h(y_i; \theta)^{\delta_i}\,S(y_i; \theta). \tag{1}$$

Proof. For one pair, with $\Delta > 0$ small, $P(y \le Y < y + \Delta, \delta = 1) = P(y \le T < y + \Delta, C \ge T) = \int_y^{y+\Delta} f(u)G(u^-)\,du$ by independence, and dividing by $\Delta$ and letting $\Delta \downarrow 0$ gives $f(y)G(y)$ ($G$ is continuous since $C$ has a density). In the same way $P(y \le Y < y + \Delta, \delta = 0) = P(y \le C < y + \Delta, T > C) = \int_y^{y+\Delta} g(u)S(u)\,du$, giving $g(y)S(y)$. Independent pairs multiply. The factor $\prod G(y_i)^{\delta_i}g(y_i)^{1 - \delta_i}$ does not involve $\theta$, so it changes no ratio of likelihoods (The Likelihood Function, the theorem that ratios do not depend on the description). The second form of (1) uses $f = hS$. $\square$

Independence is used to factor each contribution, and noninformativeness to discard the censoring factor. If the censoring distribution shared parameters with $T$, the discarded factor would carry information about them, and (1) would be a valid likelihood but not the whole one.

The same form appears under the other designs.

  • Type I censoring at a fixed $c$: a death at $y \le c$ has density $f(y)$, and a survivor has probability $S(c)$, so $L = \prod f(y_i)^{\delta_i}S(c)^{1 - \delta_i}$, which is (1).
  • Type II censoring at the $r$th death among $n$: the probability that particular subjects die in small intervals around $t_{(1)} < \cdots < t_{(r)}$ and the other $n - r$ survive past $t_{(r)}$ is $\prod_{i \le r} f(t_{(i)})\Delta \cdot S(t_{(r)})^{n-r}$. The $r$ who die can be chosen and ordered in $n!/(n-r)!$ ways, so the density of the first $r$ order statistics is $\frac{n!}{(n-r)!}\prod_{i \le r}f(t_{(i)})\,S(t_{(r)})^{n-r}$: (1) up to a constant, although $C$ here depends on the lifetimes.
  • Left truncation at $v_i$: conditioning on $T_i > v_i$ divides subject $i$'s contribution by $S(v_i; \theta)$.
  • Interval censoring in $(l_i, r_i]$ contributes $S(l_i; \theta) - S(r_i; \theta)$, and left censoring at $c_i$ contributes $F(c_i; \theta)$.

The discrete form. For lifetimes on the grid $t_1 < t_2 < \cdots$, with the convention that a subject censored at $t_j$ survived $t_j$, a death at $t_j$ contributes $P(T = t_j) = h_j\prod_{i<j}(1 - h_i)$ and a censoring at $t_j$ contributes $S(t_j) = \prod_{i \le j}(1 - h_i)$ (Theorem 2). Collecting the factors by time, with $d_j$ deaths at $t_j$ and $n_j$ subjects at risk just before it,

$$L = \prod_j h_j^{\,d_j}(1 - h_j)^{\,n_j - d_j}. \tag{2}$$

Each subject at risk at $t_j$ contributes $1 - h_j$ if it passes $t_j$ and $h_j$ if it dies there, and nothing after leaving the risk set. Each factor is a binomial likelihood, maximized at $\hat h_j = d_j/n_j$; The Kaplan–Meier Estimator builds the estimator from these and studies it.

Example (the exponential, and the person-time rate). For $S(t) = e^{-\lambda t}$, (1) gives $\ln L = d\ln\lambda - \lambda\sum_i y_i$, where $d = \sum\delta_i$ is the number of deaths. Its derivative $d/\lambda - \sum y_i$ vanishes at

$$\hat\lambda = \frac{d}{\sum_i y_i},$$

deaths over the total time observed: the incidence rate per person-time of Rates in Public Health, now derived as a maximum likelihood estimate. The EM Algorithm reaches the same estimate for type I censoring as the fixed point of its iteration (its example of censored lifetimes). The second derivative is $-d/\lambda^2$, so the observed information gives the standard error $\hat\lambda/\sqrt d$ (Maximum Likelihood Estimation). For the NHANES 2017–2018 adults: 116 deaths in 10,187 person-years, $\hat\lambda = 11.39$ deaths per 1,000 person-years, with standard error 1.06. Under this model the two-year survival is $e^{-24\hat\lambda}$ with $\hat\lambda$ per month, 0.9775, close to the product-limit 0.9791. The single rate hides very different hazards: 33.4 deaths per 1,000 person-years among the 1,314 adults aged 65 and over (83 of the deaths), and 4.3 among the younger. Models that let the hazard depend on age and other covariates are the subject of The Cox Proportional Hazards Model and Parametric Survival Models, which also develops the exponential's inference.

Example (a piecewise-constant hazard). If the hazard is $\lambda_k$ on the $k$th of several fixed intervals, (1) splits into one factor per interval, $\ln L = \sum_k (d_k\ln\lambda_k - \lambda_k E_k)$, where $d_k$ counts the deaths in interval $k$ and $E_k$ is the total time all subjects spent in it. Each factor is maximized at $\hat\lambda_k = d_k/E_k$: deaths over person-years in the interval. That is the death rate ${}_nm_x$ of a life table, which is therefore the maximum likelihood estimate of a hazard taken as constant within each age interval (Life Tables, Theorem 3).

What naive analyses do

Back to the opening. Each naive analysis replaces the censoring by something else, and under independent censoring each converges to a definite wrong answer, which the next theorem identifies. Throughout, the sample is large, so each proportion is close to its probability.

Theorem 9 (what naive analyses estimate). Let $T$ and $C$ be independent, $T$ with a density, and $t$ fixed with $S(t) > 0$.

  1. Treating every censoring as a death estimates $P(Y > t) = S(t)G(t)$, which is below $S(t)$ whenever $G(t) < 1$.
  2. Dropping the subjects censored before $t$ and taking the share of the rest alive at $t$ estimates $\frac{S(t)G(t)}{S(t)G(t) + F_1(t)}$, which is at most $S(t)$, and equal to it when there is no censoring before $t$ ($G(t) = 1$).
  3. Dropping every censored subject and taking the share of the deaths that came after $t$ estimates $P(T > t \mid T \le C)$, which is at most $S(t)$.
  4. Taking the share of deaths so far among everyone, $1 - P(Y \le t, \delta = 1)$, estimates the upper bound of Theorem 7, at least $S(t)$.
  5. Restricting to the subjects whose censoring time exceeds $t$ (possible when, as under administrative censoring, $C$ is known for everyone) estimates $P(T > t \mid C > t) = S(t)$.

Proof. (1) $P(Y > t) = P(T > t)P(C > t)$ by independence. (2) The subjects kept are those with $Y > t$, of probability $S(t)G(t)$, and those who died by $t$ before being censored, of probability $F_1(t) = \int_0^t f(u)G(u)\,du$. The share alive is the stated ratio. It is at most $S(t)$ exactly when $G(t) \le S(t)G(t) + F_1(t)$, that is, when $G(t)F(t) \le \int_0^t f(u)G(u)\,du$; this holds because $G(u) \ge G(t)$ for $u \le t$, and $\int_0^t f = F(t)$. If $G(t) = 1$, then $G(u) = 1$ for $u \le t$ and both sides equal $F(t)$. (3) $P(T > t \mid T \le C) = E[a(T)b(T)]/E[b(T)]$ with $a(u) = \mathbf{1}\{u > t\}$, nondecreasing, and $b(u) = P(C \ge u)$, nonincreasing. For an independent copy $T'$ of $T$, $(a(T) - a(T'))(b(T) - b(T')) \le 0$ always, since the two differences have opposite signs. Taking expectations, $2E[a(T)b(T)] - 2E[a(T)]E[b(T)] \le 0$, so $E[ab]/E[b] \le E[a] = S(t)$. This is Chebyshev's association inequality. (4) Theorem 7. (5) Independence. $\square$

Part 5 is legitimate but wasteful: it discards the deaths and survival of everyone with a short potential follow-up, which is what the product-limit estimator of The Kaplan–Meier Estimator uses. Part 3 is the analysis that reports "the mean time to death among those who died": in NHANES 2017–2018 that mean is 11.4 months, a number set by the length of follow-up, not by the lifetime.

An experiment: censoring a cohort whose deaths are all known

The NHANES 1999–2000 adults with a blood pressure reading who could be linked, 4,802 people, were followed to the end of 2019. Their follow-up was at least 226 months for everyone still alive, so up to month 225 every death is known and there is no censoring: the share alive at 60, 120 and 180 months, 0.9371, 0.8561 and 0.7830, is the cohort's true survival. Of the cohort, 1,397 died during follow-up.

We censored this cohort artificially and asked each method for the share alive at 120 months, whose true value is 0.856. Each of 30 seeds (20261120 to 20261149) draws new censoring times for the same 4,802 people, so the only randomness is the censoring. Two schemes:

  • Independent: each person's censoring time is a whole number of months, uniform on 0 to 240, whatever their age or health. Like the real follow-up, it is recorded in whole months, so some subjects are censored at exactly 120 months and count as alive then. The "drop" analysis below drops them along with those censored earlier, since nothing is known of them after 120 months.
  • Dependent on age: people aged 60 or over at examination, 0.324 of the cohort, are censored uniformly on 0 to 120 months and the rest on 0 to 240. Within each age group censoring is still independent of the lifetime, but age predicts both: the share alive at 120 months is 0.967 under 60 and 0.624 at 60 or over.

Medians over the 30 seeds, with the interquartile range for the methods that aim at the truth:

Method Independent censoring Censoring depends on age
Product-limit (Kaplan–Meier) 0.855 (0.854 to 0.860) 0.898 (0.896 to 0.900)
Product-limit within age groups, averaged 0.859 (0.855 to 0.866)
Only those with censoring time beyond 120 0.858 (0.854 to 0.861) 0.967 (0.966 to 0.969)
Exponential fit, $e^{-120\hat\lambda}$ 0.857 0.906
Drop those censored at or before 120 0.804 0.821
Censoring counted as death 0.426 0.326
Lower bound of Theorem 7 0.430 0.330
Upper bound of Theorem 7 0.896 0.928

The true value is 0.856 in both columns. Under independent censoring the product-limit estimate and the complete-follow-up subset were on target in this experiment, as Theorems 5 and 9 predict, while the naive analyses fell where part 1 and part 2 of Theorem 9 put them: far below. The exponential fit happens to land near the truth at 120 months, but not elsewhere: at 60 months it gave 0.926 against a product-limit 0.936 and a truth of 0.937, and at 180 months 0.794 against 0.784 and 0.783. A constant hazard is too high early and too low late for a cohort whose hazard rises as it ages.

When censoring depends on age, the overall product-limit estimate was too high in every seed, by 0.035 to 0.051. The older, frailer members leave the risk sets early, and the risk sets that remain are increasingly young. Theorem 5 still holds within each age group, where censoring is independent of the lifetime, and the product-limit estimates computed within the two groups and averaged with the groups' shares recovered the truth. Restricting to those with long potential follow-up failed badly here: everyone with a censoring time beyond 120 months is under 60. The bounds of Theorem 7 hold in both columns, whatever the dependence; they are wide because a large share of the cohort is censored before 120 months: 0.466 under the independent scheme and 0.599 under the dependent one.

What is proved here, what is cited, what the experiment shows

Proved here: the relations among survival function, density and hazard for continuous (Theorem 1), discrete (Theorem 2) and mixed lifetimes (Theorem 3), with the comparison of the product and the exponential of the summed discrete hazards; the formula for $E[T^2]$ from the tail formula of Expectation, Variance and Covariance; the mean residual life, its differential equation and its inversion (Theorem 4); identification of the hazard and survival function under independent censoring, where follow-up reaches (Theorem 5); Tsiatis's nonidentifiability theorem for two risks (Theorem 6, following his proof); the condition $\theta \le \lambda\mu$ for his example to be a distribution; the sharp bounds without assumptions (Theorem 7); the likelihood of censored data under independent noninformative censoring, its forms under type I and type II censoring, truncation and interval censoring, and its discrete form (Theorem 8); the exponential and piecewise-constant maximum likelihood estimates; the limits of five naive analyses, including Chebyshev's association inequality (Theorem 9).

Cited. Tsiatis (1975) for the theorem and the example, both read in the original. The product integral for general distributions is mentioned, not used. The NCHS description of the public-use linked mortality files: the follow-up period, the eligibility rules and the perturbation of follow-up times (NCHS, 2022). The life expectancies of the NCHS 2019 life table (Arias and Xu, 2022).

What the computations show. On NHANES 2017–2018, the naive proportions and the bounds at 24 months, the product-limit estimate and the exponential rate. On NHANES 1999–2000 with artificial censoring (30 seeds), that the product-limit estimator and the complete-follow-up subset recovered the true survival under independent censoring, that the naive analyses fell where Theorem 9 predicts, and that censoring dependent on age biased the product-limit estimator upward while stratification by age removed the bias. These are observations of this cohort and these censoring schemes.

Worked problems

These problems were written for this article, on small exact cases and on the examples above; the article's script computes every answer.

Problem 1 (discrete hazards). A component is inspected yearly. Given that it is working at the start of year $j$, it fails during year $j$ with probability $1/10$, $1/9$ and $1/8$ for $j = 1, 2, 3$. Find the probability it survives three years, and compare with the exponential of minus the summed hazards.

By Theorem 2, $S(3) = (1 - \tfrac{1}{10})(1 - \tfrac19)(1 - \tfrac18) = \tfrac{9}{10}\cdot\tfrac89\cdot\tfrac78 = \tfrac{7}{10}$; the product telescopes. The summed hazards are $\tfrac{1}{10} + \tfrac19 + \tfrac18 = 0.336$, and $e^{-0.336} = 0.715$, larger than 0.7 as Theorem 2 requires. With hazards this large, treating a discrete hazard as if it were continuous overstates survival by 0.015.

Problem 2 (a linear hazard). In Tsiatis's example, the independence analysis reports the hazard $h(t) = 0.1 + 0.02t$. Find its survival function, density, median and mean.

By Theorem 1, $H(t) = 0.1t + 0.01t^2$, $S(t) = e^{-0.1t - 0.01t^2}$ and $f(t) = (0.1 + 0.02t)e^{-0.1t - 0.01t^2}$. The median solves $0.01t^2 + 0.1t = \ln 2$, so $t = (-0.1 + \sqrt{0.01 + 0.04\ln 2})/0.02 = 4.71$. The mean is $\int_0^\infty S$. With $a = 0.1$ and $b = 0.02$, complete the square: $at + bt^2/2 = \frac b2(t + a/b)^2 - \frac{a^2}{2b}$. Substituting $s = \sqrt{b/2}\,(t + a/b)$,

$$E[T] = e^{a^2/(2b)}\sqrt{\frac2b}\int_{a/\sqrt{2b}}^\infty e^{-s^2}\,ds = \sqrt{\frac{\pi}{2b}}\,e^{a^2/(2b)}\,\operatorname{erfc}\Big(\frac{a}{\sqrt{2b}}\Big) = 5.46,$$

where $\operatorname{erfc}(z) = \frac{2}{\sqrt\pi}\int_z^\infty e^{-s^2}ds$. The true lifetime in that example is exponential with rate 0.1, mean 10: the independence assumption roughly halves the mean.

Problem 3 (a small censored sample). Five subjects give $(y, \delta) = (2, 1), (5, 0), (7, 1), (8, 0), (12, 1)$. Fit an exponential model by maximum likelihood, and compare with the rates obtained by dropping the censored subjects and by treating censoring as death.

By (1), $\ln L = 3\ln\lambda - 34\lambda$, since there are 3 deaths and the times add to 34. So $\hat\lambda = 3/34 = 0.0882$, and the maximized log-likelihood is $3\ln(3/34) - 3 = -10.283$. Dropping the censored keeps 3 deaths in 21 units of time, a rate of $3/21 = 0.1429$: the two censored subjects' 13 units of survival are thrown away. Treating censoring as death gives 5 deaths in 34 units, $5/34 = 0.1471$. Both naive rates exceed the estimate by more than half, in the same direction as the analyses of Theorem 9: each makes the lifetimes look shorter.

Problem 4 (two exponential clocks). Let $T$ be exponential with rate $\lambda = 1/10$ and $C$ independent exponential with rate $\mu = 1/40$. Find $P(\delta = 1)$, the survival at $t = 10$, and what analyses 1 and 3 of Theorem 9 estimate at $t = 10$.

$P(\delta = 1) = P(T \le C) = \int_0^\infty \lambda e^{-\lambda u}e^{-\mu u}\,du = \lambda/(\lambda + \mu) = 0.80$. The true $S(10) = e^{-1} = 0.368$. Analysis 1 estimates $P(Y > 10) = e^{-(\lambda + \mu)10} = 0.287$. Analysis 3 estimates $P(T > 10 \mid \delta = 1) = \int_{10}^\infty \lambda e^{-(\lambda + \mu)u}du \big/ \frac{\lambda}{\lambda + \mu} = e^{-(\lambda + \mu)10} = 0.287$ too: for exponential clocks the deaths that are seen have the distribution of $Y$, exponential with rate $\lambda + \mu$, and dropping the censored and counting them as deaths give the same wrong answer. Theorem 5 recovers the truth: the observed events have sub-density $\lambda e^{-(\lambda + \mu)t}$, and dividing by $P(Y \ge t) = e^{-(\lambda + \mu)t}$ gives the hazard $\lambda$.

Summary

  • The survival function, density and hazard determine one another: $S = e^{-H}$ and $f = hS$ for continuous lifetimes (Theorem 1), $S(t) = \prod(1 - h_j) \le e^{-\sum h_j}$ for discrete ones (Theorem 2), and a product of the two for mixed ones (Theorem 3). The mean is $\int S$, and the mean residual life determines the distribution (Theorem 4).
  • Censoring hides part of a lifetime; truncation hides whole subjects. NHANES follow-up is administrative right censoring with staggered entry, and on the scale of age it is also left truncated.
  • Under independent censoring the hazard is the rate of observed events among those at risk, and the survival function is identified wherever follow-up reaches (Theorem 5). Without independence nothing is identified: every dependent model has an independent twin with the same data (Theorem 6, Tsiatis), and only the bounds $P(Y > t) + P(Y = t, \delta = 0) \le S(t) \le 1 - P(Y \le t, \delta = 1)$ remain (Theorem 7).
  • Under independent noninformative censoring the likelihood is $\prod f(y_i)^{\delta_i}S(y_i)^{1-\delta_i}$, and the same form holds for type I and type II designs (Theorem 8). For a constant hazard the estimate is the person-time rate; for a piecewise-constant hazard, the life table's rates.
  • Naive analyses estimate a bound or something below the truth (Theorem 9). On NHANES 2017–2018 at 24 months the sharp bounds are 0.5037 and 0.9805, treating censoring as death gives 0.4692, and the product-limit estimate is 0.9791; on artificially censored NHANES 1999–2000 data the product-limit estimate recovered the truth under independent censoring and overestimated it by 0.035 to 0.051 when censoring depended on age.

What we have learned

A survival function, a density and a hazard are three descriptions of one lifetime, and censored data speak most directly to the hazard: among those still observed, how fast do events occur? That rate is the population's hazard only if the censoring has not selected people by their prognosis, and the data cannot say whether it has. The assumption of independent censoring is therefore part of every survival estimate, and the bounds of Theorem 7 measure how much it is doing.

Under that assumption, Theorem 5 says that the ratio of deaths to the number at risk estimates each discrete hazard, and Theorem 2 says how to multiply such ratios into a survival curve. The Kaplan–Meier Estimator turns those two facts into an estimator, derives its variance by Greenwood's formula, and draws the curves of the NHANES cohorts. Life Tables is the same product with yearly steps and a whole population.

References