Logistic Regression

Modeling the probability of a yes-or-no outcome. Why a straight line fails, odds and log-odds and the logistic function derived from them, maximum likelihood and Newton's method worked step by step, odds ratios and the divide-by-4 rule, standard errors and the likelihood-ratio test, classification cutoffs, and the danger of extrapolation, worked on the 23 shuttle flights before the Challenger accident.

On 28 January 1986, the space shuttle Challenger was destroyed shortly after launch. The accident was traced to the failure of a field joint on one of the two solid rocket boosters; each such joint is sealed by two O-rings, a primary and a secondary (Draper, 1993). On the night before, the discussion about whether to launch included concern that the O-rings were more likely to fail at low temperature, and the temperature at launch was forecast to be 31 °F (Draper, 1993). The record of 23 earlier flights gives the launch temperature and whether any of the six primary O-rings, one in each field joint, showed thermal distress, meaning erosion or blowby. Dalal, Fowlkes and Hoadley (1989) later published a statistical risk analysis of these flights using only data available before the accident. The data are in the UCI Machine Learning Repository (Draper, 1993).

The outcome here is not a number on a continuous scale. It is yes or no: distress or none. The previous articles in this topic, from Introduction to Regression to Multiple Linear Regression, modeled numerical outcomes. Logistic regression models the probability of a yes, and it is the standard method for binary outcomes in statistics and a standard baseline classifier in machine learning.

The data

Seven of the 23 flights had at least one O-ring with thermal distress. Ordered by launch temperature:

Temperature (°F) Flights with distress Flights without
53 to 63 4 (53, 57, 58, 63) 0
66 to 69 0 6
70 2 2
72 to 81 1 (75) 8

All four flights launched below 65 °F had distress, against 3 of the 19 at 65 °F or above. The coldest launch was 53 °F. The forecast of 31 °F was 22 degrees colder than any flight in the record.

The outcome is coded $y = 1$ for distress and $y = 0$ for none. Which outcome is called 1 is a choice; reversing it reverses the sign of every coefficient below and changes nothing else. (For a categorical input, the coding is done with indicator variables, as described in Multiple Linear Regression; the term one-hot encoding refers to that.)

Why not a straight line?

The mean of a 0/1 outcome is the proportion of ones, so a regression line for $y$ estimates the probability of distress at each temperature. Fitting one by least squares, the linear probability model, gives

$$\hat{p} = 2.9048 - 0.03738 \times \text{temperature}.$$

It predicts a probability of 0.288 at 70 °F and 0.924 at 53 °F, plausible values. But it predicts more than 1 for every temperature below 51.0 °F, including 1.746 at 31 °F, and less than 0 above 77.7 °F, where three flights in the data were launched. A probability outside $[0, 1]$ is meaningless, and James, Witten, Hastie and Tibshirani (2021, §4.2) use exactly this defect to motivate logistic regression. The deeper problem is the form: a straight line assumes each degree changes the probability by the same amount, while a probability must flatten as it approaches 0 or 1.

Thermal distress (1) or none (0) for the 23 flights, against launch temperature; tied points are stacked. The orange curve is the logistic regression fit, with a 95% band; the dashed gray line is a straight line fitted to the same 0/1 values, which leaves the range from 0 to 1 at both ends. The shaded region has no data, and the yellow line marks the 31 °F forecast.
Thermal distress (1) or none (0) for the 23 flights, against launch temperature; tied points are stacked. The orange curve is the logistic regression fit, with a 95% band; the dashed gray line is a straight line fitted to the same 0/1 values, which leaves the range from 0 to 1 at both ends. The shaded region has no data, and the yellow line marks the 31 °F forecast.

From odds to the logistic function

Logistic regression keeps the linear predictor $b_0 + b_1 x$ but changes what it predicts. A probability $p$ lies between 0 and 1. The odds

$$\text{odds} = \frac{p}{1 - p}$$

compare the chance of a yes with the chance of a no, and lie between 0 and $\infty$: a probability of 0.2 is odds of 1 to 4, or 0.25, and 0.9 is odds of 9 (James et al., 2021, §4.3.1). The log-odds, or logit,

$$\operatorname{logit}(p) = \ln \frac{p}{1 - p},$$

ranges over the whole real line, and is 0 when $p = 0.5$. A quantity that can take any real value can be modeled by a linear function without ever producing an impossible value. Logistic regression sets

$$\ln \frac{p(x)}{1 - p(x)} = b_0 + b_1 x.$$

Solving for $p$: exponentiate to get $p/(1-p) = e^{b_0 + b_1 x}$, multiply out, $p = (1 - p)\,e^{b_0 + b_1 x}$, and collect the terms in $p$:

$$p(x) = \frac{e^{b_0 + b_1 x}}{1 + e^{b_0 + b_1 x}} = \frac{1}{1 + e^{-(b_0 + b_1 x)}}.$$

This is the logistic function (or sigmoid) applied to the linear predictor (James et al., 2021, eqs. 4.2 to 4.4). It traces an S-shaped curve. When $b_0 + b_1 x$ is large and positive, $p$ approaches 1; when it is large and negative, $p$ approaches 0; and $p = 0.5$ where $b_0 + b_1 x = 0$, at $x = -b_0/b_1$. The sign of $b_1$ sets the direction of the S and its size sets the steepness. The intercept fixes the log-odds at $x = 0$; the probability there is $1/(1 + e^{-b_0})$, above one half when $b_0 > 0$ and below when $b_0 < 0$.

The name and the curve are old. Pierre-François Verhulst introduced the logistic function in the 1830s and 1840s as a model of population growth that levels off, and Joseph Berkson proposed it in 1944 for dose-response experiments, coining the term "logit" (Cramer, 2002). Cox (1958) developed the regression analysis of sequences of 0s and 1s whose probability depends on explanatory variables, the setting of this article.

Fitting by maximum likelihood

Least squares is no longer the natural criterion, because the outcomes are 0s and 1s and their variance depends on $p$. Logistic regression is fitted by maximum likelihood (James et al., 2021, §4.3.2): choose the coefficients under which the observed outcomes are most probable. If flight $i$ has predicted probability $p_i = p(x_i)$, the model gives its observed outcome probability $p_i$ if $y_i = 1$ and $1 - p_i$ if $y_i = 0$, which can be written $p_i^{y_i}(1 - p_i)^{1 - y_i}$. Treating flights as independent, the likelihood of all 23 outcomes is the product, and its logarithm is

$$\ell(b_0, b_1) = \sum_{i=1}^{n} \bigl[\, y_i \ln p_i + (1 - y_i) \ln (1 - p_i) \,\bigr].$$

The negative of $\ell$, averaged over observations, is the cross-entropy or log loss, the same quantity used to train the character model in The Bigram Language Model. Maximizing the likelihood and minimizing cross-entropy are the same thing.

The score equations

Write $z_i = b_0 + b_1 x_i$, so $p_i = 1/(1 + e^{-z_i})$. Two facts make the derivative simple. The derivative of the logistic function is $dp/dz = p(1 - p)$. And substituting $\ln p_i = z_i - \ln(1 + e^{z_i})$ and $\ln(1 - p_i) = -\ln(1 + e^{z_i})$ gives each term of $\ell$ as $y_i z_i - \ln(1 + e^{z_i})$. Differentiating with respect to $z_i$ gives $y_i - p_i$, and by the chain rule,

$$\frac{\partial \ell}{\partial b_0} = \sum_{i} (y_i - p_i) = 0, \qquad \frac{\partial \ell}{\partial b_1} = \sum_{i} x_i\,(y_i - p_i) = 0.$$

These score equations have the same shape as the normal equations of least squares, with $y_i - p_i$ in place of the residual: the "residuals" must sum to zero and be uncorrelated with the input. The first equation means the fitted probabilities add up to the number of observed ones. For the Challenger data they sum to 7.0000, the number of flights with distress.

Unlike the normal equations, these are not linear in the coefficients, because $p_i$ depends on them through the logistic function, and there is no closed-form solution. They are solved numerically.

Newton's method

Newton's method approximates $\ell$ near the current coefficients by a quadratic and jumps to that quadratic's maximum. The second derivatives of $\ell$ form the matrix $-X^{\mathsf{T}} W X$, where $X$ is the design matrix (a column of ones and a column of temperatures) and $W$ is diagonal with entries $p_i(1 - p_i)$. With $\mathbf{p}$ the vector of current probabilities, each step is

$$\mathbf{b}_{\text{new}} = \mathbf{b} + (X^{\mathsf{T}} W X)^{-1} X^{\mathsf{T}} (\mathbf{y} - \mathbf{p}).$$

Each step can be rewritten as a weighted least-squares regression on an adjusted response, which is why the algorithm is also called iteratively reweighted least squares. Because $X^{\mathsf{T}}WX$ is positive definite, the log-likelihood is concave and has at most one maximum. Starting from $b_0 = b_1 = 0$, where every $p_i = 0.5$:

Iteration $b_0$ $b_1$ Log-likelihood
0 0 0 −15.94239
1 9.61905 −0.14952 −10.59252
2 13.65574 −0.21125 −10.17932
3 14.93829 −0.23060 −10.15771
4 15.04229 −0.23215 −10.15760
5 15.04290 −0.23216 −10.15760

Newton's method converges in about five steps, as it typically does near the maximum, where the number of correct digits roughly doubles each step. The fitted model is

$$\ln \frac{\hat{p}}{1 - \hat{p}} = 15.0429 - 0.23216 \times \text{temperature}.$$

One situation breaks the method. If some temperature had separated the data perfectly, every flight below it with distress and every flight above it without, the likelihood would keep increasing as the curve became a step, and the slope estimate would run off to infinity. This is called separation (Gelman, Hill and Vehtari, 2020, §14.6). The Challenger data are not separated, because of the flights with distress at 70 and 75 °F.

What scikit-learn fits by default

scikit-learn's LogisticRegression does not fit this model by default. Its default adds an L2 penalty on the coefficients, with strength set by C=1.0 (scikit-learn developers, 2024), which shrinks them toward zero. On the Challenger data in degrees Fahrenheit, the default returns $14.8618$ and $-0.22950$, close to the maximum likelihood values only because the coefficients are large relative to the penalty. With temperature standardized to mean 0 and standard deviation 1, the maximum likelihood slope is $-1.6024$ and the default gives $-1.1210$, a third smaller. For the model in this article, pass penalty=None. Penalized fitting is useful with many inputs, but its coefficients are not the ones the standard errors and odds ratios below describe.

Interpreting the coefficients

The model is linear on the log-odds scale, so each scale gives the coefficients a different reading.

The fitted model on three scales, from 45 to 85 °F. The probability of distress follows an S-curve, crossing 0.5 at 64.8 °F (dotted). The odds fall by the same factor, 0.793, with each degree, a straight line on a logarithmic axis. The log-odds fall by the same amount, 0.232, with each degree.
The fitted model on three scales, from 45 to 85 °F. The probability of distress follows an S-curve, crossing 0.5 at 64.8 °F (dotted). The odds fall by the same factor, 0.793, with each degree, a straight line on a logarithmic axis. The log-odds fall by the same amount, 0.232, with each degree.
Temperature (°F) Log-odds Odds Probability
53 2.738 15.46 0.939
60 1.113 3.044 0.753
65 −0.048 0.953 0.488
70 −1.208 0.299 0.230
75 −2.369 0.094 0.086
81 −3.762 0.023 0.023

On the log-odds scale, the slope is the change in log-odds per degree: each degree warmer lowers the log-odds of distress by 0.232.

On the odds scale, a one-degree difference multiplies the odds by a constant factor. The odds at $x + 1$ divided by the odds at $x$ are

$$\frac{e^{b_0 + b_1 (x+1)}}{e^{b_0 + b_1 x}} = e^{b_1},$$

whatever $x$ is. This ratio is the odds ratio. Here $e^{-0.23216} = 0.793$: each degree warmer multiplies the odds of distress by 0.793. Read the other way, each degree colder multiplies them by $e^{0.23216} = 1.261$, and ten degrees colder by $e^{2.3216} = 10.19$. An odds ratio above 1 means the odds rise with the input, below 1 that they fall, and exactly 1 that the input makes no difference. The odds ratio is constant, which makes it the most compact summary, but it is a ratio of odds, not of probabilities. At 70 °F the odds are 0.299; ten degrees colder they are 10.19 times higher, 3.044, which is a probability of 0.753, not ten times 0.230.

On the probability scale, the effect of a degree depends on where on the curve it is. The derivative of $p$ with respect to $x$ is $b_1\,p(1 - p)$, largest in magnitude at $p = 0.5$, where it equals $b_1/4$. Gelman, Hill and Vehtari (2020, §13.2) call this the divide-by-4 rule: a logistic coefficient divided by 4 is an upper bound on the change in probability per unit of the input, and a good approximation near the middle of the curve. Here $-0.23216/4 = -0.058$: near 64.8 °F, where the probability is one half, one degree changes the probability of distress by about 5.8 percentage points. At 60 °F the slope is $-0.043$ per degree, and the actual change from 60 to 61 °F is $-0.046$.

With $k$ inputs, $\ln[p/(1-p)] = b_0 + b_1 x_1 + \cdots + b_k x_k$, and each $e^{b_j}$ is the odds ratio for a one-unit difference in $x_j$ with the other inputs held fixed (James et al., 2021, §4.3.4), read with the same care as in Multiple Linear Regression.

How sure are we?

The standard errors of maximum likelihood estimates come from the curvature of the log-likelihood at its peak: a sharply peaked likelihood pins the coefficients down, a flat one does not. The estimated covariance matrix is $(X^{\mathsf{T}} W X)^{-1}$, evaluated at the fitted probabilities, the same matrix Newton's method inverts. For the Challenger fit, the standard error of the slope is 0.10824, so the Wald statistic is $z = -0.23216/0.10824 = -2.145$ with a two-sided $p$-value of 0.032. The 95% interval for the slope is $-0.23216 \pm 1.96 \times 0.10824$, or $[-0.4443, -0.0200]$, and exponentiating its ends gives an interval for the odds ratio per degree of $[0.641, 0.980]$.

A second test compares likelihoods directly. The deviance of a model is $-2\ell$ at its fitted coefficients. The null model, with only an intercept, predicts the overall rate $7/23$ for every flight and has deviance 28.267; the model with temperature has deviance 20.315. The difference, 7.952, is the likelihood-ratio statistic, approximately chi-squared with one degree of freedom if temperature had no effect, giving $p = 0.0048$. The two tests agree that temperature matters, but with 23 observations they disagree by a factor of seven on how surprising the data would be without an effect. Both rely on large-sample approximations, and with this little data neither $p$-value should be taken to more than one significant figure. The logic of $p$-values, and their limits, is discussed further in Comparing Systems with Significance Tests.

Extrapolating to 31 °F

At the forecast temperature, the fitted model gives log-odds of $15.0429 - 0.23216 \times 31 = 7.846$, odds of about 2,555 to 1, and a probability of distress of 0.9996. That number is an extrapolation of the kind warned against in Introduction to Regression: no flight had launched below 53 °F, and the model's S-shape is an assumption, not an observation, over the 22 degrees between. The uncertainty is correspondingly large, as the widening band in the first figure shows. The standard error of the log-odds at 31 °F is 4.041, and the 95% interval for the probability, computed on the log-odds scale and transformed, runs from 0.482 to above 0.999. Leaving out the coldest flight, at 53 °F, changes the fitted coefficients to $14.18$ and $-0.2198$ and the predicted probability at 31 °F to 0.9994.

What the data could support is narrower than a single number and still clear: every flight below 65 °F had distress, the estimated risk rises steadily as temperature falls, and a launch at 31 °F would have been far outside all experience. The model cannot say what happens there with any precision. It can say that nothing in the data offered reassurance.

From probabilities to classes

A probability becomes a classification by comparing it with a cutoff. With the conventional cutoff of 0.5, the model flags the four flights at 63 °F or colder and misses the three with distress at 70 and 75 °F:

Cutoff Distress flagged (TP) False alarms (FP) Distress missed (FN) Correctly cleared (TN) Accuracy
0.5 4 0 3 16 0.870
0.2 6 8 1 8 0.609

A classifier that always predicts "no distress" scores 0.696, so accuracy alone says little here. Nothing about 0.5 is special. When a missed failure costs far more than a false alarm, a lower cutoff is appropriate even though accuracy falls: at 0.2 the model flags every flight at 70 °F or colder, catching six of the seven distress flights at the cost of eight false alarms. Choosing the cutoff is a decision about costs, not a property of the model. The trade-off between the two kinds of error, and measures such as precision, recall and ROC curves, are covered in Evaluating Classifiers and Binary Classification Metrics.

Because the log-odds are linear in the inputs, the boundary where $\hat{p}$ equals the cutoff is a single point with one input and a straight line or hyperplane with several. Logistic regression is therefore a linear classifier, like the Naive Bayes and Rocchio classifiers of the course wiki (Vector Space Classification); they differ in how they choose the boundary.

Adding an input

Each flight also has the pressure used in a leak check of the joints before launch: 50 psi for six flights, 100 for two and 200 for fifteen. Adding it to the model gives

$$\ln \frac{\hat{p}}{1 - \hat{p}} = 13.292 - 0.22867 \times \text{temperature} + 0.01040 \times \text{pressure}.$$

Holding pressure fixed, each degree colder multiplies the odds of distress by $e^{0.22867} = 1.257$, almost the same as before. Holding temperature fixed, 100 psi more multiplies them by $e^{1.040} = 2.83$, but with a standard error of 0.00898 for the pressure coefficient the data cannot distinguish this from no effect: the likelihood-ratio test for adding pressure gives 1.533, $p = 0.216$. With seven events among 23 flights, there is little information for more than one input.

In Python

import numpy as np
from sklearn.linear_model import LogisticRegression

temp = np.array([66, 70, 69, 68, 67, 72, 73, 70, 57, 63, 70, 78,
                 67, 53, 67, 75, 70, 81, 76, 79, 75, 76, 58])
distress = np.array([0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
                     0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1])

model = LogisticRegression(penalty=None).fit(temp.reshape(-1, 1), distress)
b0, b1 = model.intercept_[0], model.coef_[0, 0]
print(b0, b1)                          # 15.043, -0.2322
print(np.exp(b1))                      # odds ratio per degree, 0.793
print(model.predict_proba([[53], [70]])[:, 1])   # 0.939, 0.230
print(model.predict([[53], [70]]))     # [1 0] with the 0.5 cutoff

predict_proba returns two columns, the probabilities of class 0 and class 1, and predict applies a cutoff of 0.5. For a different cutoff, compare the second column of predict_proba with it directly. The flights are listed in flight order, as in the UCI file. The numbers in this article, including the Newton iterations, standard errors and tests, are computed by tools/experiments/data-science/logistic-regression.py in the site's repository.

References

  • Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society, Series B, 20(2), 215–232. record
  • Cramer, J. S. (2002). The Origins of Logistic Regression. Tinbergen Institute Discussion Paper 02-119/4. record
  • Dalal, S. R., Fowlkes, E. B., and Hoadley, B. (1989). Risk analysis of the space shuttle: Pre-Challenger prediction of failure. Journal of the American Statistical Association, 84(408), 945–957. record
  • Draper, D. (1993). Challenger USA Space Shuttle O-Ring [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5PW2T record · 4 pages
  • Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. record · 6 pages
  • James, G., Witten, D., Hastie, T., and Tibshirani, R. (2021). An Introduction to Statistical Learning with Applications in R (2nd ed.). Springer. record · 30 pages
  • scikit-learn developers (2024). LogisticRegression. scikit-learn 1.6 documentation. https://scikit-learn.org/1.6/modules/generated/sklearn.linear_model.LogisticRegression.html record