Regression
Logistic Regression: Understanding the Basics and Beyond
The model and likelihood from first principles, a real fitted trial, a full diagnostic toolkit, and a genuine separation failure worked end to end with Firth's correction.
Introduction
Logistic regression is the default model whenever the outcome is binary — objective response vs. no response, event vs. no event by a fixed timepoint, biomarker-positive vs. negative. It's simpler than the Cox model in one respect (no risk sets, no partial likelihood) but shares many of the same underlying machinery: maximum likelihood estimation, Wald/Score/LR testing, and an effect measure (the odds ratio) that is just as easy to misinterpret as a hazard ratio.
This tutorial goes deep: the model and its likelihood derived from first principles, a real Newton-Raphson/IRLS fit with a genuine convergence table, a full battery of model-fit diagnostics (deviance, pseudo-R², Hosmer-Lemeshow, ROC/AUC, calibration), the linearity-of-logit assumption tested via Box-Tidwell, and — treated with equal weight — the model's real failure modes, including a genuine complete-separation example where standard logistic regression breaks down entirely.
The Model
For a binary outcome \(Y \in \{0,1\}\), logistic regression models the log-odds (logit) of the outcome as a linear function of covariates:
Solving for the probability directly gives the familiar S-shaped curve:
The logit link is what makes the model tractable: it maps probabilities (bounded between 0 and 1) onto the entire real line, so an ordinary linear predictor on the right-hand side always produces a valid probability once transformed back.
Maximum Likelihood Estimation
Unlike the Cox model, logistic regression has a genuine full likelihood — each observation contributes a Bernoulli probability:
where \(p_i = 1/(1+e^{-X_i^\top\beta})\). The score (gradient) and information (negative Hessian) have clean closed forms:
Because the weights \(W\) depend on \(\beta\) through \(p\), this is solved iteratively — Newton-Raphson here is exactly Iteratively Reweighted Least Squares (IRLS), one of the cleanest examples of the general Newton-Raphson-equals-IRLS relationship that shows up throughout generalized linear models.
A Real Convergence Table
Below is the actual IRLS iteration history fitting a univariate model (objective response ~ treatment) on a simulated 250-subject oncology trial (143 responders, 57.2%) — a from-scratch implementation of the formulas above, not illustrative numbers:
| Iteration | Intercept | βtreatment | ‖Score‖ | Log-Likelihood |
|---|---|---|---|---|
| 0 | 0.000000 | 0.000000 | 28.040150 | −173.2868 |
| 1 | −0.115702 | 0.782369 | 1.075483 | −165.7886 |
| 2 | −0.115832 | 0.808865 | 0.004613 | −165.7785 |
| 3 (converged) | −0.115832 | 0.808979 | 0.000000 | −165.7785 |
Three iterations to convergence, typical for logistic regression's well-behaved (globally concave) log-likelihood. The fitted odds ratio: \(\exp(0.808979) = 2.246\), SE(β)=0.2609, 95% CI for OR = (1.347, 3.745), Wald \(Z=3.101\), \(p=0.0019\). This matches a production implementation (Python's statsmodels) to six decimal places on the identical data.
A Real Multivariate Fit
Extending to three covariates — treatment, baseline tumor burden (centered at 45mm), and biomarker status — on the same simulated trial:
| Covariate | β̂ | OR = exp(β̂) | SE(β̂) | 95% CI for OR | p-value |
|---|---|---|---|---|---|
| Treatment (vs. control) | 0.9524 | 2.592 | 0.275 | (1.513, 4.441) | 0.0005 |
| Tumor burden (per mm) | −0.0270 | 0.973 | 0.0095 | (0.955, 0.992) | 0.0047 |
| Biomarker-positive | 0.8959 | 2.450 | 0.285 | (1.402, 4.281) | 0.0017 |
Odds ratios and 95% confidence intervals from the multivariate model above. The dashed line at OR=1 marks no association; all three covariates' intervals exclude 1 in this sample.
The Predicted-Probability Curve
Converting the fitted model back to the probability scale, holding treatment and biomarker status fixed at their best- and worst-prognosis combinations, recovers the classic S-shaped curve as a function of tumor burden:
Predicted probability of response vs. baseline tumor burden, for the best-prognosis group (treated, biomarker-positive, gold) and worst-prognosis group (untreated, biomarker-negative, slate). Both curves share the same logistic shape and the same slope on the logit scale — only their intercept differs — which is itself a visual demonstration of what the additive-on-the-logit-scale model specification actually implies.
Interpreting the Odds Ratio — Carefully
OR = 2.592 for treatment means: holding tumor burden and biomarker status fixed, the odds of response are 2.592 times higher on treatment than off it. Two things worth being precise about:
- Odds are not probabilities, and an odds ratio is not a risk ratio — the two are close only when the outcome is rare. This is explored numerically below, because it's one of the most common misinterpretations of logistic regression output in applied papers.
- Like the hazard ratio, the odds ratio is non-collapsible — adding a prognostic (but non-confounding) covariate to the model can change the OR for treatment even with no confounding present, for purely mathematical reasons unrelated to bias.
Three Ways to Test a Logistic Coefficient
Exactly as with the Cox model, every coefficient can be tested three ways — computed here on the univariate treatment model above:
| Test | Statistic | χ² | p-value |
|---|---|---|---|
| Wald | \(Z = \hat\beta/SE(\hat\beta)\) | 9.617 | 0.00193 |
| Score | evaluated at β=0, no refitting needed | 9.757 | 0.00179 |
| Likelihood Ratio | \(2[\ell(\hat\beta)-\ell(0)]\) | 9.814 | 0.00173 |
The three agree closely here because the sample is reasonably large and the effect isn't extreme; they can diverge more in small samples or with strong effects, and the likelihood ratio test is generally the most trustworthy of the three when they disagree.
Model Fit: Deviance and Pseudo-R²
Deviance is \(-2\) times the log-likelihood, and differences in deviance between nested models are exactly the likelihood ratio χ² test statistic. For the multivariate model above:
| Quantity | Value |
|---|---|
| Null deviance (intercept only, df=249) | 341.372 |
| Residual deviance (full model, df=246) | 313.622 |
| LR χ² (df=3) | 27.749, p=4.1×10⁻&sup6; |
Unlike ordinary linear regression, there is no single agreed-upon R² for logistic regression — several pseudo-R² measures exist, none directly comparable to a linear-model R² in magnitude:
| Measure | Formula | Value |
|---|---|---|
| McFadden's R² | \(1-\ell(\hat\beta)/\ell(0)\) | 0.0813 |
| Cox & Snell R² | \(1-\exp\big(\tfrac{2}{n}[\ell(0)-\ell(\hat\beta)]\big)\) | 0.1051 |
| Nagelkerke R² | Cox & Snell, rescaled to a 0–1 maximum | 0.1411 |
McFadden's values of 0.08–0.14 look low by linear-regression intuition but are unremarkable for logistic regression on individual-level clinical data — McFadden himself suggested that values as low as 0.2–0.4 already represent an excellent fit for this class of model, since binary outcomes are inherently noisier per observation than continuous ones.
Hosmer-Lemeshow Goodness-of-Fit Test
The Hosmer-Lemeshow test groups subjects into deciles of predicted risk and compares observed vs. expected event counts within each group, summed into a χ² statistic with (number of groups − 2) degrees of freedom:
| Decile (predicted risk range) | n | Observed | Expected |
|---|---|---|---|
| 0.238–0.343 | 25 | 7 | 7.43 |
| 0.343–0.405 | 25 | 11 | 9.44 |
| 0.405–0.477 | 25 | 10 | 10.86 |
| 0.477–0.544 | 25 | 11 | 12.88 |
| 0.544–0.592 | 25 | 18 | 14.15 |
| 0.592–0.622 | 26 | 16 | 15.87 |
| 0.622–0.679 | 24 | 13 | 15.57 |
| 0.679–0.718 | 25 | 16 | 17.40 |
| 0.718–0.782 | 25 | 20 | 18.55 |
| 0.782–0.913 | 25 | 21 | 20.86 |
\(\chi^2 = 5.572\), df=8, \(p=0.695\) — no evidence of poor fit (a large, significant statistic would indicate the model's predictions systematically diverge from observed outcomes within risk strata; here they don't).
Discrimination: The ROC Curve and AUC
Where Hosmer-Lemeshow asks "are the predicted probabilities well-calibrated," the ROC curve asks a different question: "how well does the model rank subjects who had the event above those who didn't," across every possible classification threshold simultaneously.
ROC curve for the multivariate model, computed from real predicted probabilities against actual outcomes. Area under the curve (AUC) = 0.687 — interpretable as: a randomly chosen responder has a higher predicted probability than a randomly chosen non-responder 68.7% of the time. An AUC of 0.5 is chance; 1.0 is perfect discrimination. 0.687 reflects a real, moderate signal with three prognostic covariates and a genuinely noisy binary endpoint — consistent with the modest pseudo-R² values above.
Calibration: Are the Predicted Probabilities Trustworthy?
Discrimination (AUC) and calibration are different properties — a model can rank subjects well while still producing probabilities that are systematically too high or too low. Calibration is checked by comparing mean predicted probability to observed event rate within groups of similar predicted risk:
Predicted probability (x-axis) vs. observed response rate (y-axis) by decile of predicted risk, against the 45° line of perfect calibration. Points scatter around the line without a systematic bend, consistent with the non-significant Hosmer-Lemeshow result above — the same conclusion reached two different ways.
Checking the Linearity-of-Logit Assumption: The Box-Tidwell Test
The model assumes each continuous covariate enters linearly on the logit scale — not that the relationship between the covariate and the raw probability is linear (it isn't, by construction), but that it's linear after the logit transformation. The Box-Tidwell test checks this by adding an interaction term between the covariate and its own natural log, and testing whether that interaction is significant:
| Term | Coefficient | SE | p-value |
|---|---|---|---|
| tumor_burden | −0.3402 | 0.212 | 0.109 |
| tumor_burden × ln(tumor_burden) | 0.0653 | 0.044 | 0.139 |
The interaction term is not statistically significant (\(p=0.139\)) — no evidence against linearity in the logit for tumor burden in this model. Had it been significant, the standard remedies are a transformation (log, polynomial) of the covariate or a more flexible functional form (restricted cubic splines).
Key Assumptions, Stated Explicitly
- Linearity of the logit for continuous predictors (checkable via Box-Tidwell, above, or by visual inspection of smoothed residual plots).
- Independence of observations — violated by repeated measurements per subject or clustering within sites, which requires GEE or a mixed/random-effects logistic model.
- No (or minimal) multicollinearity among predictors — checkable via Variance Inflation Factor, exactly as in linear regression.
- Correct model specification — all relevant predictors included with the correct functional form; omitted-variable issues interact with the odds ratio's non-collapsibility discussed above.
- Adequate events per variable — a common rule of thumb calls for at least 10 events (and 10 non-events) per estimated parameter to keep bias and variance of the MLE reasonable; this trial's 143 responders and 4 parameters comfortably clears that bar.
Shortcomings and Limitations
Complete Separation: Where Maximum Likelihood Breaks
This is worth demonstrating directly rather than just describing. When a predictor (or combination of predictors) perfectly predicts the outcome — every subject with \(X \gt 0\) has the event, every subject with \(X \lt 0\) doesn't — the maximum likelihood estimate does not exist as a finite number. The likelihood keeps increasing as \(|\beta| \to \infty\), so the optimizer drives the coefficient toward infinity without ever converging in any meaningful sense. On a genuinely, deterministically separated 20-point dataset:
| Method | β̂x | SE(β̂x) | OR |
|---|---|---|---|
| Standard MLE (failed to converge) | 64.95 | 314,435,974 | essentially ∞ |
| Firth's penalized likelihood | 1.5546 | 0.7681 | 4.73 |
The standard fit's coefficient and standard error are both computational nonsense — a real, if extreme, illustration of what "the software still printed numbers" can hide. Firth's bias-reduced logistic regression fixes this by adding a penalty term (equivalent to a Jeffreys-invariant prior) to the score equation:
where \(h\) is the diagonal of the model's hat matrix. This small modification guarantees a finite estimate even under complete or quasi-complete separation, at the cost of a modest, well-characterized bias reduction rather than the effectively unbounded bias of the failed standard fit. Firth's method (or an equivalent weakly-informative-prior Bayesian approach) is the standard recommendation whenever separation is detected or suspected — common with rare events, small samples, or a strong single predictor.
The Odds Ratio Diverges from the Risk Ratio as Outcomes Become Common
Holding a true OR of 3.0 fixed and varying the baseline (control-arm) risk shows exactly how much the odds ratio and risk ratio can diverge:
| Baseline risk | OR (fixed) | Implied RR | Exposed-arm risk |
|---|---|---|---|
| 5% | 3.0 | 2.73 | 13.6% |
| 20% | 3.0 | 2.14 | 42.9% |
| 40% | 3.0 | 1.67 | 66.7% |
| 60% | 3.0 | 1.36 | 81.8% |
At a 5% baseline risk, OR and RR are reasonably close (3.0 vs. 2.73). At a 60% baseline risk — entirely plausible for a treatment-response endpoint in oncology — the same OR of 3.0 corresponds to an RR of only 1.36, a substantially smaller-sounding effect. Reporting "the odds of response tripled" when the outcome is common, without clarifying it's an odds ratio rather than a risk ratio, is one of the most common sources of effect-size overstatement in applied clinical literature.
Non-Collapsibility
As with the Cox hazard ratio, adding a prognostic (but non-confounding) covariate to a logistic model changes the odds ratio for treatment even under perfect randomization with zero confounding — a mathematical property of the odds ratio itself, not evidence that the unadjusted estimate was biased. This is a genuine source of confusion when comparing "crude" vs. "adjusted" odds ratios across studies or across models with different covariate sets.
Sensitivity to Rare Events
Beyond outright separation, ordinary maximum likelihood logistic regression is known to be finite-sample biased (away from the null) when the number of events is small relative to the number of parameters — a bias in the same direction that Firth's correction addresses more generally, which is why Firth's method is often recommended by default for rare-event studies even when separation isn't formally present.
Extensions Worth Knowing About
- Firth's penalized logistic regression — the fix for separation and rare-event bias, demonstrated above.
- GEE / mixed-effects (random-intercept) logistic regression — the fix for clustered or repeated-measures binary data, relaxing the independence assumption.
- Ordinal logistic regression (proportional odds model) — for ordered categorical outcomes with more than two levels.
- Multinomial logistic regression — for unordered categorical outcomes with more than two levels.
- Ridge/Lasso-penalized logistic regression — for high-dimensional predictor sets where ordinary MLE overfits or fails to converge.
Summary
Logistic regression's IRLS fitting is exactly Newton-Raphson on a genuine Bernoulli likelihood, which is what makes deviance-based testing, the three classical hypothesis tests, and a wide diagnostic toolkit (Hosmer-Lemeshow, ROC/AUC, calibration, Box-Tidwell) all fall out cleanly from the same underlying machinery. But two things deserve just as much attention as the mechanics: the odds ratio is not a risk ratio and diverges further from one as outcomes become more common, and standard maximum likelihood breaks down completely — not gracefully, completely — under separation, which is common enough with rare events or strong single predictors that Firth's correction is worth defaulting to rather than reaching for only after a fit visibly fails.
References
Cox, D.R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society, Series B.
Hosmer, D.W. & Lemeshow, S. (1980). Goodness of fit tests for the multiple logistic regression model. Communications in Statistics.
Box, G.E.P. & Tidwell, P.W. (1962). Transformation of the independent variables. Technometrics.
Firth, D. (1993). Bias reduction of maximum likelihood estimates. Biometrika.
Heinze, G. & Schemper, M. (2002). A solution to the problem of separation in logistic regression. Statistics in Medicine.
Peduzzi, P. et al. (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology.