Why Diagnose Fit Before Interpreting an Exposure-Response Relationship?
An exposure-response model can produce a precise-looking exposure effect while still describing the observed outcomes poorly. For example, predictions may be systematically too low at high exposure, binary event rates may exceed predicted probabilities in a clinically important subgroup, or a small number of participants may determine the estimated exposure slope. These problems matter in development decisions because they can distort estimates of benefit, risk, and the exposure range over which those estimates are used.
Fit diagnostics examine whether the model's predictions agree adequately with the data in ways that matter for its intended use. They do not establish that exposure causes an outcome, and good in-sample fit does not establish predictive performance in a new population. This tutorial focuses on diagnosing model fit for continuous, binary, and time-to-event exposure-response outcomes, building on the model structures introduced in the earlier material on continuous, binary, and time-to-event outcomes.
Residuals and the Logic of a Diagnostic
Residuals are useful because a model that captures the relevant mean structure should not leave a systematic relationship between its prediction errors and exposure, time, fitted value, or other modeled covariates. A trend in residuals against exposure can indicate that the fitted exposure-response relationship is inadequate; a trend against time can indicate an unmodeled time pattern. A fan-shaped spread may indicate that variability changes with the mean or exposure. These are clues, not automatic proof of a particular cause.
The interpretation depends on the outcome model and its assumptions. For a continuous model with conditionally independent errors, the mean model should be correctly specified and the assumed variance should reasonably describe residual spread. For binary outcomes, the conditional event probabilities and their link structure should be adequate; individual residuals are discrete and have variance that depends on the predicted probability. For time-to-event models, the event-time distribution or hazard structure, risk sets, and censoring assumptions must be considered. Repeated measurements from one participant are not independent observations: plots should be interpreted with that dependence in mind, and case-influence checks should generally remove a participant's whole record rather than a single visit.
Continuous Outcomes: Residual Patterns and Calibration
A raw residual \(e_i=Y_i-\hat{\mu}_i\) is expressed in the outcome's units. When conditional standard deviations \(\hat{\sigma}_i\) are available, a standardized residual can be formed as \(r_i=e_i/\hat{\sigma}_i\). Under a correctly specified conditional mean and variance, residuals should fluctuate around zero without a systematic pattern; standardized residuals should have approximately comparable spread across the range of fitted values. Approximate normality is relevant when the model assumes normally distributed errors, but a normal-looking residual histogram alone does not establish that the mean structure is adequate.
The mean-zero expectation follows directly from the model. If \(E(Y_i\mid X_i)=\mu_i\), where \(X_i\) denotes the modeled predictors, then \(E(Y_i-\mu_i\mid X_i)=E(Y_i\mid X_i)-\mu_i=0\). Thus, a systematic residual pattern against a predictor is evidence to investigate: the conditional mean may be misspecified, an important covariate may be absent, or some other assumption may fail. The residuals are not expected to be exactly zero, and random scatter is compatible with ordinary sampling variation.
Worked Example: Reading Continuous-Outcome Residuals
Suppose a hypothetical exposure-response analysis predicts a continuous efficacy score at four participant-visits. Scores are in points. The observed and predicted scores are 10.2 and 10.0, 12.1 and 12.0, 15.0 and 14.0, and 17.2 and 16.0, respectively. Assume for illustration that these are independent observations with a common conditional standard deviation and that the model's mean predictions are the quantities being checked.
| Exposure rank | Observed score | Predicted score | Residual (points) |
|---|---|---|---|
| Lowest | 10.2 | 10.0 | 0.2 |
| Second | 12.1 | 12.0 | 0.1 |
| Third | 15.0 | 14.0 | 1.0 |
| Highest | 17.2 | 16.0 | 1.2 |
For each row, \(e_i=Y_i-\hat{\mu}_i\); for example, the highest-exposure residual is \(17.2-16.0=1.2\) points. The root mean squared residual is
This summary gives an overall discrepancy scale, not a test of model adequacy. The two larger positive residuals occur at the higher exposure ranks, suggesting possible underprediction there. With only four observations, this is a signal to investigate rather than a reliable conclusion about the exposure-response shape. A residual-versus-exposure plot, checked alongside the assumed variance and participant-level dependence, would provide more context.
Observed-versus-predicted summaries complement residual plots. For a continuous outcome, compare observed and predicted values across clinically meaningful ranges of predicted outcome or exposure. Summaries such as group means can reveal systematic under- or overprediction that a single overall error measure hides. Grouping is descriptive: its result depends on how groups are formed, and it can conceal patterns within groups.
Binary Outcomes: Probability Calibration
For a binary outcome \(Y_i\in\{0,1\}\), a model predicts an event probability \(\hat{p}_i\). The individual Pearson residual is \((Y_i-\hat{p}_i)/\sqrt{\hat{p}_i(1-\hat{p}_i)}\). Its denominator reflects the model-based standard deviation of a Bernoulli outcome, since the variance is \(p_i(1-p_i)\). These residuals can be difficult to read individually, especially when predicted probabilities are near zero or one, so calibration summaries are often more informative.
Calibration asks whether outcomes occur at about the rates the model predicts. In a group of participants with similar predictions, the observed event proportion can be compared with the average predicted probability. A calibration-in-the-large comparison concerns overall predicted event frequency versus observed frequency. A calibration curve compares observed proportions with predictions over a range. A regression calibration summary may relate outcomes to the logit of predicted probabilities; ideal calibration has intercept zero and slope one. Such summaries are diagnostics, not proof that the model is correct.
Worked Example: Comparing Predicted and Observed Event Rates
In a hypothetical binary efficacy analysis, 20 participants in a lower-predicted-risk group each have predicted event probability 0.20; three experience the event. In a higher-predicted-risk group, 20 participants each have predicted probability 0.60; 14 experience the event. Assume each participant contributes one binary outcome and the groups are used only as descriptive calibration summaries.
For the lower group, the mean predicted probability is 0.20 and the expected event count is \(20(0.20)=4\). The observed event proportion is \(3/20=0.15\), five percentage points below the prediction. For the higher group, the expected count is \(20(0.60)=12\), while the observed proportion is \(14/20=0.70\), ten percentage points above the prediction.
The discrepancies point in different directions across the two groups, so the overall event rate alone could obscure the pattern. As a descriptive check, the group Pearson discrepancies are \((3-4)/\sqrt{20(0.20)(0.80)}=-0.559\) for the lower group and \((14-12)/\sqrt{20(0.60)(0.40)}=0.913\) for the higher group. These values do not by themselves establish lack of fit; the grouping, sample size, and model assumptions matter. In particular, examine whether the groups represent meaningful ranges of predicted probability and whether there are enough participants and events for stable summaries.
Time-to-Event Outcomes: Survival Calibration and Residuals
For a time-to-event model, compare predicted event-time behavior with observed survival experience at clinically relevant times. For example, within an exposure range, a model may predict 12-month event-free probability of 0.80. A Kaplan-Meier estimate of 0.72 in that range suggests the model may overpredict event-free survival by 0.08 at 12 months. This comparison must account for censoring: participants censored before 12 months do not have known 12-month outcomes. The Kaplan-Meier summary also relies on noninformative censoring within the group being summarized.
A complementary diagnostic uses Cox-Snell residuals. For participant \(i\), let \(\hat{H}_i(t)\) be the model-predicted cumulative hazard through time \(t\), and let \(T_i\) be the observed event or censoring time. The Cox-Snell residual is \(r_i=\hat{H}_i(T_i)\). Under a correctly specified event-time model, the latent cumulative-hazard transformation has an exponential distribution with mean one; with right censoring, the residuals are censored at the corresponding transformed censoring times. A cumulative hazard estimate for the residuals can therefore be compared with the reference line \(H(r)=r\). Departures may signal model misfit, though censoring and finite-sample variability affect the comparison.
Worked Example: Interpreting a Cox-Snell Residual
Consider two hypothetical participants from a fitted time-to-event exposure-response model. The model predicts cumulative hazard 0.40 by the first participant's observed event time; that participant's Cox-Snell residual is therefore \(r_1=0.40\). For a second participant censored at a time by which the predicted cumulative hazard is 0.20, the residual is \(r_2=0.20\), and it is censored at 0.20 rather than treated as an event.
These two residual values are not sufficient to assess fit. With a suitable sample, plot the estimated cumulative hazard of the residuals against residual time and compare it with the line \(H(r)=r\). Systematic departure over a range of residual values would prompt examination of the time-to-event model and its assumptions. The censored participant contributes information up to 0.20; treating that participant as if an event occurred at 0.20 would incorrectly add an event and distort the diagnostic.
Influential Observations and a Practical Diagnostic Sequence
A participant is influential if removing that participant materially changes an estimated parameter or a prediction relevant to the analysis. Influence is not the same as an unusual residual: a point can have a large residual but little effect on the fitted exposure slope, while a point with a moderate residual may be influential if its exposure or covariate values are unusual. In nonlinear models, assess influence by refitting after case deletion and comparing key estimates and predictions. A scaled change can be reported as the case-deletion change divided by the original standard error, but there is no universal cutoff that automatically identifies a problematic case.
Worked Example: Checking a Case-Deletion Change
In a hypothetical continuous exposure-response model, the estimated exposure slope is 0.18 score-points per exposure unit, with standard error 0.06 score-points per exposure unit. After removing one participant's entire record and refitting, the slope is 0.11 score-points per exposure unit. The scaled change is
The change is about 1.17 standard errors downward, so this participant merits review for data quality, unusual covariates, and scientific context. It does not justify deleting the participant automatically. Report whether conclusions change with and without the case, and retain the primary analysis according to a defensible prespecified approach rather than choosing the result that is more convenient.
Confirm outcome coding, time origin, units, censoring indicators, exposure summaries, and that each prediction corresponds to the correct participant and visit.
Use residuals against fitted values, exposure, time, and key covariates for continuous outcomes; probability calibration summaries for binary outcomes; and survival calibration or transformed residual checks for time-to-event outcomes.
Check whether apparent discrepancies reflect data errors, dependence, variance misspecification, censoring, or influential participants. Refit after participant-level deletion where appropriate and report changes in estimates and predictions.
Summarize persistent discrepancies and uncertainty. In-sample diagnostics assess fit to the analyzed data; they do not replace assessment of predictions in new data or establish causal interpretation.
Failure Modes, Alternatives, and Limits
- Reading a random-looking plot as proof of adequacy. Small samples may not reveal important misspecification. Use several complementary diagnostics and consider the precision of the summaries.
- Over-interpreting a residual threshold. A standardized residual or case-deletion change is a flag for review, not a universal pass/fail rule. Check context, data quality, and sensitivity of relevant conclusions.
- Ignoring dependence. Repeated observations from the same participant can create structured residuals. Do not treat every visit as an independent case when assessing influence; consider participant-level patterns.
- Ignoring censoring in survival checks. A censored participant's event time is unknown beyond the censoring time. Do not count censoring as an event when constructing observed-versus-predicted comparisons.
- Confusing calibration with discrimination or validation. Calibration concerns agreement of predicted and observed outcome levels. It does not show whether the model ranks individuals well, nor whether it predicts accurately in another population.
Residual plots are most direct for locating patterns in prediction errors; grouped observed-versus-predicted summaries are often easier to communicate clinically but depend on grouping choices; calibration regression compresses patterns into a small number of summaries; and case-deletion refits assess sensitivity to particular participants. None is sufficient alone. Formal goodness-of-fit tests may be available for particular models, but their results can depend strongly on sample size and grouping and should not replace graphical and context-specific checks.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A continuous outcome has residuals that become increasingly positive as exposure rises. What feature of the model should be investigated first?
- A binary model predicts event probability 0.30 for 40 participants in a group, and 16 events are observed. Calculate the predicted event count and observed event proportion. Describe the calibration discrepancy.
- Why should a censored participant not be counted as an event at the censoring time in a Cox-Snell residual check?
- A case-deletion refit substantially changes the exposure slope. Does this prove that the participant should be excluded? Explain.
- What is the key distinction between calibration and predictive validation in new data?
Solutions. (1) Investigate the conditional mean structure, including whether the exposure-response form captures the trend; also check data, covariates, and other assumptions before attributing the pattern to that form. (2) The predicted count is \(40(0.30)=12\). The observed proportion is \(16/40=0.40\), ten percentage points above the predicted probability, indicating underprediction in this group. (3) Censoring means the event time is unknown after the censoring time; counting it as an event invents an event and misrepresents the observed information. (4) No. Influence is a reason to examine data quality and report sensitivity, not an automatic exclusion rule; exclusion requires a defensible basis. (5) Calibration compares predicted and observed outcome levels in the data being assessed. Predictive validation asks how well predictions perform in data not used to fit the model.