From Fitting an Association to Evaluating Predictions
An exposure-response model may describe the data used to estimate it yet still predict poorly for new patients, a new trial, or a clinically important exposure range. A fitted curve can look plausible overall while systematically underpredicting adverse events at high exposure, for example. Predictive checking asks a distinct question: does the model produce outcomes that resemble the outcomes actually observed under the relevant exposure conditions?
This is not a test of whether exposure causes an outcome. It evaluates the model’s predictive performance for a specified population, endpoint, and prediction task. Predictive performance can inform confidence in using a model to support development decisions, but it does not by itself establish causal interpretation or guarantee performance after the population or clinical context changes.
Before applying a check, specify the prediction target: for example, a binary event during a defined risk window, a continuous change from baseline, or an event count. Also specify the exposure range and population in which predictions will be used. The earlier material on diagnostics for exposure-response model fit concerns agreement with the fitting data; validation adds the requirement that predictive performance be assessed beyond those data.
Prediction Checks and Their Assumptions
For a binary outcome, let \(Y_i=1\) denote that patient \(i\) experiences the event and let \(\hat{p}_i\) be the model’s predicted event probability. A probability prediction is calibrated when, among patients assigned a given probability, the event occurs at approximately that frequency. For example, among patients with predicted probability near 0.20, about 20% would be expected to experience the event over repeated comparable samples.
Calibration should be assessed overall and across relevant exposure ranges. Grouping predictions into bins is easy to communicate, but results depend on bin boundaries and can hide within-bin patterns. Calibration plots can display observed event proportions against predicted probabilities, with uncertainty intervals. For a more compact summary, a calibration regression can relate observed outcomes to predicted log-odds: the ideal intercept is 0 and slope is 1. A nonzero intercept indicates systematic under- or overprediction; a slope below 1 often indicates predictions that are too extreme. These summaries do not replace inspection of calibration across exposures and important patient subgroups.
The Brier score summarizes squared error in binary probability predictions. For \(n\) validation patients, it is the mean squared difference between each observed outcome and predicted probability:
A lower score indicates better probability predictions for the same validation outcomes; zero is perfect. The score combines calibration and the ability to assign different probabilities to patients with different outcomes, so it should not be interpreted alone. Its magnitude also depends on event prevalence and the population being evaluated.
For a continuous outcome, let \(\hat{\mu}_i\) be a predicted mean and define the prediction error as \(e_i=Y_i-\hat{\mu}_i\). Mean error reveals directional bias, mean absolute error summarizes typical absolute deviation, and root mean squared error gives larger errors extra weight. If a model produces predictive intervals, their coverage should be compared with their stated nominal level. For example, a 90% predictive interval should contain about 90% of future individual outcomes across repeated comparable predictions. It is not a confidence interval for a population mean.
These comparisons require outcomes and predictions to refer to the same endpoint definition, time window, and measurement scale. In a held-out or external dataset, the model and its prediction procedure should be fixed before examining validation outcomes. The validation observations should be sufficiently independent of the fitting process, and outcome measurement and follow-up should be comparable to the intended use. If the exposure distribution in validation data does not cover the clinically relevant range, observed performance cannot establish reliable prediction in the unsupported range.
Worked Example: Brier Score Across Two Exposure Groups
Hypothetical data. A fitted binary-event model is evaluated in 100 held-out patients, split into two exposure groups. In the lower group, 50 patients each have predicted event probability 0.10 and 7 experience the event. In the higher group, 50 patients each have predicted probability 0.30 and 18 experience the event. Assume each patient has a recorded binary outcome for the same risk window and the predictions were fixed before those outcomes were examined.
For patients with a common predicted probability \(p\), the sum of squared errors is \(k(1-p)^2+(n-k)p^2\), where \(k\) is the number of events. In the lower group, the score is:
The observed event proportion is \(7/50=0.1400\), compared with the predicted probability 0.10, so this group shows underprediction. In the higher group, the score is:
Here the observed event proportion is \(18/50=0.3600\), also above the prediction of 0.30. The overall Brier score is the total squared error divided by all 100 patients:
The score summarizes prediction error, but the group-specific comparisons reveal that events occurred more often than predicted in both exposure groups. This check does not determine why the model underpredicts or whether a different exposure-response shape would fix the discrepancy.
Simulation-Based Predictive Checks
A simulation-based predictive check asks whether a chosen feature of the observed outcomes is unusual under the fitted model. First define what is held fixed and what is simulated. In a conditional check, the observed exposures and covariates are held fixed and outcomes are simulated from the fitted model. This tests whether the model’s outcome distribution is plausible at the exposure and covariate values actually observed. If the intended prediction includes the distribution of exposures or covariates as well, those quantities must also be simulated under a specified design.
Choose a discrepancy that matches the concern: event counts by exposure band, extreme laboratory values, or the number of outcomes above a clinically meaningful threshold. A general-purpose comparison can miss a local failure. For each simulated dataset, calculate the same discrepancy as for the observed data and compare the observed value with the simulated distribution. This is a model check, not automatically a conventional hypothesis test with a universal significance threshold.
Worked Example: Simulated Event Counts at High Exposure
Hypothetical data. In a held-out set of 40 patients at high exposure, a fitted binary model assigns each patient event probability 0.20. The observed event count is 15. A prespecified predictive check uses the total event count at high exposure, because underprediction in this range could affect the intended safety assessment. Assume conditional independence of patients’ outcomes for this calculation and treat the fitted probabilities as fixed.
Under the model, the count \(X\) follows a binomial distribution with 40 trials and event probability 0.20. Its expected value is \(40(0.20)=8\), and its standard deviation is \(\sqrt{40(0.20)(0.80)}=\sqrt{6.4}=2.530\). The observed count exceeds the expected count by 7, or \(7/2.530=2.767\) standard deviations. This comparison is a useful scale check, but the distribution is discrete, so simulation can provide a direct reference.
Suppose 1,000 hypothetical datasets are simulated from the fitted model at those same exposures, and 43 have at least 15 events. The estimated upper-tail fraction is \(43/1{,}000=0.0430\). Thus, an event count at least as large as the observed count occurs in about 4.3% of these simulations. The observed high-exposure count is unusually large relative to this fitted model’s predictions, warranting investigation of high-exposure calibration and relevant model assumptions. It does not prove that the model is false, nor does it establish what alternative model should replace it.
Held-Out and External Validation
A held-out validation set comes from the same study or data source but is kept out of model fitting and selection. It provides a useful check on optimism: a model often predicts its fitting data better than new observations because fitting has adapted it to random features of those data. If sample size permits, cross-validation or bootstrap procedures can estimate internal predictive performance more efficiently than a single split. However, if model choices are repeatedly made using the same resampling results, those results can become optimistic too; a final untouched evaluation set or a carefully nested resampling design may be needed.
External validation evaluates predictions in a separate study, site, or cohort. It is particularly informative about transport to new settings, but a difference in performance may reflect differences in population, endpoint ascertainment, follow-up, treatment, or exposure distribution. Report the setting and the extent of exposure overlap. Strong performance in a cohort whose exposures stop below the clinically important high-exposure range cannot validate predictions in that range.
| Approach | Strength | Main limitation |
|---|---|---|
| In-sample fit checks | Use all observed fitting data and help locate lack of fit. | Performance can look overly favorable because the model was fit to these outcomes. |
| Held-out validation | Directly tests predictions on unused observations from the study source. | Reduces data available for fitting and may represent only a narrow setting. |
| Cross-validation or bootstrap | Uses repeated resampling to estimate internal predictive performance. | Can be computationally demanding and does not establish transport to another cohort. |
| External validation | Tests performance in an independent setting or cohort. | Differences in population or outcome measurement complicate interpretation. |
Worked Example: Continuous Predictions and Interval Coverage
Worked Example: Summarizing Held-Out Continuous Prediction Errors
Hypothetical data. A model predicts change in a laboratory measure, in mg/L, for eight held-out patients. The predicted means are 2, 4, 5, 6, 7, 8, 9, and 10 mg/L; the observed changes are 1, 5, 4, 7, 9, 7, 11, and 12 mg/L. The model also provides a central 90% predictive interval of \(\hat{\mu}_i\pm 3.29\) mg/L for each patient. Assume the outcome definition matches the prediction target and these predictions were fixed in advance.
The errors \(e_i=Y_i-\hat{\mu}_i\), in mg/L, are \(-1,1,-1,1,2,-1,2,2\). Their sum is 5, so mean error is \(5/8=0.6250\) mg/L. The positive mean indicates that observations exceed predictions on average in this small validation set. The absolute errors sum to 11, giving mean absolute error \(11/8=1.375\) mg/L.
The squared errors sum to \(1+1+1+1+4+1+4+4=17\) \((\text{mg/L})^2\). Therefore root mean squared error is:
All eight observed values fall within their respective intervals because the largest absolute error is 2 mg/L, which is less than 3.29 mg/L. The observed coverage is therefore \(8/8=100\%\), compared with the nominal 90%. Eight observations are too few to conclude that the intervals are systematically too wide: sampling variation can readily produce coverage above or below 90%. The error summaries and interval coverage answer different questions and should be reported together with adequate uncertainty information in a larger validation exercise.
Diagnosing Failure and Avoiding Overinterpretation
A useful validation report does more than give one overall score. Examine calibration and error across exposure ranges that matter for the decision, and consider whether performance changes across clinically relevant patient groups. A good average score can conceal poor predictions in a small high-exposure subgroup. Conversely, a noisy subgroup estimate should not be treated as definitive evidence of failure without considering its uncertainty and size.
- Predictions systematically too low or too high: compare observed and predicted event frequencies or inspect mean errors. Check whether the discrepancy persists across exposure ranges.
- Excessively extreme probabilities: observed proportions may be closer to the overall event frequency than predictions imply; a calibration slope below 1 is one possible signal.
- Intervals miss too often: predictive intervals may be too narrow, or the assumed outcome distribution may not represent the data. Check coverage and the pattern of misses, not only the total.
- Predictions fail at relevant exposures: limited exposure support or differences between fitting and validation settings may make extrapolation unreliable. State explicitly where validation data provide no evidence.
- Outcome definitions or follow-up differ: apparent predictive failure may not be interpretable as a model comparison if the validation outcome is not the same target.
Calibration and discrimination are distinct. Calibration asks whether predicted probabilities agree with observed frequencies; discrimination asks whether patients who experience the event tend to receive higher predicted probabilities than those who do not. A model may rank patients reasonably well and still give probabilities that are systematically too low. Conversely, a model predicting nearly the same event frequency for everyone can be calibrated overall but have little ability to distinguish individual risks.
Validation has limits. Finite samples make performance estimates uncertain, particularly for uncommon events and narrow exposure bands. A single score cannot summarize all prediction tasks, and performance in one cohort does not guarantee performance in another. Recalibrating or refitting a model after observing validation outcomes may improve predictions, but the revised model has then used those outcomes; its performance requires a further independent assessment or an appropriately designed resampling evaluation.
Check Your Understanding
Answer each question, then compare with the solutions.
- A validation group has 20 patients with predicted event probability 0.25 each; 8 experience the event. Is the group calibrated at this probability, and what is its Brier score?
- What does a calibration slope below 1 commonly suggest about predicted probabilities?
- A central 90% predictive interval covers 17 of 20 held-out outcomes. Give the observed coverage and explain why this alone does not establish a calibration problem.
- A model is validated in patients whose exposures are all below the target clinical range. What can and cannot be concluded about predictions in that target range?
Solutions:
- The observed event proportion is \(8/20=0.400\), higher than the predicted 0.25, so this group shows underprediction rather than close calibration. The Brier score is \([8(1-0.25)^2+12(0.25)^2]/20=[8(0.5625)+12(0.0625)]/20=(4.50+0.75)/20=0.2625\).
- A slope below 1 commonly indicates predictions that are too extreme: high predicted probabilities tend to be too high and low ones too low. It is a diagnostic summary, not by itself an explanation of the cause.
- Observed coverage is \(17/20=0.850\), or 85%. With only 20 outcomes, coverage is imprecisely estimated; the result alone does not establish that the nominal 90% intervals are systematically miscalibrated.
- The validation supports conclusions only over the exposure range represented, subject to its population and outcome conditions. It cannot establish reliable predictions in the unrepresented target range; those predictions are extrapolations beyond the validation evidence.