Why Check an Adjusted External-Control Model Predictively?
An adjusted outcome model can produce a plausible overall control rate while missing important patterns in the data. For example, it might underpredict outcomes among high-risk patients, fail to reproduce a calendar-time increase, or fit one endpoint while missing another. Such local failures matter in an external-control analysis: a model that does not reproduce the external observations may yield unreliable adjusted estimates even if its overall average looks reasonable.
A posterior predictive check asks a direct question: if the fitted model and its uncertainty described the external controls adequately, what outcomes could it generate for these same patients? The comparison is between observed outcomes and outcomes simulated from the fitted model, keeping relevant design information—such as covariates, risk groups, and calendar periods—aligned. The check can reveal detectable lack of fit; it cannot establish that the external controls are exchangeable with a trial population or remove unmeasured confounding.
For a binary endpoint, a common conditional model assumes independent Bernoulli outcomes given the covariates and parameters, with each patient’s event probability supplied by the fitted model. For a time-to-event endpoint, replication must follow the specified event-time model; if the analysis model includes censoring, the check must represent that process as well. These conditional assumptions, the endpoint definition, and the covariate information used for adjustment must be appropriate before a resulting check is informative.
Choose Discrepancies That Match the Concern
A discrepancy statistic \(T(y,\theta)\) reduces a dataset to a feature relevant to a particular way the model might fail. A global discrepancy can summarize overall fit, but may conceal localized problems. For external controls, useful discrepancies are often calculated separately by baseline risk group, calendar period, and endpoint. Examples include an event count, a difference between observed and model-predicted event counts, or a measure of variation in event rates across groups.
The comparison must be defined consistently for observed and replicated outcomes. For each posterior draw, calculate \(T(y,\theta^{(m)})\) using the observed outcomes and parameters, and calculate \(T(Y^{\mathrm{rep},(m)},\theta^{(m)})\) using the replicated outcomes and the same parameters. Pairing both calculations at the same draw accounts for parameter uncertainty in the model-based reference distribution.
Here \(M\) is the number of posterior predictive simulations and \(I\{\cdot\}\) equals 1 when the condition inside braces holds and 0 otherwise. This is a posterior predictive tail area for the chosen discrepancy and direction. A value near 0 or 1 indicates that observed behavior is unusual relative to the specified replications in that direction; it is a diagnostic signal, not the probability that the model is true or false.
How the Check Works
State the outcome distribution, the covariates held fixed, and how risk groups, calendar periods, and endpoints enter the model. Check that the fitted model corresponds to the observed endpoint and ascertainment process.
For each posterior draw, simulate outcomes for the external-control observations at their recorded covariate values. Do not simulate new covariates if the purpose is to check outcome fit conditional on the observed external population.
Calculate the same discrepancies for observed and replicated data, including stratified summaries where clinically relevant. Inspect both the distribution across replicates and which strata drive any discrepancy.
The justification follows from the predictive distribution. The model’s uncertainty about \(\theta\) is represented by its posterior, and uncertainty in outcomes conditional on each parameter draw is represented by simulation. Integrating over both gives the distribution of outcomes the fitted model predicts for the fixed covariates:
The simulation procedure approximates this integral by averaging predictions over posterior draws. Therefore, a replicated dataset is not merely generated at a single fitted estimate: it incorporates both parameter uncertainty and outcome variability. A check that omits posterior uncertainty can make the predictive range too narrow.
Worked Examples
Worked Example: A Risk-Stratified Binary Endpoint
Suppose a hypothetical external-control cohort has 40 patients in each of three baseline-risk strata. The observed event counts are 2, 5, and 14. The fitted adjusted model gives posterior mean event probabilities of 0.075, 0.150, and 0.225 in those strata. As a simple descriptive discrepancy, compare the observed counts with expected counts and calculate a Pearson-style statistic using these probabilities.
The expected counts are \(40(0.075)=3\), \(40(0.150)=6\), and \(40(0.225)=9\). The corresponding variance terms under independent Bernoulli outcomes at those probabilities are \(40p(1-p)\). Thus,
The first two strata are close to their expected counts; the high-risk stratum contributes most of the discrepancy. This plug-in calculation is a useful descriptive summary, not by itself a posterior predictive tail area: it uses posterior mean probabilities and does not integrate over posterior draws. For a full check, simulate probabilities from the posterior, generate replicate counts in each stratum, and compare the observed discrepancy with the resulting replicated discrepancies. A large contribution in one stratum points to where the model should be examined rather than proving why it misses.
Worked Example: Predictive Replication for a Single Event Count
Consider a hypothetical group of 40 external-control patients with a binary endpoint. In a simplified model, the event probability \(p\) has a Beta(1, 1) prior, and 8 events are observed. Conditional on \(p\), outcomes are independent Bernoulli trials. The posterior is Beta(9, 33), since the posterior parameters are \(1+8=9\) and \(1+(40-8)=33\).
The posterior predictive mean of a replicated event count \(Y^{\mathrm{rep}}\) is \(40\) times the posterior mean of \(p\):
Because this replication integrates over uncertainty in \(p\), its variance is larger than the variance of a Binomial distribution evaluated at the posterior mean alone. For a Beta-Binomial predictive distribution with \(n=40\), \(a=9\), and \(b=33\),
The predictive standard deviation is therefore \(\sqrt{12.84}\approx 3.583\) events. The observed count, 8, is close to the predictive mean of 8.571 and is not an obvious discrepancy on this scale. To assess a chosen tail area, one would still compare the observed discrepancy with simulated predictive counts; closeness to the mean alone is not a complete check. This simplified model also omits covariate adjustment, so it illustrates predictive mechanics rather than a full external-control analysis.
Worked Example: Finding Calendar-Time and Endpoint Misfit
Suppose an adjusted model is fitted to a hypothetical external cohort of 300 patients in three calendar periods, with 100 patients per period. For a primary binary endpoint, the observed event counts are 8, 12, and 24. Posterior predictive simulations produce median counts of 10, 13, and 16, respectively, with central 90% predictive intervals of 4–17, 6–21, and 8–25. The observed count in the final period is within its interval, but near the upper edge; the first two periods are also within their intervals.
Now consider a secondary endpoint in the same cohort. Suppose its third-period observed count is 21, while the simulations have a median of 11 and a 90% predictive interval of 5–18. Since 21 exceeds the upper bound, this endpoint-period combination is a clear candidate for investigation. The appropriate conclusion is that the fitted model’s replications rarely reproduce this observed summary under the stated check—not that the endpoint difference is caused by calendar time.
For example, if 80 of 2,000 posterior predictive replications have a discrepancy at least as large as the observed one, the estimated tail area is \(80/2000=0.0400\). This value summarizes the selected discrepancy and direction. Inspecting definitions, ascertainment, model specification, and other relevant data features is necessary before deciding what the signal means. Checking each endpoint and period separately also raises the chance of noticing an unusual result by chance, so the pattern and scientific context matter.
Interpretation, Diagnostics, and Limitations
A good predictive check is designed before looking at a long list of summaries: choose discrepancies tied to plausible failure modes. For example, a risk-stratified count checks whether the model reproduces outcome levels across risk groups; a period-specific count checks temporal patterns; and endpoint-specific summaries can expose differences in event ascertainment or model fit across outcomes. A global count remains useful, but it may average away a severe underprediction in one stratum and overprediction in another.
Several patterns warrant investigation. Replicated values that consistently miss the observed value in one stratum suggest systematic mismatch. Excessively narrow predictive distributions suggest that important variability or parameter uncertainty may be missing. A model can also reproduce counts while missing another feature, such as the distribution of event times. Diagnostics are only as informative as their discrepancy statistics and the fidelity of the replication process.
Posterior predictive tail areas are not ordinary frequentist \(p\)-values with a uniform distribution under the null model. The observed data are used to form the posterior and are then compared with predictions from that posterior; this reuse can make posterior predictive checks conservative for some discrepancies. Thus, a moderate tail area does not certify fit, and a small one is not a calibrated probability that the model is wrong. Graphical comparisons and stratified summaries complement tail areas.
Most importantly, conditional outcome fit is not the same as causal validity. Replicating external outcomes well does not establish that treatment and external-control populations are comparable, that all confounders were measured, or that selection into available records is ignorable. A model may fit observed controls while still extrapolating poorly to trial patients. Conversely, an observed discrepancy can reflect data harmonization or ascertainment differences as well as outcome-model misspecification. The earlier material on outcome-model calibration and selection bias provides context for these distinct concerns.
Compared with checking only fitted values, posterior predictive checks include both posterior parameter uncertainty and outcome variability. Compared with a single global goodness-of-fit statistic, stratified checks are easier to connect to clinical and data-source features, though they can produce many noisy comparisons. They complement, rather than replace, direct inspection of data quality, model diagnostics, sensitivity analyses, or comparisons with randomized control data.
Common Mistake
Correction: An interval covering one summary indicates only that this summary is not unusually extreme under that particular predictive check. It does not establish adequate fit for other risk groups, periods, or endpoints, and it cannot establish exchangeability or rule out unmeasured confounding. Check targeted summaries and interpret any apparent fit within the limits of the model and data.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a conditional posterior predictive check, why are observed covariates usually held fixed?
- A model has a posterior predictive mean of 20 events in a risk stratum. The observed count is 20. Does that alone show adequate fit? Explain.
- For 1,500 replications, 45 discrepancies are at least as large as the observed discrepancy. Calculate the estimated posterior predictive tail area and interpret it.
- Why is a posterior predictive tail area not the probability that the model is true?
- A model reproduces the external-control event counts by calendar period. What important question about using these controls for a causal comparison remains unanswered?
Solutions. (1) Holding covariates fixed checks the outcome model conditional on the observed external population, rather than mixing outcome fit with a separate check of the covariate distribution. (2) No. The mean alone does not show the predictive spread or fit in other strata, periods, or endpoints; compare the observed summary with the full predictive distribution and inspect targeted discrepancies. (3) \(45/1500=0.0300\), or 3.00%. Under the chosen discrepancy and direction, relatively few replications were at least as extreme as observed, which is a diagnostic signal to investigate, not proof of model failure. (4) The tail area compares replicated and observed discrepancies under a posterior predictive distribution; it is not a posterior probability assigned to the model being true. (5) Good outcome fit does not establish that external and trial patients are exchangeable or that confounding and selection bias have been adequately addressed.