Why Check Conflict With Replicated Outcomes?
Historical information can improve precision when historical and current trial outcomes are sufficiently compatible for the intended model. A discrepancy between them may instead signal a changed population, endpoint definition, follow-up process, or chance variation. A posterior predictive check makes that comparison explicit: it asks whether a chosen measure of discrepancy would be unusual for data replicated under a specified model.
The choice of discrepancy matters. A comparison of means may detect a location shift but miss a difference in variability or an unusual tail. A comparison of event rates may miss different patterns across clinically important strata. The check is therefore a diagnostic for a particular feature under a particular model, not a universal test of whether historical borrowing is appropriate.
For an upper-tail check, the posterior predictive tail area is \(P\{T(Y_H^{rep},Y_C^{rep},\theta)\geq T(Y_H,Y_C,\theta)\mid Y_H,Y_C\}\). A lower-tail check reverses the inequality; a two-sided check needs a clearly specified measure of extremeness. The probability is conditional on the fitted model and observed data. It is not the probability that the model is true, nor the probability that the null hypothesis is true.
Choose the Reference Distribution Deliberately
There are two closely related checks that are useful in historical borrowing, but their conditioning differs. In a joint posterior predictive check, parameters are drawn from the posterior given both historical and current data, and both datasets are replicated. This assesses whether the fitted model can reproduce the observed joint pattern. Because the current data help determine the posterior, a substantial discrepancy can pull the fitted parameter toward itself and make the check less sensitive to conflict.
A historical-to-current predictive check instead draws parameters from the posterior given the historical data, then predicts current outcomes. It asks whether the current outcomes are surprising relative to historical information under the specified model. It does not use current data to update the historical predictive distribution. This can make it more directly informative about whether historical outcomes predict current ones, though the predictive distribution still depends on the model and historical-data assumptions.
These checks are not interchangeable. State which data were used to form the posterior, what is replicated, and how the discrepancy is defined. For either approach, the assumptions include a suitable sampling model, correct handling of outcome scales and denominators, and a replication scheme that preserves design features such as sample sizes, stratification, and follow-up. If subjects or outcomes are clustered, the replication must preserve the relevant dependence structure.
Worked Examples
Worked Example: Historical Prediction for a Normal Outcome
Suppose, hypothetically, that a continuous endpoint is normally distributed with common mean \(\theta\) in historical and current patients and known standard deviation \(\sigma=4\) units. Assume independent observations conditional on \(\theta\). The historical trial has \(n_H=25\) and mean \(\bar{x}_H=10\); the current trial has \(n_C=16\) and mean \(\bar{x}_C=13\). Use the prior \(\theta\sim N(10,100)\), where the second parameter is variance. Assess the current mean by predicting it from the historical data.
Given the historical data, the posterior variance and mean are
A replicated current sample mean has variance \(\sigma^2/n_C=16/16=1\) conditional on \(\theta\). Integrating over the historical posterior gives predictive mean \(10\) and predictive variance \(1+0.6360=1.6360\). The observed mean is 3 units above that predictive mean, so its standardized discrepancy is
Under this normal model, the two-sided predictive tail area is approximately \(2\{1-\Phi(2.345)\}=0.0190\). Thus, a current sample mean at least this far from 10 in either direction would be unusual under the historical predictive distribution. This is a diagnostic of tension with the specified historical model; it does not by itself establish why the outcomes differ or determine how much to borrow.
Worked Example: A Beta-Binomial Check With Very Little Current Data
Consider a hypothetical binary endpoint. In the historical data, 4 of 20 patients experienced the event. Assume a common event probability \(p\), independent Bernoulli outcomes conditional on \(p\), and a uniform prior \(p\sim\mathrm{Beta}(1,1)\). The historical posterior is \(\mathrm{Beta}(5,17)\). Suppose the current trial has only one observed patient, who experienced the event. Use the historical posterior to predict that patient's outcome.
For a replicated current outcome \(X^{rep}\), the predictive probability of an event is the posterior mean of \(p\):
Choose the upper-tail discrepancy \(T(X)=X\), which flags an event as more adverse than no event. The observed value is \(T=1\), so \(P\{T(X^{rep})\geq T(1)\mid Y_H\}=P(X^{rep}=1\mid Y_H)=0.2273\). The observed outcome is not rare under this historical predictive distribution. However, one current observation gives very little information about a difference in event rates. A non-small tail area here is not evidence that the historical and current event rates are identical; it may reflect low diagnostic power.
Worked Example: A Multidimensional Discrepancy
Suppose, hypothetically, that investigators prespecify seven independent standardized contrast coordinates comparing current with historical outcomes across clinically relevant features. Under the assumed predictive model, each replicated coordinate is standard normal, so the replicated discrepancy \(Q^{rep}\), the sum of their squared values, has a chi-square distribution with 7 degrees of freedom. The observed coordinates are \(2,1,2,1,2,1,\sqrt{6}\), giving
Because larger values indicate a more extreme overall contrast, use the upper tail. Under the stated calibration,
This is a small tail area: the observed combined discrepancy is unusual if the seven standardized predictive contrasts really follow the assumed independent standard-normal reference. The conclusion depends on that calibration. Correlated coordinates, estimated scales, or a different replication model can change the reference distribution; a chi-square distribution should not be assigned to a discrepancy merely because it is a sum of squares.
Simulation and Interpretation
For complex models, the posterior predictive distribution of a discrepancy is usually evaluated by simulation. The basic calculation is direct: draw a parameter value from the relevant posterior, simulate replicated data conditional on it, compute the replicated discrepancy, and compare it with the observed discrepancy. For an upper-tail check with \(B\) replicates and \(R\) values at least as large as the observed value, estimate the tail area by \(R/B\). A common finite-simulation adjustment is \((R+1)/(B+1)\), which avoids reporting a tail estimate of exactly zero.
Choose a feature that would matter for the borrowing model: for example, a mean shift, a difference in event rates, a variance difference, or a contrast within a prespecified stratum.
State whether parameters are drawn from the posterior given both datasets or from the historical-data posterior alone, and state which outcomes are replicated.
Generate replicated data under the stated sampling model, compute the same discrepancy for every replicate, and estimate the appropriate tail area.
Check Monte Carlo error, alternative clinically meaningful discrepancies, and whether the model reproduces other important outcome features.
Simulation error can matter when the estimated tail area is small. Its approximate Monte Carlo standard error is \(\sqrt{\hat{p}(1-\hat{p})/B}\), where \(\hat{p}=R/B\). For example, if \(\hat{p}=0.01\) from \(B=1{,}000\) replicates, the approximate standard error is \(\sqrt{0.01(0.99)/1000}=0.00315\), which is large relative to 0.01. More simulation draws would be needed to estimate that tail area precisely.
Posterior predictive tail areas are not generally calibrated like classical \(p\)-values under a null hypothesis. In a joint check, the data are used both to fit the posterior and to assess discrepancies, which can make checks conservative or insensitive. A historical-to-current predictive check avoids using current outcomes to fit the historical predictive distribution, but it can still have weak power with small current samples or broad predictive uncertainty. Neither type of tail area is a posterior probability that borrowing is safe or unsafe.
Diagnostics, Alternatives, and Limitations
A useful conflict assessment checks more than one prespecified feature, while avoiding an unstructured search for whichever discrepancy looks most alarming. For a continuous endpoint, compare both center and spread if both are relevant. For a binary endpoint, consider whether the overall event rate is adequate or whether clinically important strata need separate checks. Plotting observed outcomes against replicated distributions can reveal patterns that one scalar tail area hides.
A small tail area can arise from genuine historical-current conflict, but also from an incorrect likelihood, unmodeled heterogeneity, data-quality problems, or a discrepancy definition that overemphasizes an irrelevant feature. A large tail area can reflect compatibility, weak information, a broad predictive distribution, or a check insensitive to the kind of conflict present. Investigate the data-generating and measurement processes before attributing a discrepancy to treatment or population differences.
A historical-to-current predictive tail area emphasizes whether current outcomes were predicted by historical information. A joint posterior predictive check emphasizes whether the fitted model reproduces a feature of the combined data. Conflict evidence from predictive densities is a related approach that compares how well current outcomes are predicted under specified models; it answers a different question from checking replicated discrepancy statistics. The method should match the diagnostic question, and none of these summaries alone decides the borrowing model.
The practical limitation is that conclusions depend on the chosen model, discrepancy, and data scale. A single global statistic may hide subgroup conflict; many exploratory checks can create a misleading impression of extreme evidence. Prespecify central checks where possible, report the direction and clinical units of observed differences alongside tail areas, and examine sensitivity to reasonable model choices. A diagnostic should inform review of the borrowing assumptions, not function as an automatic borrowing threshold.
A Short Justification for the Simulation Tail Estimate
Conditional on the fitted model and observed data, let \(I_b=1\) when replicated discrepancy \(T_b^{rep}\) is at least as large as the observed \(T_{\mathrm{obs}}\), and \(I_b=0\) otherwise. If the \(B\) replicated datasets are generated independently from the specified posterior predictive procedure, then each indicator has expectation equal to the target upper-tail probability \(p\). Consequently,
Thus \(R/B\) estimates the simulated tail probability, and its standard error is approximately \(\sqrt{p(1-p)/B}\), estimated in practice by substituting \(\hat{p}\) for \(p\). This quantifies simulation uncertainty only; it does not account for model misspecification or uncertainty about whether the discrepancy is clinically relevant.
Common Mistake
A common error is to interpret a posterior predictive tail area below a chosen cutoff as proof that historical data must not be borrowed. That reasoning treats a diagnostic probability as an automatic decision rule and ignores the discrepancy's target, power, and model assumptions. The correction is to report what feature was unusual under which predictive model, investigate plausible explanations, and assess implications for borrowing using the broader design and sensitivity analysis. A tail area is evidence about model-data fit for the chosen check, not a decision by itself.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In the normal example, why is the predictive variance of the current sample mean \(1+0.6360\), rather than just \(1\)?
- What does the beta-binomial tail area \(0.2273\) say about the one observed current event, and what does it not establish?
- For seven independent standard-normal predictive contrasts, what is the discrepancy when the coordinates are \(1,1,1,1,1,1,1\), and what is its reference distribution?
- Why can a joint posterior predictive check be less sensitive to conflict than prediction based on historical data alone?
- What additional information is needed before interpreting a small simulated tail area as a reason to reconsider borrowing?
Solutions. (1) The variance includes sampling variation for a future mean, \(16/16=1\), plus uncertainty in \(\theta\) after observing historical data, \(0.6360\); the two sources add. (2) Under the historical posterior predictive distribution, the chance of an event for this one patient is \(5/22=0.2273\), so the event is not unusual by this check. One observation cannot establish equality of historical and current event rates. (3) The sum of squares is \(7\); under the stated independent standard-normal assumption, the reference distribution is \(\chi^2_7\). (4) The current data help determine the posterior in the joint check, potentially shifting fitted parameters toward the current outcomes and making the observed discrepancy less extreme. (5) Identify the discrepancy and its clinical relevance, verify model and replication assumptions, assess Monte Carlo error and other outcome features, and consider plausible data or population differences before changing the borrowing approach.