Why Prior-Data Conflict Matters
Historical controls can contribute information when a current trial has few participants, but the contribution is trustworthy only to the extent that the historical evidence is informative for the current setting. A current control group will not usually reproduce the historical estimate exactly: even under a sound model, sampling variation creates differences. The important question is whether the observed difference is plausible under the model that connects the historical evidence to the current trial.
A conflict matters because borrowing can pull the posterior estimate of the current control outcome toward the historical experience. Since a treatment effect often compares the treatment outcome with the control outcome, that pull can change the posterior treatment-effect estimate as well as its uncertainty. A narrow posterior is not reassuring if it is narrow partly because it has borrowed information that does not fit the current data.
The predictive distribution must include the uncertainty in a future current-trial sample and, when relevant, uncertainty about the parameter informed by historical data. A raw difference between two estimates does not do this. Nor does conflict by itself identify its cause. It may reflect chance, differences in populations or outcome ascertainment, changes in care, or a model that understates variability. An unusually large discrepancy is evidence that the borrowing model deserves scrutiny; it is not proof that the historical data are invalid or that the model is false.
A Normal-Model Assessment
Consider a continuous endpoint for which larger values are better. Let \(\theta\) be the mean outcome for the current control population. Suppose the historical information is summarized as a normal distribution for \(\theta\), and the current control-arm sample mean has a normal sampling model. For the calculations below, assume the historical summary is \(N(m_H,v_H)\), the current control mean \(Y_C\) has sampling variance \(v_C\) conditional on \(\theta\), and the two sources are independent given \(\theta\). These assumptions include a common, comparable control-mean parameter and correctly specified variances.
Before observing the current control mean, its predictive distribution is obtained by averaging its sampling distribution over the historical distribution for \(\theta\). Equivalently, write \(Y_C=\theta+\varepsilon\), where \(\theta\sim N(m_H,v_H)\) and \(\varepsilon\sim N(0,v_C)\) independently. The sum of independent normal variables is normal, so:
Here \(D_H\) denotes the historical data or the historical summary that has been carried forward. Under the model, \(Z\) has a standard normal predictive distribution. Thus the denominator includes both uncertainty about the historical-informed mean and ordinary sampling variation in the current control mean. If parameters are estimated, variances are uncertain, or the predictive distribution is not normal, this simple standardized discrepancy may need a different reference distribution.
Worked Examples
Worked Example: A Difference Consistent With Sampling Variation
Suppose a historical control analysis gives a mean score of 100 points with standard error 4 points. Represent its information about the current control mean by \(\theta\sim N(100,16)\). In a hypothetical current trial, the control group has mean 110 points with standard error 5 points. Assume independent normal sampling with known standard errors and a common control-mean parameter under the borrowing model.
The predictive standard deviation of the current control mean is \(\sqrt{16+25}=\sqrt{41}=6.403\) points. Therefore:
A discrepancy of 1.562 standard deviations is not especially surprising under the normal predictive model; the approximate two-sided normal tail probability is 0.118. This does not establish that the historical and current settings are exchangeable. It says only that this difference, considered through this model and these variances, is compatible with ordinary predictive variation.
The posterior for the current control mean combines the historical information with the current control likelihood. Its variance is the reciprocal of the sum of the precisions, and its mean is the precision-weighted average:
For illustration, let the current treatment-arm mean be 120 points with standard error 5 points, independently of the control arm. Use a flat prior for the treatment-arm mean, so its posterior is \(N(120,25)\). The treatment effect is \(\Delta=\mu_T-\theta\). The posterior mean of \(\Delta\) is \(120-103.902=16.098\) points, and its variance is \(25+9.756=34.756\) square points. A normal 95% posterior interval is \(16.098\pm1.96\sqrt{34.756}\), approximately 4.543 to 27.653 points. With no historical borrowing and the same flat-prior approach, the current control mean would be 110 and the estimated effect 10 points. The borrowed estimate is higher because the historical mean is lower than the current control mean.
Worked Example: Strong Conflict and a Shifted Effect Estimate
Keep the historical information and standard errors from the previous hypothetical example, but suppose the current control mean is 120 points and the current treatment mean is 130 points. Assume the same independent normal models and that higher scores are better.
The predictive discrepancy for the control mean is:
A result this far from the predictive center is unusual under the specified normal model; its approximate two-sided normal tail probability is 0.0018. This is a warning about the model's use for borrowing in this trial, not a posterior probability that the model is wrong. The data and assumptions should be reviewed before interpreting a borrowed estimate as robust.
If borrowing is nevertheless applied as specified, the current control posterior variance remains \(400/41=9.756\), and its mean is:
The treatment-arm posterior is \(N(130,25)\). Thus the posterior mean treatment effect is \(130-107.805=22.195\) points, with variance \(25+9.756=34.756\) square points. Its normal 95% posterior interval is \(22.195\pm1.96\sqrt{34.756}\), approximately 10.640 to 33.750 points. By contrast, using only the current randomized arms gives an effect estimate of \(130-120=10\) points, with variance \(25+25=50\) and interval \(10\pm1.96\sqrt{50}\), approximately \(-3.859\) to 23.859 points. Borrowing pulls the control estimate toward 100, increasing the estimated benefit and reducing the interval width. The conflict makes clear why that apparent gain in precision should not be accepted without examining the borrowing assumptions.
Worked Example: Conflict for a Binary Control Outcome
Suppose a hypothetical historical control group has 70 responses among 100 participants. Starting with a \(\operatorname{Beta}(1,1)\) prior for the response probability \(p\), the historical posterior is \(\operatorname{Beta}(71,31)\). In a new hypothetical control group, 18 of 40 participants respond. Assume a common response probability, independent binomial sampling, and that the historical posterior is the distribution used to predict the new group.
The beta-binomial predictive distribution for the new response count \(X\) has mean \(n a/(a+b)\) and variance \(n ab(a+b+n)/[(a+b)^2(a+b+1)]\), where \(a=71\), \(b=31\), and \(n=40\). Substitution gives:
The predictive standard deviation is approximately \(\sqrt{11.67}=3.416\) responses. The observed count is 18, so the standardized discrepancy is \((18-27.84)/3.416=-2.882\). This is substantial discordance under the predictive model. Because the count distribution is discrete, skewed, and bounded, this standardized value is a useful summary rather than an exact normal tail calculation.
Updating the beta distribution with the current observations gives \(\operatorname{Beta}(71+18,31+22)=\operatorname{Beta}(89,53)\), with posterior mean \(89/142=0.6268\). With a uniform prior and only the current data, the posterior would instead be \(\operatorname{Beta}(19,23)\), with mean \(19/42=0.4524\). Historical borrowing therefore moves the estimated response probability substantially upward despite the current response rate being \(18/40=0.4500\). The discrepancy is a reason to assess the comparability and model assumptions, not a reason to report the borrowed estimate without qualification.
Why Conflict Changes Treatment-Effect Inference
In a randomized trial, the concurrent treatment and control arms provide the direct within-trial comparison. Historical borrowing changes the control estimate by combining current control information with information carried over from earlier data. When the historical and current control estimates agree reasonably well, the combination can improve precision under the model. When they disagree, the combined posterior can still be mathematically well-defined, but it may place substantial weight on a historical estimate that does not describe the current control experience.
For a treatment effect defined as \(\Delta=\mu_T-\mu_C\), a borrowed estimate of \(\mu_C\) that is lower than the current control estimate increases the estimated \(\Delta\); a higher borrowed control estimate decreases it. The direction depends on the effect definition and endpoint scale. In either case, the posterior uncertainty may shrink because historical information contributes precision. That shrinkage does not remove bias or make the model's assumptions more credible.
The normal derivation above follows directly from the model: write the future observation as the sum of the uncertain historical-informed mean and independent sampling error. Their variances add. This also explains a key diagnostic limitation: if either variance is understated, the predictive distribution is too narrow and conflict will appear more extreme than it should. Conversely, a very broad predictive distribution can make important differences look unremarkable.
Assumptions, Diagnostics, and Alternatives
A conflict assessment is only as credible as the model that produces the predictive distribution. Check that the endpoint definitions, follow-up, eligibility criteria, treatment era, and control care are sufficiently aligned for the intended borrowing model. Confirm that uncertainty in the historical estimate and current sampling error have both been represented. For binary outcomes, account for discreteness and the range of possible counts; for continuous outcomes, inspect whether the normal approximation and variance assumptions are reasonable. Where multiple outcomes or subgroups are examined, an isolated large discrepancy should be interpreted in light of how many comparisons were considered.
A diagnostic can identify that observed data strain a borrowing model, but cannot determine the cause on its own. A discrepancy can arise from chance, systematic differences between sources, data quality, or model misspecification. Nor does a small discrepancy demonstrate that historical and current patients are interchangeable. In particular, an aggregate match can conceal important differences across subgroups or covariates.
The principal alternatives differ in how much historical information they permit to influence the estimate. A concurrent-data-only analysis avoids historical borrowing for the treatment comparison, usually at the cost of greater uncertainty. A fixed historical prior borrows according to its prespecified precision and can be sensitive to conflict. Robust borrowing approaches are designed to reduce the influence of discordant historical information; their construction and tuning require separate attention. These choices should be prespecified where possible, and conclusions should be checked against reasonable alternatives rather than selected after seeing which result is most favorable.
| Approach | How it treats historical evidence | Main caution |
|---|---|---|
| Concurrent-data-only analysis | Uses current randomized arms for the treatment comparison | May be imprecise with a small current control arm |
| Fixed historical borrowing | Uses the prespecified historical distribution and its full stated precision | Can shift and over-precisely estimate effects under conflict |
| Robust borrowing | Allows the influence of historical evidence to diminish when evidence conflicts | Depends on protective choices and modeling assumptions |
Common Mistakes
- Calling any difference conflict. Historical and current estimates are expected to differ by chance. Compare the difference with a predictive distribution that includes both historical uncertainty and current sampling variation.
- Treating a standardized discrepancy as a probability the model is wrong. A value such as \(Z=3.123\) describes how unusual the observation is under the assumed model. It is not \(P(\text{model wrong}\mid D)\).
- Assuming a small discrepancy proves comparability. A limited sample can fail to reveal genuine differences, and aggregate agreement does not establish that all relevant features align.
- Equating a narrower posterior with a more reliable conclusion. Precision depends on the model and the borrowed information; conflict can make a precise estimate misleading.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A historical mean is 50 with variance 9, and the current sample mean has sampling variance 16. What is the predictive standard deviation of the current mean, and why is it not just 4?
- In a normal model, the historical mean is 80, the current mean is 86, and the predictive standard deviation is 3. Calculate the standardized discrepancy and explain what it does and does not establish.
- For \(\Delta=\mu_T-\mu_C\), what happens to the estimated effect if borrowing lowers the control estimate while the treatment estimate stays fixed?
- Does a small predictive discrepancy prove that historical and current patients are interchangeable? Give one reason.
Solutions. (1) The predictive standard deviation is \(\sqrt{9+16}=5\). It includes uncertainty in the historical-informed mean as well as current sampling variation; using only 4 would omit the historical variance. (2) \(Z=(86-80)/3=2\). The observation is two predictive standard deviations above the historical center under the specified model. This describes its relative discrepancy; it does not prove the model is wrong or assign a probability to the model being true. (3) The estimated \(\Delta\) increases because subtracting a lower control estimate from the unchanged treatment estimate produces a larger difference. (4) No. Limited data may not expose differences, and agreement in an overall mean can conceal differences in relevant patient characteristics or outcome measurement.