Two Different Questions About Historical Evidence
When a clinical trial uses historical evidence, two questions are easy to conflate. First: Do the observed historical and current data look compatible under a stated predictive model? Second: Is it reasonable to model the historical and current treatment effects as exchangeable? The first concerns the observed data relative to a reference distribution. The second is a modeling assumption about how effects arise across studies. Neither question answers the other automatically.
For example, similar estimated effects do not establish exchangeability. The trials might differ in population, treatment implementation, endpoint ascertainment, or calendar-time care in ways that make their effects systematically different. Conversely, exchangeable effects need not be identical: a hierarchical model allows genuine between-trial variation, and finite samples can produce estimates that differ substantially.
Compatibility is not a model-free verdict. A comparison of two point estimates, a predictive probability, and a posterior predictive check can emphasize different features of the data. A finding of compatibility means only that the selected discrepancy does not look unusual under the chosen reference. It does not verify that the model is correct or that borrowing is justified.
Exchangeability Is a Modeling Assumption
Let \(\theta_H\) denote a historical treatment effect and \(\theta_C\) a current-trial treatment effect, defined on the same scale and for comparable estimands. Under an exchangeable-effects model, these effects are treated as members of a common distribution, often conditional on measured study characteristics. A simple illustration is
Here \(\mu\) is the mean of the study-effect distribution and \(\tau\) describes between-study variation. If \(\tau=0\), this particular model reduces to a common-effect model. If \(\tau>0\), effects may differ while remaining exchangeable draws from the same distribution. In practice, exchangeability is conditional on the model’s specification: pooling studies without accounting for important differences can make the assumption implausible.
If data summaries \(Y_H\) and \(Y_C\) estimate the corresponding effects with sampling standard errors \(s_H\) and \(s_C\), a simple likelihood model is \(Y_j\mid\theta_j\sim N(\theta_j,s_j^2)\). The normal approximation is most reasonable when the estimators are approximately unbiased, their sampling distributions are close to normal, and their standard errors adequately reflect uncertainty. The two study estimates are treated as conditionally independent when the studies do not share participants or other sources of sampling error. These assumptions must be reconsidered for overlapping data, strongly skewed estimates, or poorly estimated standard errors.
The observed-data comparison is different. One might compare \(Y_C-Y_H\) with its uncertainty, or ask whether \(Y_C\) falls in a predictive distribution informed by historical data. The comparison needs a defined scale and estimand; a risk difference cannot be directly compared with an odds ratio as if they were the same quantity. Any predictive assessment also depends on which sources of variation it includes, including sampling error and, where modeled, between-study heterogeneity.
Worked Examples: What the Two Questions Do and Do Not Show
Worked Example: Estimates That Are Compatible but Do Not Establish Exchangeability
Suppose two hypothetical trials report treatment effects on the 12-week response risk difference, in percentage-point units expressed as proportions. The historical estimate is \(0.16\), with standard error \(0.06248\); the current estimate is \(0.05\), with standard error \(0.05895\). Assume the trials are independent and the estimates are approximately normal. What does their difference indicate?
The estimated difference, historical minus current, is \(0.16-0.05=0.11\), or 11 percentage points. Under the stated assumptions, its standard error is
The standardized difference is \(Z_{\mathrm{diff}}=0.11/0.08590\approx1.280\). A two-sided normal-reference p-value is approximately \(0.200\). The approximate 95% interval for the difference is
These data do not provide strong evidence of a difference by this normal-reference comparison. But the interval also includes differences that could matter clinically: it is compatible with the historical effect being about 5.8 percentage points smaller or about 27.8 points larger than the current effect. Thus, “not statistically distinguishable” is not equivalent to “close enough for the intended borrowing.” Nor does this comparison establish exchangeability; trial design and population assumptions require separate justification.
Worked Example: Exchangeable Effects Can Produce Unequal Estimates
Consider two hypothetical trial effects drawn independently from the same normal distribution, \(\theta_j\sim N(\mu,0.20^2)\). Suppose each trial’s estimated effect has standard error \(0.05\), and the sampling errors are independent. The observed estimates are \(0.10\) and \(0.35\), a difference of \(0.25\). Is that difference necessarily inconsistent with exchangeability?
Under this model, the difference in true effects has variance \(2(0.20^2)\). The difference in sampling errors has variance \(2(0.05^2)\). Adding these independent sources gives the predictive variance for the observed difference:
The observed difference is \(0.25/0.2915\approx0.858\) predictive standard deviations from zero. Under this specified model, the two-sided normal-reference probability of a difference at least this large in absolute value is about \(0.391\). The estimates differ, but the difference is not unusual relative to the model’s predicted variation.
This calculation assumes the common distribution and its heterogeneity scale are appropriate; it does not prove they are. It illustrates why exchangeability does not require identical estimates. If one instead imposed a common-effect model with \(\tau=0\), the same data would be judged against a much narrower reference distribution.
Worked Example: Similar Marginal Effects Can Conceal Study Differences
Suppose treatment effects on response risk differ by a baseline biomarker group. In a hypothetical historical trial, the low-biomarker group makes up half the population and has a risk difference of \(0.30\); the high-biomarker group also makes up half and has a risk difference of \(0.10\). Its marginal effect is \(0.5(0.30)+0.5(0.10)=0.20\).
In a hypothetical current trial, the low-biomarker group makes up one quarter of the population and has a risk difference of \(0.40\); the high-biomarker group makes up three quarters and has a risk difference of \(0.1333\). Its marginal effect is approximately \(0.25(0.40)+0.75(0.1333)=0.20\). The marginal effects match, even though the subgroup effects differ between trials.
Matching marginal estimates therefore do not establish that study effects are exchangeable under a model that ignores biomarker composition and effect modification. The estimand also matters: marginal effects for different patient mixes are not automatically comparable. Investigators should examine the population, treatment, endpoint, and analysis features relevant to the intended effect, rather than treating numerical agreement as evidence that those features are ignorable.
A Short Derivation: Why Exchangeability Does Not Mean Equality
The normal hierarchical model makes the distinction explicit. If \(\theta_H\) and \(\theta_C\) are independent draws from \(N(\mu,\tau^2)\), then each has mean \(\mu\) and variance \(\tau^2\). Subtracting their means gives \(E(\theta_C-\theta_H)=0\). Independence gives
Thus the expected difference is zero, but the difference itself varies whenever \(\tau>0\). With independent normal sampling errors of variances \(s_C^2\) and \(s_H^2\), the observed estimate difference has variance \(2\tau^2+s_C^2+s_H^2\). It is therefore possible to observe unequal estimates even when the model treats the underlying effects symmetrically. This derivation relies on the stated common-distribution and independence assumptions; dependence between studies or a non-normal effect distribution changes the calculation.
Assessing Compatibility Without Overclaiming
A useful analysis makes the reference model and discrepancy explicit. For example, investigators can compare the current estimate with a predictive distribution that accounts for uncertainty in the historical estimate and relevant between-study variation. A posterior predictive check can ask whether a chosen discrepancy, such as the difference in treatment effects, is unusual under the model. This connects to the earlier material on Prior-Data Conflict in Historical Borrowing: a compatibility assessment is conditional on the predictive model, not a general test of whether borrowing is safe.
When interpreting a comparison, consider:
- Estimand alignment. Are the outcome definition, follow-up time, population target, and effect scale comparable?
- Sources of uncertainty. Does the reference include sampling error and plausible study-to-study variation, or does it unrealistically treat the historical estimate as fixed?
- Clinical relevance. Could the uncertainty interval include differences large enough to affect decisions, even if a test does not flag a discrepancy?
- Design and data quality. Could changes in eligibility, treatment delivery, missingness, outcome ascertainment, or standard of care explain apparent similarity or difference?
- Model sensitivity. Do conclusions change materially when reasonable assumptions about heterogeneity or included studies change?
A failure to detect a difference may result from low precision rather than genuine similarity. Conversely, a predictive discrepancy can reflect chance, a poor likelihood approximation, an omitted effect modifier, or a genuinely different treatment effect. Compatibility diagnostics can point to questions that need investigation; they cannot by themselves identify which explanation is correct.
Choosing the Right Interpretation
| Approach | What it assumes or assesses | What it does not establish |
|---|---|---|
| Compare observed estimates | Whether a selected observed difference is large relative to its estimated uncertainty | That the studies are exchangeable or clinically equivalent |
| Common-effect model | One shared underlying treatment effect for all included studies | That this strong equality assumption is appropriate merely because estimates are similar |
| Exchangeable-effects model | Study effects arise symmetrically from a shared distribution, potentially with heterogeneity | That every study effect is equal, or that study labels and design differences are irrelevant |
| Separate-study analysis | Study effects are estimated without assuming a shared distribution | Precise estimates when data are sparse; it may leave substantial uncertainty |
The choice is substantive as well as statistical. A highly precise compatibility check may still not justify exchangeability if treatment versions or target populations differ systematically. A plausible exchangeability model may also allow noticeable data differences, provided those differences are expected under its uncertainty. The appropriate conclusion should state both the observed comparison and the assumptions needed to interpret it.
Common Mistakes and Limitations
- “The difference is not significant, so the effects are the same.” A non-significant comparison does not establish equality. Inspect the uncertainty interval and decide whether it excludes differences that matter clinically.
- “The estimates match, so the studies are exchangeable.” Similar summaries can conceal different populations, effect modifiers, or measurement processes. Exchangeability needs a scientific and design-based argument.
- “Exchangeable means identical.” Exchangeable effects can vary across studies. A common-effect model is a stronger assumption than a hierarchical model with positive heterogeneity.
- “A compatibility check validates the borrowing model.” The check is conditional on the reference model and the discrepancy chosen. It may not detect differences that the model or summary fails to represent.
Compatibility and exchangeability are useful to assess together but should be reported separately. Describe what data were compared and how unusual the discrepancy was under the stated reference. Then explain why the studies are, or are not, plausibly members of a common effect distribution, including any limitations that remain. This avoids turning a limited statistical comparison into a stronger scientific claim.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In one sentence each, distinguish observed-data compatibility from exchangeability.
- Two trials have similar estimated effects, but use different endpoint definitions. What does their numerical agreement establish?
- Under an exchangeable normal-effects model with \(\tau>0\), must two study effects be equal? Explain using the distribution of their difference.
- A compatibility comparison has a wide interval that includes both zero and clinically important differences. What is the appropriate interpretation?
Solutions. (1) Compatibility asks whether observed data look reasonable under a specified predictive reference; exchangeability assumes the study effects have a joint distribution invariant to relabeling, conditional on the modeled features. (2) It establishes only that the reported summaries are numerically similar; differing endpoint definitions may mean they are not estimates of the same estimand, and agreement does not establish exchangeability. (3) No. For independent effects drawn from \(N(\mu,\tau^2)\), their difference has mean zero and variance \(2\tau^2\), so it varies when \(\tau>0\). (4) The data are inconclusive about the direction or size of the difference at that precision. Do not interpret the comparison as proof of equality; assess whether the included clinically important differences would change the decision.