Diagnosing Conflict Through Commensurability

Why Compatibility Is a Joint Question

Historical evidence can improve precision in a current clinical trial when historical and current effects are sufficiently comparable. But compatibility is not directly observed: both the difference between the effects and the amount of borrowing are uncertain. A useful diagnosis therefore considers the posterior distribution of the historical-current difference together with the posterior distribution of the commensurability precision. A small estimated difference alone does not establish compatibility, especially if its standard error is large; nor does a low estimated precision prove that the effects differ.

This tutorial develops that joint assessment for a normal commensurate model. The model is deliberately simple enough to derive its relevant posterior distributions, while making clear which assumptions give those calculations their meaning. It concerns compatibility on a specified effect scale and for a specified estimand; it does not by itself establish that the historical and current populations are exchangeable.

Definition: Let \(\theta_H\) and \(\theta_C\) denote the historical and current effects on the same scale and for the same estimand. Define the historical-current effect difference as \(\delta=\theta_C-\theta_H\). The commensurability precision \(\tau\) is the precision in the conditional prior \(\delta\mid\tau\sim N(0,1/\tau)\). Larger \(\tau\) concentrates the prior difference more tightly around zero and represents stronger prior commensurability; smaller \(\tau\) allows larger differences. This is a model parameter, not a direct measure of compatibility.

Model and Posterior Quantities

Suppose the historical and current data yield approximately normal estimates \(y_H\) and \(y_C\), with known sampling variances \(V_H\) and \(V_C\). Assume, conditional on their effects, that the estimates are independent: \(y_H\mid\theta_H\sim N(\theta_H,V_H)\) and \(y_C\mid\theta_C\sim N(\theta_C,V_C)\). Let \(V=V_H+V_C\), and let \(D=y_C-y_H\) be the observed difference in estimates. Use the commensurate prior \(\delta\mid\tau\sim N(0,1/\tau)\), with a specified prior distribution \(\pi(\tau)\) supported on positive values. For this derivation, assign \(\theta_H\) a flat prior, so that the data about the difference are represented by \(D\mid\delta\sim N(\delta,V)\). The posterior for the difference is proper under this normal likelihood and proper prior for \(\delta\).

These assumptions require that both estimates refer to comparable effects, that their stated sampling variances are appropriate, and that their sampling errors are independent. If study estimates are correlated, or the endpoints, populations, or adjustment sets differ in ways that change the estimand, the simple variance \(V_H+V_C\) and the interpretation of \(\delta\) may not be appropriate. The normal approximation also deserves scrutiny for very sparse data or highly skewed estimates.

Conditional on \(\tau\), combining the normal likelihood for \(D\) with the normal prior for \(\delta\) gives the posterior for the effect difference. Integrating over \(\delta\) gives the likelihood of the observed difference for \(\tau\):

\delta\mid D,\tau\sim N\!\left(\frac{D}{1+\tau V},\frac{V}{1+\tau V}\right), \qquad D\mid\tau\sim N\!\left(0,V+\frac{1}{\tau}\right).

Thus the marginal posterior for \(\tau\) is proportional to its prior density or mass multiplied by the normal density of \(D\) with variance \(V+1/\tau\). In a continuous model, the posterior for \(\delta\) is a mixture of the conditional normal distributions over this posterior for \(\tau\). In a discrete model, it is a weighted mixture over the allowed precision values.

\pi(\tau\mid D)\propto \frac{1}{\sqrt{2\pi(V+1/\tau)}} \exp\!\left[-\frac{D^2}{2(V+1/\tau)}\right]\pi(\tau).

Why the Posterior Formulas Hold

The posterior precision for \(\delta\) conditional on \(\tau\) is the sum of its likelihood precision \(1/V\) and prior precision \(\tau\). Completing the square in the product of the two normal densities gives both the posterior variance and the posterior mean:

\begin{aligned} \frac{(D-\delta)^2}{V}+\tau\delta^2 &=\left(\frac{1}{V}+\tau\right) \left(\delta-\frac{D/V}{1/V+\tau}\right)^2+\text{constant in }\delta,\\ \operatorname{Var}(\delta\mid D,\tau) &=\left(\frac{1}{V}+\tau\right)^{-1} =\frac{V}{1+\tau V},\\ E(\delta\mid D,\tau) &=\frac{D/V}{1/V+\tau} =\frac{D}{1+\tau V}. \end{aligned}

For the precision likelihood, \(D=\delta+\varepsilon\), where \(\varepsilon\sim N(0,V)\), while \(\delta\mid\tau\sim N(0,1/\tau)\). Independent normal variables add to a normal variable, so integrating out \(\delta\) gives \(D\mid\tau\sim N(0,V+1/\tau)\). Bayes’ rule then yields the marginal posterior for \(\tau\) above.

The conditional mean shrinks \(D\) toward zero by the factor \(1/(1+\tau V)\). This factor is not itself a conflict probability. It depends on the sampling variance as well as on \(\tau\), and a posterior for \(\tau\) must account for the observed \(D\) and its uncertainty.

Worked Example: A Small Observed Difference with Uncertain Precision

Suppose a hypothetical trial has \(V_H=0.08\) and \(V_C=0.05\), so \(V=0.13\). The estimated current effect exceeds the historical effect by \(D=0.10\) units. For transparent arithmetic, assign equal prior probabilities to \(\tau=0.25\) and \(\tau=4\).

The marginal variance of \(D\) is \(4.13\) at \(\tau=0.25\) and \(0.38\) at \(\tau=4\). The corresponding normal likelihood densities are:

\begin{aligned} f(0.10\mid\tau=0.25) &=\frac{\exp\{-0.10^2/(2(4.13))\}}{\sqrt{2\pi(4.13)}} \approx 0.1961,\\ f(0.10\mid\tau=4) &=\frac{\exp\{-0.10^2/(2(0.38))\}}{\sqrt{2\pi(0.38)}} \approx 0.6387. \end{aligned}

With equal prior probabilities, the posterior probabilities are proportional to these densities. Using the unrounded densities gives:

P(\tau=4\mid D=0.10) =\frac{0.6387}{0.1961+0.6387}\approx 0.7651, \qquad P(\tau=0.25\mid D=0.10)\approx 0.2349.

Conditional on \(\tau=0.25\), the posterior for \(\delta\) has mean \(0.10/(1+0.25(0.13))=0.09685\) and variance \(0.13/(1+0.25(0.13))=0.1259\). Conditional on \(\tau=4\), its mean is \(0.10/(1+4(0.13))=0.06579\) and variance is \(0.13/(1+4(0.13))=0.08553\). The data favor the higher precision among these two choices, but the difference remains uncertain. In particular, \(P(\tau=4\mid D)\approx0.7651\) is not a 0.7651 probability that the effects are exactly equal.

Assessing Clinically Meaningful Differences

A compatibility assessment should ask what size of difference matters scientifically. Let \(c>0\) be a prespecified clinically meaningful difference in the units of the effect. Posterior probabilities such as \(P(|\delta|<c\mid D)\) or \(P(\delta>c\mid D)\) directly address practical questions, while retaining uncertainty in both \(\delta\) and \(\tau\). The choice of \(c\) should follow clinical and scientific considerations, not be selected after inspecting the posterior to obtain a preferred conclusion.

For a discrete prior on \(\tau\), the probability of a directional or bounded difference is a weighted sum of conditional normal probabilities. For example:

P(\delta>c\mid D) =\sum_j P(\tau=\tau_j\mid D) \left[ 1-\Phi\!\left( \frac{c-D/(1+\tau_j V)} {\sqrt{V/(1+\tau_j V)}} \right) \right],

where \(\Phi\) is the standard normal cumulative distribution function. For a continuous prior, the corresponding expression is an integral over \(\pi(\tau\mid D)\). The uncertainty in \(\tau\) is retained by this averaging; substituting a single posterior mean or mode for \(\tau\) generally does not produce the same probability.

Worked Example: A Larger Difference and a Directional Concern

Keep \(V=0.13\) and the equal prior probabilities on \(\tau=0.25\) and \(\tau=4\), but suppose a hypothetical trial observes \(D=1.20\) effect units. The marginal variances remain \(4.13\) and \(0.38\). The likelihood densities are approximately \(0.1649\) and \(0.0973\), respectively, so the posterior probabilities are approximately \(P(\tau=0.25\mid D)=0.6289\) and \(P(\tau=4\mid D)=0.3711\).

The conditional posterior distributions are \(N(1.162,0.1259)\) at \(\tau=0.25\), and \(N(0.7895,0.08553)\) at \(\tau=4\). Suppose a difference greater than \(0.50\) units is clinically important in the positive direction. The conditional probabilities are approximately \(0.969\) and \(0.839\), respectively. Averaging them with the posterior probabilities for \(\tau\) gives:

P(\delta>0.50\mid D) \approx (0.6289)(0.969)+(0.3711)(0.839) \approx 0.921.

Under this model, there is about a 0.921 posterior probability that the current effect exceeds the historical effect by more than \(0.50\) units. The estimate is evidence against assuming near-equality without qualification, but it is not proof of a population shift: the conclusion depends on the model, prior, data, and clinically chosen threshold.

Use the Joint Posterior, Not One Number

The posterior for \(\tau\) describes how the data update the model’s distribution of allowable differences; the posterior for \(\delta\) describes the size and direction of the difference. Their joint distribution matters because \(\delta\mid D,\tau\) changes with \(\tau\). A useful summary may report posterior quantiles for \(\delta\), a clinically meaningful probability such as \(P(|\delta|<c\mid D)\), and posterior probabilities for prespecified regions of \(\tau\). These summaries answer different questions and should be interpreted together.

Worked Example: High Precision Probability Does Not Guarantee a Small Difference

Return to the hypothetical data \(D=0.10\), \(V=0.13\), and the two-point prior from the first example. Suppose clinical experts regard differences within \(0.50\) units as practically small. Conditional on \(\tau=0.25\), \(\delta\) has mean \(0.09685\) and standard deviation \(0.3548\). The probability that it lies between \(-0.50\) and \(0.50\) is approximately \(0.827\). Conditional on \(\tau=4\), the mean is \(0.06579\), the standard deviation is \(0.2925\), and this probability is approximately \(0.905\).

Integrating over the posterior probabilities for \(\tau\), rather than selecting the more probable value, gives:

P(|\delta|<0.50\mid D) \approx (0.2349)(0.827)+(0.7651)(0.905) \approx 0.887.

The posterior probability of the joint event \(\{\tau=4,\ |\delta|<0.50\}\) is instead approximately \((0.7651)(0.905)=0.692\). These are not interchangeable summaries: the first concerns a practically small difference under either precision value, while the second requires both high precision and a practically small difference. The data favor \(\tau=4\) in this two-point prior, yet uncertainty about the effect difference remains visible in the probability calculations.

Diagnostics, Alternatives, and Limitations

Before interpreting these summaries, check whether the assumptions and prior make the diagnosis credible. Useful checks include:

Estimand alignment: verify that historical and current effects use the same endpoint definition, time point, population target, and effect scale. A difference in estimands can masquerade as conflict.
Sampling uncertainty: confirm that the reported variances reflect the trial designs and that dependence between estimates has not been ignored.
Prior sensitivity: compare plausible priors for \(\tau\), including their implications for the scale of \(\delta\). With limited data, the posterior may be strongly influenced by the prior.
Scale and model fit: inspect whether the normal approximation for the estimates is reasonable. For binary or time-to-event endpoints, the analysis must be carried out on the specified model scale, such as the log-odds or log-hazard scale.
Clinical threshold: state how \(c\) was chosen. A posterior probability for a threshold has little clinical meaning if the threshold is not justified in the effect’s units.

One alternative is to compare the observed \(D\) with its sampling standard error \(\sqrt{V}\). This can flag a large discrepancy relative to sampling noise, but it does not incorporate the commensurability prior or uncertainty in \(\tau\). A fixed-precision commensurate model is simpler to summarize, but treats borrowing strength as known. A model that ignores the historical-current difference entirely can conceal conflict by imposing equality. These approaches answer different questions; none replaces checking the joint posterior under the chosen commensurate model.

The model’s conditional prior centers \(\delta\) at zero and uses a symmetric normal distribution. Directional protection or a prior that favors one sign requires a different specification. Likewise, a low posterior precision can arise because the data support a broad range of differences, but it is not a calibrated probability that historical evidence is invalid. The posterior probability of a precision region depends on the prior scale and its support. When multiple historical sources, covariate differences, or data-dependent study selection matter, this two-effect normal model may be inadequate.

Common pitfall: “The posterior probability of high \(\tau\) is 0.77, so there is a 77% chance the effects are compatible.” This conflates a model precision parameter with a clinically defined difference event. Instead, define a meaningful threshold \(c\), calculate \(P(|\delta|<c\mid D)\) using the full posterior mixture, and report a posterior summary for \(\tau\) separately. Neither quantity alone eliminates uncertainty about model assumptions.

Key takeaway: Diagnose commensurability by examining both the posterior precision and the posterior effect difference. Use the observed discrepancy and its sampling uncertainty to update \(\tau\), then average clinically relevant probabilities over the posterior uncertainty in \(\tau\).

Check Your Understanding

Answer each question using the model and notation above. The solutions follow the questions.

If \(V=0.20\), \(D=0\), and \(\tau\) is fixed, what are the conditional posterior mean and variance of \(\delta\)?
In the two-point model, why is the posterior probability of \(\delta>c\) a weighted sum over \(\tau\), rather than the probability calculated at the posterior mean of \(\tau\)?
What does a posterior probability \(P(\tau=4\mid D)=0.80\) establish, and what does it not establish?
Name two assumptions whose failure could make a large observed difference misleading as a diagnosis of effect conflict.

Solutions. (1) The formulas give posterior mean \(0/(1+0.20\tau)=0\) and variance \(0.20/(1+0.20\tau)\); the variance depends on the specified \(\tau\). (2) The conditional distribution of \(\delta\) changes with \(\tau\), so its tail probability must be averaged over the posterior distribution of \(\tau\); evaluating it at one summary value generally differs. (3) It establishes that, under the specified data model and prior, 0.80 posterior probability is assigned to the point \(\tau=4\). It does not establish an 0.80 probability of exact equality or of clinically defined compatibility; those require a probability about \(\delta\) and a specified threshold. (4) Examples include mismatched estimands or effect scales, incorrect sampling variances, dependence between estimates treated as independence, or a poor normal approximation.