Why a Conflict Diagnostic Is Not a Model-Selection Rule
When historical information is borrowed for a clinical trial, a diagnostic may indicate that the current trial data are unusual under the historical model. That warning matters: unexamined conflict can make a borrowing analysis overconfident or pull an estimate toward a poorly matched historical target. But a warning is not, by itself, a rule for choosing a different model. Switching to a no-borrowing model only when the observed data look discordant is a data-dependent procedure, with behavior that the analysis must account for.
The central distinction is between assessing compatibility and specifying how information will be borrowed. A diagnostic addresses the first. The borrowing model should be specified before examining the current outcomes, using design knowledge and a clear account of uncertainty. Robustification can be built into that model in advance, so that the analysis is less reliant on historical information when the data provide evidence against it. Sensitivity analysis then shows whether substantive conclusions depend on the borrowing assumptions.
A Normal-Mean Setup for Seeing the Distinction
For the calculations below, suppose current-trial patient outcomes are independent and normally distributed with unknown mean \(\theta\) and known standard deviation \(4\) outcome units. With \(n=4\) patients, the current sample mean \(Y=\bar{x}\) has sampling distribution \(Y\mid\theta\sim N(\theta,4)\), because its variance is \(4^2/4=4\). The sampling standard error is therefore \(2\) units. Assume the historical information is summarized by a borrowing prior \(\theta\sim N(10,4)\), with mean \(10\) units and variance \(4\) square units.
A simple conflict diagnostic compares the observed current mean with the historical prior predictive distribution. Integrating over \(\theta\), the prior predictive distribution of \(Y\) is normal with mean \(10\) and variance \(4+4=8\). A two-sided prior-predictive tail area for an observed mean \(y\) is
This is a tail-area diagnostic under the specified prior and likelihood, not the probability that the historical model is true. Its numerical value depends on the chosen summary, predictive distribution, and tail definition. A small value says the observation is unusual under those assumptions; it does not identify which assumption failed or prescribe what model to use instead.
Worked Example: A Small Conflict Tail Area
Suppose the current mean is \(18\) outcome units. Calculate the prior-predictive diagnostic under the setup above and interpret it.
The predictive standard deviation is \(\sqrt{8}=2.828\) units, so the standardized discrepancy is
The two-sided tail area is \(2\{1-\Phi(2.828)\}\approx 0.00468\), where \(\Phi\) is the standard normal cumulative distribution function. Thus, means at least this far from \(10\) have prior-predictive probability about \(0.468\%\) under this particular model.
That small tail area warrants investigation of comparability, outcome measurement, and the data model. It is not a \(0.468\%\) probability that the historical information is valid, nor does it establish that borrowing should be turned off. The diagnostic has identified a discrepancy relative to a reference distribution; the analyst must still use a prespecified analysis plan.
Robustification Specified Before the Outcomes
One way to make an analysis less brittle is to specify a robust mixture prior in advance. In a simple version, the prior assigns probability \(w\) to an informative historical component and probability \(1-w\) to a broader component:
Here \(m_H\) and \(\tau_H^2\) describe the historical component, \(m_R\) and \(\tau_R^2\) describe the broader component, and \(w\) is fixed in the prior specification. The mixture is part of the model for every possible current data set, not an alternative activated only after a diagnostic crosses a threshold. The posterior probability of each component can change in response to the data; that is Bayesian updating within a prespecified model, rather than a separate diagnostic-driven model switch.
The following calculation makes that distinction concrete. Suppose the prespecified prior places probability \(0.8\) on \(N(10,4)\) and probability \(0.2\) on \(N(10,100)\). The current-data model remains \(Y\mid\theta\sim N(\theta,4)\). For each prior component, the marginal distribution of \(Y\) is normal, with variance equal to the prior variance plus the sampling variance. Bayes’ rule then updates the component probabilities using the marginal likelihoods.
Worked Example: Updating a Prespecified Robust Mixture
Use the observed mean \(y=18\) units. Calculate the posterior probability of the informative component and the posterior mean of \(\theta\).
The marginal distributions of \(Y\) under the informative and broad components are \(N(10,8)\) and \(N(10,104)\), respectively. Their density ratio at \(18\) is
The posterior probability of the informative component is therefore
Within the informative component, normal conjugate updating gives posterior variance \((1/4+1/4)^{-1}=2\) and posterior mean \((10/4+18/4)/(1/4+1/4)=14\) units. Within the broad component, the corresponding variance is \((1/100+1/4)^{-1}=3.846\), and the mean is \((10/100+18/4)/(1/100+1/4)=17.69\) units. The mixture posterior mean is consequently
The observed data have shifted posterior weight away from the informative component, as the specified mixture allows. No rule first tested for conflict and then chose a new model. The prior, likelihood, and updating procedure were all defined before the current mean was observed.
Why Switching After a Diagnostic Can Mislead
Suppose an analyst uses the historical prior when the diagnostic is not small, but discards it when the diagnostic is small. This creates a procedure whose output depends on a data-based branch. Reporting only the selected branch as though it had been fixed in advance ignores the selection step. For example, the no-borrowing estimate will tend to be used precisely in datasets that disagree with the historical center. The resulting collection of estimates and uncertainty summaries cannot be understood by examining either model in isolation.
A conflict diagnostic also cannot tell the analyst which explanation is correct. A mismatch could reflect a genuine population shift, different covariate composition, measurement changes, an outlier, or a misspecified sampling model. Those explanations have different implications for borrowing. Automatically changing models based on one tail area hides rather than resolves that ambiguity.
A formal comparison of models is a different task. It requires a defined set of candidate models and a specified way to compare them, such as a prior over model indicators and posterior model probabilities, or a decision rule with stated consequences. A conflict tail area alone is not such a comparison: it measures extremeness under a reference distribution, rather than posterior support for one borrowing model over another.
Use historical relevance, trial design, and substantive knowledge to define the prior and likelihood before inspecting current outcomes.
If historical mismatch is plausible, include an appropriate robust component or other planned protection as part of the analysis rather than activating it only after a diagnostic warning.
Check whether the flagged discrepancy is sensitive to the chosen summary and whether data quality, covariates, endpoint definitions, or likelihood assumptions offer an explanation.
Compare the prespecified primary analysis with defensible alternatives, such as stronger borrowing and weaker borrowing, and assess whether the clinical interpretation changes.
Sensitivity Analysis and Clinical Interpretation
Sensitivity analysis addresses a question that a conflict diagnostic cannot answer: does the conclusion depend materially on the borrowing assumptions? The comparisons should be defined in clinically interpretable terms, not reduced to a search for whichever model gives the preferred result. The next example uses the same hypothetical data and compares posterior probabilities for a clinically meaningful threshold of \(15\) outcome units.
Worked Example: Comparing Conclusions Across Borrowing Assumptions
For the observation \(Y=18\), calculate the posterior probability that \(\theta>15\) under (1) the informative historical prior alone, (2) the broad prior alone, and (3) the prespecified robust mixture.
With the informative prior alone, the posterior is \(N(14,2)\). Thus
With the broad prior alone, the posterior is \(N(17.69,3.846)\), so
Under the mixture, average the component probabilities using the posterior component weights:
These results show substantial sensitivity to the borrowing assumptions: the posterior probability exceeds the threshold with probability about \(0.240\), \(0.915\), or \(0.737\) under the three analyses. They do not establish which analysis is correct. The primary model and the interpretation of the sensitivity results should be determined by the prespecified plan and clinical context, not by selecting the most favorable posterior probability after seeing the data.
Assumptions, Diagnostics, and Failure Modes
The normal calculations rely on independent, normally distributed current outcomes with the stated known standard deviation, and on the specified normal priors. In applications, assumptions may be more complex, but the same logic applies: a diagnostic is conditional on its model. A predictive discrepancy can be misleading if the likelihood omits important features of the data, or if the historical and current endpoints, populations, or covariate distributions are not aligned.
Useful checks include verifying the analysis population and endpoint definitions, examining data quality and influential observations, and considering whether observed differences are explained by measured covariates. A single prespecified summary may miss conflict in another clinically important feature; conversely, searching many summaries and reporting only the most extreme one can exaggerate the appearance of conflict. Diagnostics should therefore be interpreted with their reference distribution and scope made explicit.
| Approach | Question it addresses | What it does not establish by itself |
|---|---|---|
| Conflict diagnostic | Is an observed feature unusual under a stated prior-predictive model? | Which model is correct, or whether borrowing should be switched off |
| Prespecified robust model | How should borrowing be moderated within the planned analysis? | That historical information is comparable or that the model fits every feature |
| Sensitivity analysis | How dependent is the conclusion on defensible borrowing assumptions? | Which alternative is true merely because it gives a preferred result |
| Formal model comparison | How do specified candidate models compare under an explicit comparison framework? | A conclusion without the assumptions and comparison rule used |
A common mistake is to reason, “The conflict tail area is small, so the historical model has failed; I will report the no-borrowing analysis instead.” The correction is that the tail area measures extremeness under a particular reference model; it is not a posterior probability that the model is false and does not select an alternative. Investigate the discrepancy, retain the prespecified robust analysis, and examine sensitivity to defensible alternatives. If a protocol permits outcome-triggered changes, the change rule and its inferential consequences must be specified and evaluated as part of the analysis plan.
Check Your Understanding
Answer each question using the distinction between a conflict diagnostic and a borrowing decision.
- In the normal example, what does a two-sided prior-predictive tail area of \(0.00468\) mean, and what does it not mean?
- Why is switching to no borrowing only after a small conflict tail area a data-dependent procedure?
- In the robust-mixture example, why is the decrease in informative-component probability not the same as selecting a new model after a diagnostic test?
- Suppose the diagnostic indicates conflict, but the discrepancy is explained by a change in endpoint measurement. What should the analyst investigate before interpreting the diagnostic as evidence about population comparability?
Solutions. (1) It is the probability, under the specified prior-predictive distribution, of a mean at least as far from \(10\) as the observed mean. It is not the probability that the historical model is true or false. (2) The observed data determine which analysis is reported, so the analysis procedure includes an outcome-dependent branch; treating the chosen branch as fixed in advance ignores that selection. (3) The mixture and its prior weight were specified before observing the data; Bayes’ rule updates the component probabilities within that model. (4) Check whether the endpoint definitions and measurement procedures are comparable, and assess whether the likelihood and data processing represent the measurement change appropriately.