Sensitivity Analysis for Residual Confounding in Bayesian Trials

Why Examine Residual Prognostic Differences?

An external control cohort can add useful information when a randomized trial has a small or absent concurrent control arm. Adjustment for measured baseline characteristics does not guarantee that the trial patients and external patients have the same prognosis, however. Unmeasured differences, imperfect measurement, or imperfect adjustment can leave a residual difference in expected control outcomes. A treatment-effect estimate that uses external controls may then partly reflect that mismatch rather than treatment.

Sensitivity analysis makes this concern explicit. Instead of treating the adjusted estimate as unbiased by definition, we specify a range of plausible residual differences and calculate the posterior treatment effect under each value. The exercise does not discover the true residual difference from the observed treatment contrast. It shows how much the conclusion depends on assumptions about that difference.

The sign convention matters. For an outcome where lower values are better, suppose \(\delta>0\) means that, after adjustment, the external controls would have had a lower expected outcome than the trial controls in the absence of treatment. Subtracting those lower external outcomes from trial treatment outcomes tends to make the observed contrast more positive. This is why the model uses \(y=\theta+\delta+\varepsilon\). If a different contrast or outcome coding is used, define the sign of \(\delta\) anew before calculating anything.

Posterior Conditional on a Fixed Sensitivity Value

For a transparent calculation, suppose the adjusted estimate has an approximately normal sampling distribution with known standard error \(s\), and use a normal prior for the treatment effect. Conditional on a specified \(\delta\), assume

Here \(s^2\) is the sampling variance of \(y\), \(\mu_0\) and \(\tau^2\) are the prior mean and variance for \(\theta\), and \(\delta\) is held fixed in this analysis. The likelihood and prior are normal, so the conditional posterior is normal. Completing the square gives

The conditional posterior variance \(V\) does not depend on \(\delta\) in this model: changing the assumed residual difference shifts the estimate, but supplies no new sampling information. The posterior mean moves linearly with \(\delta\), at rate \(-V/s^2\). This also means the posterior credible interval shifts with the mean while retaining the same width.

Deriving the Adjustment

The adjustment follows directly from the normal likelihood. For fixed \(\delta\), the part of the posterior density that depends on \(\theta\) is proportional to

Expanding the two squared terms, the coefficient of \(\theta^2\) in the exponent is \(-\tfrac{1}{2}(1/\tau^2+1/s^2)\), and the coefficient of \(\theta\) is \(\mu_0/\tau^2+(y-\delta)/s^2\). Completing the square therefore yields posterior precision \(1/\tau^2+1/s^2\) and posterior mean equal to that precision’s reciprocal times the coefficient of \(\theta\), which are \(V\) and \(m_\delta\) above.

If the prior is centered at zero, \(\mu_0=0\), then \(m_\delta=(V/s^2)(y-\delta)\). In the limiting case of a very diffuse prior, \(\tau^2\) becomes large, \(V\) approaches \(s^2\), and \(m_\delta\) approaches \(y-\delta\). With a more informative prior, the corrected estimate is also shrunk toward the prior mean.

Posterior Probabilities and Sensitivity Thresholds

For a normal posterior, probabilities about clinically meaningful effects can be evaluated at each fixed \(\delta\). For example, if lower blood pressure is beneficial, then

where \(\Phi\) is the standard normal cumulative distribution function. A threshold analysis solves for the \(\delta\) at which this probability reaches a chosen level. Such a value is not an estimate of the actual residual difference; it identifies the assumption required for a particular posterior conclusion.

Choosing Scenarios and Checking the Analysis

A useful sensitivity analysis chooses values that have an interpretable clinical meaning and are plausible on the outcome scale. The values need not be equally spaced, but should cover the range that decision-makers consider credible, including \(\delta=0\) as the no-residual-difference reference when appropriate. Report the posterior mean, interval, or decision-relevant probability across that range, not only the scenario that gives the most favorable result.

Several checks help identify a misleading analysis:

• Check the sign. Write out how a higher external-control outcome changes the treatment-minus-control contrast. Confirm that the chosen sign of \(\delta\) produces the stated correction.
• Check the scale and units. A difference in mmHg cannot be inserted directly into a log-odds model. Transform or define sensitivity values on the effect scale used by the likelihood.
• Check the uncertainty model. The simple normal calculation assumes an approximately normal estimate and a suitable standard error. If the estimate’s sampling distribution is strongly skewed, bounded, or based on sparse data, the normal likelihood may be a poor approximation.
• Check for double counting. The sensitivity value should represent residual mismatch after the measured-covariate adjustment used to obtain \(y\), not the entire baseline difference before adjustment.
• Check the range against the data and context. A range that is implausibly narrow can create false reassurance; an arbitrary extreme range may obscure which assumptions are genuinely relevant.

A fixed-\(\delta\) analysis is conditional: it treats each assumed value as a separate model. If the posterior conclusion changes within a plausible range, the result is sensitive to residual confounding. If it remains stable across a well-justified range, that is evidence of robustness to this particular concern, not proof that confounding is absent.

How This Analysis Differs From Other Approaches

The adjusted estimate at \(\delta=0\) is the usual reference analysis: it assumes no remaining prognostic mismatch after adjustment. The sensitivity analysis retains that estimator and its sampling uncertainty but asks what follows if this assumption is relaxed by specified amounts. Earlier material on outcome regression, weighting, and standardization concerns how to adjust for measured covariates; varying \(\delta\) addresses what those procedures may leave unresolved.

A fixed-value grid is different from assigning a probability distribution to the residual difference. The grid gives conditional posterior results for explicit scenarios and does not average over uncertainty about \(\delta\). A model that assigns \(\delta\) a distribution requires additional assumptions about that distribution and answers a different question. Neither approach can compensate for an unjustified outcome model, poor overlap, or an implausible range of residual differences. The purpose here is narrower: quantify the effect of specified residual prognostic shifts on the treatment-effect inference.

Common Mistakes and Limitations

• Wrong: “The external controls have a worse prognosis, so I add a positive correction to the effect estimate.” Correction: Start from the defined contrast and derive the sign. Under the convention here, \(\delta\) is the contribution added to \(y\), so the corrected center is based on \(y-\delta\).
• Wrong: “The posterior probability of benefit at \(\delta=0.45\) tells us the probability that the residual difference is 0.45.” Correction: It is conditional on that fixed value; it does not assign probability to the value itself.
• Wrong: “A stable result proves the external controls are unbiased.” Correction: Stability applies only to the specified form and range of prognostic mismatch. Other sources of bias may remain.
• Wrong: “Any chosen \(\delta\) can be used regardless of outcome scale.” Correction: The sensitivity parameter must be expressed on the same scale as the model’s treatment-effect estimate.

The analysis is limited by the chosen model, prior, and scenario range. The normal conjugate calculation is valuable because its shift and uncertainty are easy to inspect, but complex adjusted estimators may require fitting a model directly rather than treating \(y\) as normally distributed with a fixed standard error. A sensitivity analysis clarifies dependence on assumptions; it does not identify residual confounding from the observed data alone.