Why Control Uncertainty Belongs in the Contrast
A treatment response is not interpreted in isolation: its benefit is measured relative to a control response. When the control group is small, its estimated response may be imprecise. A posterior decision that treats the control response as known can therefore give a misleading probability of treatment benefit. The uncertainty matters both for a single treatment-control comparison and, because several arms may use one control group, for comparisons across treatments.
Let \(\theta_j\) be the response parameter for treatment arm \(j\), and let \(\theta_0\) be the response parameter for the control arm. Suppose higher values represent better outcomes. Define the treatment-control contrast as \(\Delta_j=\theta_j-\theta_0\). A decision may require not merely a positive contrast, but a benefit exceeding a prespecified margin \(\delta\), measured in the same units as the response.
If lower responses are better, define the contrast in the direction that makes benefit positive, such as \(\theta_0-\theta_j\). State that direction and the outcome units before interpreting a probability. A posterior probability is conditional on the specified model, prior distributions, and observed data; it is not a probability that the model or a null hypothesis is true.
Posterior Contrasts and Their Uncertainty
The essential input is a joint posterior distribution. If treatment and control parameters are independent in that posterior, the probability of benefit can be obtained from their separate posterior distributions. Independence is an assumption, not a requirement: shared parameters, hierarchical models, or correlated outcomes can induce posterior dependence. The earlier material on correlated outcomes addresses dependence between outcome parameters; here the central issue is that every treatment-control contrast contains the same uncertain control response.
A useful derivation follows from the variance of a difference. For any joint posterior with finite variances,
This identity follows by expanding the variance of a difference. Under posterior independence, the covariance is zero, so both arm variances contribute. In a multi-arm trial with conditionally independent arm parameters and a shared control, two contrasts are nevertheless dependent: for distinct treatments \(j\) and \(k\),
because the common term \(-\theta_0\) appears in both contrasts. Thus, even if treatment parameters are independent of each other and of control, contrasts that share control are positively correlated whenever the control posterior has nonzero variance. This matters when a decision compares several arms jointly.
Worked Example: A Normal Contrast With Limited Control Information
Suppose a continuous improvement score is measured in points, with higher values better. For each arm \(a\), observations are independent and normally distributed with known variance \(1\) point squared. Use independent priors \(\theta_a\sim N(0.5,1)\). Hypothetical data give \(n_T=n_0=4\), with sample means \(\bar{x}_T=0.75\) and \(\bar{x}_0=0.50\) points. Find the posterior probability that treatment improves the mean by more than \(\delta=0.10\) points.
The normal likelihood and normal prior give posterior variance \(1/(1+n_a)\) and posterior mean \((0.5+n_a\bar{x}_a)/(1+n_a)\), since the observation variance is \(1\). Hence
The second parameter of \(N(\mu,\sigma^2)\) is the variance. The independent priors and independent arm observations imply posterior independence between these two parameters. Therefore the contrast is normal, with mean \(0.70-0.50=0.20\) points and variance \(0.20+0.20=0.40\) points squared. Its standard deviation is \(\sqrt{0.40}=0.6325\) points. The desired probability is
Under this model, the posterior probability that the mean improvement exceeds 0.10 points is about 56.3%. For comparison, if the control mean were incorrectly fixed at its posterior mean of 0.50, the contrast variance would be only 0.20, giving \(\Phi((0.20-0.10)/\sqrt{0.20})\approx0.5885\). Fixing the control changes the answer because it removes control uncertainty; it does not make that uncertainty disappear from the data.
Worked Example: Two Treatments Sharing One Control
Consider the same hypothetical normal model and priors, now with three arms. Each arm has four observations, and the sample means are 0.75 points for treatment A, 0.65 points for treatment B, and 0.50 points for control. The posterior means are 0.70, 0.62, and 0.50 points, respectively, and each posterior variance is \(1/5=0.20\) points squared.
The posterior mean contrasts are \(\Delta_A=0.70-0.50=0.20\) points and \(\Delta_B=0.62-0.50=0.12\) points. With independent arm parameters, their covariance is the control posterior variance:
Each contrast has variance \(0.20+0.20=0.40\) points squared, so their posterior correlation is \(0.20/\sqrt{0.40(0.40)}=0.50\). The two treatment contrasts are not independent, despite the independent treatment-arm posteriors: both move together when the uncertain control response changes. Consequently, separate probabilities of benefit do not by themselves determine the probability that both treatments exceed their margins or which treatment has the larger benefit.
Binary Outcomes: Integrating Over the Control Response
For a binary outcome, let \(p_T\) and \(p_0\) be the success probabilities, with higher success probability regarded as beneficial. Under a beta-binomial model, assume independent Bernoulli observations within each arm and independent beta priors \(p_a\sim\operatorname{Beta}(a_a,b_a)\). If arm \(a\) has \(y_a\) successes among \(n_a\) participants, its posterior is \(\operatorname{Beta}(a_a+y_a,b_a+n_a-y_a)\). If the arm priors and likelihoods are independent, the treatment and control probabilities are posterior independent. This model does not account for clustering or other dependencies unless they are explicitly added.
For independent beta posteriors and a benefit margin \(0\leq\delta<1\) on the absolute risk-difference scale, integrating out the control probability gives
where \(f_T\) is the posterior density of \(p_T\) and \(F_0\) is the posterior cumulative distribution function of \(p_0\). To see the result, condition on \(p_T=x\): the required event is \(p_0<x-\delta\), which has posterior probability \(F_0(x-\delta)\). Integrating that conditional probability over treatment values \(x>\delta\) gives the displayed expression. For non-independent posteriors, use the joint posterior probability directly rather than this product-of-marginals formula.
Worked Example: A Clinically Meaningful Risk Difference
In a hypothetical sparse binary-outcome comparison, use independent \(\operatorname{Beta}(1,1)\) priors for treatment and control. Treatment has two successes and no failures; control has one success and no failures. The posteriors are \(p_T\mid D\sim\operatorname{Beta}(3,1)\) and \(p_0\mid D\sim\operatorname{Beta}(2,1)\). Calculate the probability that the treatment success rate exceeds control by more than 0.10.
The treatment density is \(f_T(x)=3x^2\), and the control cumulative distribution function is \(F_0(y)=y^2\) for \(0\leq y\leq1\). Thus
The posterior probability of a risk difference greater than 0.10 is approximately 46.0%. The computation retains uncertainty in both success probabilities. In particular, the small control sample does not justify replacing \(p_0\) with a single observed or estimated value and treating that value as certain.
Diagnostics, Alternatives, and Limits
A practical check is to compare the control posterior uncertainty with the scale of the decision margin and the posterior uncertainty of the treatment response. If the control posterior is broad, a benefit probability calculated after fixing control at a point estimate may differ materially from the joint-posterior result. This discrepancy can be especially important when a decision threshold is near the calculated probability. Sensitivity to the control prior is also worth examining when the control data are sparse: with few observations, the prior can make a substantial contribution to the control posterior.
Several approaches are possible, but they answer different questions:
| Approach | What it does | Main caution |
|---|---|---|
| Joint-posterior contrast | Calculates benefit probability while retaining uncertainty and dependence in treatment and control parameters. | Depends on the chosen likelihood, priors, and dependence structure. |
| Plug-in control estimate | Substitutes a single control estimate into the contrast. | Omits control uncertainty and can misstate a decision probability. |
| Hierarchical or external-control model | May share information across arms or incorporate information beyond the concurrent control group. | Requires defensible exchangeability and model assumptions; borrowed information is not automatically equivalent to concurrent evidence. |
A broader control posterior does not invariably lower a probability of benefit. Depending on the posterior mean, threshold, and distribution shape, adding uncertainty can increase or decrease the probability of exceeding a margin. The reliable conclusion is that the calculation changes to account for that uncertainty—not that it changes in one fixed direction.
The calculation is only as credible as its model. Check that the outcome direction and benefit margin are defined correctly, that the likelihood reflects the outcome type, and that the assumed independence or dependence is appropriate. Very small samples can make posterior conclusions prior-sensitive. A continuous normal model with known variance, for example, may be unsuitable if outcomes are strongly skewed or the variance is not adequately represented. Posterior uncertainty also does not repair bias from non-comparable treatment and control groups or an inappropriate data-collection process.
A common mistake is to calculate \(P(\theta_j>\widehat{\theta}_0+\delta\mid D)\) while treating the estimated control response \(\widehat{\theta}_0\) as fixed, then describe the result as the posterior probability of treatment benefit. That is a different calculation: it conditions on a plug-in value and leaves out uncertainty in \(\theta_0\). The correction is to define the contrast as a random variable under the joint posterior and evaluate \(P(\theta_j-\theta_0>\delta\mid D)\). For multiple treatments sharing control, preserve the joint draws or joint distribution when evaluating combined selection events.
Check Your Understanding
Answer each question, then check the solutions below.
- If the posterior means are 0.60 for treatment and 0.45 for control, is the posterior mean contrast automatically enough to determine the probability of benefit beyond 0.10? Explain.
- Under posterior independence, treatment variance is 0.09 and control variance is 0.16. What is the contrast variance?
- Why can two contrasts sharing the same control be dependent even when treatment parameters are independent?
- In a binary-outcome analysis, what does \(F_0(x-\delta)\) represent in the integral for \(P(p_T-p_0>\delta\mid D)\)?
Solutions: (1) No. The mean contrast is \(0.60-0.45=0.15\), but the probability also depends on the posterior spread and shape of the contrast distribution. (2) Under independence, the variance is \(0.09+0.16=0.25\), so the contrast standard deviation is \(0.50\). (3) Both contrasts contain the same random term \(-\theta_0\); under independent arm parameters their covariance equals \(\operatorname{Var}(\theta_0\mid D)\). (4) It is the posterior probability that the control success probability is less than \(x-\delta\), conditional on treatment probability \(p_T=x\).