Posterior Probability That an Arm Is Superior

What Does “Superior to Control” Mean?

In a multi-arm trial, a useful probability statement can concern one clearly specified arm: given the observed data and the statistical model, how probable is it that this arm’s treatment effect exceeds the control effect? The calculation is a posterior probability for a contrast, not a comparison of two separate point estimates. It depends on both the posterior means and the uncertainty and covariance of the arm and control effects.

Throughout, define each treatment effect on a scale oriented so that larger values are better. For example, an outcome may be transformed from time to recovery to negative time, so an increase represents faster recovery. Let \(\theta_0\) denote the control effect parameter and \(\theta_j\) the effect parameter for the specified treatment arm \(j\). Define the treatment-minus-control contrast as \(\Delta_j=\theta_j-\theta_0\). Superiority on this scale means \(\Delta_j>0\).

This tutorial assumes that the joint posterior distribution of the treatment and control effects is bivariate normal, or that a normal approximation is being used. Write its posterior means as \(\mu_j\) and \(\mu_0\), posterior variances as \(V_j\) and \(V_0\), and posterior covariance as \(C_{j0}\). These are posterior quantities, not necessarily the prior means, sampling variances, or standard errors. In particular, the shared-control setting can make \(C_{j0}\) nonzero.

From the Joint Posterior to a Probability

A linear contrast of jointly normal random variables is normal. Thus, under the stated posterior model, the contrast \(\Delta_j\) has posterior mean \(\mu_j-\mu_0\) and posterior variance \(V_j+V_0-2C_{j0}\). Provided this variance is positive, standardizing the contrast gives the probability formula below. Let \(\Phi\) be the cumulative distribution function of a standard normal random variable.

The argument of \(\Phi\) is the posterior contrast mean divided by the posterior standard deviation of that contrast. A positive mean gives a probability above 0.5, and a negative mean gives a probability below 0.5. Neither the sign nor the magnitude of the mean alone is enough: uncertainty also matters. If the contrast variance is zero, the posterior contrast is a point mass; the probability is 1 if its fixed value is positive and 0 if it is nonpositive.

The variance formula follows directly from the variance of a difference. For any two random variables with finite variances,

The minus sign before twice the covariance is essential. If the posterior effects move together, some of their uncertainty cancels in the difference. Treating the two effects as independent when they are not can therefore overstate or understate uncertainty in the contrast and distort the probability.

Shared Control and the Contrast Calculation

When several arms share one control, each arm-versus-control contrast includes the same control parameter. The earlier material on shared-control posterior covariance establishes why these contrasts can be dependent. For the probability about one specified arm, use that arm’s own posterior covariance with control in the variance calculation; do not omit the covariance simply because the target is a single contrast.

For example, if the posterior treatment and control effects are independent, then \(C_{j0}=0\), and the contrast variance is the sum of their posterior variances. But independence is an assumption, not a default rule. In a shared-control analysis, posterior dependence may arise from the likelihood and the prior structure. Use the joint posterior covariance supplied by the model or calculate it from the fitted joint posterior.

Reading the Result in Context

The probability is about a specified event under the posterior distribution. Its interpretation requires naming the arm, control, outcome scale, direction of benefit, and conditioning information. For instance: “Given the observed trial data and the stated model and prior, the posterior probability that arm A has a higher mean response than control is 76.7%.”

This statement does not say that arm A is the best of all arms. It also does not quantify the chance of exceeding a clinically important margin: the event here is strictly \(\Delta_j>0\). A positive but very small contrast and a large positive contrast both count as superiority for this calculation. The posterior probability of a clinically meaningful benefit is a different target.

A high probability should not be read as proof of a true effect or as a model-free fact. It can be sensitive to the prior, likelihood, outcome model, missing-data assumptions, and handling of the trial design. In a randomized trial, a causal interpretation additionally relies on the design and analysis supporting the stated treatment-effect estimand. The posterior probability itself is computed conditional on those choices.

Calculation Checks and Common Pitfalls

A reliable calculation follows a short sequence: define the direction of the contrast, obtain the joint posterior summaries, calculate the contrast variance including covariance, standardize the mean contrast, and evaluate the normal CDF. Each step has a distinct failure mode.

A common mistake is to calculate \(\Phi((\mu_j-\mu_0)/\sqrt{V_j+V_0})\) automatically. This silently assumes posterior independence. The correction is to subtract twice the posterior covariance, unless independence is justified. Another mistake is to compare posterior means alone and call the sign of their difference a probability; the mean difference does not contain enough information to determine posterior uncertainty.

A useful diagnostic is to verify that the calculated contrast variance is nonnegative and that its units are squared outcome units. A negative value indicates inconsistent or invalid covariance inputs or arithmetic, since a variance cannot be negative. Also check the CDF calculation: \(z=0\) must give probability 0.5, and increasing \(z\) must increase the probability. A direction error often shows up as a probability on the wrong side of 0.5.

How This Probability Differs From Other Summaries

A posterior probability is not a frequentist p-value. The posterior probability answers how much posterior mass lies in the event \(\Delta_j>0\), conditional on the model, prior, and observed data. A p-value is calculated under a null sampling model and describes the probability of data at least as extreme as observed under that null; it is not the probability that the treatment is superior.

A credible interval for \(\Delta_j\) is another posterior summary. It describes an interval containing a specified proportion of posterior probability under the model, while the probability calculated here gives posterior mass on one side of zero. For a normal posterior, both summaries use the same posterior distribution, but they answer different questions. Neither removes the need to state the outcome direction or assess the model’s adequacy.

This computation is focused on one specified arm-versus-control event. It does not itself establish a decision threshold for advancing an arm, correct or adjust a frequentist error rate, or identify the top-ranked arm. Those are distinct decision and comparison questions. In particular, a posterior probability should not be presented as a clinical recommendation without considering the target, consequences, and assumptions relevant to that decision.