From Superiority to Meaningful Benefit
An arm can have a positive treatment effect and still offer too little benefit to matter to patients or clinicians. For example, a treatment might improve a symptom score by a fraction of a point when a larger change is needed to justify its cost, inconvenience, or risks. The question here is not only whether an arm is better than control, but whether its benefit exceeds a clinically meaningful threshold.
Let \(\delta_j\) denote the effect of treatment arm \(j\) relative to its control, measured in the outcome’s units. For the main development below, assume that larger values of \(\delta_j\) are beneficial. A clinically meaningful threshold, denoted by \(\Delta_{\mathrm{MC}}\), is a specified effect size that represents the minimum benefit considered clinically important. The threshold should be justified for the outcome and population, rather than chosen merely because it makes a probability look favorable.
This differs from the posterior probability of superiority, \(P(\delta_j>0\mid\text{data})\). Superiority asks whether the effect is positive; clinically meaningful benefit asks whether it is larger than a positive threshold. The two events are not interchangeable. When \(\Delta_{\mathrm{MC}}>0\), the event \(\delta_j>\Delta_{\mathrm{MC}}\) is a subset of \(\delta_j>0\), so its posterior probability cannot exceed the probability of superiority.
Calculating the Probability From a Normal Posterior
Suppose the posterior distribution for an arm-specific contrast is normal, \(\delta_j\mid\text{data}\sim N(m_j,s_j^2)\), where \(m_j\) is the posterior mean and \(s_j\) is the posterior standard deviation. This form arises, for example, from a normal likelihood for a treatment contrast combined with a normal prior. The probability calculation standardizes the threshold relative to the posterior distribution.
Here \(\Phi\) is the standard normal cumulative distribution function. The final expression is convenient when using a normal probability table or software. If the posterior is not normal, the event remains the same, but its probability must be calculated from that posterior distribution, for example by integrating its density above the threshold or by counting posterior draws that exceed it.
In a shared-control analysis, each arm’s contrast can be correlated with contrasts for other arms because they use the same control data. That dependence does not change the definition above: the marginal posterior distribution for arm \(j\) is sufficient to calculate its individual probability. It does matter for questions about several arms jointly, which are distinct from the arm-specific probability considered here.
Worked Example: A Posterior Probability Just Above One Half
Hypothetical data and assumptions. A continuous outcome is scored in points, and higher values indicate improvement. A treatment-control contrast estimate is 3.0 points with standard error 1.2 points. Assume a normal likelihood for this estimate, \(\hat{\delta}\mid\delta\sim N(\delta,1.2^2)\), and a normal prior \(\delta\sim N(0,2^2)\). For illustration, suppose clinical experts have specified a minimum meaningful benefit of \(\Delta_{\mathrm{MC}}=2.0\) points. The normal likelihood and prior give a normal posterior.
The posterior variance is the reciprocal of the sum of the prior and likelihood precisions:
The posterior mean is the posterior variance times the precision-weighted sum of the prior mean and estimate:
Thus the posterior standard deviation is \(\sqrt{1.0588}=1.029\) points. The probability of exceeding the 2.0-point threshold is
Under the stated model, the posterior probability that the benefit exceeds 2.0 points is about 57.9%. The posterior probability of superiority would instead use zero as the threshold and would be higher. The estimate being close to the clinical threshold does not itself determine the probability; posterior uncertainty also matters.
Threshold Choice and Interpretation
A threshold needs a clear clinical interpretation and the same units as the treatment effect. It may be based on patient or clinician judgment, a validated instrument, or prior clinical evidence. A threshold of 2 points on one scale cannot be transferred automatically to another scale or population. If an outcome is defined so that lower values are better, either define the treatment effect with the beneficial direction reversed or formulate the event with the appropriate inequality. The direction must be settled before interpreting the probability.
A posterior probability is conditional on the data, likelihood, prior, and analysis model. Saying that the probability is 0.8 means that, under those assumptions and given the data, the posterior probability of exceeding the threshold is 80%. It does not mean that there is an 80% chance the model is correct, nor that the treatment will benefit 80% of individuals. It also does not establish that a treatment should be selected: costs, harms, and other consequences are not represented by this probability alone.
Worked Example: Superiority Is Not Meaningful Benefit
Hypothetical result. For a symptom score in which higher values indicate benefit, suppose the posterior distribution for an arm’s treatment effect is \(N(1.1,0.3^2)\) points. The clinically meaningful threshold is 1.5 points. These posterior parameters summarize the hypothetical analysis; assume the normal likelihood and prior used to obtain them are appropriate for the design and data.
The posterior probability of exceeding the meaningful-benefit threshold is
So the posterior probability of a benefit greater than 1.5 points is about 9.12%. By contrast, the probability of simple superiority is
The posterior strongly supports a positive effect, but places little probability above the larger clinical threshold. Reporting only the superiority probability would obscure this distinction.
Comparing Related Summaries
Different summaries answer different questions. A posterior mean describes the center of the posterior distribution; superiority and clinically meaningful benefit are probabilities of particular events. A credible interval describes posterior uncertainty across a range of effect values. None should be substituted for another without explaining the question each addresses.
| Summary | Event or quantity | Question answered |
|---|---|---|
| Posterior mean | \(E(\delta_j\mid\text{data})\) | What is the posterior average effect? |
| Posterior probability of superiority | \(P(\delta_j>0\mid\text{data})\) | How much posterior probability is assigned to a positive effect? |
| Posterior probability of meaningful benefit | \(P(\delta_j>\Delta_{\mathrm{MC}}\mid\text{data})\) | How much posterior probability is assigned to an effect exceeding the clinical threshold? |
| Posterior credible interval | An interval containing a stated amount of posterior probability | Which effect values are supported across a chosen range? |
A frequentist confidence interval can also be compared with a clinically meaningful threshold, but it is not the same probability statement. For example, the proportion of a confidence interval lying above the threshold is not generally the posterior probability of exceeding it. The earlier material on posterior probability that an arm is superior addresses the zero-threshold event; changing the threshold changes the clinical event being assessed.
Worked Example: A More Probable Meaningful Benefit
Hypothetical result. A treatment effect is measured in points, with higher values indicating benefit. Suppose the posterior distribution is \(N(0.8,0.5^2)\) points, and the clinically meaningful threshold is 0.5 points. Assume this posterior distribution is an adequate summary of the specified normal-normal analysis.
The posterior probability of exceeding the clinical threshold is
Thus, under the model, there is about 72.6% posterior probability that the effect exceeds 0.5 points. The probability of superiority is \(\Phi(0.8/0.5)=\Phi(1.6)\approx0.9452\), which is larger because it concerns the less demanding event \(\delta>0\). Both calculations are valid, but they answer different questions.
Diagnostics, Failure Modes, and Limitations
The arithmetic is simple once a posterior distribution and threshold are specified. The more consequential checks concern whether those inputs are appropriate. In a multi-arm trial, arm-specific estimates should use the intended control data and account for the design features relevant to the analysis. If a contrast’s standard error omits dependence induced by a shared control, the posterior uncertainty can be wrong, and so can the tail probability.
Check that the outcome direction and units are consistent across the effect estimate, posterior standard deviation, and threshold. Confirm that the threshold is clinically justified for the population and outcome, and that the prior and likelihood are plausible. For a normal posterior, inspect whether strong skewness, heavy tails, outliers, or sparse data make the normal approximation questionable. In a simulation-based analysis, verify adequate posterior sampling in the tail above the threshold; a small number of draws can give a noisy probability estimate.
A common mistake is to conclude that an arm has a high probability of clinically meaningful benefit because its posterior probability of superiority is high. The correction is to calculate the probability using \(\Delta_{\mathrm{MC}}\), not zero. Another mistake is to treat the threshold as a statistical constant discovered from the data. It is a clinical definition that should be supported and specified independently of the desired result. The probability remains model-dependent, and one arm’s marginal probability alone does not express the probability that several arms simultaneously exceed their thresholds.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- An arm has posterior distribution \(N(1.4,0.4^2)\) points, and meaningful benefit is defined as more than 1.0 point. Calculate the posterior probability of meaningful benefit and distinguish it from superiority.
- If higher outcome values indicate harm rather than benefit, what must be changed before interpreting a probability of clinically meaningful benefit?
- Why is the fraction of a frequentist confidence interval above a clinical threshold not automatically the posterior probability of exceeding that threshold?
- Two arms share a control group. Does dependence between their posterior contrasts prevent calculation of each arm’s individual probability of exceeding its threshold? Explain.
Solutions. (1) Standardize the threshold: \((1.4-1.0)/0.4=1\). Therefore \(P(\delta>1.0\mid\text{data})=\Phi(1)\approx0.8413\), or 84.13%. Superiority uses zero, giving \(\Phi(1.4/0.4)=\Phi(3.5)\approx0.9998\), a different event. (2) Define the effect so its positive direction represents benefit, or write the event using the inequality that corresponds to benefit on the original scale; retain consistent units. (3) A frequentist confidence interval has a repeated-sampling coverage interpretation, whereas the posterior probability is calculated from a probability distribution for the effect conditional on the data and model. The interval’s geometric fraction is not generally that posterior probability. (4) No. Dependence matters for joint events, but each arm’s individual probability is calculated from its marginal posterior distribution.