Posterior Probabilities of Treatment Rankings

Why Summarize a Posterior by Rank?

An estimated treatment effect gives a direction and magnitude, but it does not by itself show how uncertain the ordering of several arms is. An arm with the largest posterior mean may have only a modest probability of being best if its posterior distribution overlaps substantially with those of other arms. Posterior ranking probabilities make that uncertainty visible: they describe how often each arm occupies first place, second place, and so on across the posterior distribution.

The ranking depends on the outcome scale and direction. In this tutorial, let \(\theta_i\) denote the treatment effect for arm \(i\), expressed on a common scale where larger values are more beneficial. If the original outcome is one where smaller values are better, the effect should be defined or transformed so that this convention holds. A ranking is a description of the posterior distribution of these effects; it is not a claim that an arm has a clinically important benefit or that it should automatically be selected.

These probabilities refer to uncertainty about the effects conditional on the observed data and the specified Bayesian model. They are not probabilities that the model is true, and they are not frequentist error rates. The likelihood, prior, and posterior must be appropriate for the trial, and the ranking is meaningful only when the arm effects are comparable on a shared estimand and scale.

Computing Rankings From Joint Posterior Draws

A practical general method is to obtain draws from the joint posterior distribution of all arm effects. For each draw, order the effects from largest to smallest and record each arm’s rank. If \(\theta_i^{(m)}\) is the effect for arm \(i\) in draw \(m\), then, when there are no ties, the estimated probability that arm \(i\) ranks first is the fraction of draws in which it has the largest effect.

Here \(M\) is the number of posterior draws and \(I(\cdot)\) equals 1 when the condition is true and 0 otherwise. For rank \(r\), replace the first-place indicator with an indicator that arm \(i\) occupies position \(r\). A key requirement is that each draw represents a joint draw across arms. Drawing each arm separately from its marginal posterior and pairing the values arbitrarily can destroy posterior dependence and produce incorrect ranking probabilities. This matters, for example, when treatment contrasts share a control.

Deriving a Rank Probability From a Normal Contrast

With two arms and no ties, the ranking problem reduces to a single posterior contrast. Let \(\Delta=\theta_A-\theta_B\). Arm A ranks first exactly when \(\Delta>0\). If the posterior for this contrast is normal, the desired probability follows by standardizing the contrast. This calculation also shows why a positive posterior mean alone does not determine how likely A is to rank first: its uncertainty matters.

where \(\Phi\) is the standard normal cumulative distribution function. The equality follows because \((\Delta-\mu_\Delta)/\sigma_\Delta\) has a standard normal posterior distribution, so the event \(\Delta>0\) is equivalent to \((\Delta-\mu_\Delta)/\sigma_\Delta>-\mu_\Delta/\sigma_\Delta\).

For more than two arms, the probability that arm \(i\) ranks first is a joint probability involving all contrasts \(\theta_i-\theta_j\). Those contrasts can be correlated, especially with a shared control. In general, multiplying separate pairwise probabilities is not valid: the events that arm \(i\) beats each competitor are dependent. Joint posterior draws handle this dependence directly. For a specified multivariate normal posterior, the same probability can also be expressed as a multivariate probability over the region where all those contrasts are positive.

Exact Ties, Near-Ties, and Rank Conventions

When posterior effects have a joint density, every pairwise difference has a continuous distribution, and the probability of an exact tie is zero. More generally, it is enough that each pairwise difference has a continuous distribution. Having continuous marginal distributions for each arm, or merely having a nonzero variance for a pairwise difference, is not enough to rule out a tie: a difference can have positive probability mass at zero even when both marginal distributions are continuous.

If the model permits point masses, such as a discrete component that sets two effects equal, an exact tie can have positive posterior probability. The analyst must then state how ties affect the reported rankings. One option is to report the probability of being tied for first; under this convention, two arms tied for first each receive that draw as a first-place event, so the probabilities across arms can sum to more than 1. Another option is to split the tied first-place draw equally among the tied arms, so the first-place probabilities sum to 1. Neither convention is universally right for every decision; the convention must be clear and match the question being answered.

A tie created only by rounding displayed values is not evidence of a posterior point mass. Rank using the unrounded draws when available, and avoid declaring effects equal just because two reported posterior means have the same number of decimal places. Conversely, if exact ties are possible under the model, silently breaking them by arm label produces arbitrary rank probabilities. When close but unequal effects are a concern, report the rank distribution and relevant pairwise posterior probabilities rather than treating “near-tie” as an exact tie. A clinically meaningful difference is a separate question from which effect is numerically largest.

Interpretation, Diagnostics, and Alternatives

Ranking summaries are most informative when reported as a distribution, not only as a list ordered by posterior mean. A first-place probability of 0.45 indicates substantial uncertainty about which arm is best, even if that arm has the largest posterior mean. Likewise, an arm can often rank second while having a small chance of ranking first. The rank probabilities do not communicate the magnitude of differences: an arm may rank first by a negligible amount, or by a clinically important amount.

With posterior simulation, assess whether the chains adequately explore the joint posterior and whether rank indicators have stabilized. Rank indicators can be more variable than posterior means, so adequate convergence of means alone does not guarantee precise rank probabilities. A rough Monte Carlo standard error for an independently sampled first-place indicator with estimated probability \(\hat p\) is \[ \sqrt{\frac{\hat p(1-\hat p)}{M}}. \] For dependent draws, use an effective sample size for the rank indicator or another method that accounts for autocorrelation. Report sufficient precision: many decimal places do not compensate for too few effective draws.

The earlier material on posterior probability that an arm is superior concerns comparisons such as whether one arm exceeds a specified reference. A ranking answers a different question: which position an arm occupies relative to all the other arms. Sorting posterior means is a simple alternative, but it provides only one ordering and hides uncertainty. Pairwise superiority probabilities can clarify individual comparisons, but they do not by themselves give the joint probability of an arm ranking first. Posterior ranking probabilities likewise do not replace a clinically defined benefit threshold or a decision rule that accounts for other considerations.

Common Mistakes and Failure Modes

• Ranking posterior means and reporting the result as certainty. A fixed order of means is not a posterior probability. Count ranks across posterior draws or calculate the appropriate joint posterior probability.
• Multiplying pairwise probabilities to get a first-place probability. Pairwise comparisons involving one arm are generally dependent. Use joint draws or a valid joint probability calculation.
• Assuming continuous marginals eliminate ties. This conclusion requires, for example, that every pairwise difference has a continuous distribution. A point mass at zero can remain even when each marginal is continuous.
• Ignoring shared-control dependence. The arm effects or contrasts may be correlated. Preserve that dependence in joint draws rather than simulating each arm independently.
• Equating rank with importance. First place identifies the largest effect in a draw; it does not imply a clinically meaningful difference. Interpret the magnitude and clinical threshold separately.