Why a Ranked List Is Not the Whole Result
A trial with several clinically coherent subgroups may produce a posterior estimate for each subgroup’s treatment effect. Sorting those estimates from largest to smallest creates a simple summary, but it hides how uncertain their ordering is. If two subgroup effects are close, small changes across plausible parameter values can reverse their order. A single list does not show that uncertainty, nor does it show whether any difference is large enough to matter clinically.
Posterior rank distributions provide a more complete description. They summarize how often each subgroup occupies each position across the joint posterior distribution. The ranks describe relative ordering among the subgroups included in the comparison; they do not, by themselves, establish that the subgroups have different effects.
The treatment-effect parameter and the direction of “better” must be stated before ranking. For example, a larger response probability may indicate greater benefit, whereas a smaller adverse-event probability may be preferable. Ranking effect estimates on different scales, or using inconsistent directions, has no coherent interpretation. A rank also depends on which subgroups are included: adding or removing a subgroup can change the positions even when the other posterior effects are unchanged.
The draws must represent the joint posterior distribution of the subgroup effects under the fitted model. This matters in hierarchical models, where subgroup estimates can be dependent because they share information through common model parameters. Ranking independently shuffled marginal draws would discard this dependence and can give incorrect rank probabilities. Model assumptions and fit matter as well: a rank distribution summarizes the posterior under the model; it does not diagnose or repair a poorly fitting model.
Calculating Rank Probabilities
For each retained joint posterior draw, rank the subgroup effects using the same rule. Then count, for each subgroup, how many draws assign it each rank. With \(S\) equally weighted retained draws, the Monte Carlo estimate is $$ \widehat{P}(R_k=r\mid \text{data})= \frac{1}{S}\sum_{s=1}^{S} I\!\left(R_k^{(s)}=r\right). $$ The probabilities for a given subgroup sum to 1, apart from rounding. Rank 1 has the largest effect under the chosen direction; rank 2 has the second largest, and so on.
This calculation treats the ranks as summaries of posterior uncertainty, not as new treatment-effect estimates. It is often useful to report the complete rank distribution or a few clearly labeled quantities, such as the probability of being ranked first. The posterior expected rank, calculated as \(\sum_r rP(R_k=r\mid\text{data})\), can be a compact summary, but it can conceal very different distributions. For instance, a subgroup with substantial probability at both rank 1 and the last rank can have the same expected rank as one that is almost always in the middle.
Worked Example: A Three-Subgroup Rank Distribution
Suppose a hypothetical basket trial has three subgroups, A, B, and C. Let \(\theta\) be the posterior response probability in each subgroup, so larger values are better. The following ten equally weighted joint posterior draws are illustrative; they are not observations from a real trial. The last column gives the ordering in that draw, from rank 1 to rank 3.
| Draw | \(\theta_A\) | \(\theta_B\) | \(\theta_C\) | Ordering |
|---|---|---|---|---|
| 1 | 0.30 | 0.28 | 0.25 | A, B, C |
| 2 | 0.22 | 0.26 | 0.30 | C, B, A |
| 3 | 0.27 | 0.31 | 0.29 | B, C, A |
| 4 | 0.32 | 0.24 | 0.28 | A, C, B |
| 5 | 0.25 | 0.27 | 0.26 | B, C, A |
| 6 | 0.29 | 0.30 | 0.23 | B, A, C |
| 7 | 0.24 | 0.23 | 0.28 | C, A, B |
| 8 | 0.31 | 0.29 | 0.30 | A, C, B |
| 9 | 0.26 | 0.25 | 0.27 | C, A, B |
| 10 | 0.28 | 0.32 | 0.24 | B, A, C |
For A, rank 1 occurs in draws 1, 4, and 8, so \(\widehat{P}(R_A=1\mid\text{data})=3/10=0.30\). Rank 2 occurs in draws 6, 7, 9, and 10, giving \(4/10=0.40\); rank 3 occurs in draws 2, 3, and 5, giving \(3/10=0.30\). The same counting gives B probabilities \(0.40, 0.20, 0.40\) for ranks 1, 2, and 3, and C probabilities \(0.30, 0.40, 0.30\).
Thus, the posterior probability that A is best is 0.30 in this small illustrative draw set. The draw-wise ordering in draw 1, for example, is A, B, C, but that one ordering does not characterize the uncertainty: each subgroup appears in every rank across the draws. A ranked list based on posterior means would omit this variation.
Rank Probabilities, Pairwise Comparisons, and Meaningful Differences
A rank probability answers a relative question: how often is a subgroup in a given position among the specified subgroups? A pairwise probability answers a more focused question, such as \(P(\theta_A>\theta_B\mid\text{data})\). These quantities are related but not interchangeable. A subgroup can have a modest probability of being ranked first while having a high probability of exceeding one particular subgroup, especially when more than two subgroups are compared.
Neither quantity says whether the effect difference is clinically important. For that, specify a clinically meaningful threshold \(\delta>0\) on the effect scale and examine a contrast such as \(P(\theta_A-\theta_B>\delta\mid\text{data})\). The threshold should be justified clinically and expressed in the same units as the contrast. It is not a threshold chosen merely to make a preferred subgroup appear better.
Worked Example: Separating Superiority From Clinical Importance
In a separate hypothetical comparison, let \(\theta_A-\theta_B\) be the difference in response probabilities, expressed in percentage points. Suppose ten equally weighted posterior draws of this difference are \(-2, 0, 1, 3, -1, 4, 2, -3, 5,\) and \(1\) percentage points. Assume clinicians have specified a difference greater than 3 percentage points as the threshold for a meaningfully greater response probability.
Six of the ten draws are positive (draws 3, 4, 6, 7, 9, and 10). Therefore, $$ \widehat{P}(\theta_A>\theta_B\mid\text{data})=\frac{6}{10}=0.60. $$ In this illustrative posterior draw set, A has the higher response probability in 60% of draws.
Only two draws exceed 3 percentage points: the differences of 4 and 5. Thus, $$ \widehat{P}(\theta_A-\theta_B>3\text{ percentage points}\mid\text{data}) =\frac{2}{10}=0.20. $$ The probability of any positive difference is not the probability of a clinically meaningful advantage. These proportions are Monte Carlo summaries of a deliberately small draw set; a real analysis would use enough well-mixed posterior draws to estimate them with adequate precision.
Why the Calculation Works and How to Check It
The rank probability is a posterior expectation of an indicator. For a fixed subgroup \(k\) and rank \(r\), define \(Z_k^{(s)}=I(R_k^{(s)}=r)\). Its posterior expectation is \(E(Z_k\mid\text{data})=P(R_k=r\mid\text{data})\), because an indicator equals 1 exactly when the event occurs and 0 otherwise. Averaging the indicators over posterior draws therefore estimates the desired event probability:
This justification assumes the retained draws represent the target joint posterior. If the draws are serially dependent, the simple independent-draw Monte Carlo standard error is not appropriate without adjustment. Check the convergence and mixing of the model’s chains, and assess Monte Carlo error for rank indicators or rank probabilities of interest. A chain that rarely explores changes in ordering may produce unstable rank summaries even when marginal effect summaries seem plausible.
Worked Example: Quantifying Monte Carlo Precision
Suppose a hypothetical analysis retains 10,000 joint posterior draws and subgroup A is ranked first in 6,200 of them. The estimated probability of rank 1 is \(6{,}200/10{,}000=0.620\). Suppose the effective sample size for the rank-1 indicator is estimated as 4,000, reflecting dependence among draws. An approximate Monte Carlo standard error is $$ \sqrt{\frac{0.620(1-0.620)}{4{,}000}} =\sqrt{0.0000589} \approx 0.00768. $$
About 1.96 Monte Carlo standard errors is \(1.96(0.00768)\approx0.0150\). This indicates numerical uncertainty of roughly 1.5 percentage points around the estimate from finite posterior simulation, under the approximation used. It is not a credible interval for the rank probability as an unknown clinical parameter. If the estimated probability changes materially across chains or with longer runs, the rank calculation should not be reported as stable.
Reporting, Diagnostics, and Limitations
A useful report states the ranked quantity, its direction, the included subgroups, and the posterior probabilities that support the summary. For example: “Among the three prespecified subgroups, A had the largest posterior response probability in 30% of draws.” This is more informative than saying simply “A ranked first,” which can sound like a certain conclusion. If a clinical threshold is relevant, report the posterior probability of exceeding that threshold separately from the rank probability.
Check that each subgroup’s rank probabilities sum to approximately 1, and that each draw assigns each subgroup one rank. Inspect convergence and mixing for the effect parameters and rank indicators, not only the overall model output. If tied effect values are possible, specify a consistent rule; for exact ties, average ranks are a common descriptive convention. In many continuous-effect models exact ties have posterior probability zero, but rounded values in a table can appear tied. Do not create ties by ranking rounded values; rank the unrounded draws.
Ranks discard magnitude. A difference of 0.001 and a difference of 0.20 produce the same ordering, despite potentially having very different clinical implications. Ranks also depend on the chosen effect scale, model, subgroup definitions, and set of groups being compared. They cannot establish that effects differ, explain why a difference arose, or substitute for subgroup-specific effect estimates and uncertainty intervals. A posterior rank distribution is a supplement to those quantities, not a replacement for them.
- Overreading rank 1: “A is ranked first, so A is clearly better.” The correction is to report \(P(R_A=1\mid\text{data})\) and the relevant effect contrast with its uncertainty.
- Confusing order with importance: “A exceeds B in most draws, so the advantage matters.” The correction is to evaluate a clinically justified threshold on the difference scale.
- Ignoring the model: “The rank probabilities are objective because they are calculated from draws.” The correction is that they are conditional on the model, subgroup definitions, prior assumptions, and adequate posterior computation.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Four subgroups are ranked using an effect for which larger values indicate more benefit. What does \(P(R_C=1\mid\text{data})=0.25\) mean?
- In 200 joint posterior draws, subgroup D exceeds subgroup E in 130 draws. Estimate \(P(\theta_D>\theta_E\mid\text{data})\). Does this establish a clinically important difference?
- Why should subgroup effects not be ranked using independently shuffled marginal posterior draws when the model induces dependence?
- Suppose a subgroup has posterior rank probabilities 0.45 for rank 1, 0.10 for rank 2, and 0.45 for rank 3. What is its expected rank, and why does that summary not tell the whole story?
Solutions. (1) Under the fitted model and specified set of four subgroups, C has the largest effect in 25% of posterior draws; it is not a statement that C is certainly best. (2) The estimate is \(130/200=0.65\). It is the probability of any positive pairwise difference, not of a clinically important one; a meaningful-difference threshold must be evaluated separately. (3) Independent shuffling destroys the posterior dependence among subgroup effects, changing the joint draws on which their ordering depends and potentially changing rank probabilities. (4) The expected rank is \(1(0.45)+2(0.10)+3(0.45)=2.00\). That average hides substantial probability at both extremes, so the full rank distribution is more informative.