Why an Ordering Is Not a Difference
A posterior ranking answers an ordering question: which basket has the larger estimated effect, and how often is that ordering present across the posterior distribution? It does not by itself answer the clinical question: how large is the difference, and how often is it large enough to matter? Two baskets can be ordered with high posterior probability even when their effects are nearly indistinguishable in clinical terms.
The distinction matters whenever treatment effects are reported across baskets or subgroups. A claim that one basket “ranks first” can sound like evidence that treatment works better there. That conclusion requires more than an ordering: it requires a defined comparison, an effect scale, uncertainty for the comparison, and a clinical standard for judging its size. The earlier material on ranking subgroups without overclaiming motivates the caution; here the focus is the pairwise contrast and its uncertainty.
The effect scale is part of the claim. If \(\theta_j\) is a response probability, then \(D_{ij}\) is a risk difference in percentage points or probability units. If \(\theta_j\) is a log odds ratio, the contrast is a difference on the log-odds-ratio scale; it is not a risk difference. The threshold \(\delta\) must match the chosen scale and be justified clinically. A threshold should not be selected after looking at which contrast appears most favorable.
Posterior Contrasts Preserve Uncertainty and Dependence
In a Bayesian analysis, each basket effect has a posterior distribution conditional on the model, prior, and observed data. The pairwise contrast also has a posterior distribution. Its uncertainty depends not only on uncertainty in the two effects, but also on their posterior dependence. This is especially important in hierarchical models, where basket estimates may be related through shared model structure.
These identities follow directly from subtracting random variables: the expectation of a difference is the difference of expectations, and the variance of a difference includes the negative twice-covariance term. Ignoring that term can either overstate or understate uncertainty. In particular, subtracting two marginal interval endpoints does not generally give a credible interval for the difference.
If the joint posterior is approximately normal, the contrast may also be approximately normal. Then its posterior mean and variance can be used to calculate probabilities and an interval. But normality is not required for the definition of a contrast: when available, paired draws from the joint posterior provide a direct route. For each draw \(m\), calculate \(D_{ij}^{(m)}=\theta_i^{(m)}-\theta_j^{(m)}\). The collection of these differences represents the posterior distribution of the contrast while retaining dependence.
Worked Contrasts
Worked Example: A Likely Ranking but Uncertain Clinical Difference
Suppose a hypothetical trial’s posterior summaries on a response-probability scale are \(E(\theta_A\mid\text{data})=0.30\) and \(E(\theta_B\mid\text{data})=0.24\), with posterior standard deviations \(0.06\) for each effect and posterior correlation \(0.50\). Assume the joint posterior is adequately approximated by a bivariate normal distribution. Consider a clinically meaningful difference of \(\delta=0.10\), or 10 percentage points.
The posterior mean contrast, in response-probability units, is \(0.30-0.24=0.06\), a 6-percentage-point advantage for A. The contrast variance is
Thus the contrast standard deviation is \(\sqrt{0.0036}=0.06\). Under the normal approximation, the posterior probability that A has the larger effect is \(P(D_{AB}>0\mid\text{data})=\Phi(0.06/0.06)=\Phi(1)\approx0.8413\). The posterior probability that A's advantage exceeds 10 percentage points is \(P(D_{AB}>0.10\mid\text{data})=\Phi((0.06-0.10)/0.06)=\Phi(-0.6667)\approx0.2525\).
A 95% equal-tailed posterior credible interval is approximately \(0.06\pm1.96(0.06)=[-0.0576,\ 0.1776]\), or −5.76 to 17.76 percentage points. The posterior favors A's ordering, but the interval includes zero and the probability of an advantage exceeding the stated clinical threshold is only about 25%. The ranking alone would conceal this uncertainty.
Worked Example: A High Probability of Superiority Is Not the Whole Claim
Consider another hypothetical pair of baskets on a risk-difference scale. Let the posterior means be 0.42 and 0.27, the posterior standard deviations be 0.07 and 0.06, and the posterior correlation be 0.25. Assume a bivariate normal posterior approximation and use \(\delta=0.10\) as the clinical threshold.
The mean contrast is \(0.42-0.27=0.15\), or 15 percentage points. Its variance and standard deviation are
Therefore \(P(D>0\mid\text{data})=\Phi(0.15/0.08)=\Phi(1.875)\approx0.9696\). The posterior probability of exceeding the 10-percentage-point threshold is \(P(D>0.10\mid\text{data})=\Phi((0.15-0.10)/0.08)=\Phi(0.625)\approx0.7340\). The approximate 95% credible interval is \(0.15\pm1.96(0.08)=[-0.0068,\ 0.3068]\), or about −0.68 to 30.68 percentage points.
The posterior strongly favors the ordering, and the estimated difference exceeds the threshold. Nevertheless, uncertainty remains substantial: the posterior probability of an advantage greater than 10 percentage points is about 73%, not 97%. A report should distinguish these statements rather than use the ranking probability as a substitute for the clinically thresholded probability.
Worked Example: Reading a Ranking Among Several Baskets
Suppose a hypothetical analysis produces 10,000 paired draws from the joint posterior for three basket effects, all measured as response probabilities. Basket A has the largest effect in 6,200 draws. In 1,800 draws, its effect exceeds basket C's by more than 0.10. These are Monte Carlo summaries from the same joint draws, and the clinical threshold is 10 percentage points.
The estimated posterior probability that A ranks first is \(6{,}200/10{,}000=0.620\), or 62%. The estimated posterior probability that A exceeds C by more than the clinical threshold is \(1{,}800/10{,}000=0.180\), or 18%. For a posterior probability estimated as a proportion \(\hat q\) from \(M\) draws, the approximate Monte Carlo standard error is \(\sqrt{\hat q(1-\hat q)/M}\). For the 18% estimate this is \(\sqrt{0.18(0.82)/10{,}000}\approx0.00384\), about 0.38 percentage points.
Thus A is the top-ranked basket in a majority of draws, but a clinically important advantage over C appears in fewer than one-fifth of draws. The ranking is a relative ordering across all three baskets; the threshold probability is a direct pairwise statement about A and C. They answer different questions and should be reported separately.
How to Make and Report the Comparison
A defensible subgroup comparison starts by stating the estimand and direction. Specify what \(\theta_j\) measures, its units, and whether larger values mean more benefit. Then identify the pair of baskets and form \(D_{ij}\) on that scale. If the clinical question concerns a meaningful advantage, define \(\delta\) in advance and report the posterior probability that the contrast exceeds it, alongside the contrast estimate and uncertainty.
Name the two baskets, the effect scale, the direction of benefit, and the clinical threshold in matching units.
Use the posterior dependence: subtract the effects within each joint posterior draw, or use the covariance formula when its assumptions are appropriate.
Report the posterior contrast, a credible interval, \(P(D_{ij}>0\mid\text{data})\), and, when relevant, \(P(D_{ij}>\delta\mid\text{data})\).
Describe what the posterior supports and what remains uncertain; do not convert an ordering into a claim of clinically important difference without evidence about magnitude.
When many baskets are compared, pairwise probabilities are not independent pieces of evidence: they share basket effects and are derived from the same joint posterior. Select comparisons that answer the clinical question, and make clear whether they were prespecified or exploratory. A list of ranks can be useful descriptively, but it should not be presented as a set of established treatment differences.
Assumptions, Diagnostics, and Alternatives
The calculations are meaningful only if the basket effects refer to comparable estimands and the contrast is on a clinically interpretable scale. The model must adequately describe the data and the relevant dependence among basket effects; posterior summaries are conditional on the model and prior, not free of those choices. A normal approximation to a contrast additionally requires that its posterior shape be sufficiently close to normal for the intended approximation. If this is doubtful, calculate summaries directly from joint posterior draws and inspect the contrast distribution for skewness, heavy tails, or multiple modes.
Diagnostics should include checking the computation of the model’s posterior draws and the contrast draws, verifying that every subtraction uses matched draws from the same joint iteration, and confirming that the threshold uses the same units as the contrast. For simulation-based summaries, Monte Carlo error should be small enough for the precision claimed. Also check whether the effect scale makes the direction and threshold intelligible: a difference on a transformed scale may not translate into a constant difference in response probabilities.
The main alternative to pairwise contrasts is to report rankings or rank probabilities. Rankings efficiently describe relative position, but omit distance. Marginal basket summaries with separate credible intervals are another option, but visual overlap or non-overlap of those intervals is not a direct calculation of the posterior contrast. The contrast distribution directly answers the pairwise question. A decision analysis can go further by incorporating consequences of benefits, harms, and uncertainty, but a ranking alone does not provide those consequences.
A posterior probability is not a guarantee that a decision is correct, nor does a high probability of a positive contrast establish that the difference matters to patients. Conversely, a low probability of exceeding a demanding threshold does not prove exact equality. The posterior can support a statement about the range and probability of differences while leaving the clinical decision to the relevant decision criteria.
Common Pitfall
A related computational mistake is to treat marginal basket uncertainties as if they were independent. The variance of a difference includes the covariance term. Use the joint posterior draws when possible; otherwise state and justify the covariance assumptions used in the calculation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A posterior analysis gives \(P(D_{AB}>0\mid\text{data})=0.95\). Does this alone establish that A is clinically better than B? What additional quantity would directly address a 5-percentage-point threshold?
- For basket effects with posterior standard deviations 0.04 and 0.05 and posterior correlation 0.40, calculate the posterior variance of their difference.
- Why is subtracting the lower endpoint of one basket’s marginal credible interval from the upper endpoint of another basket’s interval not generally a 95% credible interval for the contrast?
- In 5,000 joint posterior draws, A exceeds B by more than a prespecified clinical threshold in 1,250 draws. Estimate the posterior probability of exceeding the threshold and explain its meaning.
Solutions. (1) No. The probability describes the chance, conditional on the model and data, that the contrast is positive, not whether its magnitude is clinically important. Calculate \(P(D_{AB}>0.05\mid\text{data})\) on the response-probability scale. (2) The variance is \(0.04^2+0.05^2-2(0.40)(0.04)(0.05)=0.0016+0.0025-0.0016=0.0025\). (3) The marginal intervals do not encode the joint posterior dependence, and endpoint subtraction does not in general preserve the desired posterior probability. Form the contrast from joint draws or derive its distribution using the joint covariance. (4) The estimate is \(1{,}250/5{,}000=0.25\). About 25% of the posterior draws have an advantage larger than the threshold; this is a posterior probability for that thresholded contrast, not a probability that the null hypothesis is true.