Posterior Probabilities Of Basket-Specific Benefit

What a Basket-Specific Benefit Probability Answers

In a basket trial, each basket represents a clinically defined subgroup, and a hierarchical model may allow information to be shared across baskets. That borrowing can affect each basket’s posterior distribution, but it does not remove the need to ask a separate clinical question for each subgroup: how much posterior support is there that this basket’s treatment effect exceeds a meaningful benefit threshold?

The answer is a posterior probability about a basket-specific treatment-effect parameter. It is not the probability that a basket is “the best,” and it does not by itself establish that effects differ between baskets. The quantity depends on the effect scale, the direction chosen to represent benefit, the threshold, the observed data, and the fitted model.

Definition: Let $\theta_j$ denote the treatment effect for basket $j$, defined on a specified clinical scale and oriented so that larger values represent greater benefit. Let $\delta_j$ be the clinically meaningful benefit threshold on that same scale. The basket-specific posterior probability of benefit beyond threshold is $$ P(\theta_j>\delta_j\mid y), $$ where $y$ denotes the observed trial data. If a common threshold is appropriate, write $\delta$ in place of $\delta_j$.

The event must match the clinical question. For example, if $\theta_j$ is an improvement in a symptom score measured in points, then $\theta_j>3$ asks whether the improvement exceeds 3 points. If an effect is initially defined so that smaller values mean greater benefit, first state that direction and use the corresponding event, such as $P(\theta_j<\delta_j\mid y)$, or redefine the parameter so that benefit is positive.

From Posterior Draws to a Probability

In practice, posterior distributions are often represented by draws from a simulation algorithm. For basket $j$, let $\theta_j^{(m)}$ be draw $m$, for $m=1,\ldots,M$. Each draw is checked against the prespecified threshold. The fraction of draws that exceed it estimates the posterior probability:

\widehat{P}(\theta_j>\delta_j\mid y) = \frac{1}{M}\sum_{m=1}^{M} I\!\left(\theta_j^{(m)}>\delta_j\right),

Here $I(A)$ equals 1 when event $A$ is true and 0 otherwise. This calculation preserves subgroup identity: the indicator is evaluated using the draws for basket $j$, not by replacing its effect with an overall average or a pooled effect.

The model and computation need to be credible before the probability is useful. In particular, the basket definitions and endpoint must be clinically coherent; the effect scale and threshold must be specified consistently; and the likelihood, hierarchical structure, and prior assumptions must be suitable for the data and scientific setting. The posterior computation must also have converged adequately. As in the earlier material on posterior predictive checks and shrinkage, a numerical output does not rescue a poorly fitting model or an unsuitable exchangeability assumption.

Worked Example: Counting Exceedances in Two Baskets

Suppose a hypothetical analysis defines $\theta_j$ as the absolute increase in response probability for treatment versus control, so positive values favor treatment. The clinically meaningful threshold is $\delta=0.10$, an improvement of 10 percentage points. Assume for this calculation that the posterior draws are independent. In basket C, 354 of 400 draws exceed 0.10; in basket D, 247 of 400 do.

For basket C, the estimated posterior probability is $$ \widehat{P}(\theta_C>0.10\mid y)=\frac{354}{400}=0.885. $$ Under the independent-draw assumption, the Monte Carlo standard error is approximately $$ \sqrt{\frac{0.885(1-0.885)}{400}} = \sqrt{0.0002544375} \approx 0.01595. $$ Thus, the estimated posterior probability for basket C is 88.5%, with Monte Carlo error of about 1.60 percentage points. This error describes numerical uncertainty in estimating the posterior probability from a finite simulation, not uncertainty about the treatment effect itself.

For basket D, $$ \widehat{P}(\theta_D>0.10\mid y)=\frac{247}{400}=0.6175, $$ and $$ \sqrt{\frac{0.6175(1-0.6175)}{400}} = \sqrt{0.000590484375} \approx 0.02430. $$ The estimated posterior probability is 61.75%, with Monte Carlo standard error about 2.43 percentage points. Under the fitted model, the posterior mass above the 10-percentage-point threshold is greater for basket C than for basket D. These marginal probabilities do not establish that the underlying treatment effects differ between the baskets.

Why the Draw-Based Estimate Works

The calculation has a simple justification. Define $Z_j=I(\theta_j>\delta_j)$. Since $Z_j$ is 1 when the effect exceeds the threshold and 0 otherwise, its posterior expectation is exactly the probability of that event:

E(Z_j\mid y) = 1\cdot P(\theta_j>\delta_j\mid y) + 0\cdot P(\theta_j\leq\delta_j\mid y) = P(\theta_j>\delta_j\mid y).

A posterior sample of the indicators $I(\theta_j^{(m)}>\delta_j)$ therefore estimates this expectation by its sample mean. If the draws are independent, the indicators are Bernoulli with success probability $p_j=P(\theta_j>\delta_j\mid y)$, so the variance of their sample mean is $p_j(1-p_j)/M$. Replacing the unknown $p_j$ with its estimate gives the familiar Monte Carlo standard error:

\widehat{SE}_{MC} \approx \sqrt{\frac{\hat p_j(1-\hat p_j)}{M}}, \qquad \hat p_j=\widehat{P}(\theta_j>\delta_j\mid y).

This standard error concerns simulation precision only. It should not be confused with the posterior standard deviation of $\theta_j$, which describes posterior uncertainty about the effect. Nor is it a standard error for a treatment-effect estimate in the frequentist sampling sense.

Worked Example: Calculating a Tail Probability From a Posterior Summary

Suppose a hypothetical basket-specific posterior distribution for improvement on a symptom score is approximately $N(4.2,2.0^2)$, in points. Larger improvements are better, and clinicians specify 3 points as the meaningful-benefit threshold. This is an approximation to a posterior distribution, not a claim that every basket model has a normal posterior.

Standardizing the threshold gives $$ P(\theta_j>3\mid y) = P\!\left(\frac{\theta_j-4.2}{2.0}>\frac{3-4.2}{2.0}\,\middle|\,y\right) = P(Z>-0.60) = \Phi(0.60) \approx 0.7257, $$ where $Z\sim N(0,1)$ and $\Phi$ is its cumulative distribution function. The posterior probability that the improvement exceeds 3 points is therefore about 72.6%.

In context, this is the model-based probability, conditional on the observed data and assumptions, that the basket’s average treatment effect on the defined score exceeds 3 points. It does not mean there is a 72.6% probability that a particular patient will improve by more than 3 points; that would be a predictive question about an individual outcome.

Thresholds, Decision Cutoffs, and Reporting

The clinical threshold $\delta_j$ and a decision cutoff for the posterior probability are different quantities. The threshold defines what counts as a clinically meaningful effect. A probability cutoff, say $q$, might be used in a decision rule such as “consider further study if $P(\theta_j>\delta_j\mid y)>q$.” The choice of $q$ depends on the consequences of acting or not acting and should not be presented as a universal statistical constant.

A probability near 0.5 indicates that the posterior distribution places similar mass on effects above and below the threshold. A probability near 1 indicates that most posterior mass is above it; a probability near 0 indicates that little is above it. These statements are conditional on the model and data. Report the threshold, units, effect direction, posterior probability, and enough information about the model to make the result interpretable.

Worked Example: Reporting Basket-Specific Results Without Pooling Them

Suppose a hypothetical trial uses improvement in a laboratory measure as its effect scale, in units of the measure, with $\delta=5$ units. A converged posterior simulation produces the following estimated probabilities: basket A, 0.94; basket B, 0.78; and basket C, 0.43.

A suitable report is: “Under the fitted hierarchical model, the posterior probabilities that the treatment effect exceeds a 5-unit improvement are 0.94 in basket A, 0.78 in basket B, and 0.43 in basket C.” This retains each basket’s own effect parameter and makes the clinical threshold explicit.

It would be incorrect to average these probabilities and report a single value as the probability of benefit for every basket. It would also be incorrect to infer from the difference between 0.94 and 0.43 alone that the effects in A and C are credibly different. The probabilities answer separate threshold questions; a direct comparison of effects would require a posterior quantity for their contrast.

Diagnostics and Common Failure Modes

A posterior exceedance probability can be numerically precise yet scientifically misleading. Before interpreting it, check both the event being estimated and the computation used to estimate it.

Threshold or direction mismatch. Confirm that positive values really represent benefit and that the threshold is in the same units as $\theta_j$. A threshold of 5 points cannot be compared directly with a log-odds effect.
Insufficient simulation precision. Inspect the effective sample size for the threshold indicator, not only the total number of saved iterations. If Monte Carlo error is material to the interpretation, run longer simulations or improve sampling and report uncertainty in the estimate.
Correlated MCMC draws. The independent-draw formula $\sqrt{\hat p_j(1-\hat p_j)/M}$ is not appropriate when draws are autocorrelated. Use an effective sample size or a suitable Monte Carlo error estimate based on the indicator sequence. Check chain mixing and convergence as well; a large nominal draw count does not ensure reliable tail estimation.
Unstable estimates near a decision cutoff. If the estimated probability is close to a chosen $q$, Monte Carlo error and model sensitivity may change whether it crosses that cutoff. Avoid presenting a borderline numerical comparison as decisive.
Posterior mass near the threshold. A probability alone does not show how far effects lie beyond the threshold or how broad the posterior is. A basket may have appreciable probability of exceeding the threshold while still having substantial probability of small or negative effects. Pair the probability with an appropriate summary of the basket-specific posterior.
Model dependence hidden by a single number. Hierarchical borrowing, priors, and the exchangeability structure can affect each basket’s posterior probability. Use diagnostics and sensitivity analyses appropriate to the model rather than treating the reported probability as assumption-free.

The main alternative summaries answer different questions. A posterior mean or median describes a location of the effect distribution, while a credible interval describes a range of posterior values. The exceedance probability directly addresses the chosen threshold event and can be easier to connect to a clinical criterion. A frequentist $p$-value is not the probability, given the data, that the effect exceeds a clinical threshold; it is based on a different inferential framework and null hypothesis. None of these summaries alone replaces the others in every setting.

Common Mistake

Common mistake: “The posterior probability is 0.90, so there is a 90% chance the treatment works in this basket and the basket should be declared successful.”

Correction: State the event precisely: for example, “Given the observed data and fitted model, the posterior probability that the basket-specific effect exceeds the prespecified 5-unit threshold is 0.90.” Whether that probability supports a decision depends on the decision rule, clinical consequences, model adequacy, and other relevant evidence. The probability is not a guarantee of benefit for individual patients, and it does not by itself establish differences among baskets.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For basket E, 180 of 250 independent posterior draws exceed a threshold of 2 units. What is the estimated posterior probability, and what event does it describe?
A beneficial effect is defined as a decrease, and the threshold is a decrease of at least 4 units. Express the event using an effect parameter defined as treatment minus control.
Why can the independent-draw Monte Carlo standard error formula be misleading for autocorrelated MCMC draws?
If basket A has $P(\theta_A>\delta\mid y)=0.90$ and basket B has $P(\theta_B>\delta\mid y)=0.60$, does this establish that $\theta_A>\theta_B$? Explain.
What is the difference between the clinical benefit threshold and a posterior-probability cutoff used for a decision?

Solutions. (1) The estimate is $180/250=0.72$; it estimates the posterior probability that basket E’s effect exceeds 2 units, conditional on the data and model. (2) With treatment minus control as the parameter, a decrease of at least 4 units means $\theta_j\leq -4$, so the event is $P(\theta_j\leq -4\mid y)$. (3) Autocorrelation reduces the information in a fixed number of draws; the independent-draw formula treats them as more independent than they are and can understate Monte Carlo error. (4) No. These are separate marginal threshold probabilities; a direct posterior comparison requires evaluating a contrast such as $P(\theta_A>\theta_B\mid y)$. (5) The clinical threshold defines which effect sizes count as meaningful, in effect units; the probability cutoff defines how much posterior support is required for a specified decision.