Multiplicity Of Subgroup Claims Under Partial Pooling

Why Multiple Basket Claims Need Joint Interpretation

A basket trial may report a posterior probability of benefit for each subgroup. Those probabilities answer separate questions: for basket \(j\), how much posterior probability lies above the chosen efficacy threshold? They do not, by themselves, answer whether the evidence supports making all the proposed subgroup claims together. Under partial pooling, basket effects are connected through shared model parameters, so their posterior distributions—and often their efficacy events—are dependent.

This distinction matters whenever a trial team plans to identify several baskets as responsive. A high marginal probability for each basket can coexist with materially lower posterior probability that every selected basket is truly above threshold. Conversely, dependence can affect the probability of at least one basket being below threshold. The relevant quantity depends on the claim being made.

Definition: Let \(\theta_j\) be the efficacy parameter for basket \(j\), with larger values representing greater efficacy, and let \(\Delta\) be the prespecified threshold for clinically meaningful efficacy. The marginal posterior probability of benefit in basket \(j\) is \(P(\theta_j>\Delta\mid y)\). For a set \(S\) of baskets, the joint posterior support for benefit in every basket is \(P(\theta_j>\Delta\text{ for all }j\in S\mid y)\). The posterior probability that at least one such claim is unsupported is \(1-P(\theta_j>\Delta\text{ for all }j\in S\mid y)\).

Where Dependence Comes From

Consider a normal random-effects model with a common mean \(\mu\), basket effects \(\theta_j\), and observed estimates \(y_j\):

y_j\mid\theta_j\sim N(\theta_j,s_j^2),\qquad \theta_j\mid\mu,\tau\sim N(\mu,\tau^2),\qquad \mu\sim N(m_0,v_0).

Assume the basket observations are conditionally independent given their effects, the basket effects are conditionally independent given \(\mu,\tau\), and the sampling variances \(s_j^2\) and heterogeneity variance \(\tau^2\) are treated as known in this derivation. A prior for \(\mu\) is specified before observing the data. These assumptions describe a simple hierarchical model; in an applied analysis, the heterogeneity structure and prior choices also need clinical and predictive scrutiny.

Conditional on \(\mu\), the posterior distribution for each basket effect is normal. Define \(w_j=\tau^2/(\tau^2+s_j^2)\). Completing the square in the normal likelihood and prior gives:

\theta_j\mid\mu,y_j\sim N\bigl(w_jy_j+(1-w_j)\mu,\;w_js_j^2\bigr).

Thus the conditional posterior mean is a weighted average of the basket estimate and the shared mean. The weights depend on the relative sampling and heterogeneity variances. But conditional independence of the \(\theta_j\)'s given \(\mu\) does not imply independence after \(\mu\) is integrated out: uncertainty about the common mean is shared.

A Short Derivation of Posterior Dependence

For two distinct baskets \(j\) and \(k\), apply the law of total covariance conditional on the observed data. Conditional on \(\mu\), their posterior covariance is zero under the model above. Their conditional posterior means both depend on \(\mu\), so:

\begin{aligned} \operatorname{Cov}(\theta_j,\theta_k\mid y) &=E\{\operatorname{Cov}(\theta_j,\theta_k\mid\mu,y)\mid y\}\\ &\quad+\operatorname{Cov}\{E(\theta_j\mid\mu,y),E(\theta_k\mid\mu,y)\mid y\}\\ &=(1-w_j)(1-w_k)\operatorname{Var}(\mu\mid y). \end{aligned}

The first term is zero; the second is nonnegative and is positive if \(\mu\) remains uncertain and both baskets retain some shrinkage toward it. This derivation establishes dependence between the continuous effects in this model. It does not, by itself, calculate the dependence between any particular threshold events; those event probabilities should be computed from the joint posterior, not inferred from the correlation alone.

Worked Example: Shared-Mean Uncertainty Couples Two Baskets

Suppose two hypothetical baskets have estimates \(y_1=y_2=1\) efficacy unit, with \(s_1^2=s_2^2=1\). Use \(\tau^2=1\) and \(\mu\sim N(0,1)\). The model assumptions above hold for this calculation: independent normal sampling errors, normal random effects conditional on \(\mu\), and fixed variances.

Here \(w_1=w_2=1/(1+1)=0.5\). After integrating out each \(\theta_j\), \(y_j\mid\mu\sim N(\mu,2)\). The posterior variance of \(\mu\) is \(1/(1+2/2)=0.5\), and its posterior mean is \(0.5(1+1)/2=0.5\).

The posterior mean of either basket effect is therefore \(0.5(1)+0.5(0.5)=0.75\) efficacy units. Its posterior variance is the conditional variance \(0.5\), plus the variance in its conditional mean: \(0.5+(0.5)^2(0.5)=0.625\). The covariance between the two basket effects is \((1-0.5)^2(0.5)=0.125\). Their posterior correlation is \(0.125/0.625=0.20\).

The positive covariance arises because uncertainty about \(\mu\) moves both posterior effects in the same direction. This calculation does not say that the baskets have equal effects; each retains its own effect and its own sampling uncertainty.

Compute the Claim You Intend to Make

If decisions are based on posterior draws, joint claim probabilities are straightforward to estimate. For each draw, check whether each basket exceeds \(\Delta\), then count the draws satisfying the event of interest. The same draws must be used across baskets: independent resampling of each marginal posterior would erase the dependence the hierarchical model represents.

Worked Example: Marginal Probabilities Versus a Joint Claim

In a hypothetical two-basket analysis, a researcher obtains 20 paired draws from the fitted joint posterior. For a clinically meaningful threshold \(\Delta=0.2\), the draw outcomes are: both effects exceed \(\Delta\) in 10 draws; only basket 1 exceeds it in 2; only basket 2 exceeds it in 2; and neither exceeds it in 6. Treat these counts as an intentionally small illustration of the counting calculation, not as a precise posterior estimate.

The marginal posterior probability for basket 1 is \((10+2)/20=0.60\); for basket 2 it is also \((10+2)/20=0.60\). The posterior probability that both baskets exceed the threshold is \(10/20=0.50\). The probability that at least one fails to exceed it is \(1-0.50=0.50\).

If the two events were independent, their joint probability would be \(0.60\times0.60=0.36\). The observed paired draws instead give a joint probability of \(0.50\), indicating positive dependence in this illustration. As a separate independent benchmark, if two marginal probabilities were each \(0.50\), their joint probability would be \(0.50\times0.50=0.25\). Neither product calculation should replace counting from the actual joint posterior unless independence is justified.

Worked Example: Three Claims and the Probability All Are Supported

Consider three hypothetical baskets and 100 paired draws from their joint posterior. Suppose the counts of threshold failures are as follows: no basket fails in 82 draws; only basket 1 fails in 5; only basket 2 fails in 6; only basket 3 fails in 4; baskets 1 and 2 fail in 1; baskets 1 and 3 fail in 1; and baskets 2 and 3 fail in 1. No draw has all three failing. These categories are mutually exclusive and sum to 100; each draw’s classification checks whether \(\theta_j>\Delta\).

Basket 1 fails in \(5+1+1=7\) draws, so its marginal probability of exceeding the threshold is \(93/100=0.93\). Basket 2 fails in \(6+1+1=8\) draws, giving \(0.92\); basket 3 fails in \(4+1+1=6\) draws, giving \(0.94\). Yet the probability all three exceed the threshold is only \(82/100=0.82\). The posterior probability that at least one of the three claims is unsupported is \(18/100=0.18\).

Multiplying the marginal probabilities would give \(0.93\times0.92\times0.94=0.8043\), not the joint estimate \(0.82\). The discrepancy is expected when the events are dependent. If a decision rule required every marginal probability to exceed 0.90, all three baskets would pass that rule, but the joint posterior support for all three claims would still be 0.82. Whether that is sufficient is a decision and reporting question, not an arithmetic consequence of the marginal cutoff.

Interpretation, Diagnostics, and Alternatives

A Bayesian joint posterior probability is conditional on the model, prior, and observed data. For example, \(P(\theta_j>\Delta\text{ for all }j\in S\mid y)=0.82\) means that, under this specified posterior, the effects in every basket in \(S\) exceed the threshold together with posterior probability 0.82. It is not a frequentist family-wise error rate, and it does not mean there is an 82% long-run probability that a procedure will make all claims correctly.

Partial pooling does not guarantee positive dependence in every model. In the simple normal hierarchy above, a shared uncertain mean induces nonnegative covariance. Other structures, constraints, or parameterizations can produce different dependence. Nor does effect correlation alone determine the dependence of threshold events: the threshold, posterior means, and marginal uncertainty all matter. Compute the joint event directly.

For posterior simulation, estimate a joint probability using the proportion of paired draws satisfying the joint condition. Check that the chain has mixed adequately, that effective sample sizes for the joint indicator are sufficient, and that Monte Carlo error is small enough for the decision. A low effective sample size for the indicator can make a reported joint probability unstable even when marginal summaries look smooth. Also examine model fit and sensitivity to heterogeneity and hyperprior assumptions; a precise joint calculation from a poorly fitting model is not reliable evidence.

The main alternatives answer different questions. Independent basket analyses avoid borrowing and can make separate marginal calculations, but they do not represent the shared-parameter dependence of a hierarchical model. Complete pooling imposes a common effect and can make subgroup distinctions disappear. Partial pooling allows differences while borrowing information, but joint claims must reflect the resulting joint posterior. These approaches should be compared on the scientific question and assumptions, not treated as interchangeable multiplicity corrections.

A useful conservative check follows from the union bound. For a selected set \(S\), the posterior probability of at least one failure is no greater than the sum of the marginal failure probabilities:

P(\text{at least one }\theta_j\leq\Delta\mid y) \leq \sum_{j\in S}P(\theta_j\leq\Delta\mid y).

This bound can be loose, especially when events overlap, and is not a replacement for evaluating the joint posterior. It does show why several individually small failure probabilities can accumulate when making a collection of claims. A common mistake is to multiply marginal probabilities, or to treat each basket’s probability cutoff as an automatic guarantee for the whole set. The correction is to specify whether the claim concerns one basket, at least one basket, or every selected basket, then calculate that event from joint posterior draws.

Check Your Understanding

Answer each question using the distinction between marginal and joint posterior probabilities.

Two baskets each have marginal posterior probability 0.8 of exceeding \(\Delta\). What additional assumption is needed to multiply these probabilities to obtain the probability both exceed \(\Delta\)?
In the normal hierarchy, what model feature induces posterior covariance between two basket effects after integrating over \(\mu\)?
If \(P(\theta_j>\Delta\text{ for all }j\in S\mid y)=0.87\), what is the posterior probability that at least one selected basket fails to exceed \(\Delta\)?
Why should joint claims be counted from paired posterior draws rather than from independently sampled marginal draws?

Solutions. (1) The two threshold events must be posterior independent; partial pooling does not generally justify that assumption. Under independence the product would be \(0.8\times0.8=0.64\). (2) The uncertain shared mean \(\mu\) enters both conditional posterior means, so variation in \(\mu\) induces covariance. (3) The complement is \(1-0.87=0.13\). (4) Paired draws preserve the posterior dependence between baskets; independently sampled marginal draws impose independence and can give the wrong joint probability.