Interpreting Overall And Subgroup Effects

One Hierarchy, Several Effect Summaries

A basket trial may estimate a treatment effect for each clinically coherent subgroup while also using a hierarchical model to share information across subgroups. Those results invite an apparently simple question: “What is the overall effect?” There is no single answer until the target of the summary is specified. The center of a distribution of basket effects, the effect in one basket, and an average effect for a defined patient population are related quantities, but they are not interchangeable.

This tutorial uses a continuous endpoint to make the distinctions explicit. Let $\theta_j$ be the treatment effect in basket $j$, defined as the mean outcome under control minus the mean outcome under treatment. Suppose the outcome is systolic blood pressure measured in mmHg. Under this convention, a positive $\theta_j$ represents a reduction in blood pressure with treatment and therefore favors treatment. The sign convention and units must be stated whenever effects are reported.

Definition: In a normal random-effects hierarchy, the basket-specific effects are modeled as draws from a population distribution, for example $$ \theta_j \mid \mu,\tau \sim N(\mu,\tau^2). $$ Here, $\theta_j$ is the treatment effect in basket $j$, $\mu$ is the population-level mean effect (the center of the modeled distribution of effects), and $\tau$ describes between-basket variation. These are distinct from the posterior distribution of each quantity after observing trial data.

A common accompanying sampling model is

y_j \mid \theta_j \sim N(\theta_j,s_j^2),

where $y_j$ is an estimate of the basket-specific treatment effect and $s_j$ its standard error. This illustration assumes that the effect estimates are approximately normal and conditionally independent given their true effects, that the standard errors are adequately estimated, and that the baskets are sufficiently coherent and exchangeable for the hierarchical distribution to be scientifically reasonable. The normal model for effects is an approximation; its suitability, particularly for implausible tails, needs assessment in context.

The model’s $\mu$ describes the center of the distribution from which the modeled basket effects are drawn. It is not automatically the treatment effect for every basket, the mean effect among the trial’s enrolled patients, or the effect in a clinically defined target population. A basket-specific $\theta_j$ answers a different question: what is the effect in basket $j$? A target-population summary requires specifying how much each basket contributes to that population.

Interpreting the Population-Level Mean

The population-level mean is useful when the trial’s baskets are viewed as members of a broader population of clinically comparable subgroups. In that interpretation, $\mu$ summarizes the center of the modeled distribution, while $\tau$ describes how widely subgroup effects vary around it. A posterior for $\mu$ quantifies uncertainty about that center, conditional on the likelihood, hierarchy, and prior assumptions.

For the blood-pressure example, $\mu=3$ mmHg means a mean treatment effect of a 3 mmHg reduction across the modeled distribution of baskets. It does not mean that each basket has a 3 mmHg effect. Nor does it assert that every future or unobserved basket will benefit: subgroup effects vary, and the size of that variation matters. If $\tau$ is substantial, a positive $\mu$ can coexist with some basket effects near zero or below zero.

A credible interval for $\mu$ is an interval for this model parameter. Under the specified Bayesian model and data, a 95% posterior credible interval contains 95% of the posterior probability for $\mu$. It is not a guarantee that the same effect occurs in all baskets, and it does not remove uncertainty about the model’s assumptions or the applicability of the modeled basket population.

Worked Example: Interpreting the Hierarchy Mean and Three Baskets

Suppose a hypothetical hierarchical analysis of three blood-pressure baskets reports a posterior mean for $\mu$ of 3.2 mmHg, with a 95% credible interval from 0.8 to 5.6 mmHg. The posterior means for the basket-specific effects are 5.0 mmHg in basket A, 2.1 mmHg in basket B, and 1.0 mmHg in basket C. The reported 95% credible intervals are, respectively, 1.4 to 8.6, −0.5 to 4.7, and −2.0 to 4.0 mmHg.

The interval for $\mu$ lies above zero, so under this model the posterior supports a positive mean effect across the modeled distribution of baskets. In context, that is evidence for an average reduction in blood pressure at the distribution level, not evidence that every basket benefits.

Basket A has a positive posterior mean and an interval above zero. Baskets B and C have positive posterior means, but their intervals include zero and negative values. Those intervals retain uncertainty about their individual effects. The population-level result does not replace their basket-specific summaries, and it does not turn their uncertain estimates into certain benefit.

From a Hierarchy Mean to a Target-Population Effect

A clinical decision may concern a particular population rather than a hypothetical distribution of exchangeable baskets. In that case, define weights $w_j$ that represent the share of the target population in basket $j$, with $w_j\geq 0$ and $\sum_j w_j=1$. The weighted effect for that population is

\Theta_w=\sum_{j=1}^J w_j\theta_j.

The weights define the estimand: equal basket weights describe the average of the baskets, while weights proportional to target-population prevalence describe an average for that particular population, provided the basket-specific effects are appropriate for its members. Trial enrollment proportions are not automatically the right weights. A trial may enroll some baskets more heavily than they occur in the target population.

A useful relationship follows directly from the hierarchy. Conditional on $\mu$ and $\tau$, suppose the basket effects are independent draws from $N(\mu,\tau^2)$. Then

E(\Theta_w\mid\mu,\tau) =\sum_{j=1}^J w_jE(\theta_j\mid\mu,\tau) =\sum_{j=1}^J w_j\mu =\mu\sum_{j=1}^Jw_j =\mu.

Thus, before conditioning on the observed basket effects, the weighted effect has conditional expectation $\mu$. Its conditional variance is

\operatorname{Var}(\Theta_w\mid\mu,\tau) =\sum_{j=1}^Jw_j^2\operatorname{Var}(\theta_j\mid\mu,\tau) =\tau^2\sum_{j=1}^Jw_j^2.

The variance is generally positive: a weighted collection of realized basket effects can differ from the distribution center. After observing trial data, posterior information about the individual $\theta_j$ also matters. Consequently, the posterior mean of $\Theta_w$ need not equal the posterior mean of $\mu$, and its uncertainty must reflect the joint posterior distribution of the effects.

Worked Example: Equal Basket Weights Versus the Hierarchy Mean

Use the three posterior means from the hypothetical example: 5.0, 2.1, and 1.0 mmHg. The equal-weight average of these basket-specific posterior means is

\frac{5.0+2.1+1.0}{3} =\frac{8.1}{3} =2.7\text{ mmHg}.

The hierarchy mean’s posterior mean was 3.2 mmHg, not 2.7 mmHg. These summaries need not match: $\mu$ describes the center of the modeled distribution, whereas the equal-weight summary uses the estimated effects of these three particular baskets. Their posterior distributions also depend on how the model has combined information.

Suppose instead that the target population is represented by weights 0.50, 0.30, and 0.20 for baskets A, B, and C. Substituting the basket posterior means gives the posterior mean of the weighted effect, because expectation is linear:

0.50(5.0)+0.30(2.1)+0.20(1.0) =2.50+0.63+0.20 =3.33\text{ mmHg}.

This is a 3.33 mmHg posterior mean reduction for the specified weighting, not a universal overall effect. To report a credible interval for $\Theta_w$, calculate $\Theta_w$ for each joint posterior draw of the basket effects and summarize those draws. The point estimates alone do not provide the covariance information needed to calculate that interval.

Computing and Reporting a Weighted Posterior Summary

For posterior draw $m$, let $\theta_j^{(m)}$ be the draw for basket $j$. With fixed target-population weights, calculate

\Theta_w^{(m)}=\sum_{j=1}^Jw_j\theta_j^{(m)}.

The mean of these weighted draws estimates the posterior mean of $\Theta_w$; posterior quantiles give a credible interval. Using joint draws preserves posterior dependence among the basket effects. This matters because basket effects share hyperparameters and therefore are generally not independent in their posterior distribution.

Worked Example: Changing the Target Population

Continue with posterior mean effects of 5.0, 2.1, and 1.0 mmHg. Suppose a second hypothetical target population has weights 0.20, 0.50, and 0.30. Its weighted posterior mean is

0.20(5.0)+0.50(2.1)+0.30(1.0) =1.00+1.05+0.30 =2.35\text{ mmHg}.

For the first target population, the corresponding mean was 3.33 mmHg. Both summaries use the same basket effects; their difference comes from the target populations’ different basket weights. If a joint posterior calculation gave a 95% credible interval from −0.2 to 4.9 mmHg for the second population, that interval would express posterior uncertainty about its weighted mean effect. It would not mean there is a 95% probability that treatment benefits every basket in that population.

The example shows why “overall treatment effect” should be accompanied by the target population or weighting rule. Changing the weights changes the question being answered.

Choosing Among Effect Summaries

Summary	What it describes	Key interpretation requirement
$\mu$	Center of the modeled distribution of basket effects	Describe the population of baskets to which the hierarchy is intended to apply.
$\theta_j$	Effect in basket $j$	State the basket, effect scale, direction, and uncertainty.
$\Theta_w$	Weighted mean effect for a specified mix of baskets	State the target population and justify the weights.
A common-effect estimate	One assumed shared treatment effect across baskets	It imposes a stronger equality assumption than a random-effects hierarchy.

An independent analysis that estimates each basket separately avoids borrowing across baskets, but it does not provide the same model-based distribution center and may yield less precise estimates. A common-effect model directly assumes one shared effect; it should not be described as a random-effects mean when the model has no between-basket variation. A hierarchical model permits variation while sharing information, but its $\mu$ and $\tau$ depend on the modeled basket population and the exchangeability assumptions. The right comparison is determined by the scientific question, not by which estimate is largest or most precise.

Diagnostics, Failure Modes, and Limitations

Interpretation is most defensible when the basket definitions are clinically coherent and the effect scale is meaningful across them. Earlier material on exchangeability and posterior predictive checks provides relevant diagnostics: examine whether the model reproduces the observed pattern of basket estimates and whether any basket has a residual pattern that suggests a different mechanism or an inadequate likelihood. A narrow posterior for $\mu$ does not by itself establish that the baskets are exchangeable or that the model fits.

Unclear effect direction: If some results define benefit as treatment minus control while others use control minus treatment, signs cannot be compared until the scales are aligned.
Unclear target for “overall”: A mean across baskets is not automatically a mean for trial participants or a population of future patients. State the weights and their intended interpretation.
Over-reading the hierarchy mean: A positive posterior for $\mu$ does not imply a positive effect in every subgroup. Inspect the subgroup posteriors and the estimated heterogeneity as well.
Using point estimates to invent interval precision: Weighted posterior uncertainty depends on joint posterior draws, not only the marginal posterior means.
Incoherent or nonexchangeable baskets: If treatment effects arise from materially different mechanisms, a single distribution center may be scientifically unhelpful even if it can be computed.

An effect summary also depends on its scale. A mean difference in mmHg, a log odds ratio, and a risk difference do not have the same interpretation or aggregation properties. This tutorial’s equations are stated for an additive treatment effect on a common continuous-outcome scale. Other effect measures require scale-appropriate definitions and careful clinical interpretation; a weighted mean on one scale should not be silently translated into an overall effect on another.

Key takeaway: Report the hierarchy mean as the center of a modeled distribution, subgroup effects as basket-specific quantities, and a weighted effect as an estimand for an explicitly defined population. Keep the effect scale, benefit direction, weights, and uncertainty clear.

Check Your Understanding

Answer each question, then compare with the solutions below.

Under the stated sign convention, interpret $\mu=2.4$ mmHg when the outcome is systolic blood pressure and effects are control mean minus treatment mean.
Two basket posterior means are 4.0 and 1.0 mmHg. What is their equal-weight average?
Using those same means, calculate the weighted mean for weights 0.25 and 0.75. What changes if the weights are reversed?
Why can a 95% credible interval for $\mu$ entirely above zero fail to establish benefit in every basket?
What posterior calculation is needed to obtain a credible interval for a weighted effect?

Solutions. (1) The center of the modeled basket-effect distribution corresponds to a 2.4 mmHg mean reduction in systolic blood pressure; it does not assert that every basket has that effect. (2) $(4.0+1.0)/2=2.5$ mmHg. (3) $0.25(4.0)+0.75(1.0)=1.0+0.75=1.75$ mmHg. Reversing the weights gives $0.75(4.0)+0.25(1.0)=3.0+0.25=3.25$ mmHg, because the target mix changes. (4) The interval concerns the distribution center, while individual basket effects vary around that center and retain their own posterior uncertainty. (5) For each joint posterior draw, calculate the weighted sum of basket effects; use the resulting draws to obtain posterior quantiles.

Summary	What it describes	Key interpretation requirement
\(\mu\)	Center of the modeled distribution of basket effects	Describe the population of baskets to which the hierarchy is intended to apply.
\(\theta_j\)	Effect in basket \(j\)	State the basket, effect scale, direction, and uncertainty.
\(\Theta_w\)	Weighted mean effect for a specified mix of baskets	State the target population and justify the weights.
A common-effect estimate	One assumed shared treatment effect across baskets	It imposes a stronger equality assumption than a random-effects hierarchy.