Comparing Hierarchical And Independent Basket Analyses

Why Compare the Analyses?

A hierarchical basket model uses information across baskets to estimate basket-specific effects. An independent analysis estimates each effect using only that basket’s data and prior. Comparing the two makes the consequence of borrowing visible: it can move a basket estimate toward information shared across baskets and can reduce its posterior uncertainty. Neither change is automatically beneficial. If the baskets are not plausibly exchangeable, borrowing can make an estimate appear more precise while pulling it away from the effect relevant to that basket.

The comparison is most interpretable when the analyses target the same basket-specific estimands, use the same likelihood for each basket, and start from priors with the same marginal distribution for each effect. The example calculations below use continuous treatment-effect estimates with known sampling variances. The same comparison logic applies more broadly, but the formulas depend on the likelihood and hierarchy chosen.

Definition: Independent basket analyses estimate each basket effect separately, with no information shared across baskets. A hierarchical basket analysis assigns basket effects a common population distribution, allowing data from one basket to inform estimates in others through shared parameters. Borrowing is this cross-basket influence on an estimate or its posterior distribution.

A Normal Model for a Like-for-Like Comparison

Suppose \(y_j\) is an approximately normal estimate of the treatment effect in basket \(j\), with known sampling variance \(s_j^2\). Here \(s_j\) is the standard error of that estimate, and \(\theta_j\) is the basket-specific effect, measured in the same units as \(y_j\). The likelihood assumes that the basket estimates are conditionally independent given their effects:

y_j \mid \theta_j \sim N(\theta_j,s_j^2).

For the hierarchical analysis, use a normal population model with a shared mean \(\mu\) and a fixed between-basket variance \(\tau^2\):

\theta_j \mid \mu \sim N(\mu,\tau^2), \qquad \mu \sim N(m_0,V_0).

Conditional on \(\mu\), the model treats basket effects as exchangeable: before seeing basket-specific outcomes, the labels of the baskets do not identify which effects should be higher or lower. The model also assumes a common between-basket variance and, given the effects, no residual dependence between basket estimates. Those are substantive assumptions, not consequences of having multiple baskets.

The comparison analysis gives each basket an independent prior equal to its marginal prior under this hierarchy. Since \(\theta_j\) has marginal distribution \(N(m_0,V_0+\tau^2)\), the independent analysis uses:

\theta_j \sim N(m_0,V_0+\tau^2) \quad\text{independently for each }j.

Matching these marginal priors helps isolate the effect of cross-basket borrowing. Comparing a hierarchical model with unrelated, differently informative priors would mix the influence of borrowing with the influence of different prior information.

Deriving the Hierarchical Posterior

With \(\tau^2\) fixed, the basket estimate satisfies \(y_j\mid\mu\sim N(\mu,\tau^2+s_j^2)\). Combining these distributions across baskets with the normal prior for \(\mu\) gives its posterior. Conditional on \(\mu\) and the estimate \(y_j\), the posterior for \(\theta_j\) is also normal. Integrating over the posterior uncertainty in \(\mu\) then gives the basket-specific posterior mean and variance.

U=\left(\frac{1}{V_0}+\sum_j\frac{1}{\tau^2+s_j^2}\right)^{-1}, \qquad M=U\left(\frac{m_0}{V_0}+\sum_j\frac{y_j}{\tau^2+s_j^2}\right).

\mu\mid \mathbf{y}\sim N(M,U).

To obtain the posterior for a basket effect conditional on \(\mu\), combine the likelihood variance \(s_j^2\) with the prior variance \(\tau^2\). Define \(a_j=\tau^2/(\tau^2+s_j^2)\) and \(b_j=s_j^2/(\tau^2+s_j^2)\), so \(a_j+b_j=1\). The conditional posterior mean is \(a_jy_j+b_j\mu\), and its variance is \(q_j=s_j^2\tau^2/(s_j^2+\tau^2)\). Averaging that mean over \(\mu\mid\mathbf y\), and including uncertainty in \(\mu\), yields:

E(\theta_j\mid\mathbf{y})=a_jy_j+b_jM, \qquad \operatorname{Var}(\theta_j\mid\mathbf{y})=q_j+b_j^2U.

The second term \(b_j^2U\) matters: uncertainty about the shared mean is part of uncertainty about the basket effect. Omitting it would understate the posterior variance. The independent analysis instead gives:

v_{j,\mathrm{ind}} =\left(\frac{1}{s_j^2}+\frac{1}{V_0+\tau^2}\right)^{-1}, \qquad m_{j,\mathrm{ind}} =v_{j,\mathrm{ind}}\left(\frac{y_j}{s_j^2}+\frac{m_0}{V_0+\tau^2}\right).

These formulas assume known sampling variances and a fixed \(\tau^2\). If \(\tau\) is unknown, its posterior uncertainty must be included rather than treating a single chosen value as known. The calculations also rely on the normal approximation for the basket estimates and on the specified exchangeability structure being defensible.

Worked Example: Two Baskets With Different Observed Effects

Hypothetical data. Two clinically coherent baskets have treatment-effect estimates of \(y_1=8\) and \(y_2=2\) units, each with standard error \(s_j=2\) units. Use \(\tau^2=4\), \(m_0=0\), and \(V_0=100\). The known standard errors and normal likelihood are assumed adequate for this illustration. The large \(V_0\) gives the shared mean a broad prior, while the fixed \(\tau\) specifies the assumed basket-to-basket spread.

First calculate the posterior for the shared mean. Each \(s_j^2=4\), so \(\tau^2+s_j^2=8\):

U=\left(\frac{1}{100}+\frac{1}{8}+\frac{1}{8}\right)^{-1} =\frac{1}{0.2600}=3.846, \qquad M=3.846\left(\frac{8+2}{8}\right)=4.808.

For each basket, \(a_j=b_j=4/8=0.5\), and \(q_j=4(4)/8=2\). Thus the hierarchical posterior means are \(0.5y_j+0.5M\), and each posterior variance is \(2+0.25U\):

E(\theta_1\mid\mathbf y)=6.404,\quad E(\theta_2\mid\mathbf y)=3.404,\quad \operatorname{Var}(\theta_j\mid\mathbf y)=2+0.25(3.846)=2.962.

For the independent analyses, the matched prior variance is \(V_0+\tau^2=104\). For basket 1, for example, the posterior mean is \(104(8)/(104+4)=7.704\), and the variance is \(104(4)/(104+4)=3.852\). Basket 2 has mean \(104(2)/(104+4)=1.926\) and the same variance. Relative to independent analyses, both hierarchical means move toward one another, and each posterior standard deviation decreases from \(\sqrt{3.852}=1.963\) to \(\sqrt{2.962}=1.721\) units. The hierarchical model’s shared information raises the lower estimate and lowers the higher estimate; it does not establish that either effect is clinically meaningful.

Borrowing Depends on Precision and Agreement

The weights show why baskets do not necessarily borrow equally. In \(a_jy_j+b_jM\), \(a_j\) weights the basket’s own estimate and \(b_j\) weights the shared mean. When \(s_j^2\) is small relative to \(\tau^2\), the basket estimate is relatively precise and \(a_j\) is large. When \(s_j^2\) is large, \(b_j\) is larger and the estimate is more influenced by the shared mean. This is a feature of this model and these assumptions, not a general rule that every hierarchical model must shrink estimates by this exact formula.

Worked Example: A Discordant Basket

Hypothetical data. Keep the priors and standard errors from the previous example, but suppose the estimates are \(y_1=8\) and \(y_2=-4\) units. The normal likelihood and known standard errors remain assumed; the striking disagreement is a reason to assess whether the exchangeability assumption is credible, not a reason to ignore it.

The posterior for \(\mu\) has \(U=3.846\) as before, while its mean is:

M=3.846\left(\frac{8-4}{8}\right)=1.923.

Again \(a_j=b_j=0.5\) and \(q_j=2\), so the hierarchical posterior means are \(0.5(8)+0.5(1.923)=4.962\) and \(0.5(-4)+0.5(1.923)=-1.038\) units. Each posterior variance is \(2+0.25(3.846)=2.962\). Under the independent analyses, the means are \(104(8)/108=7.704\) and \(104(-4)/108=-3.852\) units, with variance \(3.852\) each.

The hierarchical estimates are closer together, and the uncertainty is smaller under the stated model. But the shift of the negative estimate toward zero is not evidence that the basket truly has a less negative effect. It is a consequence of borrowing under an exchangeability assumption. If biological or clinical knowledge makes the effects implausibly similar, this estimate may be misleading despite its narrower posterior distribution.

Worked Example: Unequal Basket Precision

Hypothetical data. Consider three baskets with estimates \(y_1=5\), \(y_2=1\), and \(y_3=4\) units, and standard errors \(s_1=1\), \(s_2=3\), and \(s_3=2\) units. Use \(\tau^2=4\), \(m_0=0\), and \(V_0=100\). Assume the estimates have independent normal likelihoods with the stated known variances. Basket 2 is least precise and therefore has the largest \(b_j\).

The contributions to the shared-mean precision are \(1/5\), \(1/13\), and \(1/8\). Therefore:

U=\left(0.01000+\frac{1}{5}+\frac{1}{13}+\frac{1}{8}\right)^{-1} =2.428, \qquad M=2.428\left(\frac{5}{5}+\frac{1}{13}+\frac{4}{8}\right) =3.828.

For basket 1, \(a_1=4/5=0.8\), \(b_1=0.2\), and \(q_1=4/5=0.8\). Its hierarchical mean is \(0.8(5)+0.2(3.828)=4.766\), and its variance is \(0.8+0.2^2(2.428)=0.897\) square units. For basket 2, \(a_2=4/13\), \(b_2=9/13\), and \(q_2=36/13\). Its mean is \((4/13)(1)+(9/13)(3.828)=2.958\), and its variance is \(36/13+(9/13)^2(2.428)=3.247\) square units. For basket 3, \(a_3=b_3=0.5\) and \(q_3=2\); its mean is \(3.914\) and its variance is \(2+0.25(2.428)=2.607\) square units.

Under the independent analyses, the matched prior variance is 104. For basket 2, the independent posterior variance is \(1/(1/9+1/104)=8.283\) square units, compared with \(3.247\) hierarchically; the independent mean is \(104/(104+9)=0.920\) units, compared with \(2.958\). This large difference reflects the low precision of basket 2’s own estimate and the influence of the other baskets under the hierarchy. It is not a guarantee that the hierarchical estimate is closer to the unknown basket effect.

Interpreting Diagnostics and Limitations

A useful comparison reports both analyses side by side for every basket: posterior means, posterior standard deviations or credible intervals, and the direction and magnitude of the change. For example, a narrower 95% posterior credible interval means that, conditional on the model and prior, the posterior probability that the basket effect lies in that interval is 95%. It does not mean that the interval has a 95% probability of containing a fixed effect under repeated sampling.

Differences between the analyses are not a stand-alone diagnostic of model fit. A large movement may reflect useful information sharing, substantial incompatibility between baskets, or both. Examine whether the basket definitions support exchangeability and whether outcome patterns are compatible with the assumed hierarchy. Posterior predictive checks across subgroups, leave-one-subgroup-out predictive diagnostics, and residual-pattern diagnostics can reveal problems that a narrow posterior interval cannot. Sensitivity to the hyperprior scale is also important: if conclusions change materially across plausible choices, present that dependence rather than selecting the most favorable result.

The closed-form calculations here treat \(\tau^2\) as fixed and use normal likelihoods with known variances. In an applied analysis, \(\tau\) may be uncertain, standard errors may be estimated, outcomes may be binary or time-to-event, and the model may include covariates. Those features require the corresponding likelihood and posterior computation; the fixed-\(\tau\), normal formulas should not be applied mechanically. Even when computation is reliable, the comparison quantifies consequences of the chosen model, not proof that the baskets share a common underlying distribution.

Common Mistakes

Calling every reduction in uncertainty a benefit. A smaller posterior variance is conditional on the hierarchy being adequate. If baskets differ in ways the model fails to represent, apparent precision can be misleading.
Comparing unmatched priors. If the independent analyses use much broader or narrower priors than the hierarchy’s marginal priors, the difference cannot be attributed solely to borrowing.
Dropping uncertainty in the shared mean. The hierarchical variance includes \(b_j^2U\). Reporting only \(q_j\) understates uncertainty unless \(\mu\) is genuinely known.
Interpreting shrinkage as evidence that baskets are identical. Shrinkage is produced by the model’s exchangeability assumptions; it does not establish biological sameness.
Treating the fixed heterogeneity variance as certain. When \(\tau\) is estimated, uncertainty about heterogeneity affects basket estimates and must be reflected in the posterior.

Key takeaway: Compare hierarchical and independent analyses using the same basket estimands, likelihood, and marginal prior. Report how each basket’s estimate and uncertainty change, then assess whether the borrowing assumptions are credible and robust.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the normal hierarchy, what happens to the weight \(b_j\) on the shared mean when a basket’s sampling variance increases?
Why should the independent analysis use a prior matching the hierarchy’s marginal prior?
For \(s_j^2=4\) and \(\tau^2=4\), calculate \(a_j\) and \(b_j\). What do the values mean?
Does a narrower hierarchical posterior interval prove that the hierarchy is appropriate? Why or why not?
Why does the hierarchical posterior variance include \(b_j^2U\)?

Solutions. (1) \(b_j=s_j^2/(s_j^2+\tau^2)\) increases, so the basket estimate receives more weight from the shared mean. (2) Matching marginal priors helps isolate the effect of cross-basket information rather than confounding it with different prior informativeness. (3) \(a_j=4/(4+4)=0.5\) and \(b_j=4/(4+4)=0.5\); in the posterior mean, the basket estimate and shared mean receive equal weights. (4) No. The interval is conditional on the model and prior; misspecified exchangeability can produce misleading precision. (5) The basket effect depends on the uncertain shared mean, whose posterior variance is \(U\); propagating that uncertainty adds \(b_j^2U\).