Diagnosing Under-Shrinkage From Weak Hierarchical Structure

When Does a Hierarchical Model Borrow Little?

In a basket trial, a hierarchical model allows subgroup effects to differ while connecting them through shared parameters. That connection can pull a subgroup estimate toward a common level. Sometimes, however, the posterior subgroup estimates remain close to their own data-based estimates. This is little shrinkage, or weak borrowing in the fitted analysis.

Little shrinkage is a description of the estimates, not a diagnosis by itself. It can arise because the model estimates substantial between-basket variation, because the data contain little information about shared parameters, or because the chosen model structure provides only a weak connection among baskets. It may also be appropriate: genuine differences between baskets are a reason not to force their estimates together. The diagnostic task is to decide which explanation is plausible, rather than to assume that more borrowing is always better.

Definition: For a basket-specific parameter \(\theta_i\), shrinkage describes how much its posterior estimate is pulled from its basket-specific likelihood information toward a shared or model-implied level. Little shrinkage means that this pull is small relative to the estimate supported by basket \(i\)'s own data. The amount of shrinkage depends on the model, the data, and the posterior distribution of the shared parameters; it is not a property of the data alone.

A Normal Model Shows What Controls Shrinkage

A useful diagnostic calculation comes from a normal random-effects model. Suppose \(y_i\) is the estimated treatment effect in basket \(i\), measured in the outcome's units, with known standard error \(s_i\). Assume the sampling model and hierarchical distribution are

y_i\mid\theta_i\sim N(\theta_i,s_i^2),\qquad \theta_i\mid\mu,\tau\sim N(\mu,\tau^2).

Here \(\mu\) is the shared mean effect and \(\tau\) is the between-basket standard deviation. This calculation assumes conditionally independent basket estimates, a normal likelihood with correctly specified standard errors, and normally distributed basket effects conditional on \(\mu\) and \(\tau\). It isolates the role of \(\tau\); a full analysis must also account for uncertainty in the shared parameters.

Result: Conditional on \(\mu\) and \(\tau\), the posterior mean for \(\theta_i\) is a weighted average of \(y_i\) and \(\mu\). The weight on the basket estimate is \(\tau^2/(\tau^2+s_i^2)\), and the weight on the shared mean is \(s_i^2/(\tau^2+s_i^2)\). Thus, larger \(\tau\) relative to \(s_i\) means less shrinkage toward \(\mu\).

The result follows by multiplying the normal likelihood and normal prior densities. Their combined quadratic terms in \(\theta_i\) are

\frac{(y_i-\theta_i)^2}{s_i^2}+ \frac{(\theta_i-\mu)^2}{\tau^2}.

The coefficient of \(\theta_i^2\) is \(1/s_i^2+1/\tau^2\), so the posterior variance is its reciprocal. The coefficient determining the posterior mean is \(y_i/s_i^2+\mu/\tau^2\). Dividing that coefficient by the total precision gives

E(\theta_i\mid y_i,\mu,\tau) = \frac{\tau^2}{\tau^2+s_i^2}y_i+ \frac{s_i^2}{\tau^2+s_i^2}\mu.

This derivation makes an important distinction visible. A small \(s_i\) makes the basket's own data precise and therefore reduces the pull toward \(\mu\). A large \(\tau\) says that basket effects may vary substantially, also reducing that pull. Neither fact alone establishes that a fitted analysis is defective.

Worked Example: Comparing Conditional Shrinkage Weights

Suppose two hypothetical baskets have estimated treatment effects \(y_1=0.10\) and \(y_2=0.12\) outcome points, each with standard error \(s_i=0.20\) points. For this calculation only, hold the shared mean at \(\mu=0.11\) points. Compare conditional posterior means for \(\tau=0.80\) and \(\tau=0.10\) points.

With \(\tau=0.80\), the weight on each basket estimate is

\frac{0.80^2}{0.80^2+0.20^2} =\frac{0.64}{0.68}=0.9412.

The weight on the shared mean is \(1-0.9412=0.05882\). The conditional posterior means are therefore

\begin{aligned} E(\theta_1\mid y_1,\mu,\tau)&=0.9412(0.10)+0.05882(0.11)=0.1006,\\ E(\theta_2\mid y_2,\mu,\tau)&=0.9412(0.12)+0.05882(0.11)=0.1194. \end{aligned}

Each estimate moves only \(0.000588\) outcome points toward the shared mean. With \(\tau=0.10\), the weight on the basket estimate is \(0.01/(0.01+0.04)=0.20\), so the weight on the shared mean is \(0.80\). The resulting means are \(0.20(0.10)+0.80(0.11)=0.108\) and \(0.20(0.12)+0.80(0.11)=0.112\) points. These are conditional calculations, not complete posterior summaries: in a fitted model, uncertainty about \(\mu\) and \(\tau\) must also be included.

Check Whether Shared Parameters Are Well Informed

Even if the model specifies a common hypermean, the observed baskets may not pin it down precisely. This can happen with few baskets, imprecise subgroup estimates, or large between-basket variation. To see the role of these factors, additionally assume \(\mu\sim N(m_0,v_0)\), with \(v_0\) known, and temporarily condition on \(\tau\). Integrating out each \(\theta_i\) gives \(y_i\mid\mu,\tau\sim N(\mu,s_i^2+\tau^2)\). Consequently,

\operatorname{Var}(\mu\mid \mathbf y,\tau) = \left(\frac{1}{v_0}+\sum_{i=1}^{K}\frac{1}{s_i^2+\tau^2}\right)^{-1}.

When \(\tau\) is large, each basket contributes little precision about \(\mu\). With few baskets, the sum may remain small even if the individual estimates look similar. Uncertainty about \(\mu\) can then make the shared level itself a weak anchor for the subgroup estimates. This is a distinct issue from a large \(\tau\): both can contribute to limited borrowing, but they describe different parts of the hierarchy.

Worked Example: Few Baskets and an Uncertain Shared Mean

Use the hypothetical data \(y_1=0.10\), \(y_2=0.12\) outcome points and \(s_1=s_2=0.20\) points. Set the prior for the shared mean to \(\mu\sim N(0,1^2)\), with all quantities in outcome points. First condition on \(\tau=0.80\). The marginal sampling variance for each \(y_i\) given \(\mu\) is \(0.20^2+0.80^2=0.68\) square points. The posterior precision and mean of \(\mu\) are

\begin{aligned} \text{precision}(\mu\mid\mathbf y,\tau) &=1+\frac{2}{0.68}=3.941,\\ E(\mu\mid\mathbf y,\tau) &=\frac{0/1+0.10/0.68+0.12/0.68}{3.941} =0.08209\text{ points}. \end{aligned}

For each basket, the conditional weight on its own observation is \(0.9412\), as in the previous calculation. Averaging the conditional posterior mean over this posterior for \(\mu\) gives

\begin{aligned} E(\theta_1\mid\mathbf y,\tau) &=0.9412(0.10)+0.05882(0.08209)=0.0990\text{ points},\\ E(\theta_2\mid\mathbf y,\tau) &=0.9412(0.12)+0.05882(0.08209)=0.1178\text{ points}. \end{aligned}

For comparison, if we instead condition on \(\tau=0.10\), the marginal variance is \(0.05\) square points, the posterior precision for \(\mu\) is \(1+2/0.05=41\), and its posterior mean is \((0.10/0.05+0.12/0.05)/41=0.1073\) points. The basket-level posterior means become \(0.20(0.10)+0.80(0.1073)=0.1059\) and \(0.20(0.12)+0.80(0.1073)=0.1099\) points. The stronger connection reflects both the smaller assumed \(\tau\) and the resulting greater information about \(\mu\). The comparison holds \(\tau\) fixed in each calculation; it does not estimate \(\tau\) from the data.

Diagnose Before Calling the Result Under-Shrinkage

In a fitted hierarchical analysis, the conditional formula is a guide, not a substitute for inspecting the joint posterior. Shrinkage depends on posterior uncertainty and dependence among \(\theta_i\), \(\mu\), and \(\tau\). In particular, substituting a posterior median for \(\tau\) into a conditional formula will not generally reproduce the posterior mean of \(\theta_i\). A useful diagnostic sequence is:

Describe the observed shrinkage.
Compare each posterior estimate with its basket-specific data estimate and with the fitted shared level. State the comparison on the outcome's scale, with units and uncertainty.

Inspect the hierarchy.
Examine posterior uncertainty for between-basket variation and shared parameters. A posterior for \(\tau\) concentrated on values large relative to typical \(s_i\) is consistent with weak shrinkage in the normal model.

Check whether baskets support the shared level.
Few or imprecise baskets may leave \(\mu\) poorly determined. Review the number of baskets, their standard errors, posterior dependence, and the influence of the hyperprior.

Check predictive fit and model structure.
Ask whether the fitted hierarchy can plausibly generate the observed cross-basket pattern. Look for systematic differences that the assumed common distribution or exchangeability structure cannot represent.

Assess sensitivity and report the conclusion cautiously.
Compare scientifically reasonable prior and structural choices. If little shrinkage persists, describe it as a feature of those analyses, and explain whether the evidence supports heterogeneity, weak information, or unresolved uncertainty.

Worked Example: Is Little Shrinkage Compatible With Basket Differences?

Consider two hypothetical baskets with estimated treatment effects \(y_1=0.10\) and \(y_2=0.70\) outcome points, each with standard error \(0.20\) points. The difference is \(0.60\) points. Under the normal hierarchical model, conditional on \(\tau\), the difference \(y_1-y_2\) has mean zero and variance \(2(s^2+\tau^2)\), because the two basket effects and sampling errors are conditionally independent.

If \(\tau=0.10\) points, the predictive standard deviation of the difference is

\sqrt{2(0.20^2+0.10^2)} =\sqrt{0.10}=0.3162\text{ points}.

The observed difference is \(0.60/0.3162=1.897\) predictive standard deviations from zero. Under this fixed-parameter model, the two-sided predictive tail probability is approximately \(0.0578\). This is a conditional predictive check, not a frequentist p-value and not a posterior probability that the hierarchy is true.

If instead \(\tau=0.80\) points, the predictive standard deviation is \(\sqrt{2(0.20^2+0.80^2)}=\sqrt{1.36}=1.166\) points. The standardized difference is \(0.60/1.166=0.5145\), giving a two-sided predictive tail probability of about \(0.607\). With this larger between-basket variation, the difference is unremarkable under the assumed model, and little shrinkage may be a sensible response. The comparison does not determine which \(\tau\) is supported; that requires fitting the model and examining its posterior and sensitivity to assumptions.

Alternatives, Failure Modes, and Limits

An independent analysis estimates each basket without borrowing, while a shared-effect analysis imposes one common treatment effect. The hierarchical model lies between these approaches in its assumptions, but it is not guaranteed to produce an intermediate estimate in every finite data set, especially after accounting for uncertainty and model structure. Comparing all three can clarify how conclusions depend on the borrowing assumption. The independent analysis is a useful reference for data-only estimates; it may be imprecise with sparse responses. The shared-effect analysis can be precise when effects truly align, but can obscure real differences. Neither is a universal benchmark for deciding the correct amount of shrinkage.

Useful warning signs include strong sensitivity of the estimated \(\tau\) or subgroup estimates to the hyperprior scale; posterior distributions for shared parameters that remain broad; and poor predictive reproduction of cross-basket variation or patterns. A model can also be weakly connected by construction, for example if it shares a parameter that is only indirectly related to the subgroup effects of interest. Conversely, if posterior predictive checks show that the hierarchy represents observed differences reasonably, low shrinkage may be an appropriate conclusion rather than a failure.

A common mistake is to reason, “The baskets are in a hierarchical model, so their estimates should be close; estimates that remain separate prove that the model did not work.” The correction is to inspect what the model learned about between-basket variation and shared parameters, then assess fit and sensitivity. Partial pooling permits differences. Forcing more similarity by tightening a prior or changing the hierarchy can produce stronger borrowing, but it does not create evidence that the baskets are exchangeable.

These diagnostics have limits. Small numbers of baskets often make between-basket variation difficult to estimate, so a posterior for \(\tau\) may reflect its prior substantially. A narrow posterior for \(\tau\) is not automatically evidence of a well-specified model, and a broad one does not identify the true amount of heterogeneity. Predictive checks can reveal aspects of poor fit but cannot prove the hierarchy correct. If basket outcomes are sparse or modeled on a nonlinear scale, the normal-model weights above are an explanatory approximation, not a formula for the actual amount of shrinkage. In all cases, interpret posterior subgroup estimates as conditional on the likelihood, hierarchy, and prior assumptions used.

Key takeaway: Little shrinkage is a finding to explain, not a defect to eliminate. Determine whether it reflects large between-basket variation, weak information about shared parameters, structural assumptions, or sensitivity to priors; then report the subgroup estimates with that uncertainty in view.

Check Your Understanding

Answer each question using the model assumptions and distinctions developed above. Brief solutions follow the questions.

In the normal model, what happens to the weight on the shared mean as \(\tau\) increases while \(s_i\) stays fixed?
Two baskets have \(s_i=0.10\) and \(\tau=0.30\). Calculate the weight on the basket estimate and explain what it implies about shrinkage.
Why can a small number of baskets and a large \(\tau\) leave the shared mean weakly informed?
Does little shrinkage alone establish that a hierarchical model is misspecified? State two other possible explanations.
Why is a conditional predictive tail probability for a basket difference not the probability that the hierarchical model is true?

Solutions. (1) The weight on the shared mean is \(s_i^2/(s_i^2+\tau^2)\); it decreases as \(\tau\) increases, so conditional shrinkage toward \(\mu\) becomes weaker. (2) The basket weight is \(0.30^2/(0.30^2+0.10^2)=0.09/0.10=0.90\). The conditional posterior mean gives 90% weight to the basket estimate and 10% to the shared mean, indicating relatively little shrinkage. (3) Each basket contributes precision \(1/(s_i^2+\tau^2)\) about \(\mu\). Few contributions and a large \(\tau\) make their sum small, leaving the posterior for \(\mu\) more dependent on its prior. (4) No. Little shrinkage may reflect genuine between-basket heterogeneity or limited information about shared parameters; it may also follow from the chosen hierarchical structure. (5) It is calculated under specified model parameters and evaluates how compatible a discrepancy is with that predictive distribution. It is not a posterior probability over competing models or a probability that one model is true.