Why Look for Over-Shrinkage?
A hierarchical basket model borrows information across baskets by treating their effects as related draws from a shared distribution. This can stabilize estimates when a basket has few patients or few responses. But if the baskets are not sufficiently similar, borrowing can pull a genuinely distinct basket effect toward the overall pattern. The result may be a posterior that understates clinically important differences.
Over-shrinkage is not diagnosed by one number. Evidence can come from a combination of predictive discrepancies—outcomes the fitted model has difficulty reproducing—and subgroup posteriors that remain concentrated near the shared pattern despite data suggesting a clinically important departure. These are warning signs to investigate, not automatic proof that the hierarchical model is invalid.
Model Assumptions and What Shrinkage Means
For binary outcomes, a common hierarchical specification is \(Y_j\mid p_j\sim\operatorname{Binomial}(n_j,p_j)\), with \(\operatorname{logit}(p_j)=\theta_j\) and \(\theta_j\mid\mu,\tau\sim N(\mu,\tau^2)\). Here \(Y_j\) is the number of responses in basket \(j\), \(n_j\) is its sample size, \(\theta_j\) is its log-odds of response, \(\mu\) is the distributional center, and \(\tau\) describes between-basket heterogeneity. Priors are also assigned to \(\mu\) and \(\tau\).
This model assumes that outcomes are conditionally independent within baskets given their response probabilities, and that basket effects are exchangeable given \(\mu\) and \(\tau\). Exchangeability means that, under the model, basket labels alone do not determine different effect distributions. It does not mean that the true effects must be identical. A small \(\tau\), however, implies that the model expects the effects to be relatively similar and therefore encourages substantial borrowing.
The amount of shrinkage also depends on how informative each basket is. For a transparent approximation, suppose an estimate \(z_j\) of a basket’s log-odds has sampling distribution \(z_j\mid\theta_j\sim N(\theta_j,s_j^2)\), and suppose \(\theta_j\mid\mu,\tau\sim N(\mu,\tau^2)\), with \(\mu\) and \(\tau\) treated as fixed. Completing the square in the product of the normal likelihood and normal prior gives:
Equivalently, the posterior mean is \(\mu+\lambda_j(z_j-\mu)\), where \(\lambda_j=\tau^2/(\tau^2+s_j^2)\). When \(\tau^2\) is small relative to \(s_j^2\), \(\lambda_j\) is small and the estimate is pulled strongly toward \(\mu\). This normal calculation is a useful explanation of the mechanism; the actual logistic model generally requires posterior computation and also accounts for uncertainty in \(\mu\) and \(\tau\).
Worked Example: How the Heterogeneity Assumption Changes a Subgroup Posterior
Worked Example: How the Heterogeneity Assumption Changes a Subgroup Posterior
Suppose a hypothetical basket has an estimated log-odds \(z_j=1.00\), with standard error \(s_j=0.30\). For illustration, use the normal approximation above and fix the shared center at \(\mu=0.20\). Compare a relatively restrictive heterogeneity value \(\tau=0.20\) with a more permissive value \(\tau=0.50\). The observed estimate corresponds to a response probability of \(\operatorname{logit}^{-1}(1.00)\approx0.731\), while the center corresponds to about \(0.550\).
With \(\tau=0.20\), the weight on the basket estimate is:
The posterior mean is \(0.20+0.3077(1.00-0.20)=0.4462\). The posterior variance is \(0.0400(0.0900)/0.1300=0.02769\), so the posterior standard deviation is \(0.1664\). The posterior probability that the basket log-odds exceeds \(0.80\) is therefore approximately \(1-\Phi((0.80-0.4462)/0.1664)=1-\Phi(2.126)\approx0.01674\).
With \(\tau=0.50\), \(\lambda_j=0.2500/(0.2500+0.0900)=0.7353\), giving posterior mean \(0.20+0.7353(0.80)=0.7882\). The posterior variance is \(0.2500(0.0900)/0.3400=0.06618\), and the standard deviation is \(0.2572\). Now the posterior probability of exceeding \(0.80\) is approximately \(1-\Phi((0.80-0.7882)/0.2572)=1-\Phi(0.0457)\approx0.482\).
The posterior probability changes substantially because the heterogeneity assumption changes. This is a sensitivity warning, not a finding that \(\tau=0.50\) is correct. The clinical threshold and the range of plausible \(\tau\) values should be justified independently, for example through prior calibration and clinical knowledge.
Predictive Discrepancies: Can the Model Reproduce the Basket Pattern?
A posterior predictive check compares observed outcomes with replicated outcomes generated from the fitted model. In each replication, draw model parameters from their posterior distribution and then generate new outcomes under the model. A discrepancy statistic \(T\) summarizes a feature of interest, such as between-basket spread, an unusually high or low response count, or the largest basket-level residual. Compare \(T\) for the observed data with its values across the replicated data.
For example, a dispersion statistic for binary outcomes with equal basket sizes is \(T(Y)=\sum_j(Y_j-n_j\bar p)^2/n_j\), where \(\bar p=\sum_jY_j/\sum_jn_j\). Large values indicate that observed basket counts are spread out relative to their pooled response rate. For each replicated data set, calculate the statistic in the same way, using that data set’s pooled rate. If observed dispersion lies far in the upper tail of the replicated distribution, the model may not represent the observed degree of basket-to-basket variation.
Worked Example: A Posterior Predictive Check for Basket Dispersion
Consider hypothetical response counts of 3, 10, and 15 among 20 patients in each of three baskets. The pooled response rate is \(\bar p=(3+10+15)/60=0.4667\), corresponding to \(20(0.4667)=9.333\) expected responses per basket under a common rate. The observed dispersion is:
Suppose posterior simulation under a deliberately restrictive hierarchical specification produces 61 replicated data sets, out of 4,000, with dispersion at least this large. The estimated upper-tail predictive frequency is \(61/4000=0.01525\), or about 0.0153. Its Monte Carlo standard error is approximately \(\sqrt{0.01525(1-0.01525)/4000}=0.00194\). This hypothetical result flags unusually large observed dispersion under that fitted specification and merits investigation.
It does not mean there is a 1.53% probability that the model is true, or a 98.47% probability that it is false. The posterior predictive frequency is a model-based diagnostic, and the data have been used both to fit the model and to form the discrepancy comparison. The discrepancy should therefore be interpreted alongside the basket-specific posteriors and sensitivity analyses.
A useful check is targeted: ask whether the model reproduces the feature that could make borrowing problematic. An overall fit check can miss one basket’s extreme count if the other baskets fit well. Conversely, a striking raw count may not be surprising after accounting for its sample size and the uncertainty in its response probability. Predictive discrepancies should be defined in advance when possible, and the replicated statistic must be computed in the same way as the observed statistic.
Subgroup Posteriors and Clinical Thresholds
Predictive checks assess whether the model can reproduce features of the data; subgroup posteriors assess what the model implies about each basket’s effect. Inspect the full posterior distribution for a basket, not only its posterior mean. Useful summaries include credible intervals and posterior probabilities for clinically meaningful thresholds. A posterior concentrated near the shared center may be concerning when the basket’s data point away from it, particularly if the conclusion changes markedly under reasonable heterogeneity assumptions.
A posterior probability answers a conditional question: given the model, prior, and observed data, what probability does the posterior assign to a stated parameter event? It is not a probability that the model is correct. The threshold must also have a clinical interpretation. For a log-odds parameter \(\theta_j\), the condition \(\theta_j<\operatorname{logit}(0.40)\) means the basket response probability is below 0.40.
Worked Example: Checking a Basket Against a Clinical Threshold
Suppose a hypothetical basket has \(z_j=-0.50\) with \(s_j=0.25\). Use \(\mu=0.20\) and compare \(\tau=0.20\) with \(\tau=0.50\), as in the normal approximation. The clinical question is whether the response probability is below 0.40, or equivalently whether \(\theta_j<\operatorname{logit}(0.40)\approx-0.4055\).
For \(\tau=0.20\), \(\lambda_j=0.0400/(0.0400+0.0625)=0.3902\). The posterior mean is \(0.20+0.3902(-0.50-0.20)=-0.0732\); its variance is \(0.0400(0.0625)/0.1025=0.02439\), giving standard deviation \(0.1562\). Thus the posterior probability of a response probability below 0.40 is approximately \(\Phi((-0.4055+0.0732)/0.1562)=\Phi(-2.127)\approx0.0167\).
For \(\tau=0.50\), \(\lambda_j=0.2500/(0.2500+0.0625)=0.8000\). The posterior mean is \(0.20+0.8000(-0.70)=-0.3600\), with variance \(0.2500(0.0625)/0.3125=0.05000\) and standard deviation \(0.2236\). The corresponding probability is \(\Phi((-0.4055+0.3600)/0.2236)=\Phi(-0.2035)\approx0.4194\).
The restrictive specification makes the low-response event appear unlikely, while the more permissive specification assigns it appreciably more posterior probability. In an actual analysis, one would use posterior draws from the fitted model rather than this fixed-hyperparameter approximation, and would examine whether the range of results is clinically consequential.
How to Investigate a Warning Sign
When predictive checks and subgroup posteriors raise concern, first verify the data, outcome definition, and basket sizes. Then examine whether a plausible clinical or design feature—such as a difference in eligibility, assessment, or follow-up—was omitted from the model. A discrepancy can reflect misspecification beyond the exchangeability assumption; it does not identify the cause by itself.
Specify the basket pattern or clinical threshold that the model may be obscuring, rather than relying only on a general impression of shrinkage.
Compare the observed targeted discrepancy with replicated values, and note whether the discrepancy is isolated to one basket or reflects broader lack of fit.
Review the posterior distributions and clinically relevant probabilities for the baskets implicated by the check.
Reassess conclusions across defensible heterogeneity priors or model structures. Do not choose a specification solely because it produces a preferred clinical conclusion.
The main alternatives answer different questions. A shared-effect model assumes one common basket effect and therefore cannot represent true basket variation. Independent analyses allow each basket to stand alone, but can be unstable with sparse data and do not borrow across baskets. A hierarchical model lies between these choices; predictive diagnostics and sensitivity analysis help assess whether its particular degree of borrowing is credible. No alternative is automatically a gold standard, and agreement across models is not proof that their assumptions are correct.
| Approach | What it assumes | Use in investigating suspected over-shrinkage |
|---|---|---|
| Shared-effect model | Basket effects are identical. | Shows the limiting case of complete pooling, but cannot describe basket-specific departures. |
| Independent analyses | Each basket is estimated separately. | Provides a no-borrowing comparison, but estimates may be imprecise, especially with sparse data. |
| Hierarchical model | Effects vary around a shared distribution. | Provides partial pooling; check predictive fit, subgroup posteriors, and sensitivity to plausible heterogeneity assumptions. |
Common Mistakes and Limitations
- Treating a small predictive tail frequency as a model-falsity probability. It is a diagnostic comparison under the fitted model, not the posterior probability that the model is wrong.
- Calling any movement toward the center over-shrinkage. Shrinkage is expected under partial pooling. Concern requires evidence that the resulting posterior obscures a clinically important pattern or depends too strongly on restrictive assumptions.
- Inspecting only posterior means or basket rankings. These omit uncertainty and do not directly answer whether a clinical threshold is crossed.
- Using a discrepancy that does not target the concern. A global fit statistic may overlook an extreme subgroup; a targeted statistic should reflect the pattern under investigation.
- Assuming sensitivity proves which model is correct. Different assumptions can produce different answers, but sensitivity analysis reveals dependence on assumptions rather than resolving that dependence automatically.
With few baskets, predictive checks have limited ability to distinguish genuine heterogeneity from chance variation, and posterior results may be highly dependent on the heterogeneity prior. Clinical thresholds can also be uncertain or context-dependent. These diagnostics support transparent judgment about the adequacy of borrowing; they do not replace clinical reasoning or establish that one basket’s effect differs causally from another.
Check Your Understanding
Answer the questions, then compare your reasoning with the solutions below.
- In a normal approximation, what happens to the shrinkage weight \(\lambda_j=\tau^2/(\tau^2+s_j^2)\) when \(\tau\) decreases while \(s_j\) stays fixed?
- What does an unusually large observed between-basket discrepancy relative to posterior predictive replicates suggest, and what does it not establish?
- Why should a basket-specific posterior probability be tied to a clinically meaningful threshold?
- Name one limitation of using independent basket analyses as a comparison with a hierarchical model.
Solutions. (1) The weight decreases, so the posterior mean moves more strongly toward the shared center. (2) It suggests the fitted model may have difficulty reproducing the observed basket variation; it does not give the probability that the model is false. (3) The threshold defines the clinical event being assessed, so the posterior probability has an interpretable decision context rather than being an unspecified statement about the parameter. (4) Independent analyses avoid borrowing but can be imprecise or unstable when basket sample sizes and response counts are small.