Why examine hyperprior scale?
In a basket trial, a hierarchical model allows treatment effects to differ across baskets while relating them through a shared distribution. The scale of the prior on between-basket variation helps determine how strongly basket estimates are pulled toward a common value. A prior scale that favors little variation can produce substantial pooling; a larger scale can permit more basket-specific estimates. When data are sparse, that choice may influence conclusions enough to matter clinically.
Sensitivity analysis asks a targeted question: if the likelihood, basket data, and structural assumptions stay fixed, how much do subgroup conclusions change under other defensible hyperprior scales? This is not a search for the scale that produces a preferred result. The alternative scales should represent plausible beliefs about heterogeneity, informed where possible by the earlier material on calibrating heterogeneity priors against plausible effects.
A simple normal hierarchy and its assumptions
Let \(y_j\) be an approximately normal estimate of the treatment effect in basket \(j\), measured in a common effect unit, and let \(s_j\) be its known standard error. A basic model is
Here \( \theta_j \) is the basket-specific effect, \( \mu \) is the population-level mean effect, and \( \tau \) is the between-basket standard deviation. The subscript \(A\) denotes the scale choice in the hyperprior family. Conditional on \( \theta_j \), the estimate \(y_j\) has sampling variance \(s_j^2\); conditional on \( \mu,\tau \), basket effects are exchangeable draws from a distribution centered at \( \mu \). This setup also assumes that the estimates are conditionally independent across baskets and that the normal likelihood is an adequate approximation.
For transparent arithmetic below, examples use a deliberately simplified version: \( \mu=0 \) is fixed, all \(s_j=s\), and the prior on \( \tau \) has two possible values. Fixing \( \mu \) is an illustrative assumption, not a general recommendation. In an analysis with uncertain \( \mu \), its prior and posterior must be included consistently when recomputing every scale scenario.
The factor \(k(\tau)\) is the weight on the basket’s own estimate: as \( \tau \) increases, \(k(\tau)\) increases and the estimate is pulled less toward zero. This conditional result alone is not the full posterior when \( \tau \) is uncertain. Each possible \( \tau \) must also be weighted by its posterior probability given all the baskets’ data.
Worked Example: The effect of a fixed heterogeneity value
Worked Example: The effect of a fixed heterogeneity value
Suppose one hypothetical basket has estimate \(y_1=1.00\) effect units and standard error \(s=0.40\), and \( \mu=0 \). First condition on \( \tau=0.20 \). Then
The conditional posterior variance is \(v(0.20)=0.04(0.16)/0.20=0.0320\), so its standard deviation is \(\sqrt{0.0320}=0.179\) effect units. Under the alternative fixed value \( \tau=0.80 \),
and \(v(0.80)=0.64(0.16)/0.80=0.128\), with standard deviation \(0.358\) effect units. These are conditional estimates under two specified heterogeneity values, not estimates averaged over a hyperprior. The calculation isolates the direction of shrinkage: a larger \( \tau \) lets this basket’s estimate carry more weight.
Recomputing posterior weights under alternative scales
When \( \tau \) has a prior distribution, data from every basket contribute information about it. In the simplified model, integrating out \( \theta_j \) gives
Thus, for a proposed value \(t\) of \( \tau \), the likelihood across \(J\) baskets is proportional to
For a discrete prior with possible values \(t_r\) and prior probabilities \(q_r\), Bayes’ rule gives posterior weights \(w_r\):
The basket-specific posterior mean is the weighted average of its conditional posterior means, \( \sum_r w_r k(t_r)y_j \). The weights must come from the same model and data as the conditional means being averaged. Mixing weights from one fit with conditional estimates from another does not produce a coherent posterior summary.
Worked Example: Recomputing three basket estimates under two hyperprior scales
Consider three hypothetical baskets with estimates \( (y_1,y_2,y_3)=(1.00,-0.20,0.10) \) effect units, each with standard error \(s=0.40\). Keep the fixed mean \( \mu=0 \), normal likelihood, and equal prior probabilities for two heterogeneity values. Compare two scale choices \(A\), each defining the prior support as \( \tau\in\{0.25A,A\} \). This is a simple discrete scale family for demonstrating the calculation; practical analyses may use continuous hyperpriors.
First take \(A=0.25\), so the possible values are \(0.0625\) and \(0.25\). The sum of squared basket estimates is \(1.00^2+(-0.20)^2+0.10^2=1.05\). The marginal variances of each estimate under the two states are \(v_1=0.40^2+0.0625^2=0.1639\) and \(v_2=0.40^2+0.25^2=0.2225\). With equal prior probabilities, the posterior odds of the higher-\(\tau\) state relative to the lower-\(\tau\) state are the likelihood ratio:
Therefore the posterior probability of the higher-\(\tau\) state is \(1.47/(1+1.47)=0.595\); the lower state has weight \(0.405\). For basket 1, the conditional means are \(k(0.0625)(1.00)=0.0238\) and \(k(0.25)(1.00)=0.281\), because the corresponding shrinkage factors are \(0.003906/0.163906=0.0238\) and \(0.0625/0.2225=0.281\). Its posterior mean is therefore
Now take \(A=0.80\), giving \( \tau=0.20 \) or \(0.80\), again with equal prior probabilities. The marginal variances are \(0.20\) and \(0.80\). The higher-state likelihood ratio and posterior weight are
The conditional basket 1 means are \(0.20(1.00)=0.200\) and \(0.80(1.00)=0.800\). Hence
The higher-scale prior does not simply force the posterior into the high-variation state: the data give that state weight \(0.472\) here. Nevertheless, the posterior mean for basket 1 rises from \(0.177\) to \(0.483\) effect units because the conditional means under both states allow more variation. The values are internally consistent: each set of weights and conditional means was computed under its own specified scale model.
The same calculation applies to the other baskets. Their conditional means are their own \(y_j\) multiplied by the same state-specific shrinkage factors. Thus, for \(A=0.25\), the posterior means are approximately \(0.177,-0.0354,0.0177\) effect units. For \(A=0.80\), they are approximately \(0.483,-0.0966,0.0483\) effect units. These differences reflect the shared posterior weights and the common shrinkage factors, not separate hand-chosen basket adjustments.
Uncertainty and clinically relevant conclusions
A posterior mean alone can conceal changes in uncertainty. For a discrete mixture, the posterior variance of \( \theta_j \) is the average conditional variance plus the variance of the conditional means:
The first term represents uncertainty within each heterogeneity state; the second represents uncertainty about which state is appropriate. This follows by writing the total variance as the expected conditional variance plus the variance of the conditional expectation. Omitting the second term understates uncertainty whenever the conditional means differ across states.
Worked Example: Does a clinically meaningful threshold conclusion change?
Use the three-basket data and models above. Suppose, hypothetically, that a positive effect exceeding \(0.30\) units is the threshold of clinical interest for basket 1. Within each heterogeneity state the conditional posterior is normal, so \(P(\theta_1>0.30\mid\mathbf y)\) is the weighted sum of state-specific normal tail probabilities.
For \(A=0.25\), the conditional standard deviations for basket 1 are \(\sqrt{0.003906(0.16)/0.163906}=0.0617\) and \(\sqrt{0.0625(0.16)/0.2225}=0.212\). The corresponding standardized thresholds are \((0.30-0.0238)/0.0617=4.48\) and \((0.30-0.281)/0.212=0.0896\). The normal tail probabilities are approximately \(0.000004\) and \(0.464\). Using the posterior weights gives
For \(A=0.80\), the state-specific posterior standard deviations are \(\sqrt{0.032}=0.179\) and \(\sqrt{0.128}=0.358\), and the standardized thresholds are \((0.30-0.200)/0.179=0.559\) and \((0.30-0.800)/0.358=-1.40\). The corresponding tail probabilities are approximately \(0.288\) and \(0.919\). Therefore
Under the smaller scale, the posterior probability that basket 1 exceeds the hypothetical threshold is about \(0.276\); under the larger scale it is about \(0.586\). If a prespecified decision rule required this probability to exceed \(0.50\), the conclusion would change across these two scale choices. This is evidence that the decision is sensitive to assumptions about between-basket variation, not evidence that either scale is correct.
How to conduct and interpret the sensitivity analysis
Specify a primary hyperprior and alternatives that span clinically plausible between-basket variation. Keep the effect scale and parameterization explicit; a scale on a log-odds effect is not numerically interchangeable with one on a response probability.
Use the same data, likelihood, basket definitions, covariates, and prior choices apart from the hyperprior scale under study. Otherwise a changed result cannot be attributed specifically to the scale.
For each scale, report subgroup estimates and uncertainty, and any prespecified clinically relevant probability or decision summary. Do not average conditional estimates with posterior weights from a different model.
Distinguish stable conclusions from sensitive ones. If a subgroup claim changes materially, state that its evidence depends on assumptions about heterogeneity and present that dependence rather than selecting the favorable scenario.
In a continuous hyperprior model, posterior summaries are integrals over \( \tau \), usually evaluated by numerical integration or posterior simulation rather than a finite weighted sum. For each scale choice, check computational convergence and adequate exploration of the posterior for \( \tau \), particularly when data are sparse. Also examine whether the prior predictive distribution places plausible probability on basket effects of clinically credible magnitudes. A mathematically converged fit can still rely on an implausible scale.
Sensitivity to hyperprior scale is distinct from a diagnostic of model fit. If residual patterns or predictive checks indicate that baskets do not share the assumed structure, changing the scale alone may not repair the model. The analysis also cannot establish that baskets are exchangeable; that assumption requires substantive justification. Nor does a stable posterior mean guarantee a stable interval, tail probability, or clinical decision.
Common mistake and comparison with alternatives
Common mistake: “The larger scale gave a more favorable estimate, so it is the better prior.” This reasoning selects an assumption by its result. The correction is to justify the range of scales independently, show the same estimand and decision summaries under each, and describe any resulting sensitivity transparently. The prior should be calibrated to plausible heterogeneity, not tuned to a desired subgroup conclusion.
A fixed-\(\tau\) analysis conditions on one heterogeneity value and is useful for understanding how pooling changes, as in the first worked example. A hyperprior analysis averages over uncertainty in \( \tau \), which includes uncertainty about the heterogeneity value as well as conditional uncertainty in each basket effect. A sensitivity analysis compares this averaging under alternative prior scales; it is not equivalent to choosing the scale with the largest posterior weight in one fit. In every approach, the likelihood and exchangeability assumptions remain important limitations.
Check Your Understanding
Answer the questions, then compare with the solutions below.
- In the normal hierarchy, what does the scale of the prior on \( \tau \) govern?
- Why must posterior weights and conditional basket estimates come from the same model?
- In the three-basket example, did the probability that basket 1 exceeded \(0.30\) units cross the stated \(0.50\) decision threshold when the scale changed?
- Give one reason that a converged computation is not sufficient to establish that a hyperprior scale is suitable.
Solutions. (1) It governs the prior distribution of between-basket standard deviation and thereby influences how much variation the model permits. (2) The posterior mean is a mixture whose weights and component estimates must describe the same posterior distribution; otherwise the weighted average has no coherent Bayesian interpretation. (3) Yes: it was approximately \(0.276\) for \(A=0.25\) and \(0.586\) for \(A=0.80\), so it crossed \(0.50\). (4) The prior predictive distribution could still imply implausible basket effects, or the exchangeability and likelihood assumptions could be unsuitable.