Hyperpriors For Between-Subgroup Heterogeneity

Why the Hyperprior Requires Calibration

In a hierarchical treatment-effect model, subgroup effects are allowed to differ, while still being drawn from a common distribution. The between-subgroup standard deviation controls how widely those effects can vary. Its prior therefore affects how much the model can share information across subgroups: a prior concentrated near zero favors effects that are similar, whereas a prior with more mass at larger values allows greater differences. Neither choice is neutral.

Let \(\theta_j\) be the treatment effect for subgroup \(j\), on a specified scale such as log odds ratio. A standard random-effects model is

\theta_j \mid \mu,\tau \sim N(\mu,\tau^2), \qquad j=1,\ldots,J,\qquad \tau\geq 0.

Here \(\mu\) is the mean treatment effect across subgroups, and \(\tau\) is the between-subgroup standard deviation. Conditional on \(\mu\) and \(\tau\), the model assumes subgroup effects are exchangeable and independent draws from the same normal distribution. A hyperprior is a prior distribution on a parameter of this subgroup-level distribution; here it is a prior on \(\tau\), or sometimes on \(\tau^2\). The choice should match the treatment-effect scale and the clinical range of subgroup differences one wants to regard as plausible.

Definition: A hyperprior for heterogeneity is a probability distribution assigned to \(\tau\), the nonnegative between-subgroup standard deviation, before observing the trial outcomes. It expresses prior uncertainty about the degree of subgroup variation; it is not a direct prior on any one subgroup effect.

The prior on \(\tau\) is meaningful only together with the model and the units of \(\theta_j\). For log odds ratios, for example, \(\tau=0.2\) means a standard deviation of 0.2 log-odds-ratio units across subgroup effects. It does not mean that effects differ by exactly 0.2, nor does it directly specify a probability that two effects are clinically different.

Common Hyperprior Forms

Several distributions are used for a nonnegative standard deviation. Parameterizations must be stated, because names such as “half-normal” or “exponential” do not determine a prior without a scale or rate.

Common forms: A half-normal prior with scale \(A>0\) is the distribution of \(|Z|\) when \(Z\sim N(0,A^2)\). A half-\(t\) prior with degrees of freedom \(\nu\) and scale \(A\) is the distribution of \(|T|\) when \(T\) has a Student \(t\) distribution with \(\nu\) degrees of freedom and scale \(A\). An exponential prior with rate \(\lambda>0\) has density \(\lambda e^{-\lambda\tau}\) for \(\tau\geq0\). A uniform prior on \((0,U)\) has density \(1/U\) there and zero outside.

The half-normal is concentrated near zero and has a relatively light tail. The half-\(t\), especially a half-\(t\) with few degrees of freedom, has a heavier tail: it can put more prior probability on large heterogeneity while retaining density near zero. The exponential has a simple monotone decay, with mean \(1/\lambda\). A uniform prior imposes a hard upper bound \(U\), but does not make all positive values equally plausible after they are combined with the likelihood and the hierarchical model.

These distributions are alternatives, not interchangeable versions of “vague.” A half-\(t\) with a small scale may still concentrate strongly on small values; a half-normal with a larger scale can allow substantial variation. For any form, the scale must be interpreted on the scale of the treatment effect. A prior on the variance \(\tau^2\), such as an inverse-gamma distribution, is another option, but its parameters do not translate as directly into plausible standard deviations. It is important to inspect the induced prior on \(\tau\), rather than infer its behavior from the prior’s name.

Worked Example: Calibrating a Half-Normal Scale

Suppose subgroup effects are measured in log odds ratios. Investigators want the 95th percentile of the prior on \(\tau\) to be 0.40 log-odds-ratio units. Find the half-normal scale \(A\), then calculate the prior probability that \(\tau\) exceeds 0.30.

For a half-normal, \(P(\tau\leq t)=2\Phi(t/A)-1\) for \(t\geq0\), where \(\Phi\) is the standard normal cumulative distribution function. Setting the 95th percentile to 0.40 gives

2\Phi(0.40/A)-1=0.95 \quad\Longrightarrow\quad 0.40/A=\Phi^{-1}(0.975)=1.96 \quad\Longrightarrow\quad A=\frac{0.40}{1.96}=0.2041.

Thus the half-normal prior has scale \(A=0.2041\) log-odds-ratio units. Its probability of exceeding 0.30 is

P(\tau>0.30) =2\left[1-\Phi\left(\frac{0.30}{0.40/1.96}\right)\right] =2[1-\Phi(1.470)] \approx 2(1-0.9292) =0.1416.

Under this prior, the probability that the between-subgroup standard deviation exceeds 0.30 is about 14.2%. The calculation is a prior statement about \(\tau\), not a probability about the treatment effect in a particular subgroup. A useful scale is one whose implications can be explained in terms of such thresholds.

How Scale Changes Plausible Variation

A scale parameter changes the range of heterogeneity the prior regards as plausible. To compare forms, it helps to calibrate them to a common feature, such as their median, and then examine other probabilities. Matching one feature does not make priors equivalent: their tails can still differ substantially.

Worked Example: Comparing Half-Normal and Exponential Tails

Consider two priors for \(\tau\), measured in log-odds-ratio units. Give each prior a median of 0.20: a half-normal with scale \(A\), and an exponential with rate \(\lambda\). Compare their probabilities of assigning \(\tau>0.40\).

For a half-normal, its median \(m\) satisfies \(2\Phi(m/A)-1=0.5\), so \(m/A=\Phi^{-1}(0.75)=0.6745\). Therefore

A=\frac{0.20}{0.6745}=0.2965.

For an exponential distribution, its median satisfies \(1-e^{-\lambda m}=0.5\). Hence

\lambda=\frac{\log 2}{0.20}=3.466 \quad\text{per log-odds-ratio unit}.

The tail probabilities are then

P_{\text{half-normal}}(\tau>0.40) =2\left[1-\Phi\left(\frac{0.40}{0.2965}\right)\right] \approx 2[1-\Phi(1.349)] \approx 0.177,

P_{\text{exponential}}(\tau>0.40) =e^{-3.466(0.40)} =0.250.

Both priors have median heterogeneity 0.20, but the exponential assigns more probability to values above 0.40 in this comparison. Matching medians alone would therefore hide a material difference in the prior beliefs about larger heterogeneity. The exact comparison depends on the chosen scale and threshold.

A useful way to make the scale clinically legible is to consider a difference between two subgroup effects. Under the model, conditional on \(\mu\) and \(\tau\), two distinct subgroup effects are independent normal draws. Subtracting them cancels the common mean and adds their variances:

\theta_j-\theta_k\mid\tau \sim N(0,2\tau^2), \qquad j\ne k.

This conditional result gives a bridge between the standard deviation and the spread of subgroup contrasts. It does not, by itself, describe the prior predictive distribution of the contrast, because that distribution must also average over uncertainty in \(\tau\).

Worked Example: Translating a Standard Deviation Into a Subgroup Contrast

Suppose two hypothetical subgroup treatment effects follow the normal random-effects model, and consider the conditional case \(\tau=0.20\) log-odds-ratio units. Find the standard deviation of their difference and its central 95% conditional interval. Express the interval on the ratio-of-odds-ratios scale as well.

The difference has conditional variance \(2\tau^2\), so its standard deviation is

\sqrt{2(0.20)^2}=\sqrt{0.08}=0.2828 \quad\text{log-odds-ratio units}.

A central 95% interval for a normal variable with mean zero is approximately plus or minus 1.96 standard deviations. Thus

0\pm1.96(0.2828) =0\pm0.5543,

giving an interval from \(-0.5543\) to \(0.5543\) log-odds-ratio units for the conditional subgroup contrast. Exponentiating its endpoints gives \(\exp(-0.5543)=0.5745\) and \(\exp(0.5543)=1.741\). On the ratio-of-odds-ratios scale, the corresponding interval is approximately 0.5745 to 1.741. This interval is conditional on \(\tau=0.20\); it is not a 95% prior predictive interval that accounts for uncertainty in \(\tau\).

Prior Predictive Checks and Sensitivity

A prior predictive check asks what outcomes or effect differences the model can generate before seeing the data. For two subgroups, the probability of a contrast larger than a clinically chosen threshold \(d>0\) is

P(|\theta_j-\theta_k|>d) = \int_0^\infty 2\left[1-\Phi\left(\frac{d}{\sqrt{2}\tau}\right)\right] \,p(\tau)\,d\tau,

where \(p(\tau)\) is the chosen hyperprior density. This expression follows by first calculating the conditional normal tail probability at each \(\tau\), then averaging over the prior for \(\tau\). It makes explicit why the prior predictive spread depends on both the conditional model and the hyperprior. For a clinical application, \(d\) should be expressed on an interpretable treatment-effect scale; if effects use the log odds ratio, exponentiating \(d\) gives a multiplicative odds-ratio contrast.

A practical assessment can include several checks:

Specify the treatment-effect scale and its units before selecting a hyperprior scale.
Calculate prior probabilities for clinically meaningful thresholds of \(\tau\) and of pairwise subgroup contrasts.
Inspect prior predictive subgroup effects and contrasts for implausibly extreme values or an unintended concentration near zero.
Repeat the analysis with defensible alternatives, such as a half-normal and a half-\(t\), and with more than one scale.

These checks are diagnostics, not proof that a prior is correct. A prior predictive distribution that includes very large effects may reflect an overly permissive scale, a heavy-tailed form, or a model that is not appropriate for the intended effect scale. Conversely, a prior that almost excludes meaningful differences may force excessive similarity. If results change materially across reasonable hyperpriors, that sensitivity should be reported rather than hidden by choosing one convenient form.

Assumptions, Alternatives, and Limitations

The hyperprior comparisons above assume the exchangeable normal random-effects model is appropriate: subgroup effects are conditionally independent draws from a common distribution with mean \(\mu\) and standard deviation \(\tau\). They also assume \(\tau\) is defined on the same scale as \(\theta_j\). If effects have different meanings across subgroups, or the normal distribution is unsuitable, changing the hyperprior alone does not repair the model.

The main alternatives differ in their compromise between concentration near zero and allowance for large values. A half-normal offers a relatively regular tail and a direct calibration through normal quantiles. A half-\(t\) offers heavier tails, which can be useful when substantial heterogeneity should remain possible, but its scale and degrees of freedom both matter. An exponential is easy to calibrate through its mean or median, while a uniform prior is governed by its upper bound and can be sensitive to that bound. Priors placed on \(\tau^2\) may be convenient in some formulations, but should be checked through the induced distribution on \(\tau\) and through prior predictive implications.

With few subgroups, the data may provide limited information about the spread of subgroup effects. In that setting the prior can have a substantial influence, even when each subgroup has many patients. A prior that behaves well in one treatment-effect scale or trial design need not be suitable in another. Hyperprior selection is therefore part of model specification and clinical calibration, not a routine technical detail.

Common Mistakes

Calling a hyperprior vague without checking its scale. A half-normal or half-\(t\) can strongly favor small heterogeneity if its scale is small. State the scale in treatment-effect units and check what values it permits.
Using a prior on the variance as though it were a prior on the standard deviation. The induced distribution on \(\tau\) need not have the same shape or concentration. Inspect the implied prior on the quantity that describes subgroup spread.
Assuming equal medians mean equal priors. As the calculation above shows, tail probabilities can differ even when the medians match. Compare probabilities at clinically relevant thresholds as well.
Reading a conditional interval as a prior predictive interval. A calculation at fixed \(\tau\) omits uncertainty about \(\tau\). Average over its prior to obtain a prior predictive contrast distribution.
Treating a broad prior as protection against model misspecification. A broad hyperprior does not remove the exchangeability and normality assumptions or guarantee reliable inference when the number of subgroups is small.

Key takeaway: Choose a hyperprior for \(\tau\) by specifying its form and scale, then checking the implied probabilities for clinically meaningful heterogeneity and subgroup contrasts. The distribution’s name alone does not say what treatment-effect variation it regards as plausible.

Check Your Understanding

Answer each question using the random-effects model and the stated prior definitions.

A half-normal prior is assigned to \(\tau\). If its scale \(A\) is doubled, what happens to the distribution of \(\tau\) and to its probability of exceeding a fixed positive threshold?
For an exponential prior with rate \(\lambda=2\) per treatment-effect unit, calculate the median of \(\tau\).
At fixed \(\tau=0.10\), what is the standard deviation of the difference between two subgroup effects under the normal random-effects model?
Why is a half-\(t\) prior not automatically less informative than a half-normal prior?

Solutions. (1) The half-normal rescales: doubling \(A\) doubles every quantile of \(\tau\), so the probability of exceeding a fixed positive threshold increases. (2) Solve \(1-e^{-2m}=0.5\), giving \(m=\log(2)/2=0.3466\) treatment-effect units. (3) The conditional difference has standard deviation \(\sqrt{2}\tau=\sqrt{2}(0.10)=0.1414\) treatment-effect units. (4) Informativeness depends on both the form and its scale (and, for a half-\(t\), its degrees of freedom). A half-\(t\) with a sufficiently small scale can concentrate strongly near zero despite its heavier tail.