Interpreting The Heterogeneity Parameter

What the Heterogeneity Parameter Measures

In a basket trial, the treatment effect may differ across clinical subgroups even when those subgroups are treated as exchangeable. A hierarchical model represents that variation with a distribution for subgroup effects. The parameter that controls the spread of this distribution is often called the between-subgroup heterogeneity standard deviation. Its interpretation depends on the scale on which the effects are modeled.

Let \(\theta_j\) denote the treatment effect for subgroup \(j\), and let \(\mu_\theta\) denote the mean of the subgroup-effect distribution. A common model is

Here \(\tau\geq 0\) is the between-subgroup standard deviation. Conditional on the shared parameters, the model treats subgroup effects as draws from a normal distribution centered at \(\mu_\theta\), with variance \(\tau^2\). Thus \(\tau\) measures the modeled spread of the subgroup effects around their mean, in the units of \(\theta_j\). It is not a standard error for an individual subgroup estimate and is not the uncertainty in \(\mu_\theta\).

That last qualification is essential. If \(\theta_j\) is a risk difference, \(\tau\) is measured in probability units. If \(\theta_j\) is a log odds ratio, \(\tau\) is measured in log-odds-ratio units—not odds-ratio units. A numerical value of \(\tau\) cannot be judged without first identifying the effect scale and coding direction.

The model describes variation across the subgroup effects, not variation among patients within a subgroup. Patient-level outcome variability is represented elsewhere in the outcome model. Likewise, \(\tau\) is not the observed standard deviation of the subgroup estimates: those estimates also reflect sampling error, which can be substantial in small baskets.

From a Standard Deviation to Effect Differences

A standard deviation is a scale summary, not a statement that every pair of subgroup effects differs by exactly \(\tau\). Under the normal model, about 68% of subgroup effects lie within one \(\tau\) of \(\mu_\theta\), and about 95% lie within roughly \(1.96\tau\), when the distributional assumptions hold. These are proportions under the modeled distribution conditional on its parameters; they are not credible intervals for a particular subgroup’s effect.

A useful translation is to compare two subgroup effects. Conditional on \(\mu_\theta\) and \(\tau\), suppose the effects for distinct subgroups are independent draws from the stated normal distribution. Their difference has mean zero and variance \(2\tau^2\). The derivation is short but important:

The variance adds because the two subgroup effects are conditionally independent. Consequently, a difference of one \(\tau\) is not the standard deviation of a pairwise difference; that quantity is \(\sqrt{2}\tau\). Under the same normal model, roughly 95% of pairwise differences fall between \(-1.96\sqrt{2}\tau\) and \(1.96\sqrt{2}\tau\). This describes the model’s conditional distribution of differences, not the posterior uncertainty about a realized pairwise contrast.

On a transformed scale, it can be helpful to translate differences back to the clinical effect scale. For a log odds ratio, for example, a difference of \(d\) between two subgroup effects corresponds to an odds-ratio ratio of \(\exp(d)\). The direction of that ratio depends on how treatment effects and subgroup contrasts are coded.

Interpreting Other Effect Scales

On an untransformed scale, interpretation is often more direct, but the scale’s allowable range matters. For a risk difference, effects are measured in probability units, so multiplying by 100 expresses them in percentage points. For a log hazard ratio, \(\tau\) is on the log hazard-ratio scale; exponentiating a difference between subgroup effects gives a ratio of hazard ratios. These translations are useful only when the underlying effect definition and model assumptions are appropriate.

Posterior Uncertainty and Model Checks

In Bayesian analysis, \(\tau\) is unknown and has a posterior distribution, informed by the data and the hyperprior. A posterior mean or median of \(\tau\) summarizes that distribution, while a posterior credible interval communicates uncertainty about its value. For example, a posterior probability such as \(P(\tau<0.20\mid\text{data})\) is a probability statement about the heterogeneity parameter conditional on the model and observed data. It is not the probability that all subgroup effects are close together.

When making a statement about a pairwise difference, uncertainty in \(\tau\), \(\mu_\theta\), and the subgroup effects also matters. The arithmetic \(\sqrt{2}\tau\) applies conditional on fixed hyperparameters and the conditional independence assumption. It is not, by itself, a posterior interval for the difference between two named subgroups. Keep the interpretation of the model parameter distinct from posterior summaries of specific subgroup contrasts.

Several checks help prevent overinterpretation:

• Confirm the effect scale. A value of 0.20 means different things for a risk difference, a log odds ratio, and a standardized mean difference.
• Check the treatment-effect coding. Reversing the treatment contrast changes signs. It does not change the standard deviation, but it changes which direction is described as beneficial.
• Read the posterior spread, not just a point summary. A wide posterior for \(\tau\) means the data do not pin down the amount of heterogeneity precisely.
• Assess whether the assumed distribution is plausible. A normal distribution on a bounded scale may permit impossible values, while a symmetric distribution may poorly represent genuinely asymmetric subgroup effects.
• Separate effect heterogeneity from estimation noise. Subgroup estimates can appear widely scattered because they are imprecise; their observed spread is not automatically an estimate of \(\tau\).

How the Interpretation Relates to Pooling

The heterogeneity standard deviation gives a model-scale description of how much subgroup effects vary; it also clarifies the assumptions behind different pooling strategies. In a complete-pooling model, all subgroup effects are constrained to be equal, so there is no between-subgroup spread. In a no-pooling analysis, each effect is estimated separately and no common distribution governs their variation. A hierarchical model instead allows variation while relating subgroup effects through a shared distribution. The familiar distinction among complete pooling, no pooling, and partial pooling is useful here, but \(\tau\) should not be treated as a direct numerical measure of how much borrowing occurs: borrowing also depends on the data and other model features.

A particularly common mistake is to say, “\(\tau=0.25\), so subgroup odds ratios differ by 0.25.” That reasoning treats a log odds ratio as though it were an odds ratio. The correction is to keep the scale explicit: \(\tau=0.25\) is a standard deviation of log odds ratios; a log-scale difference \(d\) translates to an odds-ratio ratio \(\exp(d)\). Another mistake is to call \(\tau\) the standard error of a subgroup treatment effect. It measures modeled between-subgroup dispersion; uncertainty about an individual effect is a separate posterior quantity.