Calibrating Heterogeneity Priors Against Plausible Effects

Why Calibrate a Heterogeneity Prior?

In a basket trial, the heterogeneity parameter controls how much treatment effects may differ across subgroups. A prior scale for this parameter is therefore consequential: a scale that permits implausibly large differences can make the model inconsistent with clinical knowledge, while an overly restrictive scale can make meaningful differences difficult to represent. The scale itself is not usually interpretable in clinical units. Calibration translates it into effects and contrasts that are.

The task is not to declare a particular amount of heterogeneity “correct” from a formula alone. It is to choose candidate prior scales, work out what subgroup differences they imply, and assess whether those differences are compatible with clinical understanding of the baskets and endpoint. The calculations below concern prior implications, not the posterior estimates after observing trial data.

Model and Effect Scale

Let \(\theta_j\) be the treatment effect in basket \(j\), measured on a specified scale, and let \(\mu\) be the mean treatment effect across the basket hierarchy. A normal random-effects model is

Conditional on \(\mu\) and \(\tau\), the basket deviations \(u_j\) are assumed independent and identically distributed. The parameter \(\tau\geq 0\) is the between-basket standard deviation on the treatment-effect scale. It is not the standard error of an estimated basket effect, nor does it describe patient-level variation. The model also assumes that the baskets are sufficiently coherent for this common distribution of deviations to be meaningful.

The effect scale must be fixed before interpreting \(\tau\). For example, a binary endpoint might use a log odds ratio, a time-to-event endpoint a log hazard ratio, and a continuous endpoint a difference in means or a standardized mean difference. A numerical standard deviation has different clinical meanings on each scale. When effects are expressed as log odds ratios, exponentiation converts a difference in effects into a ratio of odds ratios.

Translate a Heterogeneity Value Into Effect Ranges

For a fixed value of \(\tau\), approximately 95% of the normal distribution of basket deviations lies within \(1.96\tau\) of zero. Thus a conditional central 95% range for a basket effect is approximately \(\mu\pm1.96\tau\). This is a range for an individual basket deviation around the hierarchy mean; it does not say that all baskets in a trial with many baskets will lie inside it.

A pairwise comparison between baskets is often more directly relevant to clinical plausibility. For two distinct baskets \(j\) and \(k\), define \(\Delta_{jk}=\theta_j-\theta_k\). Because the common mean cancels, this contrast describes the difference in their treatment effects, not the difference between either treatment effect and zero.

The variance follows from conditional independence: \(\operatorname{Var}(u_j-u_k\mid\tau)=\operatorname{Var}(u_j\mid\tau)+\operatorname{Var}(u_k\mid\tau)=2\tau^2\). Consequently, a conditional central 95% range for a pairwise difference is approximately

This calculation is a useful calibration tool: if clinicians judge differences larger than \(D\) on the chosen effect scale to be implausible, compare \(D\) with \(1.96\sqrt{2}\tau\). Conversely, a value of \(\tau\) for which this half-width equals \(D\) is \(D/(1.96\sqrt{2})\). This is a conditional normal range for a specified \(\tau\), not a guarantee about every pair of baskets.

Calibrating a Prior on \(\tau\)

In many analyses \(\tau\) is unknown and receives a prior distribution. For example, a half-normal prior with scale \(A\) is defined as the distribution of the absolute value of a normal random variable with mean zero and standard deviation \(A\):

Here \(A\) is the scale of the prior distribution on \(\tau\); it is not the heterogeneity standard deviation itself. To interpret \(A\), inspect useful quantiles of the prior for \(\tau\), then translate those values into conditional effect ranges. For a half-normal prior, the 90th percentile of \(\tau\) is approximately \(1.645A\), since \(P(\tau\leq t)=2\Phi(t/A)-1\). The resulting effect range is conditional on \(\tau\) being at that quantile. It is not a 95% prior-predictive interval for treatment effects.

Calibrate Against a Clinical Difference

Clinical elicitation is often clearest when expressed as a pairwise difference that would be considered large but still plausible. Suppose the largest clinically plausible absolute pairwise difference is \(D\) on the model’s effect scale. A value of \(\tau\) whose conditional central 95% pairwise half-width equals \(D\) is

This calculation supplies a reference point for choosing a prior scale. It does not automatically identify the right prior: the whole distribution of \(\tau\) matters, and a prior can assign substantial probability both below and above this reference value.

Justification and Prior-Predictive Checks

The pairwise formula follows directly from the model. Conditional on \(\tau\), the deviations in two distinct baskets are independent with mean zero and variance \(\tau^2\). Their difference has mean zero and variance \(2\tau^2\), so for \(\tau>0\), standardizing gives \(\Delta_{jk}/(\sqrt{2}\tau)\sim N(0,1)\). When \(\tau=0\), \(\Delta_{jk}=0\) with probability one. The normal central 95% quantiles are approximately \(-1.96\) and \(1.96\), which yields the stated conditional range. The common mean \(\mu\) does not appear because it cancels in the contrast.

A useful check for a prior distribution on \(\tau\) is prior-predictive simulation: draw \(\tau\) from its prior, draw basket deviations from the hierarchical model, and examine the resulting effects and pairwise contrasts on the clinical scale. For each simulated draw, transform log odds ratios or log hazard ratios back to the ratio scale when that is how clinicians naturally describe differences. Check quantiles and the frequency of effects or contrasts beyond prespecified plausibility thresholds. This assesses the implications of the entire prior, rather than just one selected value of \(\tau\).

For a half-normal prior with scale \(A\), the conditional variance of a pairwise contrast is \(2\tau^2\). Since \(E(\tau^2)=A^2\) for this prior, the prior-predictive variance of that contrast is \(2A^2\), and its standard deviation is \(\sqrt{2}A\). But the prior-predictive contrast is a mixture of normal distributions, not generally a single normal distribution. Multiplying \(\sqrt{2}A\) by 1.96 therefore does not give its exact central 95% interval.

The number of baskets also matters when interpreting ranges. A conditional central 95% range for one basket deviation does not imply that all baskets lie inside that range, and a pairwise range for one selected pair does not bound the most extreme difference among many pairs. If the clinical concern is an extreme basket or the largest difference across the planned baskets, simulate the full set of basket effects under the proposed prior and assess that quantity directly.

Choosing Among Calibration Approaches

Different approaches answer different questions. A fixed \(\tau\) makes the implied conditional ranges direct to calculate, but treats the degree of heterogeneity as known. A prior such as a half-normal allows uncertainty about \(\tau\); its scale must be interpreted through the prior’s quantiles and predictive implications. A prior-predictive simulation handles the complete hierarchy and can assess extremes across all baskets, but it requires explicit choices about the effect scale and the clinical thresholds used to judge plausibility.

A common-effect model corresponds to \(\tau=0\), which permits no between-basket variation in this hierarchy. It may be a useful reference point, but it is not a calibration of a positive heterogeneity prior. Conversely, an interval such as \(\mu\pm1.96\tau\) is a conditional distributional range, not a bound on the possible effects. The appropriate approach depends on whether the clinical question concerns a typical basket, a pair of baskets, or the most extreme difference across a collection.

Common Mistakes and Limitations

• Treating the prior scale as an effect bound. A half-normal scale \(A\) is not a 95% maximum for \(\tau\), and neither \(A\) nor \(\tau\) is itself a maximum effect difference. Use prior quantiles and translate them through the model.
• Confusing individual and pairwise ranges. A single basket’s deviation has standard deviation \(\tau\); a difference between two independent basket deviations has standard deviation \(\sqrt{2}\tau\). Use the quantity that matches the clinical question.
• Ignoring the effect scale. A value of \(\tau\) on the log odds-ratio scale cannot be read as a difference in percentage points. Transform it or calibrate directly on an appropriate scale.
• Calling a conditional range a prior-predictive interval. A normal range calculated at a fixed \(\tau\) does not integrate over uncertainty in \(\tau\). Use simulation or an appropriate mixture calculation for prior-predictive statements.
• Assuming all baskets fall within a typical range. The chance that at least one basket or pair is more extreme increases with the number of baskets. Assess the full collection if extremes are the concern.

Calibration depends on the quality of the clinical judgments and on whether the hierarchical normal model is a reasonable representation of basket effects. If baskets have distinct biological mechanisms, a single exchangeable distribution may not be appropriate even when a prior scale appears numerically plausible. Calibration cannot repair a poorly defined subgroup structure or an unsuitable effect model.