Diagnosing Heterogeneity in Pooled Dose-Response Evidence

Why Inspect Heterogeneity?

A pooled dose-response estimate can summarize evidence across Phase 2 trials, but the summary may conceal meaningful differences between trials. If the estimated effect at a dose is similar across studies, a common summary may be useful. If effects vary substantially, the pooled mean may describe no particular trial well, and a single dose recommendation may not transfer reliably across settings.

The target of this tutorial is between-trial heterogeneity in a dose-response effect. Before assessing it, define the effect being compared: for example, the estimated change from a reference dose to a specified dose at a specified treatment duration, on a common endpoint scale. Trial estimates must refer to a sufficiently aligned estimand and be oriented in the same direction. Differences in endpoint, time point, population, dose contrast, or analysis can create apparent variation that is not simply random variation around one shared effect.

A common random-effects model represents the underlying effect in trial \(i\) by \(\theta_i\):

This model assumes that trial estimates are independent, their sampling variances are adequately estimated, and the trial-specific effects are draws from a distribution with mean \(\mu\) and variance \(\tau^2\). It also assumes that the effects being pooled are meaningfully comparable. The model does not explain why effects differ, and the normal distribution for trial effects may be an imperfect approximation, particularly when there are few trials.

The marginal model is \(y_i\sim N(\mu,v_i+\tau^2)\). The fixed-effect model is the special case \(\tau^2=0\): every trial is assumed to share one underlying effect, with observed differences attributed to sampling error. Neither assumption should be selected merely because it produces a narrower interval.

Quantifying Variation

A useful first diagnostic is Cochran’s \(Q\), which measures weighted dispersion around the fixed-effect pooled estimate. Define \(w_i=1/v_i\) and \(\bar{y}_w=\sum_i w_i y_i/\sum_i w_i\). Under independent, approximately normal estimates with known sampling variances and a common underlying effect, \(Q\) has approximately a chi-square distribution with \(k-1\) degrees of freedom.

A small p-value for \(Q\) is evidence against the common-effect assumption, not a measure of how large or clinically important the variation is. With few trials, the test can have low power; a non-significant result does not establish homogeneity.

The \(I^2\) statistic expresses the estimated share of observed dispersion that is attributed to heterogeneity rather than sampling error. For \(Q>0\), use the usual formula below; when \(Q=0\), define \(I^2=0\), so the statistic is defined in every case.

This is a descriptive, scale-free measure; it is not the percentage of trials that differ, nor does it report the size of the variation in endpoint units. For that, estimate \(\tau^2\) or \(\tau\).

One widely used moment estimator is the DerSimonian–Laird estimate. It compares observed \(Q\) with the amount expected under a common effect, then divides the excess by a weight-based factor. The estimate is truncated at zero because a variance cannot be negative.

The expression for \(C\) matters: the numerator in its second term is \(\sum_i w_i^2\), not \((\sum_i w_i)^2\). For equal sampling variances and \(k\) trials, each \(w_i=w\), so \(C=kw-kw^2/(kw)=(k-1)w\), which is positive when \(k>1\).

Why the Moment Estimator Has This Form

The estimator follows from matching the expected value of \(Q\) to its value under a random-effects model. With fixed weights \(w_i=1/v_i\), the weighted mean removes one degree of freedom. Sampling variation therefore contributes \(k-1\) to \(E(Q)\). Variation in the underlying effects contributes \(\tau^2 C\). In outline, the weighted residual sum of squares can be written as

For independent trial effects with variance \(\tau^2\), the extra contribution to the expectation of the first term is \(\tau^2\sum_i w_i\). The extra contribution to the subtracted term is \(\tau^2\sum_i w_i^2/\sum_i w_i\). Their difference is \(\tau^2 C\). Thus \(E(Q)=(k-1)+\tau^2C\). Equating this expectation to the observed \(Q\) gives the untruncated moment estimate \([Q-(k-1)]/C\); taking the maximum with zero yields the stated estimator.

Pooling Under a Random-Effects Model

Once \(\tau^2\) is estimated, random-effects weights give less relative weight to highly precise trials than fixed-effect weights do, because each trial’s total variance includes both sampling variance and between-trial variance. The pooled estimate and its plug-in standard error are

The estimate describes the mean of the assumed distribution of trial-specific effects, not necessarily the effect in every trial. The usual normal-approximation confidence interval, \(\hat{\mu}_{RE}\pm1.96\,SE(\hat{\mu}_{RE})\), quantifies uncertainty about that mean under the fitted model. It is not a range intended to contain most trial-specific effects. The plug-in standard error treats the estimated \(\tau^2\) as if it were known, an approximation that can understate uncertainty when \(k\) is small.

Influence, Diagnostics, and Decision Relevance

A heterogeneity statistic alone does not show whether variation is broadly shared or driven by one unusual trial. Inspect the trial estimates and their uncertainty in a forest plot, then repeat the pooled analysis after omitting each trial in turn. Compare the pooled mean, \(\hat{\tau}^2\), and the resulting dose implication. A large change identifies sensitivity to that trial; it does not establish that the trial is erroneous or should be excluded.

A common mistake is to treat a non-significant \(Q\) test as proof that a common dose recommendation is justified. The test can miss heterogeneity when only a few trials are available. Another mistake is to interpret a large \(I^2\) as necessarily important in clinical terms: \(I^2\) depends on both heterogeneity and sampling precision, whereas \(\tau\) is expressed in effect units. Inspect both the size of \(\tau\) and the range of effects compatible with the evidence.

The main alternative is a fixed-effect analysis, which answers a different question: it estimates one common effect under the assumption \(\tau^2=0\). A random-effects analysis allows trial effects to vary and targets their mean. Neither method repairs differences in estimands. If trials differ systematically in a way that can be represented and justified, a stratified or covariate-based analysis may be more informative than a single pooled summary; with few trials, such analyses are fragile and should be interpreted cautiously. A common recommendation is more defensible when the estimands align, the pooled result is not dominated by one trial, and plausible trial-specific effects support the same practical choice. If those conditions fail, report the pooled result with its limitations and qualify the recommendation rather than treating the average as universal.

Diagnostics should also look for unusual standard errors, inconsistent effect direction, and a relationship between trial characteristics and estimated effects. A forest plot can reveal these patterns, but neither visual overlap of confidence intervals nor a single heterogeneity statistic is a complete diagnostic. The number of trials is often too small to identify the cause of variation reliably. Sensitivity analyses can show how conclusions depend on assumptions; they cannot by themselves distinguish genuine effect modification from design or measurement differences.