Meta-Analysis of Treatment Effects Across Populations

Why Synthesize Treatment Effects Across Populations?

A treatment effect estimated in one population is not automatically the effect that will occur in another. When several populations have comparable treatment-effect estimates, meta-analysis provides a structured way to summarize the evidence and assess how consistent those estimates are. The central distinction is between estimating a shared average effect and predicting the effect for a particular new population. Those are different questions, and a narrow interval for an average does not necessarily imply a narrow range of effects across populations.

The analysis is meaningful only if the studies estimate sufficiently comparable treatment effects. Differences in endpoint definition, follow-up time, treatment contrast, or analysis population can make a numerical synthesis misleading. A meta-analysis quantifies evidence under its model; it does not make incomparable estimands comparable or establish that a new population is exchangeable with those observed.

Population-Specific Effects and the Two Models

Definition: Let \(y_i\) be the estimated treatment effect in population \(i\), with estimated within-population standard error \(s_i\) and variance \(v_i=s_i^2\), for \(i=1,\ldots,k\). The underlying true effect in population \(i\) is \(\theta_i\). A fixed-effect (common-effect) model assumes \(\theta_i=\mu\) for every population. A random-effects model assumes \(\theta_i\) varies across populations, with \(\theta_i\sim N(\mu,\tau^2)\). Here, \(\mu\) is the mean of the distribution of true population effects and \(\tau^2\) is their between-population variance.

The usual working sampling model is \(y_i\mid\theta_i\sim N(\theta_i,v_i)\), with independent estimates across populations and sufficiently well-estimated within-population variances. The random-effects model therefore separates two sources of variation: sampling error, represented by \(v_i\), and genuine variation in effects, represented by \(\tau^2\). It also assumes the included populations can reasonably be treated as draws from a relevant distribution of populations. That assumption deserves particular scrutiny when the purpose is to predict an effect for a specific target population.

Under the fixed-effect model, the inverse-variance weight for population \(i\) is \(w_i=1/v_i\). The pooled estimate is the weighted mean; its standard error follows from independent estimates with known variances. These expressions also provide the starting point for diagnosing whether a common effect is a plausible description.

\hat{\mu}_{F}=\frac{\sum_{i=1}^{k}w_i y_i}{\sum_{i=1}^{k}w_i}, \qquad SE(\hat{\mu}_{F})=\sqrt{\frac{1}{\sum_{i=1}^{k}w_i}}.

Worked Example: A Fixed-Effect Summary and Its Consistency Check

Suppose three hypothetical populations have estimated treatment effects of 1, 2, and 0 points, with standard errors 0.5, 0.5, and 1.0 points. Assume the outcome, contrast, and follow-up are comparable. Under a common-effect model, the variances are 0.25, 0.25, and 1.00 points squared, giving weights 4, 4, and 1.

The pooled estimate and standard error are

\hat{\mu}_{F}=\frac{4(1)+4(2)+1(0)}{4+4+1} =\frac{12}{9}=1.333\text{ points}, \qquad SE(\hat{\mu}_{F})=\sqrt{\frac{1}{9}}=0.3333\text{ points}.

Using a normal critical value of 1.96, the approximate 95% confidence interval is \(1.333\pm1.96(0.3333)\), or 0.680 to 1.987 points. This interval describes uncertainty about the common effect under the fixed-effect assumptions; it is not an interval intended to cover the true effects of 95% of populations.

A consistency check uses Cochran’s \(Q=\sum_i w_i(y_i-\hat{\mu}_{F})^2\). Here,

Q=4(1-1.3333)^2+4(2-1.3333)^2+1(0-1.3333)^2 \approx 4.000.

Under the common-effect model, \(Q\) is approximately chi-squared with \(k-1=2\) degrees of freedom. Its upper-tail p-value is approximately 0.135. This does not provide strong evidence against a common effect at the 0.05 level, but it does not prove that the true effects are identical; with only three populations, the test has limited ability to detect heterogeneity.

Random Effects: Estimating the Mean and Heterogeneity

When true effects vary, the random-effects estimate gives less relative weight to very precise populations than the fixed-effect estimate does. If \(\tau^2\) were known, the appropriate weights would be \(w_i^*=1/(v_i+\tau^2)\). In practice, \(\tau^2\) must be estimated. One common moment estimator is the DerSimonian–Laird estimator, which uses the fixed-effect \(Q\) statistic:

C=\sum_{i=1}^{k}w_i-\frac{\sum_{i=1}^{k}w_i^2}{\sum_{i=1}^{k}w_i}, \qquad \hat{\tau}_{DL}^{\,2}=\max\left(0,\frac{Q-(k-1)}{C}\right).

The truncation at zero prevents an estimated variance from being negative. This is a particular estimation method, not a guarantee that the resulting estimate is precise. For a chosen estimate \(\hat{\tau}^2\), define \(w_i^*=1/(v_i+\hat{\tau}^2)\). The random-effects estimate and its conventional standard error are

\hat{\mu}_{R}=\frac{\sum_{i=1}^{k}w_i^*y_i}{\sum_{i=1}^{k}w_i^*}, \qquad SE(\hat{\mu}_{R})=\sqrt{\frac{1}{\sum_{i=1}^{k}w_i^*}}.

A short justification explains the moment estimator. Under the random-effects model, each estimate contains its sampling variance plus the extra variance \(\tau^2\). The expected fixed-effect \(Q\) is therefore \(k-1+\tau^2 C\), using the fixed-effect weights in \(C\). Equating the observed \(Q\) to this expectation and solving for \(\tau^2\) gives \((Q-(k-1))/C\); truncation at zero yields the stated estimator.

A common descriptive measure is \(I^2=\max(0,(Q-(k-1))/Q)\times100\%\) when \(Q>0\). It expresses the estimated proportion of observed dispersion beyond sampling variation, not the size of the differences in treatment effects. Always report \(\hat{\tau}^2\), or \(\hat{\tau}\) in the effect’s units, as well: an \(I^2\) value alone cannot show whether the heterogeneity is clinically important.

Worked Example: Random-Effects Summary and Prediction

Consider four hypothetical populations with effects 1, 2, 4, and 5 points, each with standard error 1 point. All fixed-effect weights are 1, so the fixed-effect mean is 3 points. The deviations from that mean are −2, −1, 1, and 2 points; hence \(Q=4+1+1+4=10\), with \(df=3\).

For these data, \(\sum w_i=4\), \(\sum w_i^2=4\), and \(C=4-4/4=3\). Thus,

\hat{\tau}_{DL}^{\,2}=\frac{10-3}{3}=2.333\text{ points}^2, \qquad I^2=\frac{10-3}{10}\times100\%=70.0\%.

Each random-effects weight is \(1/(1+2.333)=0.3000\). The weighted mean remains 3 points because the weights are equal, and its conventional standard error is

SE(\hat{\mu}_{R})=\sqrt{\frac{1}{4(0.3000)}}=0.9129\text{ points}.

A conventional normal-based 95% confidence interval for the mean is \(3.000\pm1.96(0.9129)\), or approximately 1.211 to 4.789 points. It quantifies uncertainty about the mean of the population-effect distribution, not the spread of effects among populations.

For illustration, an approximate prediction interval for the true effect in a new population is \(\hat{\mu}_{R}\pm t_{k-2,0.975}\sqrt{\hat{\tau}^2+SE(\hat{\mu}_{R})^2}\). With \(k=4\), \(t_{2,0.975}=4.303\), and \(\sqrt{2.333+0.9129^2}=\sqrt{3.167}\approx1.780\) points. Using unrounded values, the margin is approximately \(4.303(1.7795)=7.657\) points. The resulting interval is approximately −4.657 to 10.657 points. Its breadth indicates that the mean estimate alone gives little precision about the effect in an individual new population; with only four populations, the estimated heterogeneity and prediction interval are themselves uncertain.

Prediction for a New Target Population

The prediction interval adds estimated between-population variation to uncertainty about the mean. In the approximate formula above, \(\hat{\tau}^2\) accounts for the distribution of true effects across populations, while \(SE(\hat{\mu}_{R})^2\) accounts for uncertainty in the estimated mean. The \(t\) critical value reflects the small number of populations in a commonly used approximation. This is an interval for a new population’s true effect under the random-effects model. It is not a confidence interval for the average effect, nor does it include the sampling error of a future trial’s estimate. If predicting a future observed estimate, its sampling variance must also be included.

Worked Example: Interpreting a Prediction Interval on a Ratio Scale

Suppose a hypothetical random-effects meta-analysis of eight comparable populations reports a mean log risk ratio of 0.400, a standard error for that mean of 0.100, and an estimated between-population variance of 0.090 on the log scale. Assume the model and population-exchangeability assumptions are reasonable. With \(k=8\), \(t_{6,0.975}=2.447\).

The prediction standard error is \(\sqrt{0.090+0.100^2}=\sqrt{0.100}=0.3162\). The margin is \(2.447(0.3162)=0.7738\), so the approximate interval on the log scale is \(0.400\pm0.7738\), or −0.3738 to 1.1738.

Exponentiating both endpoints gives a prediction interval for the true risk ratio of approximately \(\exp(-0.3738)=0.688\) to \(\exp(1.1738)=3.235\). The pooled mean corresponds to a ratio of \(\exp(0.400)=1.492\), but the prediction interval includes effects below and above 1. If the event is undesirable and the ratio is treatment relative to control, those values are compatible with benefit in some populations and harm in others. The interval is conditional on the model and estimated inputs; it is not a probability statement that the effect in this particular target population lies in the interval.

Diagnostics, Alternatives, and Limitations

Before interpreting a pooled result, check whether the evidence supports the model and whether the populations answer a common scientific question. A forest plot helps reveal outlying estimates, wide within-population intervals, and patterns that a single summary can hide. Examine the individual effect estimates and standard errors, compare the fixed- and random-effects summaries, and assess whether one population has disproportionate influence. Investigate clinical and design differences that could explain variation rather than treating heterogeneity as merely a statistical nuisance.

Neither \(Q\) nor \(I^2\) is a stand-alone decision rule. \(Q\) can have low power when few populations are available and can flag small, unimportant differences with many precise estimates. \(I^2\) depends on precision and does not report heterogeneity in the effect’s original units. A zero estimate of \(\hat{\tau}^2\) can arise from limited information; it does not demonstrate that populations have identical effects. With few populations, standard errors, heterogeneity estimates, and prediction intervals can all be unstable.

Approach	Main assumption and output	Important caution
Fixed effect	All populations share one true effect; estimates a common effect.	May understate uncertainty or misrepresent variation if true effects differ.
Random effects	True effects vary around a mean; estimates the mean and between-population variance.	Prediction depends on a credible distribution of relevant populations and a stable estimate of heterogeneity.
Bayesian hierarchical model	Models population effects and can incorporate prior distributions for parameters.	Results can depend on prior choices, especially when the number of populations is small.

The DerSimonian–Laird estimator is convenient, but other estimators, including restricted maximum likelihood, are commonly used and may behave differently, especially with few studies or unequal precisions. Small-sample methods such as Hartung–Knapp adjustments can also alter uncertainty for the pooled mean. These alternatives do not remove the need to justify the estimand, assess population relevance, and examine heterogeneity. A statistically coherent meta-analysis is not, by itself, proof that a treatment effect transports to a new target population.

Common Mistake

Common pitfall: “The pooled mean has a narrow 95% confidence interval, so the true effect in the new population is probably close to the pooled estimate.” This confuses precision about the mean effect with variation among true population effects. A prediction interval addresses the latter by incorporating \(\hat{\tau}^2\), as well as uncertainty in the mean. Even then, its relevance to a specific target depends on whether that population is plausibly drawn from the modeled distribution.

Key takeaway: Use a fixed-effect summary only when a shared true effect is a defensible assumption. When effects may vary, report the random-effects mean and heterogeneity, and use a prediction interval—not the confidence interval for the mean—to describe the model’s implications for a new population.

Check Your Understanding

Answer the questions first, then compare your reasoning with the solutions.

What two sources of variation are separated by the random-effects model?
If \(Q=12\) for five populations, what are \(df\) and \(I^2\)? Interpret the latter carefully.
Why can a prediction interval be much wider than the confidence interval for the pooled mean?
For the eight-population ratio-scale example, what is the approximate pooled mean risk ratio, and does its prediction interval exclude 1?
What does a DerSimonian–Laird estimate of zero for \(\tau^2\) establish?

Solutions. (1) It separates within-population sampling variance \(v_i\) from between-population variance \(\tau^2\) in true effects. (2) \(df=k-1=4\), and \(I^2=(12-4)/12\times100\%=66.7\%\). This is a descriptive estimate of the proportion of observed dispersion beyond sampling variation, not a measure of the absolute size or clinical importance of effect differences. (3) The prediction interval includes estimated variation in true effects across populations, whereas the confidence interval concerns the mean effect; the mean can be estimated precisely even when effects vary substantially. (4) The pooled mean ratio is \(\exp(0.400)=1.492\); the prediction interval, approximately 0.688 to 3.235, includes 1. (5) It establishes only that this estimator, after truncation, is zero for the available data. Limited information may fail to reveal heterogeneity, so it does not prove that all true effects are identical.