Prediction Intervals for Genetic Effects

Why Predict a New Cohort’s Effect?

A random-effects meta-analysis summarizes a distribution of cohort-specific genetic effects. Its mean effect describes the centre of that distribution, but it does not say how much the true effect may vary from cohort to cohort. When planning replication or describing how portable an association might be, a second question matters: what range of true effects is plausible for a new cohort drawn from the same target population?

A confidence interval for the mean and a prediction interval for a new cohort answer different questions. A confidence interval reflects uncertainty about the mean effect. A prediction interval reflects both that uncertainty and the between-cohort variation in true effects. It is therefore usually wider, and can include effects that differ in direction from the estimated mean.

Definition: In a random-effects meta-analysis, let \(\theta_i\) be the true genetic effect in cohort \(i\), measured on a common, harmonized effect scale. The model assumes \(\theta_i\) are drawn from a distribution with mean \(\mu\) and between-cohort variance \(\tau^2\). A prediction interval is an interval intended to cover the true effect \(\theta_{\text{new}}\) in a new cohort drawn from that same distribution, with a stated prediction level, such as 95%.

Model and Prediction-Interval Formula

Suppose the cohort-specific estimated effects are \(y_i\), with estimated sampling variances \(s_i^2\). The standard random-effects model represents the sampling process as

\theta_i \sim N(\mu,\tau^2), \qquad y_i \mid \theta_i \sim N(\theta_i,s_i^2).

The \(s_i^2\) values describe sampling uncertainty within cohorts; \(\tau^2\) describes variation among their true effects. A random-effects meta-analysis estimates the mean effect \(\mu\), its standard error, and the between-cohort variance. Write these estimates as \(\hat{\mu}\), \(SE(\hat{\mu})\), and \(\hat{\tau}^2\), respectively.

A commonly used approximate 95% prediction interval for the true effect in a new cohort is

\hat{\mu} \ \pm\ t_{k-2,\,0.975} \sqrt{\hat{\tau}^2 + SE(\hat{\mu})^2},

where \(k\) is the number of cohorts and \(t_{k-2,\,0.975}\) is the 97.5th percentile of a \(t\) distribution with \(k-2\) degrees of freedom. The corresponding 95% prediction interval has lower and upper limits formed by subtracting and adding the margin of error to \(\hat{\mu}\). This is an approximate, plug-in procedure: it uses the estimated between-cohort variance and a \(t\) multiplier to reflect limited information about the mean and heterogeneity. It is not an exact guarantee that the interval has 95% coverage in every setting.

The confidence interval for \(\mu\) is narrower in concept because it targets only the mean. A normal-approximation 95% confidence interval is \(\hat{\mu} \pm 1.96SE(\hat{\mu})\); an analysis may instead use a small-sample method appropriate to its estimator. Neither version includes \(\hat{\tau}^2\) as a source of variation in the target. The prediction interval does.

Worked Example: Predicting a True Effect in a New Cohort

Hypothetical data. A meta-analysis of \(k=8\) independent cohorts estimates a genetic effect of \(0.20\) standard-deviation units per effect allele, with \(SE(\hat{\mu})=0.05\) and \(\hat{\tau}=0.10\) standard-deviation units. Construct an approximate 95% prediction interval for the true effect in a new cohort from the same target population. For \(df=6\), use \(t_{6,0.975}=2.447\).

First, \(\hat{\tau}^2=(0.10)^2=0.0100\), and \(SE(\hat{\mu})^2=(0.05)^2=0.0025\). Thus the estimated predictive standard deviation is

\sqrt{0.0100+0.0025}=\sqrt{0.0125}=0.1118.

The margin of error is \(2.447(0.1118)=0.2736\). Therefore,

0.20 \pm 0.2736 = (-0.0736,\ 0.4736).

The interval is in standard-deviation units per effect allele. Under the model and approximation, it gives a range for the new cohort’s true effect. It includes zero and some negative effects, even though the estimated mean is positive.

Why the Formula Includes Two Sources of Uncertainty

The formula can be motivated by considering the difference between the new cohort’s true effect and the estimated mean. If \(\hat{\mu}\) estimates \(\mu\), then the prediction error can be written as

\theta_{\text{new}}-\hat{\mu} = (\theta_{\text{new}}-\mu)+(\mu-\hat{\mu}).

The first term is the new cohort’s deviation from the population mean; under the model, its variance is \(\tau^2\). The second term is estimation error in the meta-analytic mean; its estimated variance is \(SE(\hat{\mu})^2\). If the new cohort is independent of the cohorts used to estimate the mean, these terms are independent under the model. Their variances therefore add:

\operatorname{Var}(\theta_{\text{new}}-\hat{\mu}) = \tau^2+\operatorname{Var}(\hat{\mu}) \ \approx\ \tau^2+SE(\hat{\mu})^2.

Taking the square root gives the predictive standard deviation used in the interval. Replacing unknown quantities with estimates and using a \(t\) critical value gives the stated approximate procedure. This justification also shows why using only \(SE(\hat{\mu})\) would answer the wrong question: it would quantify uncertainty about the mean, not the additional variation of a new cohort’s true effect.

Worked Example: A Precise Mean Can Still Have a Wide Prediction Interval

Hypothetical data. In 12 independent cohorts, the estimated mean genetic effect is \(0.30\) log-odds units per effect allele, \(SE(\hat{\mu})=0.06\), and \(\hat{\tau}=0.20\) log-odds units. Use \(t_{10,0.975}=2.228\). Compare the prediction interval with a normal-approximation 95% confidence interval for the mean.

The mean’s confidence interval is

0.30 \pm 1.96(0.06) =0.30\pm0.1176 =(0.1824,\ 0.4176).

For prediction, \(\hat{\tau}^2=0.0400\) and \(SE(\hat{\mu})^2=0.0036\), so

\sqrt{0.0400+0.0036}=0.2088, \qquad 2.228(0.2088)=0.4652.

The approximate 95% prediction interval is therefore

0.30\pm0.4652=(-0.1652,\ 0.7652).

The confidence interval is entirely above zero, which indicates that the mean effect is estimated to be positive using this normal approximation. The prediction interval is much wider and includes zero and negative effects. This is not a contradiction: cohorts can have true effects that vary around a positive mean.

What Exactly Is Being Predicted?

The formula above predicts \(\theta_{\text{new}}\), the new cohort’s true effect. It does not directly predict the cohort’s observed estimate \(y_{\text{new}}\), which also has sampling error. If the new cohort’s sampling variance \(s_{\text{new}}^2\) is known or specified, a prediction interval for its observed estimate uses an additional variance term:

\hat{\mu} \ \pm\ t_{k-2,\,0.975} \sqrt{\hat{\tau}^2+SE(\hat{\mu})^2+s_{\text{new}}^2}.

That interval is wider because an observed estimate can differ from the new cohort’s true effect due to both between-cohort heterogeneity and sampling noise. The added term is appropriate only when the target is the future estimate, not when the target is the true cohort effect.

Worked Example: True Effect Versus Future Estimate

Hypothetical data. A 10-cohort meta-analysis estimates a mean effect of \(0.12\) standard-deviation units per effect allele, with \(SE(\hat{\mu})=0.04\) and \(\hat{\tau}=0.08\). Use \(t_{8,0.975}=2.306\). A planned new cohort is expected to have an effect-estimate standard error of \(s_{\text{new}}=0.06\).

For the new cohort’s true effect, the predictive standard deviation is

\sqrt{(0.08)^2+(0.04)^2} =\sqrt{0.0064+0.0016} =\sqrt{0.0080} =0.08944.

The margin is \(2.306(0.08944)=0.2061\), giving a true-effect prediction interval of

0.12\pm0.2061=(-0.0861,\ 0.3261).

For the future observed estimate, include its sampling variance, \(s_{\text{new}}^2=(0.06)^2=0.0036\). The predictive standard deviation becomes

\sqrt{0.0064+0.0016+0.0036} =\sqrt{0.0116} =0.1077.

Its interval is \(0.12\pm2.306(0.1077)=0.12\pm0.2484\), or \((-0.1284,\ 0.3684)\) standard-deviation units per effect allele. This wider interval predicts a noisy estimate, not just the underlying true effect.

Assumptions and Practical Interpretation

A prediction interval is only as useful as the model and target population supporting it. Its interpretation requires particular care in genetic meta-analysis:

Comparable effects. Cohort effects must be harmonized to the same effect allele and phenotype scale. Otherwise, apparent variation can arise from inconsistent definitions rather than cohort-level differences.
A relevant target population. The new cohort should be plausibly drawn from the same distribution of true effects represented by the cohorts in the meta-analysis. A cohort with materially different population composition or study conditions may not be represented by that distribution.
Independent cohorts. The standard formula assumes the new cohort is independent of the cohorts used to estimate \(\hat{\mu}\). Dependence can alter the uncertainty calculation.
A suitable heterogeneity model. The normal random-effects model assumes cohort effects vary around a mean with variance \(\tau^2\). Strong departures, such as distinct subgroups or extreme outliers, can make a single interval misleading.
Uncertainty in \(\hat{\tau}^2\). The formula plugs in an estimate of between-cohort variance; the \(t\) adjustment is an approximation, not a complete account of every source of uncertainty. With few cohorts, heterogeneity estimates can be imprecise and the interval may be unstable.

A prediction interval containing zero does not establish that the true effect is zero in a new cohort. It indicates that, under the model, zero lies within the approximate range of effects compatible with the meta-analysis. Likewise, an interval that excludes zero does not guarantee that every future cohort will have an effect in the same direction.

A Practical Construction Sequence

Define the target.
Specify whether the interval is for a new cohort’s true genetic effect or its future observed estimate, and state the effect scale and target population.

Check the model inputs.
Record \(k\), \(\hat{\mu}\), \(SE(\hat{\mu})\), and \(\hat{\tau}^2\) from the random-effects analysis; confirm that cohort effects are aligned and comparable.

Combine relevant variances.
For a true-effect interval, calculate \(\hat{\tau}^2+SE(\hat{\mu})^2\). Add \(s_{\text{new}}^2\) only if predicting the future observed estimate.

Calculate and interpret limits.
Take the square root, multiply by the chosen critical value, and form the lower and upper limits around \(\hat{\mu}\). Interpret the result in the stated units and for the stated target.

Common Mistake

A frequent error is to report \(\hat{\mu}\pm1.96SE(\hat{\mu})\) as the range expected for the true effect in a new cohort. That is a confidence interval for the mean under a normal approximation; it leaves out between-cohort variation. The correction is to include \(\hat{\tau}^2\) in the predictive variance and use the specified prediction procedure. A second error is to describe the resulting interval as a range for future observed estimates without adding their sampling variance. Always name the target before calculating or interpreting the interval.

Key takeaway: A confidence interval describes uncertainty about the mean effect \(\mu\); a prediction interval describes uncertainty about a new cohort’s true effect. The latter includes both uncertainty in the estimated mean and estimated between-cohort variation, so it can be substantially wider.

Check Your Understanding

Answer each question using the distinction between a population mean and a new cohort’s effect.

In the random-effects model, what does \(\tau^2\) represent, and what does \(SE(\hat{\mu})^2\) represent?
Why can a 95% confidence interval for the mean effect exclude zero while a 95% prediction interval includes zero?
A formula for predicting a new cohort’s true effect includes \(\hat{\tau}^2\), but does not include \(s_{\text{new}}^2\). Explain why.
What additional variance term is needed when the target is the new cohort’s observed effect estimate?
Name one reason a prediction interval from a set of cohorts may not be appropriate for a particular new cohort.