Prediction Intervals for Effects in New Cohorts

What a Prediction Interval Targets

A pooled estimate summarizes the average genetic effect across cohorts. It does not, by itself, say how far the true effect in a particular new cohort might lie from that average. When effects vary across cohorts, predicting the effect in a new cohort requires accounting for both this variation and uncertainty in the estimated pooled mean.

The target here is the true underlying genetic effect in a new cohort, on the same scale as the effects in the meta-analysis. For example, if the estimates are log odds ratios per allele, the target is the new cohort’s true log odds ratio per allele. This is distinct from the new cohort’s observed estimate, which would also vary because of sampling error.

Definition: Suppose the true effect in cohort \(i\) is \(\beta_i\), and cohort estimates \(y_i\) have standard errors \(s_i\). A prediction interval for a new cohort is an interval intended to cover its true effect \(\beta_{\mathrm{new}}\), under a specified random-effects model. It is not an interval for the average effect \(\mu\).

The interval is useful when the scientific question is about transportability: how plausible is a particular effect size in another cohort drawn from the same target population of cohorts? Its relevance depends on whether that new cohort is reasonably represented by the distribution of cohorts used in the meta-analysis.

Model, Assumptions, and Formula

For \(k\) independent cohorts, use the normal random-effects model

\beta_i \sim N(\mu,\tau^2), \qquad y_i\mid\beta_i \sim N(\beta_i,s_i^2), \qquad i=1,\ldots,k.

Here, \(\mu\) is the mean of the cohort effects and \(\tau^2\) is the between-cohort variance. The sampling variances \(s_i^2\) describe uncertainty in the estimated effects within cohorts. A new cohort is assumed to have an effect drawn from the same distribution: \(\beta_{\mathrm{new}}\sim N(\mu,\tau^2)\). Cohorts and their sampling errors are assumed independent, and the effects must be on a comparable scale.

Let \(\hat{\tau}^2\) be the estimated between-cohort variance from the random-effects analysis. The random-effects weights, pooled estimate and its estimated standard error are

w_i=\frac{1}{s_i^2+\hat{\tau}^2}, \qquad \hat{\mu}=\frac{\sum_{i=1}^{k}w_i y_i}{\sum_{i=1}^{k}w_i}, \qquad SE(\hat{\mu})=\sqrt{\frac{1}{\sum_{i=1}^{k}w_i}}.

A commonly used approximate two-sided \(100(1-\alpha)\%\) prediction interval is

\hat{\mu}\ \pm\ t_{k-2,\,1-\alpha/2} \sqrt{\hat{\tau}^2+SE(\hat{\mu})^2}.

The square-root term combines estimated variation among true cohort effects with uncertainty in the pooled mean. The \(t\) critical value, with \(k-2\) degrees of freedom, is a small-sample adjustment often used with this formula. Since \(\tau^2\) is estimated, this is an approximation, not an exact finite-sample interval in general. It accounts for the estimated heterogeneity in the prediction spread, but does not fully propagate every source of uncertainty in \(\hat{\tau}^2\).

Why the Prediction Spread Has Two Parts

The central justification follows by separating the difference between a new effect and the estimated pooled mean into two terms:

\beta_{\mathrm{new}}-\hat{\mu} = (\beta_{\mathrm{new}}-\mu)+(\mu-\hat{\mu}).

Under the model, the first term represents deviation of the new cohort’s true effect from the population mean, with variance \(\tau^2\). The second represents estimation error in the pooled mean, with variance approximately \(SE(\hat{\mu})^2\). The new cohort is independent of the cohorts used to estimate \(\hat{\mu}\), so these terms have zero covariance. Thus, if the parameters were known and the pooled estimate were normally distributed,

\operatorname{Var}(\beta_{\mathrm{new}}-\hat{\mu}) = \tau^2+SE(\hat{\mu})^2.

Replacing \(\tau^2\) with \(\hat{\tau}^2\) gives the plug-in prediction spread. The \(t\) multiplier widens the interval to reflect the limited number of cohorts. This adjustment is a practical approximation; it does not turn the plug-in calculation into an exact solution to the uncertainty in a variance-component estimate.

Worked Example: An 80% Prediction Interval for a Log Odds Ratio

Suppose six independent hypothetical cohorts estimate the same allele effect on the log odds ratio scale. A random-effects analysis gives \(\hat{\mu}=0.0800\), \(SE(\hat{\mu})=0.0200\), and \(\hat{\tau}^2=0.000917\). Construct an 80% prediction interval using the \(t\) adjustment with \(k-2=4\) degrees of freedom. For this confidence level, \(t_{4,0.90}=1.533\).

The prediction standard error is

\sqrt{\hat{\tau}^2+SE(\hat{\mu})^2} = \sqrt{0.000917+(0.0200)^2} = \sqrt{0.001317} \approx 0.03629.

The margin is \(1.533(0.03629)=0.05563\). Therefore,

0.0800\pm0.05563 = (0.0244,\ 0.1356).

Under the assumed model, the interval is intended to cover the true log odds ratio in a new cohort from the same cohort population at an approximate 80% rate. It is not an 80% probability statement about the fixed effect in one already identified cohort. Exponentiating the endpoints gives an odds-ratio interval of approximately \((1.025,\ 1.145)\) per allele for the new cohort’s true effect.

Prediction Is Not Estimation of the Mean

A confidence interval for \(\mu\) addresses uncertainty about the average effect across the modeled cohort population. A prediction interval for \(\beta_{\mathrm{new}}\) is wider because it also allows for genuine differences between cohorts. The distinction remains important even when both intervals are centered on \(\hat{\mu}\).

Worked Example: Comparing a Mean Interval With a Prediction Interval

Use the estimates from the previous example. For illustration, a conventional 95% normal-based interval for the pooled mean, treating its reported standard error as adequate, is

0.0800\pm1.96(0.0200) = 0.0800\pm0.0392 = (0.0408,\ 0.1192).

This interval concerns the mean log odds ratio \(\mu\), not the effect in a particular new cohort. The earlier 80% prediction interval, \((0.0244,0.1356)\), includes the additional between-cohort variation and uses a different confidence level and small-sample multiplier. The intervals should not be compared as if their coverage levels were the same. Their targets differ as well as their stated levels.

If \(\hat{\tau}^2\) is near zero, the prediction interval may be close in width to an interval for the mean, although the small-sample adjustment can still matter. If heterogeneity is substantial, the prediction interval can be much wider even when the pooled mean is estimated precisely.

Reading the Interval in Context

A prediction interval is not a guarantee that every future cohort effect will fall inside its limits. Its coverage is a repeated-sampling property under the model: across repetitions of the meta-analysis and draws of a new cohort effect from the same population, the stated fraction of intervals is intended to contain that effect. The approximation can be poor when few cohorts are available or the model’s assumptions do not describe the cohort effects.

For genetic association, the effect scale must be consistent across cohorts. The variant, outcome, effect allele, and effect definition must refer to comparable quantities. The independent-cohort assumption also matters: overlap between cohorts can invalidate the usual standard-error calculation. The earlier material on replication estimands emphasizes why the target population and effect definition need to be specified; a prediction for a different population is not justified merely by applying this formula.

Worked Example: A Negative Mean With a Prediction Interval Crossing Zero

In a second hypothetical meta-analysis of five independent cohorts, suppose the pooled effect is \(\hat{\mu}=-0.0300\) log odds-ratio units, \(SE(\hat{\mu})=0.0100\), and \(\hat{\tau}^2=0.000400\). For a 95% prediction interval, use \(df=5-2=3\) and \(t_{3,0.975}=3.182\).

First calculate the prediction standard error:

\sqrt{0.000400+(0.0100)^2} = \sqrt{0.000500} \approx0.02236.

The margin is \(3.182(0.02236)\approx0.07115\), so the interval is

-0.0300\pm0.07115 = (-0.1012,\ 0.0412).

Although the estimated mean effect is negative, the interval includes zero and positive values. Under this model and its assumptions, the data do not support a confident prediction that the true effect in a new cohort must be negative. The wide interval reflects both estimated heterogeneity and the small number of cohorts, not a contradiction with the negative pooled estimate.

Diagnostics, Alternatives, and Limitations

Before interpreting a prediction interval, check whether the inputs and model are plausible. Useful checks include reviewing the cohort-specific effects and standard errors, confirming that all estimates use a common effect scale, and looking for an influential cohort or an effect pattern that is poorly represented by a single normal distribution. The forest plot and heterogeneity summaries can help identify discrepancies; they do not establish that the random-effects distribution is correct.

Few cohorts: The heterogeneity estimate is imprecise, and the \(t\)-based correction may not fully capture that uncertainty. Prediction intervals can be unstable.
Uncertain or boundary estimate of \(\tau^2\): An estimate at zero does not establish that true heterogeneity is absent. Treating it as known can produce an overconfident prediction.
Non-normal or structured effects: A single normal distribution may poorly describe multimodal, skewed, or systematically different cohort effects.
Mismatch in target cohorts: If the new cohort differs in ancestry, phenotype definition, environment, or other effect modifiers, exchangeability with the analyzed cohorts may fail.
Sampling dependence: Overlapping participants or correlated estimates violate the independence assumptions unless that dependence is modeled.

The formula is an accessible approximation and should be reported with its assumptions and the variance estimator used. A parametric bootstrap or a likelihood-based predictive procedure can be considered when uncertainty in \(\tau^2\) is especially consequential. Such methods depend on their own modeling choices and do not repair a poor cohort population definition or non-comparable effects.

A confidence interval for \(\mu\) is the main alternative when the scientific target is the average effect rather than a new cohort. Reporting only the pooled estimate and its confidence interval may be suitable for that target, but it does not communicate the expected spread of true effects across cohorts. Conversely, a prediction interval is not a substitute for a confidence interval when inference about the mean is the goal.

Common Pitfall

Common mistake: “The confidence interval for the pooled mean excludes zero, so the effect in a new cohort will also be nonzero in the same direction.” This confuses uncertainty about \(\mu\) with the distribution of \(\beta_{\mathrm{new}}\). A mean can be estimated precisely while cohort effects vary substantially. Use a prediction interval for the new-cohort target, and interpret its width and assumptions rather than inferring new-cohort certainty from the mean interval alone.

Key takeaway: A random-effects prediction interval combines the estimated between-cohort variance with uncertainty in the pooled mean. It predicts a true effect for a new cohort assumed to come from the same cohort population; it is not a confidence interval for the pooled mean or a prediction interval for a noisy observed estimate.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the random-effects model, what quantity does a prediction interval target, and how does that differ from \(\mu\)?
If \(\hat{\tau}^2=0\), does that prove that all true cohort effects are identical? Explain.
For \(k=8\) cohorts, what degrees of freedom enter the commonly used \(t\) adjustment?
Suppose the new cohort is drawn from a population that differs materially from the analyzed cohorts. Which prediction assumption is in question?
In a prediction interval, why is \(SE(\hat{\mu})^2\) added to \(\hat{\tau}^2\)?

Solutions. (1) It targets \(\beta_{\mathrm{new}}\), the new cohort’s true effect; \(\mu\) is the mean of the cohort-effect distribution. (2) No. Zero is an estimated boundary value and may reflect limited information; it does not establish absence of true heterogeneity. (3) \(df=k-2=6\). (4) The assumption that the new cohort is exchangeable with, and drawn from the same effect distribution as, the analyzed cohorts is questionable. (5) The new true effect varies around \(\mu\) with variance \(\tau^2\), while the estimated pooled mean has its own variance; under independence these contributions add when calculating the prediction error variance.