Residual Heterogeneity in Cross-Population Predictions

Why a Mean Prediction Is Not a Prediction for Every Population

A model can predict the average treatment effect for a target population while still leaving uncertainty about what the effect will be in one particular, unstudied population. These are different questions. Uncertainty about the average prediction arises because model parameters are estimated from finite data. Residual heterogeneity describes how much population-specific effects can vary around that average after accounting for the model’s predictors.

For a bridging decision, the average may be favorable while a plausible effect in a new region is smaller, or even clinically inadequate. A confidence interval for the average does not describe that range. A prediction interval does, subject to the model’s assumptions. The distinction builds on hierarchical models and meta-analysis of treatment effects across populations: here the emphasis is on prediction for a new population and on keeping its sources of uncertainty separate.

Model and Definitions

A commonly used sampling model for the estimated effect \(y_j\) in population \(j\) is

Here \(s_j\) is the standard error of the estimated effect from population \(j\). In this formulation, \(s_j^2\) describes sampling variance within a population, whereas \(\tau^2\) describes variation among population-specific true effects that remains after conditioning on \(x_j\). The two variances have different meanings and neither substitutes for the other.

The usual prediction-interval calculation relies on several assumptions: effects are expressed on a comparable scale; the target population is represented by the model’s predictors; residual deviations are exchangeable across the studied and target populations; and the residuals are independent of the fitted mean estimator. A normal prediction interval additionally assumes that the residual distribution is adequately approximated by a normal distribution. In practice, \(\beta\) and \(\tau^2\) are estimated, so a simple plug-in interval also treats uncertainty in those estimates approximately.

Deriving the Prediction Interval

For target covariates \(x_*\), the model-predicted mean effect is \(x_*^\mathsf{T}\hat{\beta}\). Its estimation uncertainty is summarized by the covariance matrix \(\widehat{\operatorname{Var}}(\hat{\beta})\). The target population has its own deviation \(u_*\), with variance \(\tau^2\). The prediction error can be decomposed as

Under the stated independence assumptions, the variance of a sum is the sum of the variances. The first term contributes \(\tau^2\); the second contributes \(x_*^\mathsf{T}\widehat{\operatorname{Var}}(\hat{\beta})x_*\). Thus the plug-in prediction variance for the latent effect is

Writing \(SE_{\text{mean},*}=\sqrt{x_*^\mathsf{T}\widehat{\operatorname{Var}}(\hat{\beta})x_*}\), an approximate two-sided 95% prediction interval is

This interval is wider than a confidence interval for the mean prediction because it includes residual heterogeneity. If the question instead concerns a future observed effect estimate \(y_*\), its within-population sampling variance \(s_*^2\) must also be included:

The multiplier 1.96 gives a normal-approximation interval, not a guarantee of exact coverage. With few populations, uncertainty in \(\hat{\tau}^2\) can be substantial, and a plug-in normal interval may be too narrow. The method used to estimate heterogeneity, the number of populations, and any small-sample or distributional adjustments should therefore be reported.

Latent Effects and Future Estimates Are Different Targets

The prediction target must be stated before selecting the variance formula. A latent-effect interval asks what the true effect in the new population could plausibly be. A future-estimate interval asks what an effect estimate from a new study could plausibly report. The latter includes both heterogeneity and sampling noise in that study. Confusing these targets can make an interval either too wide or too narrow for the decision being considered.

Using the Distribution for a Clinical Question

A prediction interval can also support a model-based probability statement about a clinically meaningful threshold. Such a probability is conditional on the model and its parameter estimates. It is not a claim that the model has captured every source of uncertainty, and it should not be mistaken for a probability that a fixed parameter is random in the frequentist interpretation.

Diagnostics, Alternatives, and Limitations

Residual heterogeneity is a model component, not a universal measure of all transport risk. Its estimated size is meaningful only relative to the populations, effect scale, and covariates in the analysis. Review whether the target lies within the range of studied populations and whether the covariates that explain systematic differences are available and measured comparably. If the target is unlike the source populations in important ways, exchangeability may fail; a narrow interval cannot repair that mismatch.

Useful checks include examining population-specific residuals against fitted effects and relevant population characteristics, comparing observed effects with their sampling uncertainty, and assessing whether conclusions change when individual populations are omitted. A single influential population can strongly affect \(\hat{\tau}\), particularly when there are few populations. Inspecting residual patterns can reveal outliers, trends, or asymmetry that a normal random-effects distribution does not represent well. These checks do not prove the model correct, but they can expose important failures.

The normal plug-in interval is convenient but has limitations. Estimation uncertainty in \(\tau\) is often substantial with a small number of populations, while the formula treats \(\hat{\tau}\) as known. Resampling procedures or a Bayesian posterior predictive analysis can propagate more of this uncertainty, provided their assumptions and implementation are appropriate. If residuals are skewed or heavy-tailed, an alternative residual distribution may be more credible than a normal one. With few populations, however, the data may not support reliable estimation of a more flexible distribution.

The principal alternatives answer different questions. A confidence interval for the fitted mean quantifies uncertainty in the average effect and excludes residual variation in a new population. A fixed-effect model assumes no residual between-population heterogeneity (\(\tau^2=0\)); it can therefore understate predictive variation when real heterogeneity remains. A prediction interval adds that heterogeneity, and a future-estimate interval adds the new study’s sampling variance as well. These quantities should be reported with their target and units rather than treated as interchangeable summaries.