Fixed-Effect and Random-Effects Replication Claims

Why the Pooling Model Changes the Replication Claim

When several independent cohorts estimate the same molecular association, a pooled estimate can summarize evidence across them. But “the effect across cohorts” can mean two different things. It may mean one common underlying effect shared by the cohorts being analyzed, or it may mean the average of a distribution of cohort-specific effects. Those are different scientific claims, and they require different models.

The earlier material on diagnosing between-cohort heterogeneity concerns whether cohort estimates vary more than expected from sampling error alone. This tutorial uses that distinction to compare the fixed-effect and random-effects models. The key issue is not simply which model produces a smaller standard error. It is which population quantity the pooled estimate is intended to describe.

Let \(y_i\) be the estimated effect in cohort \(i\), and let \(s_i\) be its standard error, for \(i=1,\ldots,k\). The effect scale must have the same interpretation across cohorts: for example, a log fold-change per specified unit of exposure, after applying a consistent direction convention. The usual calculations assume that cohorts are independent and that each \(y_i\), conditional on its underlying effect, is approximately normally distributed with variance \(s_i^2\).

Models, Weights, and Targets

Under the fixed-effect model, the assumptions are \(y_i=\theta+\varepsilon_i\), with independent sampling errors \(\varepsilon_i\) having mean zero and variance \(s_i^2\). The cohorts must therefore be estimating a common effect on a common scale. Inverse-variance weighting gives more influence to more precise estimates:

The fixed-effect estimate targets the common \(\theta\), conditional on the assumption that it exists and applies to all included cohorts. It does not, by itself, estimate an average effect in a broader population of possible cohorts. A precise pooled estimate cannot compensate for a false common-effect assumption.

Under the random-effects model, the sampling model is \(y_i\mid\theta_i\) with mean \(\theta_i\) and variance \(s_i^2\), while the cohort effects vary around \(\mu\) with variance \(\tau^2\). If the cohort deviations and sampling errors are independent, then the marginal variance of \(y_i\) is \(s_i^2+\tau^2\). The corresponding weights and pooled estimate, conditional on a value of \(\tau^2\), are:

These standard errors treat \(\tau^2\) as known. In practice, it is usually estimated from the cohort data; the particular estimator and interval method matter, especially when there are few cohorts. The random-effects target \(\mu\) describes the mean of the modeled distribution of cohort effects, not a guarantee that every cohort has an effect equal to that mean. The amount of variation around it is part of the scientific result.

Why Random-Effects Weights Differ

In random-effects pooling, the added variance \(\tau^2\) reduces the relative dominance of the most precise cohorts. If \(\tau^2\) is large compared with the within-cohort variances, differences in \(s_i^2\) matter less to the weights. This is sensible when cohorts are viewed as draws from a broader distribution: even a very large cohort estimates its own cohort effect precisely, but it does not eliminate uncertainty about how that effect relates to the mean across cohorts.

A common moment-based estimator illustrates how heterogeneity can be estimated. First calculate the fixed-effect estimate and the weighted residual statistic \(Q=\sum_i w_i^{\mathrm{FE}}(y_i-\hat{\theta}_{\mathrm{FE}})^2\). Define \(C=\sum_i w_i^{\mathrm{FE}}-\sum_i(w_i^{\mathrm{FE}})^2/\sum_i w_i^{\mathrm{FE}}\). The DerSimonian–Laird moment estimate is \(\hat{\tau}^2=\max\{0,(Q-(k-1))/C\}\), when \(C>0\). It is one possible estimator, not a universal choice.

The justification follows from the expected size of \(Q\). If the common-effect model holds, its expected value is \(k-1\). Under a random-effects model, the expected excess is approximately \(\tau^2 C\). Equating the observed excess \(Q-(k-1)\) to \(\tau^2C\) gives the ratio above. Since a variance cannot be negative, negative estimates are set to zero. This is a moment argument; it does not mean that heterogeneity has been proven absent when the estimate reaches the boundary.

Assumptions, Diagnostics, and Failure Modes

Both models require independent cohort estimates, comparable effect definitions, and a reasonable approximation to the stated within-cohort standard errors. If participants overlap, estimates may be correlated, and the usual inverse-variance formulas do not correctly describe their joint uncertainty. If assays or variable definitions differ, a numerical average may combine effects that do not share a meaningful scale. Harmonization and cohort eligibility are therefore part of the model’s scientific justification, not merely data-preparation details.

The fixed-effect model adds the strong condition that all included cohorts share one true effect. It can be appropriate for a deliberately narrow set of comparable cohorts and a target explicitly conditional on that set. It is less convincing when important cohort-level differences plausibly modify the effect. A small observed heterogeneity statistic does not establish that effects are identical: with few or imprecise cohorts, meaningful heterogeneity can be hard to detect.

The random-effects model replaces a single true effect with a distribution of effects. Its usual form assumes that cohort effects are exchangeable draws from a distribution with mean \(\mu\) and variance \(\tau^2\); a normal distribution is often used for the random deviations. This model does not explain why effects vary, and it may be inappropriate if the analyzed cohorts are a selective set that cannot reasonably represent a broader cohort population. With very few cohorts, \(\tau^2\) can be estimated imprecisely, and the pooled mean may be sensitive to the estimator or interval method.

Diagnostics should connect the numerical heterogeneity assessment to the cohort estimates themselves. Inspect the estimates and their uncertainty, confirm that signs and units were aligned, and consider whether study design or measurement differences could account for discrepancies. A single summary of heterogeneity cannot reveal whether one cohort is influential or whether variation follows a systematic pattern. The later leave-one-cohort-out material addresses a specific sensitivity check; here, the immediate lesson is to avoid treating either pooled model as a substitute for examining the cohort-level evidence.

Making a Defensible Replication Claim

A pooled result should state both the model and its target. For example, “the fixed-effect pooled estimate was positive” is a claim about a common effect under the fixed-effect assumption. “The random-effects mean estimate was positive” concerns an average across the modeled distribution of cohort effects. Neither sentence implies that every cohort has a positive effect. The sign, magnitude, uncertainty, and between-cohort variation should be read together, using the effect’s actual units.

A confidence interval for \(\mu\) concerns uncertainty about the mean effect, not the range in which the true effect of a new cohort is expected to fall. The latter requires accounting for between-cohort variation as well as uncertainty about the mean, and can be substantially wider. In multi-omics work, that distinction matters when a pooled association is used to support a claim about replication across settings: a positive average can coexist with weak, null, or negative cohort-specific effects.

A common mistake is to say, “The random-effects estimate is the effect that will occur in every new cohort because it accounts for heterogeneity.” This is incorrect: random-effects pooling estimates the mean of a distribution, while \(\tau^2\) explicitly allows cohort effects to differ. The correction is to report the mean and the estimated heterogeneity separately, and to avoid claiming universal replication from the pooled mean alone.

The main limitation of either summary is that it compresses cohort-level evidence. A model can produce a precise pooled estimate while relying on questionable comparability assumptions, or a broad random-effects mean that masks meaningful effect modifiers. Choose the model from the scientific target and design, not from which result appears more favorable.