Leave-One-Cohort-Out Influence Analysis

Why Recalculate the Meta-Analysis?

A pooled genetic association estimate summarizes evidence across cohorts, but the summary can conceal how much individual cohorts contribute to it. One cohort may have a large effect estimate and shift the pooled result; another may be precise enough to matter despite a modest discrepancy; a third may chiefly affect the estimated heterogeneity. Leave-one-cohort-out (LOCO) influence analysis makes these roles visible by repeating the meta-analysis after excluding each cohort in turn.

LOCO analysis is a sensitivity analysis, not a procedure for deciding which cohort is “wrong.” A large change identifies dependence of the combined result on that cohort under the chosen model. It does not establish that the cohort is biased, that its result should be discarded, or that the remaining result is necessarily more credible.

Definition and Assumptions

Suppose \(k\) independent cohorts provide estimates \(\hat\theta_i\) of the same genetic association parameter, with standard errors \(SE_i\), for \(i=1,\ldots,k\). The effect scale and allele orientation must be harmonized so that the estimates refer to the same contrast. The sampling variance for cohort \(i\) is \(v_i=SE_i^2\).

For an inverse-variance fixed-effect analysis, the model assumes each cohort estimates one common true effect, and the cohort estimates are independent and approximately normal with their stated sampling variances. Thus \(\hat\theta_i\) is modeled as approximately \(N(\theta,SE_i^2)\). For a random-effects analysis, cohort-specific true effects are instead modeled as draws around a mean effect, with between-cohort variance \(\tau^2\); the sampling errors are still assumed independent, and their variances are treated as known or adequately estimated. In either case, the deletion analysis must use the same effect definition, harmonization, model, and estimation method for every subset.

If cohorts overlap or their estimates are otherwise correlated, ordinary inverse-variance weights do not correctly describe their joint information. The dependence must be represented in the meta-analysis covariance structure, and that structure must be refitted or subsetted appropriately for each deletion. Harmonizing alleles and effect scales is also essential: an apparent influence caused by a reversed allele or mismatched scale is a data-processing problem, not a meaningful sensitivity result.

Fixed-Effect Deletion Calculations

Under the fixed-effect model, the inverse-variance weight for cohort \(i\) is \(w_i=1/SE_i^2\). Define \(W=\sum_{i=1}^k w_i\). The full pooled estimate and its standard error are

After omitting cohort \(i\), its weight and contribution are removed:

These expressions provide a useful direct interpretation of influence. Since \(W\hat\theta=\sum_jw_j\hat\theta_j\), subtracting the omitted cohort’s contribution gives

Therefore a cohort moves the pooled estimate substantially when it has appreciable weight and its estimate differs from the full pooled estimate. A striking but very imprecise estimate may have little impact; a precise estimate with a moderate discrepancy may have a larger impact. Under this fixed-effect calculation, deleting a cohort reduces total weight and therefore increases the standard error. Random-effects standard errors need not behave this way, because the estimated between-cohort variance can also change.

Reassess Heterogeneity, Not Just the Pooled Effect

For a fixed-effect analysis, Cochran’s heterogeneity statistic is \(Q=\sum_i w_i(\hat\theta_i-\hat\theta)^2\), with \(k-1\) degrees of freedom under the common-effect model. In a deletion analysis, recompute the pooled estimate first and then recompute \(Q\) using only the retained cohorts. The degrees of freedom become \(k-2\), provided at least two cohorts remain. Related summaries such as \(I^2\) should likewise be recalculated for each subset rather than copied from the full analysis.

A falling \(Q\) after deleting a cohort can indicate that the omitted estimate was discordant with the others. But \(Q\) also depends on the number of cohorts and their precision; it is not a measure of how far the pooled estimate moves. Conversely, an omitted cohort may materially change the pooled effect without producing an especially dramatic change in \(Q\). Report effect influence and heterogeneity influence as distinct findings.

Random-Effects Analysis Requires Refitting

In a random-effects model, the cohort-specific true effects vary around a mean effect. A common formulation is \(\theta_i\sim N(\theta,\tau^2)\), with observed estimate \(\hat\theta_i\) having sampling variance \(SE_i^2\) around \(\theta_i\). Given an estimate of \(\tau^2\), the random-effects weight is \(w_i^*=1/(SE_i^2+\tau^2)\). The pooled mean uses these weights, and its conventional standard error is \(\sqrt{1/\sum_iw_i^*}\).

For LOCO analysis, do not merely remove a cohort while keeping the full-data \(\tau^2\) fixed unless that is an explicitly chosen, separate sensitivity analysis. The usual procedure refits the heterogeneity estimate and the pooled effect on each retained set. The standard error can then decrease after a deletion if the estimated \(\tau^2\) falls enough to outweigh the lost cohort information. This is why “one less cohort must mean a wider interval” is not generally correct for random effects.

How to Read the Influence Pattern

A useful report gives the full-data result and a row for every omitted cohort. For each row, include the pooled effect, its standard error or confidence interval, and the heterogeneity summary relevant to the chosen model. Also report \(\Delta_i\), which places the effect change directly on the scientific scale. If the effect is a log-odds ratio, for example, \(\Delta_i\) is in log-odds units; exponentiating an estimate gives the corresponding odds ratio. For other effect scales, retain their appropriate units.

Influence has no universal cutoff. A change that matters scientifically depends on the effect scale and the question: movement across a clinically or biologically relevant threshold may matter even if it is modest relative to a standard error. Likewise, a confidence interval that crosses a conventional significance threshold after one deletion does not establish that the cohort “caused” significance. LOCO results are descriptive sensitivity checks, not a replacement for prespecified replication criteria or a formal test of whether a cohort should be included.

Precision, Diagnostics, and Limitations

A forest plot helps locate estimates that are far from the pooled result, but visual distance alone does not measure influence. In the fixed-effect model, the deletion formula shows why precision matters: influence depends jointly on \(w_i\) and the discrepancy \(\hat\theta_i-\hat\theta\). A low-precision outlier may have little effect on the pooled estimate, even while contributing to heterogeneity. Conversely, removing a very precise cohort can shift the pooled estimate appreciably even if its estimate is not the most visually extreme.

Check that every deletion uses the intended cohorts and that the weight, standard error, and heterogeneity calculations are recomputed consistently. A cohort with different phenotype definitions, ancestry composition, genotyping quality, or analytic adjustment may warrant investigation, but those characteristics do not determine the direction of bias automatically. With very few cohorts, both heterogeneity estimates and deletion patterns are unstable; a single omission may leave too little information for a meaningful heterogeneity assessment.

The main alternative is to examine only the full-data forest plot or heterogeneity statistic. Those summaries describe the observed set collectively but can obscure dependence on a particular cohort. A formal outlier procedure asks a different question and entails additional assumptions; LOCO analysis itself does not label observations as outliers or supply a universal decision threshold. It is also distinct from rerunning a meta-analysis with a different model: LOCO holds the chosen model specification fixed across subsets so that changes can be attributed to cohort omission, while still refitting model parameters such as \(\tau^2\) when the model requires it.

Common Pitfall

A second mistake is to report only whether statistical significance changes. The effect estimate, its uncertainty, and heterogeneity answer different questions and should all be examined. A cohort can leave the pooled estimate nearly unchanged while substantially changing \(\tau^2\), or shift the pooled estimate while barely changing the heterogeneity summary.