Leave-One-Cohort-Out Checks for Meta-Analysis

Why Recompute the Meta-Analysis?

A pooled estimate can summarize evidence across cohorts while concealing how much each cohort contributes. One large or unusual cohort may strongly affect the estimate, its uncertainty, or whether an interval includes the null value. A leave-one-cohort-out check makes that dependence visible: omit one cohort, recompute the meta-analysis, and repeat for every cohort.

The purpose is sensitivity analysis. A substantial change after an omission flags a result that depends on that cohort under the chosen model. It does not establish that the cohort is erroneous, that the full result is invalid, or that the omitted-cohort result should replace the primary analysis. Interpret the check alongside the cohort’s design, measurements, and eligibility, as well as the meta-analysis assumptions.

Fixed-Effect Leave-One-Out Calculations

Under a fixed-effect model, the assumption is that all cohorts estimate one common true effect \(\theta\). Their observed estimates differ because of sampling error, and the estimates are independent with variances \(v_i\) that are adequately estimated. Effects must be on a comparable scale and in the same direction—for example, all log odds ratios defined with the same event and reference group.

Inverse-variance weighting assigns cohort \(i\) weight \(w_i=1/v_i\). The pooled estimate and its model-based standard error are:

After omitting cohort \(j\), use only the remaining weights and estimates:

These formulas show why precision matters: a cohort with a small standard error has a large weight and can move the pooled estimate substantially if its estimate differs from the others. For the fixed-effect model, the leave-one-out estimate can also be obtained from the full estimate by subtracting the omitted cohort’s weighted contribution, but direct recalculation is often clearer and less error-prone.

Random-Effects Checks Must Refit Heterogeneity

A random-effects model allows cohort-specific true effects to vary around a mean effect. In the usual formulation, observed estimates have sampling variance \(v_i\), while the true cohort effects have between-cohort variance \(\tau^2\). The marginal variance used for pooling is \(v_i+\tau^2\). This approach assumes independent cohorts, comparable effect definitions, and a model in which the cohort effects are reasonably represented by a distribution centered on a common mean.

The random-effects estimate uses weights \(w_i^*=1/(v_i+\tau^2)\). Because \(\tau^2\) is estimated from the included cohorts, a leave-one-out analysis should generally estimate \(\tau^2\) again after each omission, then recompute the weights, pooled estimate, and standard error. Holding the full-data \(\tau^2\) fixed answers a different, conditional sensitivity question; it does not show what the complete random-effects procedure would produce on the reduced set.

There are several estimators of \(\tau^2\). To make the next calculation explicit, use the DerSimonian–Laird estimator. For a set of \(m\) cohorts, first calculate fixed-effect weights \(w_i=1/v_i\), their fixed-effect estimate, and Cochran’s \(Q=\sum_i w_i(y_i-\hat{\theta}_{FE})^2\). Then calculate:

The truncation at zero prevents a negative estimated variance. This estimator is used here for transparent arithmetic, not because it is universally preferable; other estimation methods can yield different leave-one-out results. With \(\hat{\tau}^{\,2}_{DL}\), pool using \(w_i^*=1/(v_i+\hat{\tau}^{\,2}_{DL})\), with model-based standard error \(\sqrt{1/\sum_i w_i^*}\). With few cohorts, both heterogeneity estimation and the resulting uncertainty can be unstable; the calculation should not be presented as precise evidence about the distribution of effects.

Interpreting Influence and Diagnosing Problems

A useful first summary is the change in the pooled point estimate, \(\hat{\theta}_{(-j)}-\hat{\theta}\), for each omitted cohort, using the same model and effect scale. Also compare standard errors or confidence intervals, the direction and practical size of the estimate, and—where relevant—whether a stated decision or conclusion changes. Report the full-data result and all omission results rather than only the most dramatic omission.

There is no universal change threshold that makes a cohort “influential.” A small change in the estimate may still matter if the scientific interpretation is near a decision boundary; a larger change may have little practical meaning on a different scale. P-values can change when standard errors change, so crossing a conventional significance threshold is not, by itself, a complete account of influence. Interpret effect magnitude and uncertainty in context.

Check that estimates, standard errors, and effect directions were aligned consistently before pooling. Review whether one cohort has unusually high precision, a distinct measurement platform, a different target population, or a design feature that could explain its contribution. Inspect the cohort-specific estimates and intervals, and compare heterogeneity summaries across omissions. A marked change in estimated \(\tau^2\) can signal that one cohort strongly affects the estimated spread, but it can also reflect the instability of heterogeneity estimation with few cohorts.

The calculation also depends on the cohorts being independent. Participant overlap can make the assumed covariance structure wrong and cause the weights and uncertainty to be misleading. Similarly, an omitted cohort must not be removed simply because its result is inconvenient: the check is a planned sensitivity analysis, not a data-driven rule for selecting a preferred answer.

What the Check Can and Cannot Establish

Leave-one-cohort-out analysis is most informative when the number of cohorts, their estimates, and the changes after omission are all reported. It complements—but does not replace—other ways of assessing robustness. For example, a fixed-effect and a random-effects analysis answer different model-based questions: the former assumes one common true effect, while the latter allows variation in true cohort effects. Neither model becomes appropriate merely because its results are more reassuring.

A single-cohort omission is not the same as a subgroup analysis, an influence diagnostic based on a different statistical quantity, or an analysis that removes several cohorts at once. It can miss joint dependence: two cohorts may each have modest individual influence but substantial combined influence. It also cannot repair biased cohort estimates, incompatible effect definitions, selection bias, or incorrect standard errors. With very few cohorts, omission leaves little information, especially for estimating \(\tau^2\); changes can be large simply because the remaining evidence is sparse.

Use this check to describe robustness, not to certify it. If the conclusion changes materially, say which cohort’s omission changes which result, quantify the difference, and explain plausible design or measurement reasons without asserting causation. If it does not change, that supports stability to these particular one-at-a-time omissions, not robustness to every possible source of bias or modeling choice.