External Validation of Differential-Abundance Findings

What External Validation Is Intended to Establish

A differential-abundance result is useful only if its meaning can be carried beyond the data in which it was first found. External validation evaluates a prespecified finding in a separate cohort whose donors were not used to discover or select that finding. The question is not simply whether the second analysis produces another small p-value. It is whether the same cell population, comparison, denominator, and effect scale yield a suitably compatible estimate in independent data.

The unit of replication is the donor, not the individual cell. Cells from one donor share biological and technical features, so treating thousands of cells as independent evidence can make uncertainty appear much smaller than it is. For a donor-level proportion analysis, first calculate the proportion for each donor, then estimate the group contrast across donors. The resulting effect describes a difference in observed proportions; it does not by itself establish a difference in absolute cell number. This distinction is central to interpreting relative abundance versus absolute cell number.

Before analyzing the validation cohort, specify the cell type, the condition contrast, the population of donors to which the result is intended to apply, the denominator, and the effect measure. For example, an effect might be the case-minus-control difference in the proportion of a defined T-cell subtype among all captured immune cells. A difference in percentage points is not interchangeable with a fold change or a log-ratio effect. Whichever scale was prespecified should be used in both cohorts.

Estimating a Donor-Level Effect

Suppose each donor \(i\) has a count \(Y_i\) of cells assigned to the target type and a count \(N_i\) in the chosen denominator. The observed donor-level proportion is \(q_i=Y_i/N_i\). For a simple two-group comparison, let \(\bar{q}_1\) and \(\bar{q}_0\) be the sample mean donor proportions in cases and controls. The estimated case-minus-control effect is \(\hat{\theta}=\bar{q}_1-\bar{q}_0\), measured in proportion units. Multiplying by 100 expresses the same effect in percentage points.

This simple estimate is appropriate when donors are independent, each donor contributes one relevant profile, and the target comparison is not distorted by unaccounted-for design differences. A t-based interval additionally relies on donor-level proportions being reasonably well behaved for the sample size, without extreme influential observations; with very small samples, the approximation may be fragile. If the analysis uses covariate adjustment or a model suited to proportions, its adjusted estimate and standard error can be used instead, provided the estimand and effect scale remain comparable across cohorts.

Here \(s_1\) and \(s_0\) are the sample standard deviations of donor proportions in the two groups, and \(n_1\) and \(n_0\) are their donor counts. The displayed standard error is the independent-groups Welch standard error. It does not treat cells as independent observations. A confidence interval for the effect quantifies sampling uncertainty under the analysis assumptions; it is not a probability statement about whether a fixed parameter lies in the realized interval.

Comparing the Discovery and Validation Estimates

Let \(\hat{\theta}_D\) denote the discovery estimate and \(\hat{\theta}_V\) the estimate from an independent validation cohort. The direct comparison is their difference, \(\hat{\Delta}=\hat{\theta}_V-\hat{\theta}_D\). If the estimates are independent and their standard errors are estimated on the same scale, the standard error of this difference is the square root of the sum of their variances. A confidence interval for \(\Delta\) helps assess the size of disagreement compatible with the data.

The 1.96 multiplier gives an approximate two-sided 95% interval when the difference is approximately normal. A t-based or model-based interval may be preferable when sample sizes are small or the estimates come from a more complex analysis. Independence matters: if cohorts overlap, or if estimates share data, their covariance must be accounted for rather than using the independent-estimates formula.

A confidence interval for the difference that includes zero says the data do not clearly distinguish the two effects at that confidence level. It does not prove the effects are equal. To claim that effects are sufficiently close for a particular scientific purpose, investigators need a prespecified acceptable difference, such as a margin of \(\pm 0.02\) in proportion units, and an interval-based equivalence assessment. The margin must be justified in context; it cannot be chosen after seeing the validation result.

Align the Biological and Statistical Targets

A comparison is interpretable only if both cohorts estimate the same target. “The same cell type” requires more than matching labels in two analysis files. Check the marker evidence, annotation rules, and resolution used to define the population. If one cohort labels a broad parent population while the other separates its subtypes, the corresponding proportions need not represent the same biological quantity. When annotations cannot be harmonized confidently, report the uncertainty and avoid presenting the numerical comparison as direct validation.

The denominator must also match. A proportion among all captured cells, among immune cells, or among a specific lineage answers a different question. Even with an identical numerator label, changing the denominator changes the estimand. In addition, differences in tissue processing, capture, or cell recovery may change observed proportions. A matched observed-proportion effect is evidence about those measured proportions, not automatically about tissue abundance in absolute terms.

Finally, compare like with like in study design. Confirm that the direction of the contrast is consistent (for example, case minus control in both cohorts), that covariates and adjustments support a comparable target, and that the relevant donor populations are sufficiently similar for the intended generalization. If a substantial cohort difference is scientifically expected, report cohort-specific effects rather than treating a discrepancy as a simple technical failure.

Diagnostics, Alternatives, and Common Pitfalls

Before interpreting a validation comparison, inspect donor-level proportions and group summaries in each cohort. Look for extreme donors, strong skew, unusually large differences in donor counts, and patterns suggesting that a few donors drive the group contrast. Check that each donor contributes once to the analysis and that missing profiles or inconsistent filtering have not changed the denominator. Review estimates and intervals for all prespecified cell types, not only the one with the most favorable validation result. If many candidates are explored in the validation cohort, multiplicity and selection affect how confirmatory the resulting evidence is.

External validation is not the only way to assess reproducibility. Internal resampling or cross-validation can assess stability within the discovery data, but it does not provide an independent cohort check. A joint analysis that combines cohorts can estimate an overall effect, but by itself may obscure cohort-specific disagreement. Reporting each cohort’s estimate and uncertainty alongside any combined summary makes that disagreement visible. When cohort estimates are not comparable because of different definitions or target populations, a combined estimate is not a remedy.

• Wrong reasoning: “The discovery result has \(p<0.05\), and the validation result has \(p>0.05\), so the effect failed to replicate.” Correction: Compare the effect estimates and uncertainty. A less precise validation estimate may be compatible with the discovery effect even when its own test is nonsignificant.
• Wrong reasoning: “Both cohorts are significant, so the effects agree.” Correction: Significance in both cohorts does not show that their magnitudes are similar; directly examine the difference and its interval.
• Wrong reasoning: “The cell-type labels match, so the proportions are comparable.” Correction: Check annotation rules, cell-type resolution, denominator, contrast direction, and adjustment before comparing estimates.