Validating Markers In An Independent Dataset

Why Validate Markers in a Separate Dataset?

A marker discovered in a dataset may appear strongly different between clusters partly because the same expression values helped define those clusters. Testing that difference as if the clusters had been fixed in advance can understate uncertainty. An independent dataset offers another route: use the discovery data to choose a marker claim, then test that claim in data that played no role in choosing it.

Independence alone is not enough. A validation analysis must also specify what will be tested before examining validation expression: the gene, the cluster correspondence, the contrast, the effect scale, and the testing procedure. If the investigator searches validation results for the most favorable genes or cluster labels and reports only those, the validation data have become discovery data too.

For example, a claim might be: “Gene G has higher mean log2 normalized expression in the population corresponding to discovery cluster A than in the population corresponding to discovery cluster B.” This is more specific than “Gene G is a marker.” It states the gene, the comparison, and the expected direction. A two-sided test can assess any difference; a one-sided test is appropriate only when the direction was chosen in advance and a difference in the opposite direction would not count as support.

Designing the Validation Before Looking at Expression

First define how cells in the validation dataset will be assigned to the populations being compared. A prespecified reference-based label transfer rule or a classifier fitted on discovery data can be applied without refitting it to maximize the validation contrast. The correspondence between discovery labels and validation labels must also be fixed using information other than the expression values for the gene being tested. Otherwise, choosing the mapping that makes the candidate look strongest is another form of selection.

Next specify the gene-level outcome and the biological unit of replication. In a typical design, cells from each biological sample are aggregated within each assigned cluster to form a pseudobulk expression summary. Let \(Y_{igA}\) and \(Y_{igB}\) denote the summaries for gene \(g\) from donor \(i\) in the populations corresponding to clusters A and B. A paired donor contrast is \(D_{ig}=Y_{igA}-Y_{igB}\). If the summaries are on a log2 scale, \(D_{ig}\) is a log2 expression difference; exponentiating it gives the corresponding ratio of expression summaries.

Cells from the same donor are not independent biological replicates. Treating hundreds of cells from a few donors as hundreds of independent observations generally makes standard errors too small. The analysis should reflect the sampling design, commonly by using donor-level pseudobulk summaries or an appropriate model that accounts for donor dependence. The examples below use donor-level contrasts so that the unit of analysis is explicit.

Paired Inference for a Prespecified Contrast

For paired donors, suppose \(D_{ig}\) is the validation contrast for donor \(i\), and there are \(n\) independent donors with both cluster summaries available. The target is the population mean donor-level contrast \(\mu_D\). A one-sample t procedure assumes the donor contrasts are independent and identically distributed, and normally distributed when an exact small-sample t reference distribution is required. With a sufficiently large number of donors, approximate inference may be reasonable under less restrictive distributions, but strong skewness or influential donors still require attention.

For the two-sided null hypothesis \(H_0:\mu_D=0\) against \(H_a:\mu_D\ne0\), calculate the sample mean \(\bar{D}\), sample standard deviation \(s_D\), and standard error \(SE=s_D/\sqrt{n}\). Under the stated assumptions, the statistic has a t distribution with \(df=n-1\) when \(H_0\) holds. A confidence interval for the mean contrast is \(\bar{D}\) plus or minus the appropriate t critical value times \(SE\).

The p-value is the probability, under \(H_0\) and the model assumptions, of obtaining a t statistic at least as extreme in absolute value as the observed statistic. It is not the probability that \(H_0\) is true. The confidence interval describes values of the population mean contrast compatible with the data and model at the stated confidence level; it is not a probability statement about the fixed parameter.

Derivation and Interpretation of the Test

The t statistic follows by standardizing the sample mean contrast. Under the paired model, \(D_{ig}\) has mean \(\mu_D\) and variance \(\sigma_D^2\). If the contrasts are normally distributed, then \((\bar{D}-\mu_D)/(\sigma_D/\sqrt{n})\) is standard normal. Replacing the unknown \(\sigma_D\) with the sample standard deviation \(s_D\) adds uncertainty. Under normal sampling, \((n-1)s_D^2/\sigma_D^2\) has a chi-squared distribution with \(n-1\) degrees of freedom and is independent of the standardized mean. Their ratio therefore has a t distribution with \(n-1\) degrees of freedom. Under \(H_0:\mu_D=0\), this ratio is exactly the statistic used above.

The argument depends on the prespecified contrast and valid independent donor observations. It does not correct a wrong cluster mapping, a biased assignment rule, missingness related to expression, or a contrast selected after inspecting validation results. The inferential calculation cannot repair an invalid design.

Mapping, Multiplicity, and Failure Modes

A validation contrast only answers the intended question if the validation populations correspond to the intended discovery clusters. If the labels are reversed, the sign of the contrast reverses. If cells are assigned using the candidate gene’s validation expression and that assignment is then used to test the same gene, the analysis has reintroduced circularity. Freeze the assignment procedure and correspondence independently of the tested expression, and report how many donors contribute observations to each side of the contrast.

When several genes are prespecified, testing each at level \(\alpha\) does not keep the probability of at least one false positive among the whole list at \(\alpha\). The analysis must account for the declared family of tests, for example with a familywise-error or false-discovery-rate procedure appropriate to the inferential goal. Reporting only the genes that passed, while hiding the rest of the prespecified list, makes the validation appear stronger than it is. If only one candidate and one contrast were genuinely prespecified, the single-test p-value addresses that test; it does not license unplanned searches over alternative contrasts.

Useful diagnostics include inspecting donor-level contrasts, checking for influential donors, verifying that each donor has adequate representation in both populations, and confirming that the cluster assignment rule was applied as specified. Large between-donor differences, an outlying donor that drives the mean, or a striking imbalance in cell counts may make a simple t analysis questionable. With few donors, distributional assumptions are difficult to assess from a plot; sensitivity analyses and transparent uncertainty are preferable to treating cell counts as a substitute for biological replication.

How Independent Validation Differs From Other Approaches

Independent-dataset validation and sample splitting both separate selection from testing, but they do so differently. Sample splitting reserves part of one study for testing and may reduce the sample size available for both stages. Independent validation uses a separate cohort, which can provide a stronger check of reproducibility across samples, but it requires compatible assays, populations, and cluster definitions. The earlier material on sample splitting addresses the separate discovery and testing portions of a single dataset.

Holding out genes used for clustering can reduce direct reuse of the tested gene’s expression in forming clusters, but it does not by itself establish replication in a new cohort. Conversely, an independent cohort does not remove every source of bias: candidate choice, outcome definition, and reporting decisions still need to be controlled. Selective-inference methods address a different problem by adjusting inference for selection using the same data; they are not a substitute for a correctly designed external validation.

Independent validation has limits. Different sequencing protocols, sample handling, cell composition, or disease mix can change measured expression or the populations represented. A failure to replicate may reflect low power or a genuine lack of transportability, not necessarily a false discovery in every setting. A positive result supports the specific gene, contrast, and target population tested; it does not automatically validate all markers, all cluster comparisons, or a causal interpretation.

Common Mistake

A second frequent error is to count cells as independent replicates when cells are nested within a small number of donors. The number of cells can improve the precision of a donor’s expression summary, but it does not create additional independent donors. Inference should match the biological sampling unit and the study design.