Why Repeated Marker Evidence Can Be Circular
A gene can look like an excellent marker for a cluster for two very different reasons. It may genuinely distinguish a biological cell population, or its expression may have helped create the cluster in the first place. In the second case, finding the same gene enriched when comparing the resulting clusters is not independent confirmation: the analysis is partly reporting back a feature of its own construction.
This issue arises when researchers cluster cells using expression measurements and then compare expression between the resulting groups. It is especially clear when a gene is directly included in the features used for clustering, but it can also arise when feature selection, dimensionality reduction, or other steps depend on that gene. The central question is therefore not merely whether a gene differs between clusters, but whether the evidence for that difference is independent of the process that defined the clusters.
For a gene \(g\), a simple marker contrast is the difference in average expression between two groups, such as \(\Delta_g=\bar{x}_{1g}-\bar{x}_{0g}\). A test may compare this contrast with zero under a specified null model. If the groups were formed using \(g\), however, the usual reference distribution for that test generally does not account for the selection step. The resulting p-value can be too small, and the estimated contrast can be too large relative to the contrast one would expect from independent data.
How Selection Creates Apparent Separation
A short derivation shows the basic mechanism. Suppose a standardized gene statistic \(Z\) would follow a standard normal distribution under a null model before selection: \(Z\sim N(0,1)\). Imagine that a feature is retained for a cluster comparison only when \(Z>c\), for a fixed threshold \(c\). After selection, the relevant distribution is not the original standard normal distribution; it is the distribution conditional on \(Z>c\).
Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution functions. The second equality follows because \(\phi'(z)=-z\phi(z)\), so \(\int_c^\infty z\phi(z)\,dz=\phi(c)\). The final quantity is positive for any finite \(c\). Thus, even when the unselected statistic is centered at zero, a statistic selected for being large has a positive conditional mean. A conventional test that ignores this selection treats the selected value as if it had come from the unselected distribution.
In clustering, selection is often more complicated than a single threshold. A gene may influence distances, a low-dimensional representation, a clustering resolution, or the assignment of cells. These steps need not make every defining-gene contrast large in the same direction. But they create a dependence between the observed expression and the groups being compared. The threshold calculation explains why the ordinary test reference distribution cannot simply be assumed to apply after this data-dependent construction.
Worked Example: A Defining Gene Looks Highly Separated
Consider hypothetical data with two clusters, each containing 40 independent observations. Let \(G\) be a log-expression measurement, with mean 1.20 and standard deviation 0.40 in cluster 1, and mean 0.20 and standard deviation 0.40 in cluster 0. Suppose \(G\) was among the features used to form the clusters. For illustration, use the Welch statistic for the difference in means; this calculation does not correct for cluster selection.
The estimated contrast, standard error, and statistic are
If these observations were independent, approximately normal observations from two groups and the groups had been fixed independently of \(G\), a statistic this large would be strong evidence against equal population means. But the groups were constructed using \(G\). The calculation verifies a large observed separation; it does not establish that the nominal Welch p-value has its usual null distribution. In this example, the same expression both helped define the groups and is being offered as evidence that the groups differ. The inference is circular unless the selection mechanism is properly accounted for.
The equal sample sizes and standard deviations make the arithmetic transparent. They do not remove selection bias. Nor does a stable clustering solution establish that a conventional marker p-value is calibrated: stability concerns repeatability of the clustering, not whether the subsequent test accounts for how the groups were chosen.
What a Held-Out Gene Comparison Can Show
A useful diagnostic comparison separates genes that directly contributed to cluster construction from genes that were excluded. If defining genes show extreme contrasts while held-out genes show smaller or more variable contrasts, the pattern is consistent with reuse contributing to the apparent strength of the defining-gene results. It is a diagnostic comparison, not proof that every defining-gene result is false or every held-out-gene result is valid.
For a conventional test of a held-out gene to have its usual interpretation, several conditions matter. The gene must genuinely have been excluded from all relevant steps that formed and assigned clusters, including data-driven filtering or tuning that could depend on its expression. Under the null being tested, its expression must also be suitably independent of the clustering information, conditional on any variables included in the model. Finally, the test’s sampling assumptions must be plausible. In single-cell data, cells from one donor are not automatically independent biological replicates; ignoring donor structure can make standard errors too small.
Excluding a gene from the clustering features is therefore not equivalent to obtaining an independent sample. A held-out gene may be correlated with defining genes because of shared regulation, cell state, or technical effects. Clusters formed from other genes can then differ in the held-out gene even though that gene did not directly determine membership. Such a difference may reflect real biology, correlation with the clustering features, or both. Its interpretation depends on the scientific null and design.
Worked Example: A Held-Out Gene Has a Smaller Contrast
In a second hypothetical comparison, gene \(H\) was not used in feature filtering, dimension reduction, clustering, or assignment. Assume 40 independent observations per cluster, with means 0.30 and 0.10 log-expression units and standard deviations 0.60 and 0.50, respectively. For the stated calculation, assume the observations within each group are approximately normal; do not assume equal variances.
The estimated difference is \(0.30-0.10=0.20\) log-expression units. The Welch standard error and statistic are
The Welch degrees of freedom are approximately
For a two-sided test, these values give a p-value of approximately 0.110 under the stated fixed-group model. The data estimate a difference of 0.20 log-expression units, but they do not provide strong evidence against a zero mean difference at a 0.05 significance level under that model. This is not evidence that the means are exactly equal. More importantly, the calculation is interpretable as an ordinary test only if the held-out status and independence conditions are credible. The fact that \(H\) was omitted from the clustering features alone does not establish them.
Separating Gene Holdout From Sample Holdout
Holding out genes and holding out samples address different sources of reuse. A gene holdout asks whether expression of that gene directly contributed to cluster construction. A sample holdout asks whether the data used to evaluate a claim are independent of the data that produced the clusters or selected the claim. A stronger validation design can use discovery samples to construct clusters and select a fixed assignment rule, then apply that rule in independent validation samples. To test a particular gene there, the assignment rule must not use that gene’s validation expression.
Worked Example: Testing a Defining Gene in Independent Samples
Suppose a hypothetical discovery set was used to define two groups and a fixed assignment rule based on genes other than \(G\). In an independent validation set, that rule assigns 30 independent biological replicates to each group. Their replicate-level mean log expression for \(G\) is 1.00 (standard deviation 0.50) in group 1 and 0.40 (standard deviation 0.50) in group 0. Assume approximate normality and independence across replicates; the validation assignments do not use \(G\).
The validation contrast and Welch statistic are
The Welch degrees of freedom are 58 because the sample sizes and sample variances are equal. A two-sided test gives \(p<0.001\). Under the stated validation-sample assumptions, this is evidence of a difference in mean replicate-level log expression between groups assigned by the fixed rule. Unlike the first example, the validation observations were not used to form the original clusters. The result supports reproducibility of the association under this assignment procedure; by itself, it does not show that \(G\) caused the cell states or that the labels are biologically correct.
This design also illustrates why the unit of analysis matters. The 30 observations per group here are independent biological replicates by assumption, not 30 cells from one donor. If the observations were correlated cells from a small number of donors, treating all cells as independent would not justify the same standard error or p-value.
Comparing Evidence and Diagnosing Failure
When reviewing marker claims, make the data flow explicit: which expression measurements entered filtering, representation, clustering, and assignment; which genes were selected after examining the results; and which samples were used for evaluation. Then compare the defining-gene results with an appropriate held-out analysis, without treating the contrast itself as a calibrated test unless its assumptions are satisfied.
| Approach | What it helps diagnose or address | Important limitation |
|---|---|---|
| Test the same cells for genes used to form clusters | Describes observed separation of defining features | Ordinary fixed-group tests ignore reuse and can overstate evidence |
| Hold out genes from clustering | Checks whether separation also appears in features not directly used | Correlation, indirect filtering, or selection can still connect the gene to cluster formation |
| Hold out samples and use a fixed assignment rule | Evaluates a prespecified claim on observations independent of discovery | Requires a genuinely independent validation set and an assignment rule that does not use the tested expression |
| Use a selection-aware procedure | Can account for specified aspects of cluster or marker selection | Validity depends on correctly representing the selection procedure and its assumptions |
Several warning signs deserve attention. A tested gene may have been called “held out” even though it was used in highly variable gene selection, tuning, or a dimensionality reduction step. A cluster label may have been reassigned after viewing the candidate marker. A large number of candidate genes may have been screened, while only striking results are reported. Or a cell-level test may treat cells from the same donor as independent. These issues concern different parts of the analysis; fixing one does not automatically fix the others.
The earlier material on negative controls is relevant as a complementary diagnostic: negative-control results can reveal systematic false signal, but a successful control does not convert a reused defining-gene test into independent evidence. Likewise, correcting for multiple comparisons across genes addresses the number of tests, not necessarily the selection induced by cluster construction. The inferential target and the process that generated the tested groups must still be specified.
A practical report should identify which genes contributed to clustering, describe which genes were held out and at what stage, and distinguish descriptive contrasts from calibrated inferential results. State the unit of analysis, the assumptions behind uncertainty estimates, and whether cluster assignments were fixed independently of the tested expression. If a comparison is only diagnostic, say so rather than presenting its p-values as confirmatory evidence.
Common Mistake and Key Takeaway
- Wrong reasoning: “This gene has the largest cluster contrast and a tiny ordinary p-value, so it independently confirms the clusters.”
- Correction: If the gene helped construct or select the clusters, the contrast is partly a consequence of the selection process. The ordinary p-value generally does not account for that reuse. Treat the result as descriptive or use an inference strategy whose assumptions cover the selection.
- Wrong reasoning: “The gene was left out of the clustering matrix, so its test is automatically valid.”
- Correction: Check whether the gene influenced filtering or tuning, whether it is associated with the defining features under the null, whether the cells or replicates are independent, and whether marker selection was itself data-driven.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A gene was used in principal-component construction but omitted from the final clustering matrix. Is it necessarily a held-out gene for marker inference? Explain.
- Under the threshold-selection setup, why is the conditional mean of a selected standard normal statistic positive?
- A cluster-defining gene has a large observed contrast and a conventional p-value of 0.001. What can be concluded without a selection-aware analysis?
- What extra condition must hold for a gene-holdout comparison to give an ordinary test its usual interpretation?
Solutions. (1) No. Its expression contributed to the representation used to construct clusters, so it was not excluded from the relevant clustering process. (2) Conditioning on \(Z>c\) removes all values at or below \(c\), and \(E(Z\mid Z>c)=\phi(c)/(1-\Phi(c))>0\). (3) The observed contrast is large, but the nominal p-value does not by itself provide calibrated evidence because it ignores reuse of the gene in cluster construction. (4) Among other assumptions, the gene’s expression under the tested null must be suitably independent of the clustering information, conditional on modeled variables; it must also be excluded from other data-dependent selection steps, and the sampling assumptions must be appropriate.