Why the Null Depends on How Clusters Were Made
A marker test asks whether a gene differs across groups of cells. Its interpretation depends on what those groups represent and how they were obtained. If group labels were fixed independently of a gene’s measured expression, a test can compare the gene’s values across those labels under an appropriate sampling model. If the labels were chosen using the same expression data, however, the observed contrast may partly reflect the selection rule itself. The ordinary fixed-label reference distribution may then fail to describe how extreme the statistic would be under the relevant null.
This distinction is central to the double-dipping problem: the same observations help define a question and answer it. The earlier material on double dipping motivates the issue; here the goal is to state precisely what null is being tested and why a null distribution must reflect the route by which the clusters were selected.
A mean-equality null is not the only possible scientific null. One might instead specify equal expression distributions, or equality of a parameter in a count model. These claims are not interchangeable: equality of means does not, by itself, imply equality of variances or of the entire distributions. A test statistic and its reference distribution must match the estimand and assumptions being used. In the examples below, expression is treated as a quantitative measurement and the mean is the estimand.
In single-cell data, “independent observations” also requires care. Cells from one donor may share biological and technical features. A cell-level test that treats all cells as independent may therefore have an incorrect reference distribution even if the cluster labels were fixed in advance. The fixed-label examples below explicitly assume independent observations; they do not establish that this assumption is suitable for every study.
Fixed Labels: What the Ordinary Test Assumes
For two fixed groups, a pooled two-sample t test compares their sample means. Its exact t reference distribution requires independent observations, normally distributed expression within each group, and a common population variance. Under the null of equal population means, the statistic has a t distribution with \(n_1+n_0-2\) degrees of freedom. If the labels were selected from the tested expression values, those fixed-label assumptions alone are not enough: the test’s reference distribution does not account for selection.
Worked Example: A Mean Comparison With Prespecified Labels
Suppose a hypothetical experiment has two groups of three independent cells, with labels fixed before examining gene \(g\). The measured expression values are 1, 2, and 3 units in group 1, and 4, 5, and 6 units in group 0. Assume, for this example, that expression in each group is normally distributed and that the groups have a common population variance. Test \(H_0:\mu_{g1}=\mu_{g0}\) against a two-sided alternative.
The group means are \(\bar{x}_1=(1+2+3)/3=2\) units and \(\bar{x}_0=(4+5+6)/3=5\) units. The sample variances are both 1 square unit, so the pooled variance is \(s_p^2=\{(3-1)(1)+(3-1)(1)\}/(3+3-2)=1\) square unit. Thus \(s_p=1\) unit.
The estimated difference, group 1 minus group 0, is \(-3\) units. Its standard error and t statistic are
The degrees of freedom are \(df=3+3-2=4\). Under the stated normality, equal-variance, independence, and fixed-label assumptions, the two-sided p-value is approximately 0.0213. At \(\alpha=0.05\), this is evidence against equal mean expression for these prespecified groups. It does not establish that the group labels are biologically meaningful, nor does it support a causal claim.
The assumptions are deliberately explicit. With only three observations per group, equal observed sample variances do not establish equal population variances, and independence alone does not justify the pooled t reference distribution. If the labels had instead been selected using these six expression values, the calculated p-value would still be the fixed-label t-tail area, but it would not automatically be a calibrated post-selection p-value.
What Changes After Selection
Write \(A(X)\) for a clustering procedure applied to expression data \(X\), and \(C=A(X)\) for the resulting labels. A marker statistic \(T_g(X,C)\) is then computed using both the expression values and the labels. When gene \(g\) contributes to clustering, \(C\) can depend directly on \(X_{ig}\). Even when the clustering uses other genes, dependence between those genes and \(g\) can make the selected labels informative about \(g\).
The issue can be expressed without committing to a particular clustering algorithm. Let \(S\) denote the event that the procedure selected the observed labels or a specified class of clusterings. For a statistic whose large values are evidence against a null, the relevant conditional tail probability is \(P(T_g\geq t\mid S,H_0)\), not necessarily the fixed-label quantity \(P(T_g\geq t\mid H_0)\). By the definition of conditional probability,
The fixed-label tail probability is appropriate after selection only if the selection does not alter the statistic’s null distribution, or if the analysis otherwise accounts for that alteration. If selection and the test statistic are dependent under the null, the conditional tail area can differ from the ordinary one. This is the justification for distinguishing the hypotheses and reference distributions; it does not require that every data-adaptive clustering procedure produce the same degree or direction of distortion.
Worked Example: A Gene Defines the Cluster Boundary
Consider five hypothetical expression values for one gene: \(-2,-1,0.5,1,\) and \(2\) units. Suppose a rule assigns cells with expression at most 0 to one cluster and cells above 0 to another. This is a simplified illustration, not a recommended clustering procedure.
The first cluster contains \(-2\) and \(-1\), with mean \((-2-1)/2=-1.5\) units. The second contains \(0.5,1,\) and \(2\), with mean \((0.5+1+2)/3=1.1667\) units. The observed difference, second cluster minus first, is \(1.1667-(-1.5)=2.6667\) units.
The positive contrast is partly guaranteed by the rule: membership in the second group required a value above 0, while membership in the first required a value at or below 0. A fixed-label test that acts as if these labels were chosen independently of this gene does not represent that selection step. The contrast is a valid description of these five values and their constructed groups, but its ordinary fixed-label p-value would not, on its own, test whether the gene distinguishes independently defined populations.
This example does not say that every observed contrast is entirely an artifact, or that the gene has no biological relevance. It shows why the scientific claim and the null model must specify whether the groups were fixed independently or created using the gene’s expression.
Selection Among Candidate Results
Selection can affect the reference distribution even when the choice is among several candidate clusterings or marker statistics rather than a single threshold on one gene. The following toy calculation isolates that point. It assumes independent standard normal statistics for simplicity; real candidate partitions are usually correlated, so the numeric result is illustrative rather than universal.
Worked Example: Choosing the More Extreme of Two Candidate Partitions
Suppose two candidate partitions produce statistics \(Z_1\) and \(Z_2\) for the same gene. Under a null model, assume \(Z_1\) and \(Z_2\) are independent standard normal random variables. The procedure selects the partition with the larger absolute statistic, so the reported statistic is \(M=\max(|Z_1|,|Z_2|)\).
For one prespecified partition, the two-sided standard normal tail probability beyond 1.96 is approximately 0.05. For the selected maximum, the event \(M\geq1.96\) occurs if at least one statistic exceeds 1.96 in absolute value. Independence gives
Thus, although each candidate statistic has a 0.05 two-sided tail probability when considered alone, the chance that the selected statistic crosses 1.96 is 0.0975 under these assumptions. Reporting the single-candidate 0.05 tail area after choosing the more extreme candidate would understate the probability of such a selected result. If the two statistics are dependent, the value need not be 0.0975, but the selection rule still needs to be considered when establishing calibration.
Which Null Is the Scientific Question?
It helps to separate three claims that are sometimes blurred together:
- A descriptive contrast: the observed mean of gene \(g\) differs across the labels produced by a stated clustering procedure. This describes the analyzed data; it is not, by itself, a calibrated inferential claim.
- A fixed-label inferential claim: under a model for prespecified groups, the population means or distributions differ. A fixed-label test is justified only when its assumptions and label-generation conditions hold.
- A post-selection inferential claim: the gene differs across labels after accounting for how those labels or the reported statistic were selected. This requires a null model and reference distribution that include the selection process.
The phrase “no marker effect” is therefore incomplete unless it identifies the group definition, the expression estimand, and the sampling or selection model. For example, equal mean expression across externally specified cell types and equal mean expression across clusters defined using the tested gene are different inferential setups. A gene can be a useful descriptive marker of an algorithmically defined partition even when a naïve p-value does not support a valid population-level claim.
Diagnostics, Alternatives, and Limitations
Before interpreting a marker p-value, trace how the labels and the reported gene were chosen. Ask whether the tested gene was used directly in clustering, whether correlated expression features contributed, whether the clustering resolution or number of clusters was selected after inspecting results, and whether the reported statistic was chosen from several genes or contrasts. Also inspect whether cells are plausibly independent for the intended inference; donor-level dependence can undermine a cell-level reference distribution independently of double dipping.
Common signs of a mismatch include labels that sharply separate the tested gene because it helped define the clusters, a reported marker that was selected as the largest contrast, or a nominal p-value interpreted as if labels had been prespecified. Cluster stability does not resolve this problem: reproducible labels and a valid marker-test reference distribution are different properties.
The main alternatives answer different needs. Descriptive reporting states the observed contrast without attaching an uncalibrated inferential claim. Inference with genuinely prespecified labels uses a fixed-label test when its assumptions are defensible. Other strategies can use independent data for confirmation, or explicitly incorporate the selection procedure into calibration; these require their own assumptions and design choices. No single correction follows merely from knowing that clustering occurred. In particular, the appropriate null distribution depends on the full procedure, including preprocessing, feature selection, clustering choices, and the statistic ultimately reported.
Check Your Understanding
Answer each question by identifying the null or assumption at issue.
- A researcher defines two groups before measuring gene \(g\), then compares their mean expression. What does the fixed-label null \(H_0:\mu_{g1}=\mu_{g0}\) state?
- A cluster algorithm uses gene \(g\) to assign cells, and the resulting groups have different observed means for \(g\). Why does that contrast not automatically have a valid ordinary fixed-label p-value?
- In the two-candidate example, each fixed statistic has a two-sided 0.05 tail probability beyond 1.96. What is the probability that the selected maximum exceeds 1.96 under the stated independence assumption?
- Does observing equal sample variances establish the equal-population-variance assumption for a pooled t test? Explain.
- Name one reason a cell-level fixed-label test might have an incorrect reference distribution even if clustering did not use the tested gene.
Solutions. (1) It states that the two prespecified groups have equal population mean expression for gene \(g\), under the model defining those means. (2) The labels depend on the tested expression, so the distribution of the statistic after selection need not match its fixed-label distribution. (3) It is \(1-(0.95)^2=0.0975\), or 9.75%. (4) No. Equal sample variances are observed summaries and do not prove equal population variances. (5) Cells from the same donor may be dependent, violating the independence assumption used by a cell-level test.