Why a Small Marker p-Value May Be Circular
A marker test compares expression between groups of cells. Its usual reference distribution assumes that the groups are fixed independently of the expression values being tested. But if a clustering rule uses a gene to create the groups, that same gene is likely to differ between them by construction. A conventional test can then report an impressive-looking p-value even when there were no pre-existing populations to discover.
The example below isolates this issue with two genes. Clustering uses Gene A alone, and the resulting labels are then compared on Gene A and Gene B. The contrast is not between two competing clustering methods: it is between reusing the clustering input for a marker test and testing a gene that did not directly determine the labels.
All values below are hypothetical normalized expression measurements in log-expression units. Suppose cells are independent observations, and the clustering rule assigns a cell to group H when its Gene A value exceeds 5, and to group L otherwise. The cutoff and the gene used for clustering are specified as part of this toy setup. In real analyses, a more complicated clustering algorithm may use many genes and may select groups in less transparent ways.
Worked Example: Forming the Groups From Gene A
Worked Example: Forming the Groups From Gene A
Consider these eight hypothetical cells. The group labels follow the stated rule: Gene A greater than 5 gives H; otherwise, the label is L.
| Cell | Gene A | Gene B | Group from Gene A |
|---|---|---|---|
| 1 | 1 | 2 | L |
| 2 | 2 | 4 | L |
| 3 | 3 | 5 | L |
| 4 | 4 | 7 | L |
| 5 | 7 | 3 | H |
| 6 | 8 | 4 | H |
| 7 | 9 | 6 | H |
| 8 | 10 | 7 | H |
There are four cells in each group. For Gene A, the group means are \(\bar{x}_L=(1+2+3+4)/4=2.5\) and \(\bar{x}_H=(7+8+9+10)/4=8.5\). The observed H-minus-L difference is \(8.5-2.5=6\) log-expression units.
That difference accurately describes these assigned groups. It does not, by itself, show that Gene A independently distinguishes naturally existing cell populations: the rule made groups using Gene A in the first place.
The Naïve Test for the Gene Used in Clustering
To see the source of the problem, calculate the ordinary pooled two-sample t statistic for Gene A as if the group labels had been fixed independently of its expression. This conventional test assumes independent observations, normally distributed expression within each group, and equal population variances. Under a valid fixed-group null hypothesis of equal means, its statistic follows a t distribution with \(df=n_H+n_L-2\). Those assumptions about the reference distribution are not enough here: the labels were created from the tested values.
Worked Example: The Nominal Gene A t Test
For group L, the deviations from its mean 2.5 are \(-1.5,-0.5,0.5,1.5\). Their squared sum is \(5\), so \(s_L^2=5/(4-1)=1.667\). Group H has the same squared deviations around 8.5, giving \(s_H^2=1.667\). The pooled variance and standard error are therefore
The H-minus-L statistic is
If the groups really had been fixed independently and the usual model held, this would give a two-sided p-value of about 0.0006. In this example, however, the Gene A values determined the labels. The value 0.0006 is thus a nominal p-value from a test whose group-selection mechanism was left out; it is not a calibrated probability for the actual data-adaptive procedure. It should not be presented as valid evidence against the no-population-difference explanation.
Why the Ordinary Reference Distribution Fails
Under an ordinary two-sample t test, the null model treats the group labels as fixed, or as determined independently of the tested gene. That lets the test compare the observed mean difference with differences that could arise from random sampling under equal means. If the labels are instead functions of the tested gene, the observed difference is selected to be large, and the ordinary reference distribution does not include that selection.
A simple normal model illustrates the mechanism. Suppose, under a no-population-difference model, a single gene value \(X\) follows \(N(0,\sigma^2)\) in every cell, and labels are assigned by its sign: positive values to H and negative values to L. By symmetry, \(E[X\mid X<0]=-E[X\mid X>0]\). For a normal variable, integrating its density over the positive half-line gives
Consequently, the expected H-minus-L difference after this sign-based selection is \(2\sigma\sqrt{2/\pi}\), which is positive even though the same distribution generated every cell. This calculation is not the precise selection rule in the table; it shows why sorting or partitioning on a gene can create separation under a null model. The earlier material on selection bias in a Gaussian marker test develops this general issue; the toy data make its practical consequence visible.
Testing a Gene Excluded From Clustering
Gene B did not enter the stated clustering rule. That removes the most direct form of circularity for its test, but does not automatically make every test of Gene B valid. For the simple argument here, assume that cells are independent; that under the null Gene B has the same normal distribution in both groups; that its variance is equal across groups; and, crucially, that Gene B is independent of Gene A. Because the labels depend only on Gene A, this independence means those labels do not select Gene B values to be high or low. Given the group sizes, the ordinary pooled t reference distribution is then appropriate under the stated model.
Worked Example: Testing Excluded Gene B
For Gene B, the L values are \(2,4,5,7\), with mean \(\bar{x}_L=18/4=4.5\). The H values are \(3,4,6,7\), with mean \(\bar{x}_H=20/4=5\). The observed H-minus-L difference is \(0.5\) log-expression units.
The squared deviations from 4.5 in L sum to \(6.25+0.25+0.25+6.25=13\), so \(s_L^2=13/3=4.333\). In H, the squared deviations from 5 sum to \(4+1+1+4=10\), so \(s_H^2=10/3=3.333\). Thus,
The test statistic and degrees of freedom are
The two-sided p-value is approximately 0.73. Under the stated independent, equal-variance normal model, this is not evidence of a difference in mean Gene B expression between the two groups. It does not prove that the means are equal, and it does not establish that Gene B is biologically unrelated to cell identity. The result is conditional on the toy model and its assumptions.
Assumptions, Diagnostics, and Limits
Excluding a gene from the clustering inputs addresses direct reuse, not every source of dependence. If Gene B is strongly correlated with Gene A, then selecting groups using A can also select B values indirectly. Shared technical effects, normalization choices, or other genes used in clustering can likewise connect the tested gene to the labels. In those cases, the ordinary p-value for Gene B may still be poorly calibrated, even though B itself was not entered into the clustering algorithm.
The pooled t test also relies on its distributional assumptions. With only four cells per group, normality and variance equality cannot be assessed reliably from the toy data. In larger data sets, examine group distributions, outliers, and spread, while remembering that diagnostics do not repair selection bias. If cells from the same donor are dependent, treating every cell as an independent replicate can understate uncertainty; the simple cell-level calculation here does not handle that design.
The practical comparison is therefore conditional. A test of the clustering gene is directly circular under this setup. A test of an excluded gene can have a valid ordinary reference distribution when the gene is independent of the clustering inputs and the model assumptions hold. In an actual analysis those conditions require scientific and design-based scrutiny; exclusion alone is not proof that they hold. The alternative of treating the observed expression difference as a descriptive summary avoids a claim of calibrated statistical significance, but does not answer an inferential question. Approaches that explicitly account for selection or use independent data require additional procedures beyond this toy example.
Record which genes or measurements helped assign the cells to clusters.
A group mean difference describes the observed labels; a p-value needs a reference model that accounts for how those labels arose.
Direct exclusion helps, but correlation or shared data structure can still transmit selection.
Report the contrast and units, the assumptions supporting inference, and any unresolved dependence or selection concerns.
Common Mistake
Wrong reasoning: “The Gene A test has \(p=0.0006\), so there is strong evidence that the clusters differ in Gene A.” This treats the labels as if they had been set without looking at Gene A.
Correction: The six-unit observed difference is real as a description of these assigned groups, but the nominal p-value does not account for selecting the groups with Gene A. It cannot be interpreted using the ordinary fixed-group t-test calibration. For Gene B, a conventional test is more defensible only if its independence from the clustering inputs and the other model assumptions are plausible.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In the toy data, why is the Gene A difference of 6 log-expression units descriptive but its nominal p-value not calibrated?
- What assumptions make the ordinary Gene B t test defensible in the example?
- If Gene B was excluded from clustering but is highly correlated with Gene A, what concern remains?
- For Gene B, calculate the pooled standard error from \(s_p^2=23/6\) and four cells per group. What is the resulting t statistic for the H-minus-L difference?
- Does a p-value of approximately 0.73 establish that the two groups have equal mean Gene B expression? Explain.
Solutions. (1) The difference summarizes the values in the groups created by the threshold rule. Because Gene A determined those labels, the ordinary t reference distribution ignores selection and its nominal p-value is not calibrated for this procedure. (2) The simple test assumes independent cells, normal Gene B values with a common variance under the null, and independence of Gene B from Gene A, which determines the labels. (3) Correlation can make Gene B values indirectly selected by the clustering rule, so exclusion does not alone guarantee a valid p-value. (4) \(SE=\sqrt{(23/6)(1/4+1/4)}=\sqrt{23/12}=1.384\); hence \(t=0.5/1.384=0.361\), with \(df=6\). (5) No. The p-value indicates that this difference is not unusual under the stated null model; it does not prove equality or give the probability that the null is true.