How Reusing Expression Data Creates Double Dipping

Why Reuse Changes the Question

A cluster can look biologically distinctive because its cells have different expression profiles. But if the same expression measurements both created the cluster labels and are then used to test for differences between those labels, the analysis has used the data twice for related purposes. The observed contrast is real as a description of the data; the difficulty is treating an ordinary test statistic or p-value as if the groups had been specified independently of those data.

For a prespecified two-group comparison, a test asks whether the observed expression difference would be unusual under a null model for the expression values in those fixed groups. After clustering, the groups are not necessarily fixed in that sense. Their membership may depend on the very gene being tested, or on other expression features correlated with that gene. The clustering procedure can therefore make the eventual group contrast unusually large before the test is even calculated.

Definition: Double dipping occurs when the same data contribute to selecting or constructing a feature of an analysis and are then reused to assess that selected feature as though it had been specified independently. In marker analysis, one common form is to use expression data to define clusters and then use those same data to test for differential expression between the resulting clusters.

The distinction established in the earlier material on marker genes is important: a gene can be a descriptive marker because its observed expression differs across the reported clusters. A p-value is an inferential claim about how surprising that difference would be under a specified null model. Double dipping does not erase the descriptive difference; it challenges whether the ordinary inferential calibration applies.

The Ordinary Test and Its Assumptions

For clarity, consider a two-sided pooled two-sample t-test comparing a gene’s expression in two groups. Let \(\bar{x}_1\) and \(\bar{x}_2\) be the group sample means, \(s_1^2\) and \(s_2^2\) the sample variances, and \(n_1\) and \(n_2\) the group sizes. The pooled variance and test statistic are

s_p^2=\frac{(n_1-1)s_1^2+(n_2-1)s_2^2}{n_1+n_2-2}, \qquad t=\frac{\bar{x}_1-\bar{x}_2}{s_p\sqrt{1/n_1+1/n_2}}.

Under the null hypothesis of equal group means, the usual exact reference distribution is \(t\) with \(df=n_1+n_2-2\) when observations are independent, normally distributed within each group, and have a common variance. For the ordinary interpretation to apply after clustering, the group definitions must also be treated as fixed independently of the tested expression values under the null. If the labels were constructed from those values, that last condition is generally not met.

A p-value is the probability, under the null model and its assumptions, of obtaining a test statistic at least as extreme as the observed one. It is not the probability that the null hypothesis is true. If the assumed reference distribution is wrong because labels were selected using the data, a small nominal p-value need not correspond to a small probability of a false positive.

Worked Example: Splitting Cells on the Gene Being Tested

Suppose a hypothetical analysis has eight cells with normalized expression values for gene G of 1, 2, 3, 4, 7, 8, 9, and 10 units. A clustering step sorts the cells by this gene and assigns the four lowest values to cluster 1 and the four highest to cluster 2. The researcher then runs a pooled t-test for gene G.

The group means are \(\bar{x}_1=(1+2+3+4)/4=2.5\) units and \(\bar{x}_2=(7+8+9+10)/4=8.5\) units. Each group has sample variance \(s_1^2=s_2^2=5/3\) square units: for the first group, the squared deviations from 2.5 sum to \(5\), and division by \(4-1\) gives \(5/3\); the second group has the same pattern. Thus \(s_p^2=5/3\), \(s_p=\sqrt{5/3}\), and

SE=\sqrt{\frac{5}{3}}\sqrt{\frac14+\frac14} =\sqrt{\frac56}\approx0.9129\text{ units}, \qquad t=\frac{2.5-8.5}{0.9129}\approx-6.573.

With \(df=4+4-2=6\), the ordinary two-sided t reference gives a nominal p-value of about 0.00059. Under a prespecified, independent grouping and the test assumptions, that would be strong evidence against equal means. Here, however, the grouping was defined by sorting on gene G itself. The large separation is built into the assignment rule, so the conventional t reference does not account for how the labels were obtained. The calculation accurately summarizes the separation; its nominal p-value does not provide the stated ordinary-test evidence for a distinct biological marker.

Selection Can Enlarge Evidence

Double dipping can also arise when the clustering uses many genes and the analyst reports the gene with the strongest observed difference. Even if a conventional p-value were valid for each gene considered alone with fixed labels, choosing the smallest p-value changes the probability of seeing a small one. A simple calculation shows the effect in an idealized setting.

Suppose \(G\) independent tests are all true null hypotheses, and each test has a correctly calibrated p-value. For one test, \(P(p\leq\alpha)=\alpha\). The probability that none of the \(G\) p-values is at most \(\alpha\) is \((1-\alpha)^G\), so the probability that at least one is at most \(\alpha\) is

P\left(\min_{1\leq g\leq G}p_g\leq\alpha\right) =1-(1-\alpha)^G.

This is a justification for why searching and then reporting only the winner can make a nominal p-value overstate evidence. It is an idealized calculation: expression measurements across genes are often dependent, and clustering adds a further data-adaptive step. The exact size of the distortion therefore cannot be read off from this formula alone.

Worked Example: Reporting the Smallest of Many P-Values

Imagine a hypothetical analysis examines 20 genes. To isolate the effect of searching, suppose each null p-value is valid and the 20 p-values are independent. The analyst reports whichever gene has the smallest p-value and highlights a result below \(\alpha=0.05\).

The probability of no p-value being below 0.05 is \(0.95^{20}\approx0.3585\). Therefore,

P(\text{at least one }p_g\leq0.05) =1-0.95^{20} \approx0.6415.

So even if all 20 null hypotheses are true, there is about a 64.15% chance of finding at least one nominal p-value at or below 0.05 under these simplifying assumptions. A smallest p-value of 0.012 is a result selected from the search; it cannot be interpreted as though that gene had been named before looking at the data. In real clustering analyses, dependence among genes and the label-construction process affect the calculation, but neither makes it appropriate to ignore the search.

When Reuse Is Direct, Indirect, or Absent

The most direct case occurs when the tested gene is explicitly included in a clustering distance, score, or threshold. A second case is indirect reuse: the gene is not itself an input, but its expression is correlated with genes that drove the partition. Then the labels can still carry information about the tested gene. Merely saying “this gene was not used by the clustering algorithm” is not enough to establish independent evidence.

A contrast is different when group labels come from an external, prespecified source that is independent of the tested expression values under the null. Under the ordinary model assumptions, the test can then have its usual calibration. That independence is a substantive assumption, not a conclusion established by obtaining a small p-value. In single-cell data, dependence among cells from the same donor is another potential problem: addressing double dipping alone does not make an analysis valid if the observational units are not independent.

Worked Example: A Prespecified Grouping as a Contrast

Consider six hypothetical cells assigned to two groups before gene G is measured, with expression values 1, 2, and 3 units in group A and 4, 5, and 6 units in group B. Assume for this illustration that the observations are independent and normal with a common variance, and that the labels were not chosen using gene G. The group means are 2 and 5 units. Each sample variance is 1 square unit, so the pooled variance is \(s_p^2=1\).

SE=\sqrt{1}\sqrt{\frac13+\frac13} =\sqrt{\frac23}\approx0.8165\text{ units}, \qquad t=\frac{2-5}{0.8165}\approx-3.674, \qquad df=3+3-2=4.

The two-sided p-value from a \(t_4\) reference is about 0.0213. In this hypothetical example, the labels are prespecified independently of gene G, so the ordinary t-test assumptions support that nominal calculation. The same arithmetic applied after labels were selected using gene G would not have the same justification. The key distinction is not whether the expression difference is visible, but whether the reference distribution matches the process that produced the labels.

Diagnostics, Comparisons, and Limitations

Before interpreting a marker p-value, trace the analysis path for the tested gene. Check whether it contributed to clustering, feature selection, dimensionality reduction, tuning of the cluster number, or a later choice of which cluster contrast to report. Also ask whether the analysis searched across many genes or contrasts and then presented only the most striking result. These are not proof that a particular p-value is wrong by a known amount; they identify where the ordinary reference calculation may fail to represent the full procedure.

Warning sign: the most significant markers are also the features that visibly define the cluster separation. This is compatible with real biology, but it is also what reuse can produce by construction.
Warning sign: the reported test treats the final labels as fixed, while records or code show that labels were formed from the same expression matrix.
Warning sign: only the most different genes or cluster pairs are reported, with no account of the search that selected them.
Separate issue: cell-level independence, donor structure, count-model assumptions, and multiple testing can each affect inference. A diagnosis of double dipping does not resolve these other issues.

The main alternative for the claim being made may be descriptive reporting: state the observed mean or proportion, its units, and the cluster definition, without attaching an ordinary inferential interpretation that the design does not support. If the aim is confirmatory inference, the data used to select the groups or markers must be accounted for by the inferential design. Different strategies make different assumptions, and none can be justified merely by relabeling a descriptive result as a test.

A useful way to assess any proposed analysis is to ask what would happen if the entire procedure were repeated under the null: would the clustering and the marker-selection steps also be repeated, or would only the final test be recalculated? An ordinary p-value for the last step generally describes only that last step under fixed labels. If earlier data-dependent choices are omitted from the reference process, the reported evidence can be too optimistic. The precise correction or alternative depends on the selection procedure and is a separate inferential problem.

Key takeaway: A large expression difference between clusters is a valid description of the observed partition. An ordinary marker p-value is justified only when its null reference accounts for how the labels and tested gene were selected. Reusing measurements can make the test statistic look more extreme and its nominal p-value look smaller than the full data-adaptive analysis warrants.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A gene is used to divide cells into high-expression and low-expression clusters, then tested for a difference between those clusters. Which part of the ordinary test interpretation is threatened?
For 10 independent, valid null p-values, calculate the probability that at least one is at most 0.05.
Why does excluding a tested gene from the clustering input not, by itself, guarantee a valid ordinary marker test?
Can a cluster difference remain a useful descriptive result when the ordinary p-value is not justified? Explain.

Solutions. 1. The usual reference distribution assumes an appropriate fixed grouping under the null; here the labels depend directly on the tested values, so the ordinary p-value need not be calibrated. 2. \(1-(1-0.05)^{10}=1-0.95^{10}\approx1-0.5987=0.4013\), or about 40.13%. 3. The tested gene may be correlated with genes used to form the clusters, and the analysis may also have selected among many genes or contrasts. Independence must be justified, not inferred from omission alone. 4. Yes. The observed expression difference still describes the data and partition; what is not justified is interpreting the unadjusted p-value as if the labels and tested feature had been fixed independently of those data.