Why Stability and Validity Are Different Questions
A clustering result can be reproducible without supporting valid marker p-values. Stability asks whether a partition changes when the data or analysis is perturbed in a specified way. Marker-test validity asks whether the test statistic and its p-value have the claimed null distribution for the comparison being made. The first concerns the behavior of a clustering algorithm; the second concerns repeated-sampling error rates for an inferential procedure.
This distinction matters because marker analyses often compare expression between groups created from the same data. If a gene helped determine the groups, its observed values influenced both the labels and the subsequent comparison. Earlier material on selection bias and selective inference explains why the resulting test distribution may differ from the distribution assumed by an ordinary test. Repeating a similar partition after small perturbations does not, by itself, undo that dependence.
A stability summary might be the average fraction of cells retaining their matched cluster assignment across perturbations, or a label-invariant measure of partition agreement. Such summaries describe reproducibility relative to a chosen perturbation scheme: for example, resampling cells, changing initialization, or adding small measurement perturbations. They are not probabilities that the null hypothesis is true, and they are not estimates of a marker test's type I error unless a separate procedure explicitly evaluates that error under an appropriate null model.
A Stable Partition Can Still Select Its Own Marker
Consider a simple clustering rule with two groups: assign a cell to Group A when the value of gene \(Z\) is below zero, and to Group B when it is above zero. This is intentionally simple, but it isolates the central issue: the gene being tested also determines the labels. The partition can be insensitive to small changes in the measurements while the comparison of \(Z\) between its own groups remains selected.
Worked Example: A Stable Split With an Invalid Ordinary Marker Test
Suppose ten hypothetical cells have the following standardized values of \(Z\). The clustering rule assigns negative values to Group A and positive values to Group B.
The means are \(\bar{x}_A=-0.970\) and \(\bar{x}_B=0.970\), so the observed difference is \(1.940\) standardized units. In each group, the sum of squared deviations from its mean is \(0.0280\). Thus each sample variance is \(0.0280/4=0.00700\), and the pooled variance for the equal-variance two-sample t-test is \(s_p^2=0.00700\). Its standard error and statistic are
With \(df=8\), the ordinary two-sided t-test reports a p-value far below \(0.05\) (in fact, below \(10^{-7}\)). But its usual reference distribution presumes that the group labels were fixed independently of the tested values, under the null model for the test. Here, the labels were defined by the sign of \(Z\). The tiny p-value therefore does not establish that this test is calibrated for the data-adaptive comparison.
Now define a perturbation protocol that changes every recorded value by at most \(0.02\), without changing the threshold rule. Every Group A value remains at most \(-0.88\), because even the largest, \(-0.90\), can increase only to \(-0.88\). Every Group B value remains at least \(0.88\), because even the smallest, \(0.90\), can decrease only to \(0.88\). Thus no cell crosses zero, and the partition is unchanged throughout this perturbation protocol. This is strong local stability, but it does not alter how the labels were selected or justify the ordinary t-test reference distribution.
Why Selection Changes the Null Distribution
A short population calculation makes the problem explicit. Suppose a homogeneous null model gives each cell a marker value \(X \sim N(0,1)\), and the clustering rule labels cells by whether \(X\) is positive or negative. The overall population mean is zero. However, the conditional means within the two algorithm-defined groups are not zero.
Consequently, if many independent observations are split this way, the sample mean difference tends toward \(1.596\), not zero. Its ordinary standard error shrinks as the sample size grows. The resulting t-statistic can grow in magnitude, and the ordinary fixed-label p-value can approach zero even though the data came from a homogeneous null population. This is a direct failure of calibration for that procedure, not a matter of whether the partition is easy to reproduce.
A valid analysis must instead specify a null model and a procedure whose reference distribution accounts for the way labels were formed. Depending on the setting, that may mean conditioning on the selection event, rerunning the full clustering pipeline in a valid permutation scheme, or separating the data used to discover groups from the data used to test markers. These approaches require their own assumptions; none is made valid merely by observing a stable partition.
An Unstable Partition Does Not Repair the Test
Instability creates a different concern: the named groups may depend strongly on small details of the data or algorithm. But instability alone does not diagnose the direction or size of p-value miscalibration. If the tested marker still helps determine the labels, the ordinary test still has to account for that selection, whether the partition is stable or not.
Worked Example: Small Changes Alter Labels but Not the Inferential Requirement
Four hypothetical cells have marker values \(-0.03,-0.01,0.01,0.03\). Again, cluster by sign. Group A contains \(-0.03,-0.01\), with mean \(-0.020\); Group B contains \(0.01,0.03\), with mean \(0.020\). Each group has sample variance \(0.0002\). The equal-variance two-sample statistic is
With \(df=2\), the two-sided ordinary t-test p-value is approximately \(0.1056\). If the values for the cells initially at \(-0.01\) and \(0.01\) are each perturbed by \(0.02\), they exchange signs. Two of the four cells switch groups, so the fraction retaining their original assignment is \(2/4=0.50\). The partition is sensitive to this perturbation.
The p-value of \(0.1056\) is not evidence that the test is calibrated: a particular p-value, large or small, cannot establish calibration. The labels were still formed using the tested gene, so the ordinary fixed-label reference distribution is still unjustified. Conversely, the partition's instability does not prove that every possible marker test is invalid; it flags sensitivity in the grouping and motivates checking how the actual inference procedure handles selection.
What Can Support Marker-Test Validity?
The main alternatives target the inferential dependence directly rather than using stability as a substitute for calibration. Their assumptions differ, so the choice should follow the data-generating and study design.
| Approach | What it addresses | Key requirement or limitation |
|---|---|---|
| Independent test data | Separates the data used to define groups from the data used to test the marker. | Test observations and labels must be independent in the relevant sense; the assignment rule and test must be prespecified or otherwise accounted for. |
| Selective inference | Calibrates inference conditional on the selection event. | Requires a correctly specified model and a tractable account of the selection rule. |
| Permutation with the full pipeline rerun | Can reproduce selection under a null distribution by reclustering each permuted data set. | Permutations must satisfy the relevant exchangeability conditions, and the null must match the permutation scheme. |
| Stability analysis | Quantifies sensitivity of the partition to a declared perturbation scheme. | Does not, on its own, provide a null distribution or control marker-test error. |
Worked Example: Testing a Marker on Independent Test Cells
Suppose a hypothetical training data set defines two groups using genes other than marker \(Z\). A fixed assignment rule assigns cells in an independent test set to those groups, without using their \(Z\) values. In the test set, assume four independent cells per group and values \(1,2,2,3\) for Group A and \(3,4,4,5\) for Group B. For this illustration, assume test values are normally distributed with a known standard deviation of \(1\) unit in each group, and the null hypothesis is equal group means.
The observed means are \(2\) and \(4\), so the estimated difference, Group B minus Group A, is \(2\) units. Under the stated null model and fixed, independently assigned labels, the standard error is
The two-sided standard normal p-value is approximately \(0.00468\). At \(\alpha=0.05\), this procedure rejects equal means under the stated model. The interpretation is evidence of a difference in mean \(Z\) expression between the test cells assigned to the two groups, not proof that the groups are distinct biological cell types. Its calibration depends on the example's assumptions, including independent test observations, the known variance, and the marker's not being used to assign test labels. If assignment depends on \(Z\), the independence argument no longer applies.
Diagnostics, Failure Modes, and Limits
A useful report keeps stability diagnostics and inferential diagnostics separate. For stability, state the perturbation scheme, the clustering steps repeated, how cluster labels were matched, and the measure of agreement. For marker inference, state the null hypothesis, which data and genes determined the labels, what observations enter the test, and why the stated reference distribution applies. A high agreement score answers none of those inferential questions by itself.
- Failure mode: treating robustness as a significance guarantee. A partition may persist because the marker values are far from a threshold, even though those same values created the contrast. Report stability as a property of the partition, not as evidence that its marker p-values are calibrated.
- Failure mode: treating instability as proof of p-value invalidity. Instability signals sensitivity of the labels. It does not quantify a test's type I error. Assess the validity of the particular testing procedure under its null and selection process.
- Failure mode: perturbing only the last clustering step. If feature selection, normalization, dimensionality reduction, or tuning used the data, holding those decisions fixed may not represent the full analysis pipeline. A robustness analysis should specify which choices are repeated; a calibration method must reproduce or condition on relevant selection.
- Limitation: stability depends on the perturbations chosen. A partition stable to initialization changes may be sensitive to cell resampling; stability under either scheme does not imply stability under every plausible source of variation.
- Limitation: validity is always relative to a model and claim. A calibrated test for a specified mean contrast does not automatically validate a broader claim about cell identity, causation, or all genes. Dependence among cells, such as shared donors, may also require a model that respects the sampling unit.
The practical conclusion is not to discard stability analysis. Stable partitions can be useful evidence that a computational result is reproducible under stated perturbations. Rather, use that evidence for the question it answers, and justify marker p-values separately through an appropriate design or inferential method.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A marker gene defines a two-group split. The split is unchanged in every small-perturbation run. Does that establish that an ordinary two-sample p-value for the marker is calibrated? Why?
- Under \(X\sim N(0,1)\), what is \(E[X\mid X>0]\), and why does it matter for a sign-based split?
- In the four-cell example, two of four cells switch groups after a perturbation. What fraction retains its original assignment, and what can that fraction establish?
- Name one design or inferential approach that can address the dependence between cluster selection and marker testing, and state one requirement.
Solutions. (1) No. Stability under the chosen perturbations does not supply the ordinary test's null distribution; using the marker to form the groups makes the labels data-dependent on the tested values. (2) \(E[X\mid X>0]=\sqrt{2/\pi}\approx0.7979\), obtained by integrating the standard normal density over positive values and dividing by \(P(X>0)=1/2\). Thus the positive and negative groups have different conditional means even under a homogeneous, mean-zero population. (3) The retained fraction is \(2/4=0.50\). It describes assignment sensitivity for that perturbation, but does not establish marker-test calibration. (4) For example, independent test data can separate discovery from testing, provided the test observations are independent of the discovery process in the relevant sense and labels are assigned without using the tested marker values.