A Worked Comparison Of Honest Marker Strategies

One Clustering Problem, Three Inferential Routes

A gene can look like a strong marker partly because it helped create the clusters being compared. The central issue is not simply whether a test statistic is large, but whether its reference distribution accounts for the way the gene or cluster labels were selected. Sample splitting, full-pipeline permutation, and selective testing handle that selection in different ways, so they can answer different questions and rely on different assumptions.

Consider a hypothetical data set with two candidate genes, G1 and G2. A clustering pipeline uses expression information to divide cells into two groups, then ranks the genes by the absolute standardized difference between groups. For illustration, suppose G1 is the selected marker and the largest observed absolute statistic is 2.4. We will compare three analyses of this same general problem. The numerical summaries below describe a toy setup; they are not results from a real study.

These approaches do not make arbitrary clustering labels into known biological cell types. Their conclusions concern statistical evidence for expression differences under their respective models and null hypotheses. A small p-value does not establish a causal role for a gene or validate a biological annotation.

Strategy 1: Reserve Cells for a Held-Out Test

In sample splitting, randomly divide the cells into a discovery set and a validation set before marker testing. Use the discovery cells to fit the clustering and choose candidate markers. Then freeze the clustering rule and the candidate gene list. Assign validation cells using a rule determined from discovery data, such as a fixed classifier or fixed cluster centroids, and test the selected marker using only validation expression values.

For a simple worked test, suppose the validation data produce two groups of eight cells. Assume the per-cell expression residuals for G1 are independent normal observations with known variance 1 in each group, that under the null the validation group labels are independent of the validation G1 residuals, and that the assignment rule and choice of G1 were fixed using discovery data alone. Under the null of equal group means, the standardized difference has a standard normal distribution. The test is two-sided. These assumptions are substantive: they would not be justified merely by having split the cells.

The held-out p-value is valid for the prespecified test if the validation outcomes are independent of discovery-based selection, and if the distributional and assignment assumptions hold. If many genes are tested in the validation set, the analysis must also account for that multiplicity; a split does not by itself solve multiple testing within the held-out data.

Strategy 2: Rerun the Full Pipeline Under Permutation

A full-pipeline permutation test asks how often the observed marker statistic, or a more extreme one, would occur under a specified null randomization. The statistic should reflect the selection being assessed. If the analysis reports the strongest marker among two genes, a natural statistic is the maximum absolute standardized contrast across those genes. For every allowed permutation, rerun the clustering and marker ranking before recording that maximum. Permuting only the final labels or testing only the selected gene would not reproduce the full selection process.

Exact validity requires the observed data and the permitted randomized data to be exchangeable under the null, possibly within prespecified blocks. The transformations must preserve whatever structure is required by the null and must represent the data-generating randomization. In expression data, permuting a gene independently of all other genes can be invalid if the null does not justify that operation. The permutation scheme must also rerun any preprocessing or selection steps that are part of the tested pipeline and depend on the permuted data.

The permutation p-value is not automatically valid just because clustering was rerun. If the permitted transformations are not exchangeable under the null, rerunning the pipeline cannot repair the wrong reference distribution. Conversely, a valid randomization scheme that reruns all relevant steps can account for selection patterns that are difficult to describe with a simple parametric model.

Strategy 3: Model the Selection Event

Selective testing derives a reference distribution conditional on a stated selection event. The selection rule must be specified precisely: selecting the largest absolute statistic among two candidates is different from selecting genes after several filters, choosing a cluster resolution, or ranking across many clusters. The model must describe the joint distribution of the statistics before selection, not just the marginal distribution of the winner.

Here is a deliberately simple model for the two-gene toy problem. Under a global null, suppose the two candidate statistics \(Z_1\) and \(Z_2\) are independent standard normal variables. The selected statistic is \(T=\max(|Z_1|,|Z_2|)\). This model is appropriate only if the standardization, null distributions, independence, and two-gene selection rule are justified. It is not a generic model for statistics produced by clustering.

A useful justification for the maximum-statistic formula follows directly from independence. The event \(T< t\) occurs exactly when both \(|Z_1|<t\) and \(|Z_2|<t\). Hence, for \(t\geq0\),

Taking the complement gives \(P(T\geq t)=1-\left(1-2\Phi(-t)\right)^2\), which yields the calculation above. If the statistics are correlated, the product step is not valid. Their joint null distribution must instead be modeled or estimated with a justified procedure.

What the Three Results Do—and Do Not—Compare

In these toy calculations, the held-out test gives \(0.04550\), the full-pipeline permutation test gives \(0.2000\), and the two-statistic maximum test gives \(0.03252\). These are not competing estimates of one universal quantity. The held-out analysis uses validation cells and tests a fixed gene under a Gaussian model. The permutation analysis tests a maximum statistic against a particular exchangeable randomization distribution that includes reclustering. The selective calculation tests a maximum under an independent standard-normal model for two candidates.

The numerical ordering does not establish that one strategy is universally more conservative or more powerful. The p-values differ because their data inputs, null models, statistics, and conditioning or randomization schemes differ. For example, the held-out test statistic is 2.00, whereas the selection-adjusted maximum is 2.4; they are not the same observed statistic. The permutation p-value is determined by its discrete 20-assignment reference distribution, not by the Gaussian calculation.

Diagnostics, Failure Modes, and Practical Limits

The main diagnostic question is whether the method’s reference distribution matches the actual analysis. For a split analysis, verify that no validation expression values influenced feature selection, tuning, clustering, threshold choice, or the assignment rule. Check that the validation groups contain enough cells for the chosen approximation and that the assumed variance model is credible. A split can lose power because fewer cells are used for each stage, and its result pertains to the frozen assignment procedure rather than a clustering refit on all cells.

For permutation, inspect the allowed transformations and explain why they are exchangeable under the null. Check that every data-dependent part of the pipeline is rerun and that the statistic matches the reported claim. With few possible assignments, p-values are coarse: in the example, each additional exceedance changes the exact p-value by \(1/20=0.05\). Permutation cannot fix a null scheme that destroys relevant dependence or violates block structure.

For selective testing, verify the candidate set, ranking rule, joint null distribution, and any conditioning event. A formula for two independent normal statistics is not justified when statistics are correlated or the pipeline selected genes through additional filters. Misspecifying the selection event can make an apparently adjusted p-value anti-conservative. Selective inference may also depend on strong modeling assumptions and can be sensitive to choices made during exploratory analysis.

A frequent mistake is to call whichever p-value is smallest the “correct” one, or to interpret disagreement as evidence that one calculation is wrong. The correction is to state each method’s null hypothesis, statistic, selection handling, and assumptions first. Then report which inferential target is relevant to the scientific question. No strategy eliminates the need to describe how clusters and markers were produced.