Why Reporting Must Expose the Analysis Path
A marker table can look precise while concealing how its results were obtained. A gene may have been selected because it helped create the clusters, tested on those same expression values, and then presented with an ordinary p-value as if the contrast had been specified in advance. That p-value alone does not reveal whether selection was addressed, which other hypotheses were considered, or how large the estimated difference is.
A useful report therefore documents the path from cluster discovery to marker testing. It names the contrast, identifies the discovery and testing data, explains how selection and multiplicity were handled, and gives an effect estimate in an interpretable scale. These details let readers judge what the evidence supports without treating a marker list as a cell-type label or a causal explanation.
The report must also distinguish a descriptive effect from an inferential claim. A difference in observed expression can be reported even when the data do not support a valid post-selection p-value. Conversely, an adjusted p-value does not communicate whether the estimated difference is practically large or on what scale it was measured.
A Reporting Framework
For each reported marker contrast, make the following items recoverable from the text, table, or its caption:
- Contrast: name both clusters and the direction of subtraction. State whether the target is, for example, cluster A minus cluster B.
- Discovery and testing data: say which observations, samples, and expression measurements were used to form clusters, choose features, select candidate genes, and test the contrast. If data were split, describe the split and how held-out observations received their cluster assignments.
- Selection adjustment: describe how the analysis accounts for choosing the clusters or genes. If selection was not accounted for, state that plainly and do not present an ordinary p-value as valid confirmatory evidence.
- Multiplicity family: define the complete set of hypotheses included in correction, such as all genes for one contrast or all genes across a specified set of contrasts. Name the adjustment and report adjusted values.
- Effect and uncertainty: give an effect estimate, scale, direction, and an uncertainty measure when justified by the procedure. Distinguish a log-scale difference from a fold change.
- Limits: state what the design and procedure do not establish, including limits on generalization, causal interpretation, and biological labeling.
A concise sentence can still be informative: “In the independent test cohort, cluster A minus cluster B had an estimated mean difference of 1.20 log2-normalized expression units for G7; the paired donor-level test gave a BH-adjusted p-value of 0.050 across 100 genes for this contrast.” The methods and caption should then specify how clusters were learned, how testing labels were assigned, and how the donor-level analysis was constructed.
A p-value is interpretable only relative to its testing procedure and null hypothesis. A small value does not mean the null hypothesis is probably false, nor does it measure the probability that a reported marker is genuine. An adjusted p-value further depends on the specified multiplicity procedure and family. Neither number should be reported without that context.
Worked Reporting Examples
Worked Example: Held-Out Testing and a Defined Gene Family
Suppose a hypothetical study uses a discovery cohort to define clusters A and B and to choose a candidate marker list. A separate validation cohort contains eight independent donors. A frozen assignment rule, based on features other than the tested gene, assigns validation cells to A or B. For each donor, expression is summarized within each cluster, and a prespecified paired donor-level model estimates the contrast. The model gives mean log2-normalized expression of 3.20 in A and 2.00 in B for gene G7, with a two-sided p-value of 0.001. Assume the paired-model assumptions and the cluster-assignment procedure were checked and considered adequate for this hypothetical analysis.
The estimated contrast is defined as A minus B:
On the corresponding fold scale, the difference represents an estimated ratio of \(2^{1.20}\approx 2.297\), or about 2.30. This conversion describes the estimated scale change; it does not turn the estimate into a causal effect.
Suppose the multiplicity family is all 100 genes tested for this one contrast, and the analysis uses the Benjamini–Hochberg procedure. In the hypothetical ordered p-values, G7 has rank 2 with raw p-value 0.001; the smallest p-value is 0.0003 at rank 1, and all later values make the rank-2 adjusted value 0.050. The rank-2 Benjamini–Hochberg candidate is:
The rank-1 candidate is \(100(0.0003)=0.030\), but G7 is rank 2, so its adjusted value is the minimum of the rank-2 candidate and the candidates for ranks at or above 2. Under the stated hypothetical ordering, that minimum is 0.050. Thus the report should give the adjusted p-value as 0.050, not 0.001.
A suitable report is: “In the held-out validation cohort, G7 was higher in cluster A than B by an estimated 1.20 log2-normalized expression units (approximately 2.30-fold); the paired donor-level test gave raw p = 0.001 and Benjamini–Hochberg-adjusted p = 0.050 across 100 genes for this contrast.” The report must also say how many donors contributed usable measurements and disclose any exclusions. The held-out design supports a test on the validation data only to the extent that the split and assignment rule were genuinely independent of the tested expression measurements.
Worked Example: Pipeline Permutation and Family-Wide Adjustment
Suppose a hypothetical analysis has one prespecified cluster contrast and reports 20 candidate genes. The observed statistic for a gene is 2.4 in absolute value. A permutation procedure generates 999 null replicates. In each replicate, the complete selection-and-testing pipeline is rerun, including the steps that choose the clustering features and select the candidate gene. For each replicate, the largest absolute test statistic across all 20 genes is recorded. Assume the exchangeability conditions required by this permutation scheme are justified under the null.
In 39 of the 999 replicates, the maximum statistic is at least 2.4. Using the finite-simulation correction, the family-wise adjusted permutation p-value is:
The added one in numerator and denominator prevents reporting a simulated p-value of zero when no replicate exceeds the observed statistic. Because each replicate records the maximum over the 20-gene family and repeats the selection steps, this adjusted value addresses both the stated family and the simulated selection process, conditional on the validity of the permutation design. It would not be justified if the permutation broke important dependence or failed to reproduce a data-dependent step.
Suppose the gene’s estimated A-minus-B difference is 0.85 log2-normalized expression units. Report that estimate separately from the adjusted p-value: “The estimated difference was 0.85 log2-normalized expression units (A higher than B); the full-pipeline maximum-statistic permutation test gave family-wise adjusted p = 0.040 across 20 genes.” State the number of permutations, the statistic, the family, and which pipeline stages were rerun. The estimate is not automatically an unbiased population effect merely because the p-value is selection-aware.
Worked Example: Reused Data and an Exploratory Result
In a hypothetical exploratory analysis, the same expression matrix is used to define clusters A and B, choose gene G12 as a marker, and test its expression difference. The observed means are 4.10 and 2.80 log2-normalized expression units. A standard test applied after selection returns p = 0.0002, but the test does not account for clustering or marker selection.
The observed difference is:
It is reasonable to report the observed contrast as descriptive: G12 was 1.30 log2-normalized expression units higher in A than B in these data, corresponding to an estimated ratio of about 2.46. But p = 0.0002 is a nominal result from a test that ignores the selection process; it should not be presented as valid confirmatory evidence. Multiplying this p-value by the number of genes, or applying a false-discovery-rate adjustment to the same invalid inputs, would address neither the unmodeled data reuse nor the selection-induced bias.
A careful report would identify the result as exploratory and state that the clusters and marker were selected using the same expression data. It would reserve confirmatory language for a suitable independent test or another procedure that accounts for the relevant selection. This does not require suppressing the descriptive estimate; it requires separating what was observed from what the design can infer.
Justifying the Adjustments and Checking the Report
The distinction between selection and multiplicity can be stated precisely. Let \(S\) denote the selection procedure that determines the reported cluster contrast or genes, and let \(T\) denote the subsequent test statistic. A nominal test calibrated as though the contrast were fixed in advance generally does not have its claimed null distribution after conditioning on a selection event that depends on the same data. A multiplicity procedure instead addresses a collection of tests, under its own assumptions. It cannot reconstruct the missing post-selection calibration merely by changing the reported p-value.
A full-pipeline permutation illustrates the logic of selection adjustment. Under a valid null randomization scheme, permute the data or labels in a way that preserves the relevant design, then rerun every data-dependent operation that contributed to the reported statistic. Comparing the observed statistic to the resulting null distribution reproduces selection variability under that null. If the pipeline is not rerun, the permutation distribution may represent a different procedure from the one that generated the reported marker.
The finite-permutation formula also has a simple justification. With \(B\) random permutations, the observed arrangement and the \(B\) simulated arrangements play symmetric roles under the null. If \(b\) simulated statistics are at least as extreme as the observed statistic, a conservative Monte Carlo estimate is \((b+1)/(B+1)\), rather than \(b/B\). The correction ensures the estimated tail probability is not zero solely because a finite simulation happened to miss an extreme outcome.
Before finalizing a report, verify that its claims match the actual analysis:
Record which data formed clusters, selected features or genes, assigned test observations, and supplied expression values for testing. Check that a claimed held-out test did not leak the test gene into the assignment rule.
State the two cluster names and subtraction direction. Check that the sign of the reported estimate and the words “higher” or “lower” agree.
Give the test or resampling procedure and the assumptions that justify it. For permutations, check exchangeability and confirm that the necessary selection steps were rerun.
List what hypotheses were included and what adjustment was used. Verify that the reported adjusted values correspond to that family, including any cluster contrasts beyond the one shown.
Report the effect on its stated scale, its uncertainty when available, and the adjusted evidence. Do not let a small p-value substitute for an effect estimate or a description of limitations.
Useful diagnostics include checking whether the cluster assignment changes when the marker gene is omitted, examining whether the chosen effect is dominated by a small number of donors, and checking permutation null distributions or negative controls where appropriate. These are diagnostic evidence, not automatic proofs of validity. A stable cluster solution does not establish that a marker p-value is calibrated, and a reassuring diagnostic cannot replace the assumptions of the inferential design.
Common reporting failures include omitting the comparison direction, calling a log2-scale difference a fold change, reporting only raw p-values after a large search, and describing cell-level measurements as independent biological replication when the analysis does not account for donor structure. Another frequent error is to say “the gene is a marker” without specifying the contrast or the evidence standard. State the observed result and procedure instead. The resulting claim is narrower but assessable.
Limits of the Claim
A statistically supported post-clustering contrast is evidence for a difference under a particular data-processing and testing procedure. It does not by itself establish that a gene causes a cluster, that the clusters represent distinct biological cell types, or that the estimate generalizes to populations not represented by the samples. Technical effects, donor composition, measurement choices, and imperfect cluster assignment may all constrain interpretation.
Reports should identify these limits rather than implying that adjustment removes every source of uncertainty. A selection-aware p-value addresses the selection and null model it was designed for; a multiplicity adjustment addresses its stated family; an effect estimate summarizes a scale-specific contrast. Readers need all three pieces, along with the data and design description, to assess marker evidence.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A report gives a positive estimate for “A minus B” but says the gene is higher in B. What should be checked?
- Why does applying Benjamini–Hochberg adjustment not automatically correct a nominal p-value that ignored clustering-based gene selection?
- A full-pipeline permutation has 999 replicates, of which 9 maximum statistics equal or exceed the observed statistic. What finite-simulation adjusted p-value is reported?
- What information is needed to interpret a statement that a marker has adjusted p = 0.03?
Solutions. (1) Check the subtraction order, group means, and sign; a positive A-minus-B estimate means the estimated value is higher in A. (2) Benjamini–Hochberg addresses multiplicity across a defined family under its assumptions, but does not recalibrate a test whose null distribution fails after data-dependent selection. (3) Using the plus-one correction gives \((9+1)/(999+1)=10/1000=0.010\). (4) The reader needs the contrast and direction, testing and discovery data, selection procedure or adjustment, multiplicity family and correction, effect estimate and scale, and relevant assumptions and limitations.