Why Use Negative Controls?
A marker analysis can report many small p-values because genes truly differ between clusters, but also because of model misspecification, unaccounted dependence, or reuse of the data to define and test the clusters. Negative controls provide a check on whether a workflow produces apparent discoveries when the scientific expectation is that there should be none. They are diagnostics, not a substitute for a valid inferential design.
There are two useful but distinct control strategies. A negative-control gene is expected, on scientific grounds, not to differ for the contrast being tested. A permuted-label control replaces the observed labels with labels generated under a null scheme and repeats a specified part—or, when appropriate, all—of the analysis. The first checks behavior for a chosen set of genes; the second checks behavior under a chosen label-randomization scheme. Each is informative only if its control assumptions are credible.
Negative-Control Genes and Their Assumptions
The phrase “no difference” must refer to the actual estimand and test. For example, a gene may be expected not to differ in average expression between two clusters, while still varying across donors or responding to a technical factor associated with cluster membership. Such a gene is not a useful negative control unless the analysis and control definition account for those features. Controls should be chosen independently of the observed p-values; choosing genes because their results look reassuring makes the check circular.
A useful control set should also resemble the tested genes in properties that affect testing, such as expression level, detection rate, and variability. Otherwise, differences in rejection rates may reflect different power or calibration rather than a problem in the marker workflow. Matching does not make the genes true nulls, and biological knowledge does not guarantee that every selected control is unaffected. These are assumptions to defend, not facts established by the resulting analysis.
For a set of \(m\) true-null control genes, let \(p_g\) be the p-value for control gene \(g\). If each p-value is valid under its null, then for any threshold \(t\) in \([0,1]\), \(P(p_g \leq t) \leq t\). In particular, at a nominal threshold \(\alpha\), each control has rejection probability at most \(\alpha\). This is a marginal statement: it does not require the control-gene tests to be independent.
The expectation result follows by linearity:
No independence assumption is needed for this inequality. Independence alone is not enough to give a binomial distribution: the rejection probabilities may differ across genes or be strictly below \(\alpha\). For a binomial model, the indicators must be independent Bernoulli variables with the same rejection probability. In marker analyses, dependence between genes is common, so a binomial calculation is best treated as a reference under explicit assumptions, not an automatic calibration.
Worked Example: Interpreting Rejections Among Control Genes
Suppose a hypothetical analysis tests \(m=40\) negative-control genes at \(\alpha=0.05\), and 3 have p-values at or below 0.05. Assume for the moment that all controls are true nulls and each test is valid.
The expected number of rejections is bounded by \(E(V_C) \leq 40(0.05)=2\). The observed count, 3, is above this expected-count bound, but that alone does not show the workflow is miscalibrated: an observed count can exceed its expectation by chance.
As a more specific reference calculation, suppose additionally that the tests are independent and exact, so \(V_C \sim \operatorname{Binomial}(40,0.05)\). Then
Thus, under those stronger assumptions, 3 or more rejections would not be unusual. The calculation is not justified if the rejection indicators are dependent or their probabilities differ from 0.05. The appropriate conclusion is that the control count is a diagnostic to investigate, not a direct estimate of the workflow’s error rate.
Permuted Labels as a Control
A label-permutation diagnostic asks what the marker procedure would report under a rearrangement of labels intended to represent a null situation. The permutation unit and restrictions must match the design. For example, if cells from the same biological sample are dependent, unrestricted cell-level shuffling can destroy that dependence and create an unrealistic null. A permutation may need to occur at the sample level or within valid strata. The exchangeability requirement is that, under the null being tested, the observed arrangement and the allowed rearrangements have the same distribution.
The target of the diagnostic depends on what is rerun. Permuting labels while holding cluster assignments fixed checks the marker test for those fixed labels, under the permutation assumptions. It does not account for the fact that the same expression data may have been used to discover those clusters. To assess that adaptive workflow, the null-generating procedure must represent the relevant selection process and rerun the steps whose selection is part of the question. Simply shuffling labels after clustering does not automatically remove double dipping.
For a chosen statistic \(T\), such as the number of genes declared significant, let \(T_{\mathrm{obs}}\) be its value for the observed data and \(T_b\) its value for permutation \(b\), for \(b=1,\ldots,B\). If the observed and permuted statistics are exchangeable under the null, a Monte Carlo upper-tail p-value is
The added one includes the observed arrangement among the possible null arrangements and prevents a reported Monte Carlo p-value of zero. This p-value answers a null question about the selected statistic under the specified permutation scheme. It does not certify every gene-level p-value, nor does it show that the permutation scheme represents the actual sampling design.
Worked Example: A Permutation Check for the Discovery Count
In a hypothetical experiment, the observed workflow reports \(T_{\mathrm{obs}}=18\) marker discoveries. The analyst performs \(B=999\) valid permutations, applying the same prespecified testing and discovery-counting steps each time. In 7 permutations, the discovery count is at least 18.
The Monte Carlo permutation p-value is
Under the assumed exchangeability scheme and null, a discovery count at least as large as 18 is unusual relative to the permutation reference distribution. This does not mean the probability that the null is true is 0.008. Nor does it establish that each of the 18 genes is a true marker: the statistic concerns the total count, and the interpretation depends on whether the permutation preserves the relevant structure and repeats the relevant selection steps.
Comparing the Two Controls and Diagnosing Failure
| Control | Useful diagnostic | Key assumption | What it cannot establish |
|---|---|---|---|
| Negative-control genes | Whether a specified set of expected-null genes shows an excess or pattern of apparent discoveries | The controls are true nulls for the contrast and relevant to the tested genes | Validity for all genes, or a false discovery rate for the full tested set without additional representativeness assumptions |
| Permuted labels | Whether a statistic is extreme relative to a null distribution generated by allowed label rearrangements | Exchangeability under the null and a permutation scheme that respects the design | Validity under a different null, design, or selection process not represented by the permutations |
| Independent validation data | Whether marker patterns replicate in data not used to select the original markers | The validation data and measurement process are sufficiently comparable | Proof that the original analysis was calibrated for every gene or population |
The main advantage of negative-control genes is that they can expose gene-level problems while leaving the observed labels and much of the workflow intact. Their weakness is that their expected null status and comparability must be justified. Permutations can provide a reference distribution for a chosen statistic and can incorporate more of the workflow if the null scheme is well designed. Their weakness is that an invalid shuffle can produce a precise but irrelevant answer. Independent validation is conceptually different: it assesses replication in new data rather than the null behavior of the original analysis.
Warning signs include a control set with more small p-values than expected, a concentration of control discoveries in a particular sample or technical stratum, or a permutation distribution that changes substantially under reasonable design-respecting schemes. These signals motivate investigation. They do not identify a unique cause: possible explanations include non-null controls, mismatched expression properties, dependence, batch imbalance, or an invalid test model. Conversely, few or no control discoveries do not prove validity; a small control set may have low power to reveal a problem.
A practical diagnostic should therefore state the control definition, the contrast, the tested statistic, the assumptions, and the limitations. It should report the number of controls and the observed results, rather than only saying that the controls “passed.” A control check is strongest when its assumptions are specified before inspecting the marker results and when its scope is kept narrow.
Name the cluster contrast and the gene-level or workflow-level quantity the control is intended to assess.
Use externally justified null genes or a permutation scheme that respects the sampling and dependence structure.
For label permutations, decide whether labels are fixed or selection is rerun; for control genes, use the same relevant testing and filtering rules.
Report the observed control behavior and assumptions, and do not treat a reassuring result as proof of general validity.
Worked Example: A Control Set with Different Detection Rates
A hypothetical marker analysis tests 200 genes at \(\alpha=0.05\). Fourteen genes in a proposed negative-control set are rejected. The controls were selected because they are believed not to respond to the contrast, but they are mostly detected in very few cells, unlike the genes in the main analysis.
The observed control rejection proportion is \(14/200=0.07\), or 7%. If all 200 controls are true nulls with valid p-values, the expected rejection count is at most \(200(0.05)=10\). Fourteen is a signal to examine, but the expectation bound alone does not determine whether it is surprising, and the 7% proportion is not automatically an estimate of the false discovery rate among all tested genes.
The detection-rate mismatch weakens the comparison: sparse genes can have different test behavior from the genes the diagnostic is meant to represent. The analyst should check whether the controls are truly null, inspect the p-value distribution and any relevant strata, and consider a prespecified control set more closely matched on detection and expression. Even after matching, dependence among genes means a binomial reference calculation would require additional assumptions. The appropriate conclusion is that this control set raises a calibration question but does not, by itself, quantify the marker analysis’s error rate.
Common Mistakes and Limits
- Calling the expected count exactly \(m\alpha\). Valid tests guarantee \(P(p_g \leq \alpha)\leq\alpha\), so \(E(V_C)\leq m\alpha\). Equality requires exact rejection probability \(\alpha\) for each test.
- Using independence to claim a binomial count. Independence is insufficient if rejection probabilities differ or are below \(\alpha\). A binomial model requires independent indicators with a common rejection probability.
- Using arbitrary genes as negative controls. A gene that responds to the contrast, differs in detection properties, or was selected after viewing results may not be a meaningful control.
- Permuting cells freely without checking the design. If cells are dependent within samples or strata, unrestricted shuffling may violate exchangeability.
- Holding selected clusters fixed and claiming to test the full adaptive workflow. This checks only the fixed-label part of the procedure; it does not reproduce the cluster-selection step.
- Interpreting a reassuring control as proof. A diagnostic can fail to detect a problem, especially with few controls or a poorly matched null scheme.
Negative controls are most valuable when they are treated as targeted stress tests. They can reveal unexpected behavior and help locate where a workflow deserves scrutiny, but they cannot replace a valid inferential design or turn an invalid test into a valid one.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For 60 true-null control genes, each with a valid p-value, what is the upper bound on the expected number with \(p_g\leq0.05\)? Does this require independence?
- Under what additional assumptions is the number of rejections among \(m\) controls binomial with parameters \(m\) and \(\alpha\)?
- A researcher permutes cluster labels among cells while keeping clusters and all preprocessing fixed. What aspect of the analysis does this check, and what important issue does it leave unaddressed?
- In 499 valid permutations, a statistic is at least as large as observed in 4 permutations. Calculate the Monte Carlo permutation p-value using the plus-one formula.
- Why does a 7% rejection fraction among negative controls not automatically estimate the false discovery rate among all tested genes?
Solutions. (1) The bound is \(E(V_C)\leq60(0.05)=3\). Linearity of expectation gives the bound without independence. (2) The rejection indicators must be independent Bernoulli variables with the same rejection probability, and that probability must equal \(\alpha\) for the stated \(\operatorname{Binomial}(m,\alpha)\) model. (3) It checks marker testing for the fixed labels under the assumed exchangeability scheme. It does not account for cluster selection from the same expression data. (4) \(\hat p_{\mathrm{perm}}=(1+4)/(499+1)=5/500=0.01\). This is a tail probability under the specified permutation null, not the probability that the null is true. (5) The controls may not represent the tested genes, may not all be true nulls, and may have different dependence or test behavior. A rejection fraction in that set is a diagnostic summary, not the full-set false discovery rate without further assumptions.