Adjusting For The Largest Selected Marker Statistic

Why the Largest Marker Statistic Needs an Adjustment

Suppose an analysis considers a prespecified set of $m$ genes and reports the gene with the largest marker statistic. Even if every gene-level test is valid on its own, choosing the most extreme result makes that result more likely to look compelling than a gene named before seeing the data. The relevant question is therefore not only whether the selected gene’s statistic is large, but also how large the maximum would be under a suitable null model.

This tutorial concerns the multiplicity created by selecting the largest statistic within a fixed family. It does not make an invalid gene-level test valid. In particular, if the cluster labels were learned from the same expression values used to test markers, an ordinary single-gene p-value may not have its claimed null distribution. The earlier material on selective inference describes ways to account for selection in the gene-level test; the family-wise adjustment considered here is an additional step when choosing among genes.

Definition: For a prespecified family of $m$ hypotheses $H_{0j}$, $j=1,\ldots,m$, the family-wise error rate (FWER) is the probability of rejecting at least one true null hypothesis: $$ \mathrm{FWER}=P(\text{at least one true }H_{0j}\text{ is rejected}). $$ A procedure controls FWER at level $\alpha$ if this probability is at most $\alpha$ for every configuration of true and false hypotheses under its stated assumptions.

Let $T_j$ be the test statistic for gene $j$, oriented so that larger values are more evidence against its null. For a two-sided test, this could be an absolute standardized statistic. Let $p_j$ be its corresponding p-value, and let $p_{\min}=\min_{1\leq j\leq m}p_j$. Selecting the largest statistic is equivalent to selecting the smallest p-value only when all tests use compatible, correctly calibrated p-values and the same direction or two-sided convention.

Bonferroni and Šidák Adjustments

The Bonferroni procedure rejects a gene-level null when $p_j\leq\alpha/m$. Equivalently, for the selected gene it reports the adjusted p-value $\min(1,mp_{\min})$. This procedure does not require independence among genes. Its key requirement is that each true-null p-value is valid: for every $u$ between 0 and 1, $P(p_j\leq u)\leq u$.

Theorem: If each true-null p-value is valid, the Bonferroni procedure controls FWER at level $\alpha$, for any dependence among the tests.

The justification is the union bound. If $I_0$ is the set of indices of true null hypotheses, then

P(\text{at least one true null is rejected}) =P\left(\bigcup_{j\in I_0}\{p_j\leq \alpha/m\}\right) \leq \sum_{j\in I_0}P(p_j\leq\alpha/m) \leq \frac{|I_0|\alpha}{m} \leq \alpha.

The inequality does not require the events to be independent. Bonferroni can be conservative, especially when test statistics are strongly correlated, but its guarantee is straightforward.

When the tests are independent and all $m$ null hypotheses are true, the probability that the minimum p-value is no larger than $c$ is $1-(1-c)^m$. Setting this equal to $\alpha$ gives the Šidák cutoff $c=1-(1-\alpha)^{1/m}$, and the corresponding adjusted p-value is $1-(1-p_{\min})^m$. This is an exact complete-null calculation under independence. With some false nulls, the distribution of the minimum across all $m$ p-values is not generally this formula: false-null p-values need not be uniform.

Nevertheless, if the p-values for the true nulls are independent and valid, the Šidák cutoff based on the full family size $m$ still controls FWER. There are at most $m$ true nulls, and the chance of at least one true-null p-value falling below the cutoff is at most $1-(1-c)^m=\alpha$. In that partial-null setting, the formula based on $p_{\min}$ is not generally the exact FWER or the exact adjusted p-value for the selected gene.

Worked Example: Bonferroni for the Strongest of 12 Genes

Suppose a hypothetical analysis prespecified 12 candidate genes. Assume the gene-level p-values are valid for their null hypotheses under the analysis being used. The smallest p-value is $0.0038$, and the target FWER is $\alpha=0.05$. No independence assumption is needed for Bonferroni.

The per-gene rejection threshold is

\frac{\alpha}{m}=\frac{0.05}{12}=0.004167.

The selected gene has $p_{\min}=0.0038<0.004167$, so it passes the Bonferroni family-wise criterion. Its adjusted p-value is

\min(1,mp_{\min})=\min(1,12\times 0.0038)=0.0456.

At the 0.05 level, this procedure rejects the selected gene’s null while controlling the probability of one or more false rejections in the 12-gene family at no more than 0.05. This conclusion depends on the stated validity of the gene-level p-values; the multiplication by 12 does not establish that validity.

Worked Example: Šidák Under a Complete Null

Consider a hypothetical family of 20 independent gene tests, with all 20 null hypotheses true and each p-value exactly uniform under its null. Suppose the smallest p-value is $0.004$. Under these assumptions, the probability that none of the 20 p-values is at most $p_{\min}$ is $(1-p_{\min})^{20}$. Thus the complete-null probability of seeing a minimum this small or smaller is

1-(1-0.004)^{20}=1-0.996^{20}\approx 0.0770.

The Šidák-adjusted p-value is approximately $0.0770$, so the result does not meet a 0.05 family-wise threshold. The unadjusted p-value, $0.004$, would have met a single-test 0.05 threshold, but that comparison ignores the search over 20 genes.

The calculation is exact here because all tests are null and independent. It would be incorrect to claim that $1-(1-p_{\min})^{20}$ is always the exact probability for the minimum when some genes have false null hypotheses.

Calibrating the Maximum Statistic

A more direct approach is to calibrate the maximum statistic $M=\max_{1\leq j\leq m}T_j$ against its joint null distribution. For example, suppose under the complete null the statistics are independent standard normal variables and the alternative is one-sided toward large values. If $\Phi$ is the standard normal cumulative distribution function, then

P(M\leq t)=P(T_1\leq t,\ldots,T_m\leq t)=\Phi(t)^m.

The maximum-statistic p-value for an observed maximum $t_{\mathrm{obs}}$ is therefore $1-\Phi(t_{\mathrm{obs}})^m$. For two-sided tests based on $|T_j|$, the corresponding independent complete-null distribution is $P(\max_j|T_j|\leq t)=[2\Phi(t)-1]^m$, for $t\geq 0$. These formulas require the stated null distributions and independence; correlated gene statistics generally have a different maximum distribution.

Worked Example: Adjusting a One-Sided Maximum Statistic

In a hypothetical prespecified family of eight genes, suppose the largest one-sided standardized statistic is $2.40$. For this example, assume that under the complete null each statistic is standard normal and the eight statistics are independent. The unadjusted single-gene p-value for $2.40$ is approximately $1-\Phi(2.40)=0.00820$.

The probability of a maximum at least as large as $2.40$ is

1-\Phi(2.40)^8 =1-(1-0.00820)^8 \approx 1-0.99180^8 \approx 0.0637.

The maximum-statistic adjusted p-value is about $0.0637$, so the maximum does not pass a 0.05 threshold. The unadjusted value $0.00820$ answers a different question: how unusual this statistic would be for one prespecified gene under its null, not how unusual the largest statistic would be after considering eight genes.

When the joint null distribution is known, a maximum-statistic cutoff can account for dependence and may be less conservative than Bonferroni. For strong FWER control across configurations with both true and false nulls, calibration under the complete null needs an additional condition such as subset pivotality: the joint distribution of the test statistics for the true null hypotheses must be the same whether or not other hypotheses are false. Without such a condition, complete-null calibration alone does not automatically guarantee strong FWER control. Bonferroni avoids this particular joint-distribution requirement if the marginal true-null p-values are valid.

Selection, Assumptions, and Diagnostics

Before applying any adjustment, specify the family. The set of genes must be determined independently of the test results for the simple formulas above to describe the intended search. If the analyst examines many genes but reports only a subset, adjusting for only the reported subset does not account for the full search. Likewise, choosing the contrast, direction, or analysis based on observed results can expand the effective family beyond the gene list.

Most importantly, adjustment assumes valid gene-level tests. If cluster labels were adaptively defined from the same expression data, an ordinary p-value that treats those labels as fixed may not be uniform under the relevant null. Bonferroni, Šidák, and maximum-statistic procedures do not repair that double dipping. Use a gene-level inferential method that accounts for the relevant selection, or obtain valid tests from an appropriate held-out or independent analysis, before applying a family-wise adjustment.

Useful checks include verifying that the number $m$ includes every gene searched, confirming that statistics and p-values use consistent one- or two-sided definitions, and checking the null calibration of the underlying tests. If the tests are strongly correlated, an independence-based Šidák calculation should not be presented as exact. If the joint null distribution is estimated or specified for a maximum-statistic procedure, its assumptions and calibration need scrutiny; an assumed covariance structure that poorly represents the genes can misstate tail probabilities.

The main alternatives answer different inferential questions. An unadjusted single-gene test is appropriate when the gene and test were genuinely specified in advance and no selection from a larger family occurred. Bonferroni is robust to dependence but may sacrifice power. Šidák exploits independence for a sharper cutoff under its assumptions. A calibrated maximum-statistic procedure can use joint dependence information, but its guarantees require a defensible joint null model and, for strong control, suitable conditions such as subset pivotality. None of these changes an exploratory marker ranking into confirmatory evidence by itself.

Common Mistakes and Practical Interpretation

Treating the minimum-p formula as universal. The expression $1-(1-p_{\min})^m$ is the exact distributional calculation for independent, uniform p-values under the complete null. With false nulls among the family, it is not generally the exact probability for the minimum across all tests.
Reporting only the unadjusted p-value of the winner. That p-value does not reflect selection of the strongest result from $m$ candidates. Report the family size and the adjustment used.
Assuming adjustment fixes double dipping. Multiplicity correction addresses selection among tests with valid null p-values. It cannot make a post-clustering p-value valid if the test ignored how the clusters were selected.
Calling a non-significant adjusted result proof of no marker effect. Failure to reject at the chosen FWER level is not acceptance of the null; it indicates that the adjusted evidence did not meet that decision threshold.

Key takeaway: For the strongest result among a prespecified family, report an adjustment that matches the test dependence and null assumptions. Bonferroni gives a simple dependence-robust FWER guarantee when true-null p-values are valid; more exact maximum-based calculations require stronger assumptions. First ensure that the gene-level tests themselves account for any data-adaptive clustering.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A prespecified family contains 25 genes and has valid gene-level p-values. What Bonferroni threshold gives FWER at most 0.05, regardless of dependence?
Under the complete null, 10 independent tests have a minimum p-value of 0.01. Write the exact Šidák adjustment and state whether it is smaller or larger than 0.05.
There are two independent tests: one true-null p-value is uniform, while the other gene’s null is false and its p-value is always zero. Is $1-(1-p_{\min})^2$ the exact FWER? Explain.
Why does a family-wise adjustment not automatically validate an ordinary marker p-value after data-adaptive clustering?

Solutions. (1) The threshold is $0.05/25=0.002$. The union-bound argument controls the chance of any false rejection at no more than 0.05. (2) The adjustment is $1-(1-0.01)^{10}=1-0.99^{10}\approx 0.0956$, larger than 0.05. It is exact under the stated complete-null independence assumptions. (3) No. Here $p_{\min}=0$ always, so the formula gives zero, but the probability that neither p-value is at most any positive threshold is zero, not the complete-null expression $(1-c)^2$. The false-null p-value is not uniform, so the complete-null distribution does not apply. (4) The adjustment controls multiplicity only when the gene-level p-values are valid for their nulls. If clustering used the same expression data and the test treats the resulting labels as fixed without accounting for selection, that validity may fail before the family-wise adjustment is applied.