Why Replication Tests Need a Defined Family
A multi-omics validation study may test hundreds of molecular features, several assay modalities, and multiple outcomes. If each test is judged against the same nominal significance level, the chance of obtaining at least one false positive can be much larger than that level. A confirmatory analysis therefore needs two things specified together: the replication claims being tested and the error rate controlled across those claims.
Multiplicity is not determined simply by how many measurements appear in a data set. It is determined by the set of claims on which the study intends to make confirmatory decisions. That set must be defined from the validation protocol, not narrowed after seeing which validation results look promising. Earlier material on locked cross-cohort validation protocols provides the design context: the family should follow the prespecified scientific claims and analysis plan.
For example, if a protocol claims that each of 20 specified transcripts replicates its association with one outcome, those 20 feature-level tests form a family. If it makes claims for the same features across two outcomes, the family may contain 40 tests. If an analysis instead makes one claim about each feature-outcome pair, the hypotheses must be indexed at that level. The counting rule follows the scientific claim, not the number of rows in a software output.
A feature is not automatically one hypothesis regardless of context. Testing the same feature separately in two modalities or against two outcomes usually gives distinct hypotheses, because the claims are distinct. Conversely, if the protocol defines a single prespecified combined test as the claim for a feature, its components are not automatically separate confirmatory claims. The protocol must state what result counts as evidence for each claim.
Specify the Hypotheses Before Testing
For each claim \(j\), define a parameter \(\theta_j\) representing the validation effect in the target population, with its scale and units stated in the protocol. If the claim is directional and the expected direction was locked in advance, a one-sided formulation can be \(H_{0j}:\theta_j\leq 0\) versus \(H_{aj}:\theta_j>0\). For a nondirectional claim, use \(H_{0j}:\theta_j=0\) versus \(H_{aj}:\theta_j\neq 0\). The definition of the null and alternative must match the claim and the test used to calculate its p-value.
The multiplicity procedure assumes that each input p-value is valid for its corresponding null: under that null, the probability that its p-value is at most \(u\) is no greater than \(u\), for \(0\leq u\leq1\). The tests may be dependent. Shared participants, correlated molecular measurements, and common outcomes can all induce dependence, but they do not by themselves invalidate Bonferroni or Holm control if the individual p-values remain valid.
Use the stated significance level \(\alpha\) for the whole family, not separately for every hypothesis. A common choice is \(\alpha=0.05\). The procedures below control the probability of one or more false rejections across the family, subject to valid p-values and a correctly specified family.
Bonferroni and Holm Control
The Bonferroni procedure rejects hypothesis \(j\) when \(p_j\leq\alpha/m\), where \(m\) is the number of hypotheses in the family. Equivalently, compare the Bonferroni-adjusted p-value \(\min(1,mp_j)\) with \(\alpha\). This procedure is simple and controls FWER under any dependence among the tests.
Holm’s step-down procedure is also valid under arbitrary dependence and is never less powerful than ordinary Bonferroni for the same family and \(\alpha\). Order the p-values from smallest to largest, \(p_{(1)}\leq\cdots\leq p_{(m)}\). Compare them in order to increasingly permissive thresholds. Reject sequentially until the first comparison that fails; do not reject that hypothesis or any that follow it.
The denominator decreases as \(i\) increases. Thus, the first p-value faces the strictest threshold, \(\alpha/m\), while the last faces \(\alpha\). Holm-adjusted p-values can be computed as \(\min\{1,\max_{k\leq i}[(m-k+1)p_{(k)}]\}\) for the hypothesis at rank \(i\). The running maximum ensures adjusted p-values do not decrease as the ordered raw p-values increase.
The justification only needs each true-null p-value to be valid; it does not require independence. The same union-bound argument gives Bonferroni control: the probability that any one of \(m\) true nulls has \(p_j\leq\alpha/m\) is at most \(m(\alpha/m)=\alpha\).
Worked Example: Holm Correction Across Features
A hypothetical validation protocol specifies six directional replication claims: three RNA features and three protein features, each for the same outcome. The one-sided p-values are 0.004, 0.012, 0.018, 0.031, 0.200, and 0.600. Assume the individual tests are valid and use \(\alpha=0.05\) for the family.
Order the p-values as listed. There are \(m=6\) tests. At rank 1, the Holm threshold is \(0.05/6=0.008333\), and \(0.004\leq0.008333\), so reject the first null. At rank 2, the threshold is \(0.05/5=0.01000\), but \(0.012>0.01000\). Stop at this failure: the remaining hypotheses are not rejected by Holm.
Thus only the claim with \(p=0.004\) is rejected at familywise level 0.05. Its Holm-adjusted p-value is \(6(0.004)=0.024\). For the next claim, the untruncated adjusted value is \(\max\{6(0.004),5(0.012)\}=\max\{0.024,0.060\}=0.060\), which exceeds 0.05. The other claims also cannot be rejected. Under ordinary Bonferroni, the threshold is \(0.05/6=0.008333\), so the same first claim, and no others, would be rejected here.
Make the Family Match the Validation Claim
When features, modalities, and outcomes all vary, make the indexing explicit. Suppose a protocol tests each of two metabolites and two transcripts against each of two clinical outcomes. If each feature-outcome association is a distinct claim, the family contains \(4\times2=8\) hypotheses. If the claims are instead defined separately by modality and outcome, the protocol should make clear whether each feature remains an individual claim or whether an aggregate modality-level test is intended.
The same logic applies to cohorts. If the confirmatory claim concerns one prespecified validation estimate per feature, a single estimate may yield one p-value for that feature. If the protocol separately claims replication in each of several cohorts, those cohort-specific tests add hypotheses to the family. A pooled test and several cohort-specific claims are not interchangeable after results are known. State which is confirmatory in advance.
Worked Example: Counting Across Features, Modalities, and Outcomes
A hypothetical protocol makes separate claims for two proteins and two metabolites, each measured against both a glucose outcome and an inflammatory outcome. This yields eight claims. Suppose their ordered p-values are 0.003, 0.007, 0.011, 0.019, 0.024, 0.041, 0.220, and 0.700, and assume all eight p-values are valid.
For Holm at \(\alpha=0.05\), the rank-1 threshold is \(0.05/8=0.00625\); \(0.003\) passes. The rank-2 threshold is \(0.05/7=0.007143\); \(0.007\) passes. The rank-3 threshold is \(0.05/6=0.008333\); \(0.011\) fails. Stop at rank 3, so exactly the first two claims in the ordered list are rejected.
For clarity, suppose those first two p-values correspond to protein A with glucose and metabolite B with inflammation. The supported confirmatory conclusions are limited to those two feature-outcome claims. The remaining six are not rejected by this familywise procedure; that does not establish that their effects are zero.
Choosing an Error-Control Target
For a small, explicitly confirmatory family, Holm is often a useful default: it controls FWER under arbitrary dependence and is at least as powerful as Bonferroni. Bonferroni remains attractive when a single transparent threshold is desired, especially for a very small family. Both methods can be conservative when many tests are strongly dependent or when the family is large.
False discovery rate (FDR) control is a different objective. The Benjamini–Hochberg procedure is designed to control the expected proportion of false rejections among all rejections, under independence and under certain forms of positive dependence. It does not generally guarantee that the probability of even one false rejection is at most \(\alpha\). FDR methods can suit large-scale screening, but they should not be presented as FWER control for confirmatory claims. The protocol should state which error rate its scientific conclusions require.
| Approach | Target | Useful distinction |
|---|---|---|
| Bonferroni | FWER | Compare every p-value with \(\alpha/m\); valid under arbitrary dependence. |
| Holm | FWER | Step down through ordered p-values; valid under arbitrary dependence and at least as powerful as Bonferroni. |
| Benjamini–Hochberg | FDR | Targets the expected false-discovery proportion, not the chance of any false rejection. |
Diagnostics, Pitfalls, and Limitations
Multiplicity correction cannot repair invalid individual tests. Check that each p-value corresponds to its prespecified feature, modality, outcome, direction, and analysis; that its null distribution is appropriate; and that the validation estimate and standard error were calculated as planned. Missing measurements or failed assays should be handled according to prespecified rules. Do not quietly remove difficult or nonsignificant tests from the family after observing the results.
- Defining the family after looking at p-values. Keeping only the strongest results makes the reported family smaller than the one searched. Define the family and its exclusions before examining validation results.
- Counting only the most favorable modality. If claims were tested across modalities or outcomes and conclusions may be drawn from any of them, those claims belong in the relevant confirmatory family. Do not correct only the selected subset.
- Interpreting non-rejection as no effect. A failure to reject is not proof of a null effect. Report the estimate and its uncertainty in context; the test alone does not establish equivalence.
- Changing direction after seeing results. A one-sided p-value is justified only when the direction was prespecified. Choosing the direction that matches the observed estimate changes the testing procedure and can inflate false-positive risk.
- Treating correction as a substitute for design. Multiplicity adjustment does not address participant overlap, cohort comparability, measurement differences, or selective model choices. These require their own design and analysis checks.
The key practical limitation is that a correction only protects the family it was applied to. If the study explores many additional outcomes, feature definitions, or analysis choices and reports only a selected subset as confirmatory, the nominal guarantee no longer describes the full search. Transparent reporting should identify the family, the number of hypotheses, the procedure, the chosen \(\alpha\), and the adjusted decisions.
Worked Example: Prespecified Direction and Family Size
A hypothetical validation protocol specifies three one-sided claims that a molecular feature has a positive association with a clinical outcome. Valid one-sided p-values are 0.012, 0.028, and 0.041. With \(m=3\) and familywise \(\alpha=0.05\), Holm compares the first p-value with \(0.05/3=0.01667\). Since \(0.012\leq0.01667\), reject the first null.
At rank 2, compare \(0.028\) with \(0.05/2=0.025\). Because \(0.028>0.025\), stop: the second and third null hypotheses are not rejected. The first Holm-adjusted p-value is \(3(0.012)=0.036\). The second is \(\max\{3(0.012),2(0.028)\}=0.056\), and the third is \(\max\{0.036,0.056,1(0.041)\}=0.056\).
The decision is justified only for the positive-direction claims specified in advance. If the protocol had instead specified two-sided claims, the p-values and hypotheses would need to correspond to those two-sided tests. Reversing direction after observing a negative estimate would not be a valid way to preserve the original one-sided procedure.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A protocol tests 12 features against each of three outcomes, with one separate claim for every feature-outcome pair. How many hypotheses are in the family? If \(\alpha=0.05\), what is the Bonferroni threshold?
- For four ordered p-values, Holm has rejected the first two, but the third p-value is larger than its Holm threshold. What decision is made for the third and fourth hypotheses?
- Why can Holm control FWER even when tests are dependent?
- A researcher reports the smallest p-value from several modalities but says the family contained only the selected modality. What is the problem?
Solutions. (1) There are \(12\times3=36\) hypotheses. The Bonferroni threshold is \(0.05/36=0.001389\). (2) The third hypothesis is not rejected, and the step-down procedure stops; the fourth is also not rejected. (3) For any configuration of true nulls, if a true null is rejected, the first true null in the ordered list must have p-value at most \(\alpha/k\), where \(k\) is the number of true nulls. The union bound makes the probability of this event at most \(k(\alpha/k)=\alpha\); this argument does not require independence. (4) The family was selected after inspecting results. If claims across several modalities were eligible for reporting, they must be accounted for in the prespecified family; correcting only the winner does not provide the stated familywise guarantee.