Why Calibrate Against the Maximum?
In a genomic analysis, one scientific question may generate hundreds or thousands of related tests. A familywise error occurs if the analysis rejects even one true null hypothesis in the family. Since the test statistics can be dependent—for example, nearby genetic variants may be correlated—adjustment based on their joint behavior can be useful. The single-step permutation maxT procedure uses that joint behavior by asking how extreme each observed statistic is relative to the largest statistic in a permutation replicate.
The method begins with a family defined for the scientific question, as in the earlier material on familywise error and testing families. It does not decide which tests belong in that family. Once the family is fixed, the procedure calibrates each test against a common reference distribution of family-wide maxima.
Construction and Interpretation
Use the same permutation of the experimental labels or residual structure across all tests in a replicate. This is important: if the same subjects contribute measurements to many genomic features, a shared permutation retains the dependence among those features in the resulting vector of statistics. For two-sided tests, take the absolute value before computing each replicate maximum; for one-sided tests, use statistics oriented in the relevant alternative direction.
If \(B\) random permutations are sampled, let \(M^{(1)},\ldots,M^{(B)}\) be their maxima. A common Monte Carlo estimate of the adjusted p-value is
Here \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. The plus-one correction includes the observed data as a reference outcome and avoids reporting a Monte Carlo p-value of zero. If all allowable permutations are enumerated, use the proportion of the complete reference set that is at least as extreme as the observed statistic, counting the observed arrangement as part of that set. Ties are counted as at least as extreme, making the rule conservative when the reference distribution is discrete.
An adjusted p-value is not the probability that the null hypothesis is true. It is a tail probability calibrated against the family-wide maximum reference distribution. Comparing \(\widehat p_{j,\mathrm{maxT}}\) with \(\alpha\) gives a familywise decision rule: reject \(H_{0j}\) when \(\widehat p_{j,\mathrm{maxT}}\leq\alpha\). Thus, at level \(\alpha\), each rejection is judged against the chance of seeing a statistic at least that extreme somewhere in the family under the relevant null reference.
Worked Example: Calculating Three Adjusted P-Values
Suppose a hypothetical genomic screen tests three variants using two-sided statistics \(T_j=|t_j|\). The observed statistics are 3.4, 2.9, and 2.2. The analyst generates \(B=19\) valid shared permutations and obtains these sorted family maxima:
\(1.1, 1.3, 1.5, 1.7, 1.8, 1.9, 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.7, 2.8, 2.9, 3.0, 3.1, 3.3, 3.6.\)
For the first variant, no maximum is at least 3.4, so
For the second variant, five maxima are at least 2.9 (2.9, 3.0, 3.1, 3.3, and 3.6). Therefore \(\widehat p_{2,\mathrm{maxT}}=(1+5)/20=0.300\). For the third, eleven maxima are at least 2.2, giving \(\widehat p_{3,\mathrm{maxT}}=(1+11)/20=0.600\). At \(\alpha=0.05\), the first test meets the rejection threshold and the other two do not. This is a familywise-calibrated decision only if the permutation scheme and control conditions are appropriate; the small number of permutations also makes the p-values coarse.
Assumptions and Why the Maximum Controls the Family
A permutation distribution is valid only when the permitted rearrangements reflect the data-generating design or a justified null model. In a randomized experiment, treatment labels may be permuted according to the actual randomization scheme under a sharp null of no treatment effect for any unit. In observational or regression settings, unrestricted label shuffling is not automatically valid. Exchangeability must be justified: under the null model, the observations being permuted must have a joint distribution unchanged by the permitted rearrangements. Blocking, pairing, covariates, or repeated measurements may require restricted permutations or a different valid null-resampling scheme.
There are two levels of error control to distinguish. Under the complete null, where all hypotheses in the family are true, calibration of the maximum gives weak familywise error control if the permutation distribution correctly represents the joint null distribution. Strong control—control of the probability of at least one false rejection for every configuration of true and false nulls—requires validity for the joint distribution of the true-null statistics when other hypotheses may be false. A standard sufficient condition is subset pivotality: the joint distribution of statistics for any subset of true nulls does not depend on which hypotheses outside that subset are false. If this condition or another justification for the true-null reference distribution is absent, a global-null permutation maxT calibration should not automatically be claimed to give strong control.
The central justification is that any false rejection among the true nulls requires at least one true-null statistic to cross the maxT cutoff. When the reference distribution correctly calibrates the maximum over those true nulls, the chance of that event is at most \(\alpha\). Using the maximum over the whole family is at least as conservative as using the maximum over only the true nulls, because it includes every true-null statistic and possibly additional ones. Under a valid joint reference for the true-null statistics, this gives the familywise bound.
Specify the hypotheses and choose statistics whose larger values consistently represent more extreme evidence, using absolute values for two-sided tests.
Use rearrangements justified by the design or null model, respecting blocks, pairs, and other restrictions.
Apply each shared permutation across the family, calculate all statistics, and retain only the largest statistic for that replicate.
Count maxima at least as large as the observed statistic and use the appropriate complete-enumeration or Monte Carlo tail proportion.
Worked Example: A One-Sided Family and a Monte Carlo Tail Count
Suppose a hypothetical study tests four genes for increased expression in a treatment group. The test statistics are one-sided and oriented so that larger values support increased expression. The observed statistic for Gene A is 2.6. From \(B=99\) valid shared permutations, 7 family maxima are at least 2.6.
The adjusted p-value is
At \(\alpha=0.05\), Gene A is not rejected after single-step maxT adjustment. This does not establish that its null is true; it says that the observed statistic does not pass the chosen familywise threshold under this procedure. The permutation scheme must preserve the design, and the one-sided direction must have been chosen before interpreting the result. Taking absolute values here would answer a different, two-sided question.
Diagnostics, Alternatives, and Limitations
Check both the data structure and the computed reference distribution. Confirm that every replicate uses the same permitted permutation across all tests; otherwise the empirical maxima need not represent the joint dependence. Check that the tail direction matches the alternative, that the number of permutations is adequate for the desired p-value resolution, and that observed and permuted statistics are calculated in the same way. For Monte Carlo sampling, the smallest possible adjusted p-value is \(1/(B+1)\), so \(B=99\) cannot produce a value below 0.01.
A conspicuously narrow or implausibly light-tailed maximum distribution may indicate that dependence was destroyed—for example, independently permuting each gene rather than permuting subject labels jointly. Unrestricted permutations can also be invalid when observations are paired, clustered, or have nonexchangeable covariates. If a feature is constant under permutation, or the model fit fails repeatedly for some replicates, investigate coding, missingness, and the definition of the statistic rather than treating the resulting p-values as reliable.
The main conservative alternative is Bonferroni control, which does not require estimating the dependence structure: it compares each unadjusted p-value with \(\alpha/m\), or equivalently multiplies each p-value by \(m\), capped at 1. MaxT can be less conservative when tests are strongly dependent because it calibrates against the observed joint behavior rather than a worst-case union bound. In exchange, it requires a defensible permutation mechanism and enough computation. A maxT analysis is not automatically more powerful: a large family maximum can create a demanding reference threshold, and invalid permutations can make its calibration misleading. This is a single-step procedure; it uses the same family-wide reference distribution for every test rather than updating the reference as rejections accumulate.
Worked Example: Respecting Pairing in a Permutation Scheme
In a hypothetical paired experiment, each of 12 participants provides a control and a treated sample. The target is a two-sided treatment effect for each of several genes. Swapping treated and control labels independently within each participant is consistent with the paired design under a sharp null: for each replicate, flip the sign of each participant’s within-pair difference, using the same set of 12 sign flips for every gene. Calculate the gene statistics from those permuted differences and retain their maximum absolute statistic.
By contrast, pooling all 24 samples and shuffling labels freely would allow arrangements that do not preserve the pairing. Such shuffles do not reproduce the experiment’s assignment structure, so the resulting maximum distribution may be invalid even if the number of permutations is very large. The correction is to restrict permutations to the actual exchangeable units and apply each restricted rearrangement jointly across the gene family. This example also shows why “more permutations” cannot repair a wrong permutation design.
Check Your Understanding
Answer each question, then compare with the solutions below.
- With \(B=49\) sampled permutations, 3 maxima are at least as large as an observed statistic. Calculate its Monte Carlo maxT-adjusted p-value.
- Why should the same permutation be applied across all genomic tests within a replicate?
- A two-sided test has observed \(t=-3.1\). What statistic should be compared with maxima constructed from absolute t-statistics?
- Does control under the complete null alone establish strong familywise control? State a condition that can justify strong control.
- Why is unrestricted label shuffling potentially invalid for paired observations?
Solutions. (1) \((1+3)/(49+1)=0.080\). (2) Shared permutations preserve the dependence among test statistics, which affects the family maximum’s distribution. (3) Compare \(|-3.1|=3.1\). (4) No. Strong control requires a valid reference for the true-null statistics under configurations with false nulls; subset pivotality is one sufficient condition. (5) It breaks the pairing structure and includes rearrangements that were not permitted by the design, so the resulting observations need not be exchangeable under the null.