Why Calibrate the Maximum?
A genomic analysis may test thousands of features, such as genes, regions, or molecular measurements. If each test is judged against its own unadjusted threshold, the probability of making at least one false rejection can be much larger than the per-test error rate. The tests are often correlated, so treating them as independent is not generally justified. A single-step max-T procedure instead uses the joint behavior of all test statistics under a common resampling scheme.
The key idea is to compare each observed statistic with the largest statistic produced across the whole family in each permutation. This calibrates every test against the same reference distribution. The result is a familywise adjustment that accounts for the dependence represented by the joint permutations. Its guarantee in this tutorial is weak familywise control: control of the probability of any false rejection when all tested null hypotheses are true.
Definition and Assumptions
Suppose there are \(m\) hypotheses, with test statistics \(T_1,\ldots,T_m\). For two-sided tests, the procedure uses their absolute values. Let \(b=1,\ldots,B\) index the allowed permutations or transformations, and let \(T_j^{(b)}\) be the statistic for test \(j\) under transformation \(b\). The transformed statistics must be computed jointly: each transformation is applied consistently across all tests, preserving the dependence structure represented by the data and design.
The formula uses “greater than or equal to,” so ties with the observed statistic count in the tail. This convention makes the permutation test conservative or exact in the presence of ties; it does not make it anti-conservative. The reference set for an exact permutation test includes the observed assignment and all allowed transformations, with the relevant randomization or exchangeability assumptions in force.
The procedure relies on a valid joint reference distribution under the complete null. In a randomization test, the allowed assignments must follow the study’s randomization scheme under the null. In a model-based permutation test, the observations or residuals being permuted must be exchangeable under the null, with any restrictions required by pairing, blocks, or nuisance structure respected. The test statistics and their studentization must also be defined consistently across transformations. If these conditions fail, a well-computed maximum does not repair an invalid permutation reference distribution.
Why the Procedure Controls Weak Familywise Error
Write \(M^{\mathrm{obs}}=\max_{1\leq j\leq m}|T_j^{\mathrm{obs}}|\). If all null hypotheses are true, rejecting any hypothesis at level \(\alpha\) under the max-T rule is the same event as the observed maximum being sufficiently extreme in its null reference distribution. The joint calibration therefore targets the event of at least one rejection directly, rather than adding up separate per-test error probabilities.
For an exact permutation distribution, the rank of the observed maximum among the maxima generated by allowed transformations is uniform when there are no ties. With ties, counting all reference maxima greater than or equal to the observed value gives a conservative tail probability. Thus the complete-null probability that the maximum-based p-value is at most \(\alpha\) is no greater than \(\alpha\). Since a rejection of any individual test requires the maximum to cross the same calibrated boundary, this is weak familywise control.
The adjusted p-value for test \(j\) is not the probability that its null hypothesis is true. It is the tail proportion of the joint max-T reference distribution at least as large as that test’s observed absolute statistic. A small value means that a statistic of at least that magnitude is rare when each permutation is judged by its most extreme test.
Critical Values and Discreteness
A critical value can be used instead of calculating each adjusted p-value, but the boundary must be chosen carefully. Let \(M\) denote a random draw from the exact reference distribution of the maximum. For a discrete distribution, choose a support value \(c_\alpha\) satisfying \(P_{\mathrm{perm}}(M>c_\alpha)\leq\alpha<P_{\mathrm{perm}}(M\geq c_\alpha)\). Reject test \(j\) exactly when \(|T_j^{\mathrm{obs}}|>c_\alpha\). The strict inequality and the two tail conditions work together: they ensure the critical-value rule agrees with \(p_{j,\mathrm{adj}}\leq\alpha\), including at the boundary.
To verify this equivalence, if \(|T_j^{\mathrm{obs}}|>c_\alpha\), the reference maxima at least that large form a subset of those greater than \(c_\alpha\). Their proportion is therefore at most \(P_{\mathrm{perm}}(M>c_\alpha)\leq\alpha\). If \(|T_j^{\mathrm{obs}}|\leq c_\alpha\), every reference maximum at least \(c_\alpha\) is also at least \(|T_j^{\mathrm{obs}}|\). The adjusted p-value is at least \(P_{\mathrm{perm}}(M\geq c_\alpha)>\alpha\). This proves both directions. The strict bracket matters: merely requiring \(P_{\mathrm{perm}}(M>c_\alpha)\leq\alpha\) can produce a boundary that rejects fewer tests than the p-value rule.
Worked Example: A Discrete Critical Value
Suppose an exact reference distribution contains 10 equally weighted maximum statistics with values \(0.4, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0, 2.2,\) and \(2.4\). Set \(\alpha=0.10\). For \(c_\alpha=2.2\), one value is strictly greater than the boundary, while two are greater than or equal to it:
The boundary satisfies both required conditions. An observed absolute statistic of \(2.4\) has adjusted p-value \(1/10=0.10\), so it is rejected by both rules: \(2.4>2.2\), and its adjusted p-value is at most \(0.10\). An observed value of \(2.2\) has adjusted p-value \(2/10=0.20\), so neither rule rejects it. The observed maximum crosses the boundary only for one of the 10 equally likely transformations; under the complete null, the probability of any rejection is therefore \(0.10\).
Had the boundary been set to \(2.4\), the strict rule would reject no statistic, even though a statistic of \(2.4\) has adjusted p-value \(0.10\). The boundary must be selected to match the tail convention, not merely to make the strict upper-tail probability no greater than \(\alpha\).
Calculating and Interpreting Adjusted P-Values
In practice, the same set of maximum statistics can calibrate all \(m\) tests. For each observed feature statistic, count how many permutation maxima are at least as large as its absolute value, then divide by the number of equally weighted transformations. The adjusted p-values are monotone in the observed absolute statistics: a larger observed statistic cannot have a larger max-T adjusted p-value.
Worked Example: Adjusted P-Values Across Three Features
Consider a hypothetical analysis with three features and 20 exact permutations. Suppose the sorted permutation maxima are \(1.1,1.4,1.8,2.0,2.2,2.4,2.6,2.8,2.8,3.0,3.1,3.2,3.4,3.6,3.8,4.0,4.1,4.3,4.5,4.8\). The observed absolute statistics are \(4.5\), \(3.2\), and \(2.8\), respectively.
For the first feature, two maxima are at least \(4.5\), namely \(4.5\) and \(4.8\). Thus \(p_{1,\mathrm{adj}}=2/20=0.10\). For the second feature, nine maxima are at least \(3.2\), so \(p_{2,\mathrm{adj}}=9/20=0.45\). For the third feature, 13 maxima are at least \(2.8\), including both tied values of \(2.8\), so \(p_{3,\mathrm{adj}}=13/20=0.65\).
At \(\alpha=0.10\), only the first feature meets the adjusted-p-value criterion. A boundary satisfying the discrete quantile conditions is \(c_\alpha=4.3\): two maxima are greater than \(4.3\), giving \(2/20=0.10\), and three are at least \(4.3\), giving \(3/20=0.15>0.10\). The strict critical-value rule also rejects only the statistic \(4.5\). The values are hypothetical and illustrate the calculation, not evidence about a real genomic study.
Because the reference maximum is computed across the tests together, the procedure can reflect correlations among test statistics. No independence assumption among features is needed for the complete-null argument. The dependence must, however, be represented appropriately by the joint transformations; permuting each feature independently would generally create a different and potentially invalid reference distribution.
Worked Example: Correlated Tests and a Coarse Reference Distribution
Suppose four hypothetical genomic features are tested using 12 exact joint permutations. The sorted maxima are \(1.2,1.4,1.5,1.7,1.8,2.0,2.1,2.3,2.5,2.7,3.0,\) and \(3.4\). The observed absolute statistics are \(3.0,2.5,1.9,\) and \(1.1\). Their adjusted p-values are calculated from the maxima, not from separate independent null distributions.
For the first feature, two maxima are at least \(3.0\), so the adjusted p-value is \(2/12=0.167\). For the second, four maxima are at least \(2.5\), giving \(4/12=0.333\). For the third, seven maxima are at least \(1.9\), giving \(7/12=0.583\). For the fourth, all 12 maxima are at least \(1.1\), giving \(12/12=1.00\). At \(\alpha=0.05\), none is rejected.
This result is partly due to the coarse reference distribution: the smallest attainable adjusted p-value in this example is \(1/12\), which exceeds \(0.05\). A larger number of valid transformations may provide finer resolution, but it cannot make an invalid transformation scheme valid. In an actual analysis, the joint maxima would also preserve whatever dependence the selected permutation design implies.
Diagnostics, Alternatives, and Limitations
Before interpreting a max-T result, check that the permutation scheme preserves the design and the null exchangeability assumptions. Inspect whether restrictions for pairs, blocks, or other design features were applied, and whether the same transformation was used across all features. Verify that the maximum uses the intended family of tests and the intended statistic; for two-sided tests, that means taking absolute values before maximizing. A coding error that maximizes signed statistics, omits a feature, or breaks the joint structure changes the procedure being calibrated.
Also inspect the reference distribution’s resolution and tail. If very few permutations reach the observed maximum, the adjusted p-value is constrained by the number of transformations; it should not be reported with unsupported precision. With Monte Carlo sampling rather than exhaustive enumeration, the estimated tail proportion has simulation uncertainty. The exact equally weighted formula above describes an enumerated reference distribution; Monte Carlo p-value estimation must use an appropriate simulation-based calculation and account for that uncertainty.
Max-T is not the only familywise approach. Bonferroni control bounds the familywise error rate by testing each hypothesis at \(\alpha/m\), without needing the joint distribution, but may be conservative, especially when tests are correlated. A max-T permutation procedure uses the joint null reference and can be less conservative when that reference is valid. Unadjusted tests do not provide familywise control simply because the tests are correlated. The choice among these approaches depends on the design, assumptions, and intended error criterion.
The crucial limitation here is the scope of the guarantee. The derivation concerns the complete null, when every tested null is true. It does not by itself establish control when some hypotheses are false and others true. Thus, a single-step max-T analysis with a valid complete-null reference supports weak familywise control; do not describe that result as strong control over every possible configuration of true and false null hypotheses.
Common Mistakes
- Using the wrong boundary condition. “The probability of exceeding \(c_\alpha\) is at most \(\alpha\)” alone does not ensure equivalence with the adjusted-p-value rule. Use a boundary satisfying \(P_{\mathrm{perm}}(M>c_\alpha)\leq\alpha<P_{\mathrm{perm}}(M\geq c_\alpha)\) and reject only for values strictly greater than it.
- Ignoring ties. Counting only maxima strictly greater than an observed statistic can understate the tail probability. The stated adjusted p-value counts maxima greater than or equal to the observed absolute statistic.
- Claiming independence is required. Max-T uses joint transformations and does not require independent tests. It does require a valid joint null reference that preserves the relevant design structure.
- Overstating the guarantee. Control under the complete null is weak familywise control, not proof of control under every mixture of true and false nulls.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why does max-T calibration not require independent test statistics?
- For a reference distribution with \(P(M>c)=0.04\) and \(P(M\geq c)=0.12\), is \(c\) a suitable boundary at \(\alpha=0.05\)? What does the strict rule do at equality?
- An observed absolute statistic is tied with two of 50 permutation maxima, and three additional maxima are larger. What is its adjusted p-value?
- What error-rate guarantee follows from this procedure under the complete null, and what does that guarantee not establish?
- Why might no feature be rejected at a chosen level even when the observed statistics appear moderately large?
Solutions. (1) Each transformation produces the full vector of test statistics, so the maximum’s reference distribution reflects their joint dependence; independence is not assumed. (2) Yes. The strict upper tail is \(0.04\leq0.05\), while the inclusive tail is \(0.12>0.05\). A statistic equal to \(c\) is not rejected by the strict rule, and its adjusted p-value exceeds \(0.05\). (3) Five maxima are greater than or equal to the observed value, so \(p_{\mathrm{adj}}=5/50=0.10\). (4) The method controls weak familywise error: under the complete null, the probability of any rejection is at most the chosen level, subject to a valid reference distribution. This alone does not establish control when some null hypotheses are false and others are true. (5) The joint null maxima may often be large, or the reference distribution may be coarse with too few transformations to attain a p-value as small as the chosen level.