MaxT Permutation Tests for Genome-Wide Error

Why Calibrate the Largest Statistic?

A genome-wide association analysis may test hundreds of thousands or millions of variants. A small unadjusted p-value at one variant does not by itself show that the result is unusual after searching across the entire testing family. The relevant error rate is often the family-wise error rate (FWER): the probability of making at least one false rejection among the hypotheses in the family.

Bonferroni control uses the number of tests to set a threshold without needing to model their dependence. A maxT permutation procedure takes a different approach: it repeatedly generates data under a null arrangement, calculates a test statistic at every variant, and saves the largest statistic from each repetition. This distribution reflects the joint behavior of the tests, including the dependence induced by linkage disequilibrium, when the permutation scheme preserves that dependence.

Definition: Suppose a family contains \(m\) variant-level hypotheses, with observed test statistics \(T_1,\ldots,T_m\). For a two-sided analysis, define \(U_j=|T_j|\), and define the observed maximum as \(M_{\mathrm{obs}}=\max_{1\leq j\leq m} U_j\). In permutation replicate \(b\), calculate the corresponding statistic at every variant and save \(M_b=\max_{1\leq j\leq m}|T_{jb}|\). The collection of \(M_b\) values is the permutation reference distribution for the largest statistic.

The absolute value is appropriate for two-sided tests when large departures in either direction count as evidence against a null hypothesis. For a one-sided analysis, use the statistic in the prespecified direction rather than its absolute value. If test statistics are not on a comparable scale, the procedure must use statistics calibrated to a common null scale, or otherwise define a suitable maximum; taking the maximum of incomparable raw statistics is not meaningful.

Constructing the Permutation Distribution

A maxT analysis begins with a defined family of variants and a specified association statistic at each one. The observed data produce \(T_1,\ldots,T_m\). For each permutation, the analyst rearranges the data in a way that is valid under the null hypothesis, recalculates all \(m\) statistics, and records just their maximum. In a simple setting with exchangeable phenotype observations under the null, the phenotype labels are permuted while the genotype data are held fixed.

The same permuted phenotype arrangement must be used across all variants in a replicate. The genotype data are also kept together: this retains the observed relationships among variants, including linkage disequilibrium. Permuting each variant or each genotype column independently would change that joint structure and would no longer reproduce the dependence relevant to the observed genome-wide maximum.

The validity of the rearrangement is an assumption, not a mechanical consequence of running permutations. Under the null, the original and permuted arrangements must have the appropriate exchangeability or other justified randomization property. For example, simply shuffling phenotypes without regard to covariates, family structure, or other design features can be invalid when those features affect exchangeability. The permutation scheme must match the analysis and study design.

Specify the family and statistic.
List the \(m\) variants to be tested and define the statistic and tail direction at every variant before examining the permutation results.

Calculate observed statistics.
Analyze the original data to obtain \(T_j\) for each variant and, for two-sided tests, set \(U_j=|T_j|\).

Generate each null replicate.
Apply one valid permutation to the data arrangement, repeat the association analysis at all variants, and record the maximum statistic \(M_b\).

Compare with the maximum distribution.
Use the permutation maxima to calculate adjusted p-values or a family-wide rejection threshold, and interpret the result for the full testing family.

In a small teaching example, a few permutations make the arithmetic visible. In an actual genome-wide analysis, the number of permutations must be large enough to resolve the tail probability of interest. With only a few permutations, the smallest attainable adjusted p-value is too large to assess stringent genome-wide thresholds.

Single-Step maxT Adjusted p-Values

For a variant with observed two-sided statistic \(U_j=|T_j|\), count how often a permutation maximum is at least as large. Because every \(M_b\) is the largest statistic across the entire family, this comparison asks how often a null replicate produces an extreme result anywhere in the family at least as large as the result at variant \(j\).

\widehat{p}_{j,\mathrm{maxT}} = \frac{1+\sum_{b=1}^{B} I(M_b\geq U_j)}{B+1},

Here \(B\) is the number of random permutations, and \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. The added 1 accounts for the observed arrangement in the Monte Carlo reference set; it prevents a reported p-value of zero and gives the usual finite-permutation correction. Ties count as at least as extreme, making the comparison conservative when the statistic is discrete. A maxT adjusted p-value is a family-wise, not a single-variant unadjusted, measure of evidence.

Worked Example: Calculating an Adjusted p-Value

Suppose a hypothetical study tests a family of variants using two-sided statistics. At one variant, the observed statistic is \(T_j=-3.1\), so \(U_j=3.1\). Nine valid phenotype permutations produce these family-wide maxima: 1.2, 2.0, 2.7, 3.4, 2.9, 3.1, 1.8, 3.6, and 2.5.

There are three maxima at least as large as 3.1: 3.4, 3.1, and 3.6. Thus \(B=9\) and the exceedance count is 3. Substitution into the Monte Carlo formula gives

\widehat{p}_{j,\mathrm{maxT}} = \frac{1+3}{9+1} = \frac{4}{10} = 0.400.

The maxT-adjusted p-value is 0.400 in this deliberately small example. It is not the probability that the null hypothesis is true. It estimates how often the null permutation reference produces a family maximum at least as extreme as this variant's observed statistic. Nine permutations are far too few for a precise genome-wide tail assessment.

The adjusted p-values are ordered consistently with the observed statistics: a larger observed \(U_j\) cannot have more permutation maxima exceeding it than a smaller \(U_j\). The single-step procedure compares every variant with the same family-wide maximum distribution. It does not require testing variants in a stepwise order.

Choosing a Genome-Wide Threshold

Instead of reporting an adjusted p-value for each variant, an analyst can choose a critical value \(c_\alpha\) from the permutation maxima and reject a variant's null hypothesis when its observed statistic exceeds that value. Under the complete global null, every hypothesis in the family is true, so any rejection is a false rejection. The family-wise error event is then exactly that at least one statistic exceeds the critical value.

\{\text{at least one rejection}\} = \left\{\max_{1\leq j\leq m} U_j \geq c_\alpha\right\} = \{M_{\mathrm{obs}}\geq c_\alpha\}.

Therefore, if the critical value is chosen so that the null probability of exceeding it is at most \(\alpha\), the probability of any false rejection under the global null is at most \(\alpha\). In practice, the critical value comes from an upper-tail quantile of the permutation maxima. The exact quantile convention must account for the finite number of permutations and ties; it should agree with the comparison rule used for adjusted p-values.

Worked Example: Comparing an Observed Statistic With the Permutation Tail

In a hypothetical analysis, \(B=99\) valid permutations yield family-wide maxima. For one variant the observed two-sided statistic has magnitude \(U_j=3.05\), and exactly four of the 99 maxima are at least 3.05. The Monte Carlo adjusted p-value is

\widehat{p}_{j,\mathrm{maxT}} = \frac{1+4}{99+1} = \frac{5}{100} = 0.0500.

At a prespecified family-wise level \(\alpha=0.05\), this result is at the rejection boundary under the stated \(\leq\alpha\) rule. The conclusion is that this variant meets the maxT-adjusted threshold in this example, subject to the validity of the permutation scheme. It would not be correct to describe 0.0500 as an unadjusted single-variant p-value.

Equivalently, one can sort the permutation maxima and use an upper-tail cutoff calibrated to \(\alpha\). With discrete permutation distributions, the attainable tail probabilities occur in increments, and ties can make a cutoff conservative. Reporting the adjusted p-value directly makes the finite-permutation comparison explicit.

The maximum distribution incorporates the multiplicity cost of searching across the family. If variants are strongly dependent, the maximum may behave differently from the maximum under independent tests. This is why a valid joint permutation distribution can be less conservative than a method that ignores dependence, though the amount of gain depends on the data and testing family.

Why the Joint Structure Matters

Worked Example: One Shared Permutation Across Variants

Consider a hypothetical family of four variants with observed absolute statistics 2.9, 2.4, 1.7, and 1.1. Five valid shared phenotype permutations produce maxima of 2.2, 3.1, 2.7, 1.9, and 3.0. For the first variant, two maxima (3.1 and 3.0) are at least 2.9, giving

\widehat{p}_{1,\mathrm{maxT}} = \frac{1+2}{5+1} = 0.500.

For the second variant, three maxima (3.1, 2.7, and 3.0) are at least 2.4, so

\widehat{p}_{2,\mathrm{maxT}} = \frac{1+3}{5+1} = 0.667.

The maximum in each replicate was calculated from all four variant statistics generated under the same permutation. That preserves the joint genotype structure while assessing how extreme each observed variant is relative to the family-wide null behavior. The five-permutation calculation is only illustrative; its coarse p-value resolution would not support a stringent genome-wide claim.

A key distinction is between weak and strong FWER control. Calibration using the complete global null directly targets weak control: the chance of at least one rejection when every tested null is true. Strong control requires the FWER to be at most \(\alpha\) for every configuration of true and false null hypotheses. A standard sufficient condition for using a complete-null maxT reference distribution to obtain strong control is subset pivotality: the joint distribution of the statistics for any subset of true null hypotheses does not depend on the states of the hypotheses outside that subset. The permutation scheme must also correctly represent the relevant null distributions. These requirements should not be assumed automatically in every association analysis.

Derivation of the Family-Wide Guarantee

The core justification is a direct property of the maximum. Under the complete global null, let \(M=\max_j U_j\) have the null distribution represented by the valid permutation procedure. Rejecting when \(M\) reaches or exceeds \(c_\alpha\) makes the event of one or more rejections identical to the event that this maximum reaches or exceeds the cutoff. Thus, if the cutoff is selected so that \(P(M\geq c_\alpha)\leq\alpha\), then

\mathrm{FWER} = P(\text{at least one false rejection}) = P(M\geq c_\alpha) \leq \alpha.

For strong control under subset pivotality, consider any configuration of true and false hypotheses and let \(S\) be the set of true nulls. A false rejection can occur only if at least one statistic indexed by \(S\) crosses the cutoff. Therefore,

P(\text{at least one false rejection}) = P\left(\max_{j\in S}U_j\geq c_\alpha\right) \leq P\left(\max_{1\leq j\leq m}U_j^{(0)}\geq c_\alpha\right) \leq \alpha,

where \(U_j^{(0)}\) denotes the corresponding complete-null reference statistics. The first inequality uses the subset-pivotality condition to justify calibrating the true-null subset by the complete-null reference and the fact that the maximum over a subset cannot exceed the maximum over the full family. Without the conditions supporting that calibration, the displayed strong-control guarantee does not follow merely from having performed permutations.

Common Pitfalls and Practical Interpretation

Permuting each variant separately. This destroys the joint pattern of association statistics across variants. Apply the same valid data permutation across the entire family in each replicate.
Calling every permutation maxT result strongly controlling. Complete-null calibration provides a global-null guarantee; strong control needs additional justification, such as subset pivotality.
Using an invalid rearrangement. Permutation does not repair confounding or a mismatch between the null model and the study design. The data arrangement must be exchangeable under the null for the chosen procedure.
Reporting zero because no permutation exceeded the statistic. With \(B\) random permutations, use the finite-permutation correction; if the exceedance count is zero, the estimate is \(1/(B+1)\), not zero.
Changing the family after seeing results. The tested variants and the statistic should define the family before interpreting the maximum distribution. Changing the family changes the multiplicity adjustment.

MaxT is useful when the goal is to control the probability of at least one false discovery in a prespecified family and when a defensible joint null permutation scheme is available. The adjusted p-value answers a family-aware question: how often would a null permutation produce a largest statistic at least as extreme as the observed statistic at this variant? It does not establish a biological mechanism, measure the probability that an association is true, or replace checks of model and permutation assumptions.

Key takeaway: A single-step maxT procedure compares each observed variant statistic with the permutation distribution of the largest statistic across all variants. Its family-wide error guarantee depends on valid null permutations; strong FWER control requires additional conditions beyond global-null calibration.

Check Your Understanding

Work through these questions using the definitions and conditions in this tutorial.

For a two-sided maxT test, what quantity is saved from each permutation, and why is one shared permutation used across variants?
With \(B=499\) permutations and 7 maxima at least as large as an observed variant statistic, calculate the Monte Carlo maxT-adjusted p-value.
Explain why a critical value controlling the probability of any rejection under the complete global null gives weak FWER control.
What additional condition can justify strong FWER control using a complete-null maxT reference distribution?
Why might shuffling phenotype labels without accounting for study design or covariates give an invalid permutation reference distribution?