Constructing a Max-T Reference Distribution

Why Construct a Joint Reference Distribution?

A genomic analysis may test thousands of hypotheses on the same samples. Even when each test statistic has a valid null distribution on its own, the statistics are usually dependent: features are measured on shared individuals, and correlated biological signals can produce correlated test results. A multiple-testing reference distribution should therefore represent the behavior of the whole family of statistics together, rather than treating each feature as if it had been permuted independently of the others.

The max-T construction does this by applying each allowed permutation to all features at once, calculating the statistic for each feature, and retaining the largest absolute statistic from that permutation. Repeating this produces a reference distribution for the family’s most extreme result. A feature’s single-step adjusted p-value is the proportion of this reference distribution at least as extreme as that feature’s observed statistic.

Definition and Assumptions

Definition: Suppose a fixed family contains $m$ genomic hypotheses, with observed test statistics $t_1,\ldots,t_m$. For a permitted permutation $b$, let $T_j^{(b)}$ be the test statistic for feature $j$, calculated using that same permutation. The permutation-specific maximum absolute statistic is $$ M^{(b)}=\max_{1\leq j\leq m}|T_j^{(b)}|. $$ The collection of these maxima over the permitted permutations, or over random draws from them, is the max-T reference distribution. The single-step max-T adjusted p-value for hypothesis $j$ is the reference tail probability at its observed threshold $|t_j|$.

The absolute value makes this a two-sided construction: large positive and large negative statistics both count as extreme. For a one-sided family, the maximum must instead be defined using the direction specified by the hypotheses; the two-sided formula should not be used automatically.

The reference distribution is meaningful only if the permutation scheme is justified for the design and null being tested. The earlier material on exchangeability and exact permutation tests establishes the key principle: under an appropriate null, the permitted transformations must leave the joint data arrangement invariant. Restricted designs require restricted permutations; paired or blocked observations must remain paired or within their blocks. When nuisance covariates are present, the chosen residual-permutation procedure must also be appropriate to the model and error structure.

In addition, the family of features and the statistic calculation should be fixed across permutations. Each permutation must be applied to the same observational units across all features, and every statistic must be recomputed by the same analysis rule used for the observed data. These conditions make each permuted vector a plausible joint null realization. They do not follow merely from having many permutations.

From Permutations to Adjusted Tail Probabilities

If the permitted randomizations can be enumerated, let $\mathcal{G}$ denote their set and let $G=|\mathcal{G}|$. For each $g\in\mathcal{G}$, compute the entire vector of statistics and then its maximum $M^{(g)}$. The exhaustive-reference adjusted p-value is

\hat p_{j,\mathrm{adj}} = \frac{1}{G}\sum_{g\in\mathcal{G}} I\!\left(M^{(g)}\geq |t_j|\right).

Here $I(\cdot)$ is 1 when its condition is true and 0 otherwise. The comparison is “greater than or equal to,” so ties at the observed threshold are counted as at least as extreme. This convention is important for discrete permutation distributions and avoids making a tied statistic look more unusual than it is.

When exhaustive enumeration is too large, draw $B$ random permitted permutations and calculate $M^{(1)},\ldots,M^{(B)}$. If the observed assignment is not among those draws and the randomization procedure justifies the usual add-one Monte Carlo rank calculation, use

\hat p_{j,\mathrm{adj}} = \frac{1+\sum_{b=1}^{B}I\!\left(M^{(b)}\geq |t_j|\right)} {B+1}.

The added 1 represents the observed assignment’s place in the comparison. This calculation gives a finite Monte Carlo p-value rather than a zero estimate when no sampled maximum exceeds the observed threshold. The number of draws affects its precision; the earlier material on Monte Carlo permutation p-values discusses uncertainty from finite sampling.

Worked Examples

Worked Example: Exact Max-T Adjustment for Three Features

Hypothetical data. Suppose three genomic hypotheses are tested using an exact set of four permitted assignments, including the observed assignment. The observed statistics are $t_1=2.4$, $t_2=-1.8$, and $t_3=0.9$. The statistic vectors from the four assignments are shown below. These small numbers are chosen only to make the calculation transparent.

Assignment	$T_1$	$T_2$	$T_3$	$M=\max_j\|T_j\|$
1 (observed)	2.4	-1.8	0.9	2.4
2	1.1	2.2	0.4	2.2
3	-0.7	1.3	-2.6	2.6
4	1.8	-0.5	1.0	1.8

For assignment 1, for example, the maximum is $\max(2.4,1.8,0.9)=2.4$. The four maxima are therefore $2.4,2.2,2.6,1.8$.

For feature 1, the observed absolute statistic is $|t_1|=2.4$. Two maxima meet or exceed 2.4, so $\hat p_{1,\mathrm{adj}}=2/4=0.50$. For feature 2, $|t_2|=1.8$, and all four maxima meet or exceed 1.8, giving $\hat p_{2,\mathrm{adj}}=4/4=1.00$. For feature 3, $|t_3|=0.9$, and again all four maxima meet or exceed the threshold, giving $\hat p_{3,\mathrm{adj}}=4/4=1.00$.

The adjusted tail area uses the family maximum, not just the statistic for the feature being assessed. For comparison, the feature-1 marginal reference counts only its own values: among $2.4,1.1,-0.7,1.8$, just one absolute value is at least 2.4, giving $1/4$. The max-T tail is larger here because another feature’s statistic can make a permutation maximum extreme.

Worked Example: Monte Carlo Max-T Tail Calculation

Hypothetical data. A family contains 500 genomic hypotheses. The observed statistic for feature $j$ is $t_j=-2.10$, and $B=199$ random permitted permutations are drawn. All 500 statistics are recomputed for each draw, producing 199 maxima. Seven of these maxima are at least $2.10$.

Using the add-one calculation,

\hat p_{j,\mathrm{adj}} = \frac{1+7}{199+1} = \frac{8}{200} = 0.040.

The threshold is $|t_j|=2.10$, not $-2.10$, because this example uses a two-sided maximum of absolute statistics. The result is a Monte Carlo estimate of the reference tail probability, based on the stated permutation scheme and 199 draws. It is not a probability that the feature’s null hypothesis is true.

Worked Example: Dependence Changes the Joint Tail

Hypothetical data. Consider two features and four exact joint null statistic vectors, represented here by their absolute values: $(0.5,0.4)$, $(1.2,1.1)$, $(2.0,0.3)$, and $(0.8,1.7)$. Suppose the observed statistic for feature 1 is $t_1=1.2$, which appears in the second vector. The four maxima are $0.5,1.2,2.0,1.7$.

Three of the four maxima are at least $|t_1|=1.2$, so the feature-1 adjusted p-value is $3/4=0.75$. In the marginal reference for feature 1, two of four absolute statistics are at least 1.2, giving a marginal tail proportion of $2/4=0.50$. For feature 2, one of four is at least 1.2, giving $1/4=0.25$.

If one incorrectly treated those two marginal exceedance events as independent, the estimated chance that at least one exceeds 1.2 would be $1-(1-0.50)(1-0.25)=0.625$. But the actual joint reference gives $3/4=0.75$. The difference arises from the arrangement of the two statistics across the same four vectors. The maximum distribution directly uses that joint arrangement, without requiring the tests to be independent.

Why the Joint Construction Matters

For each permutation, $M^{(b)}$ records how extreme the most extreme feature would be under that joint null realization. Comparing $|t_j|$ with this distribution asks how often a null realization produces some feature at least as extreme as feature $j$’s observed statistic. This produces single-step adjusted tail probabilities on a common reference scale. With the same reference assignments and inclusive tie rule, the maximum tail count cannot be smaller than the corresponding feature-specific tail count, since $M^{(b)}\geq |T_j^{(b)}|$ for every $b$.

This construction is particularly relevant when genomic tests are dependent. A separate set of permutations for each feature would not preserve the observed cross-feature dependence and would not produce the joint maximum distribution defined here. Likewise, independently combining marginal tail probabilities substitutes an assumption about joint behavior for the joint behavior represented by the data and design.

There are alternatives. Bonferroni adjustment uses the union bound: for $m$ valid marginal p-values, it compares each with $\alpha/m$, without needing independence, but it does not estimate the joint maximum distribution and can be conservative. A Šidák-style calculation uses the form $1-(1-p)^m$, but its usual exact interpretation relies on independent tests with suitable common marginal behavior. Max-T instead uses the permitted joint reference distribution. Its validity depends on that reference being appropriate; the name of the adjustment does not repair an invalid permutation scheme.

Diagnostics, Pitfalls, and Limitations

Before interpreting the adjusted values, inspect how the reference was constructed. Check that the permutation unit matches the design, any pairing or blocking restrictions are retained, the same permutation is used across all features, and the statistic is recalculated consistently. Verify that each recorded maximum equals the largest absolute statistic in its permutation vector. For Monte Carlo sampling, report $B$ and the tail-count calculation so the finite reference is transparent.

Common pitfall: Permuting each gene independently and then taking the maximum is not the same as applying a shared permitted permutation to all genes. Independent permutations scramble the cross-feature relationships, so the resulting maxima need not represent the joint null distribution for the genomic family. Use one design-valid transformation per replicate across all features, then calculate the maximum.

A second error is to omit absolute values while describing a two-sided analysis. That would make large negative statistics disappear from the maximum, changing the reference distribution. Another is to use $>$ rather than $\geq$ in a discrete reference with ties; the resulting tail count can be too small relative to the stated inclusive-tail convention.

The construction also has limits. Exact enumeration may be computationally infeasible, while Monte Carlo sampling introduces uncertainty. The choice of statistic affects which features tend to dominate the maximum, and the permutation scheme must still justify the null reference for the question being asked. The tail probability itself is not a posterior probability, does not measure the size of a biological effect, and by itself does not establish the conditions under which a particular familywise error guarantee holds.

Fix the family and statistic.
Specify the hypotheses, the two-sided or one-sided convention, and the statistic calculation before generating the reference.

Generate a valid joint null replicate.
Apply one permitted assignment or transformation to the relevant observational units across all features, respecting the design.

Retain the family maximum.
Calculate every feature statistic and record the maximum absolute statistic for that replicate.

Count tail exceedances.
For each observed feature statistic, count reference maxima at least as large as its absolute value and use the exact or Monte Carlo denominator appropriate to the construction.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In one permutation replicate, three feature statistics are $-1.4$, $0.8$, and $2.1$. What maximum is recorded for a two-sided max-T reference?
Among 99 random permitted permutations, 4 maxima are at least as large as an observed absolute statistic. What is the add-one Monte Carlo adjusted p-value?
Why should the same permitted permutation be applied across all features in a replicate?
A two-feature exact reference has marginal exceedance proportions 0.40 and 0.30 at a chosen threshold. Can the joint maximum exceedance proportion be determined from these two numbers alone? Explain.

Solutions. (1) The maximum is $\max(|-1.4|,|0.8|,|2.1|)=2.1$. (2) The add-one value is $(1+4)/(99+1)=5/100=0.050$. (3) A shared permutation retains the joint arrangement of feature statistics generated by the design; independently permuting features can destroy their dependence and alter the maximum distribution. (4) No. The marginal proportions do not specify how often the two features exceed together. The joint maximum proportion requires the joint reference, because its event is that at least one feature exceeds the threshold.