Why Correlated Tests Need a Joint Reference Distribution
A genomic scan may test many variants on the same participants and against the same phenotype. The resulting test statistics are dependent: nearby variants can carry correlated genotype information, and every test also uses the same observed phenotype. A max-T procedure uses this joint dependence rather than treating the tests as independent. Its central question is: under a specified null hypothesis and a valid resampling scheme, how often would the largest test statistic across the variants be at least as large as the observed statistic for a particular variant?
The key is to resample in a way that is valid under the null while preserving the dependence that exists among tests. In a phenotype-permutation analysis, genotypes remain fixed, and each permutation reassigns the phenotype values among participants. The same reassignment is used for every variant. Permuting separately for each variant would destroy the observed dependence between tests and would not produce the joint reference distribution required by max-T.
This definition describes a two-sided analysis. For a pre-specified one-sided test, use the corresponding signed maximum and tail. The statistics must also be comparable across variants: here they are correlations, which are standardized, bounded association statistics. The same max-T construction applies to other suitable test statistics, including studentized statistics.
Choosing a Permutation Scheme
A permutation is justified by the assignment mechanism or by an exchangeability assumption under the null, not by the fact that it is easy to compute. For example, phenotype values may be permuted across participants under a global null when participants are independent and identically sampled, the phenotype is independent of the full genotype matrix under that null, and no design features require restrictions on reassignment. Conditional on the observed phenotypes and genotypes, this makes all phenotype-to-participant assignments equally likely.
That reasoning fails if participants are related, if ancestry or another covariate affects both genotype and phenotype, or if the study design imposes pairing, blocks, or other assignment constraints. Such settings require an appropriate restricted scheme or another method justified for the design. The earlier material on exchangeability and exact permutation tests, permutation units, restricted permutations, and residual permutation addresses these design-specific choices. For this worked analysis, assume a hypothetical sample of unrelated participants from a homogeneous population, no relevant covariates, and a global null under which phenotype values are exchangeable across participants.
Worked Example: Choosing the Resampling Unit
Suppose a hypothetical study has eight unrelated participants, three correlated variants measured on each participant, and one quantitative phenotype. The study has no matching, family structure, or covariates. The scientific null for this example is that the phenotype is independent of the entire three-variant genotype matrix.
Resampling choice. Keep the eight participants’ genotype vectors fixed and permute the eight phenotype values across participants. Use each resulting assignment for all three variant tests. Under the stated global null and exchangeability assumptions, each of the \(8!\) assignments is equally likely.
Why this preserves correlation. The genotype vectors, including their correlations across variants, are unchanged. Because the same permuted phenotype is used for every test, dependence due to the shared phenotype is preserved as well.
What would invalidate this choice? If two participants were siblings, unrestricted shuffling could assign their phenotypes as if all eight participants were exchangeable, despite dependence within families. If ancestry affected both genotype and phenotype, unrestricted shuffling could break the structure that generated their association. In either case, the stated permutation reference distribution would not be justified without a design-appropriate adjustment.
A Small Correlated-Variant Calculation
The following deliberately small hypothetical data set makes the arithmetic visible. The three columns \(g_1,g_2,g_3\) are centered dosage-like genotype scores for four participants; they are not intended as a realistic genotype sample. Let the phenotype values be \(y=(1,3,2,4)\). Use the sample correlation between each genotype score and the phenotype as \(T_j\), and compare absolute correlations. The genotype scores and centered phenotype are:
The genotype-score means are zero, \(\bar{y}=2.5\), and the centered sums of squares are \(20,20,4\) for the three genotype scores and \(5\) for the phenotype. Thus the correlation denominators are \(10,10,\sqrt{20}\), respectively. The first two genotype scores have correlation \(12/20=0.6\); the third is orthogonal to each of them. This is a small example of correlated tests: the first two tests are related through their genotype patterns, and all three use the same phenotype.
Worked Example: Observed Test Statistics
For each variant, calculate \(T_j=\sum_i g_{ij}(y_i-\bar{y})/\sqrt{\sum_i g_{ij}^2\sum_i(y_i-\bar{y})^2}\). For variant 1, the numerator is \((-3)(-1.5)+(-1)(0.5)+(1)(-0.5)+(3)(1.5)=8\), so \(T_1=8/\sqrt{20\cdot5}=0.800\).
For variant 2, the numerator is \((-1)(-1.5)+(-3)(0.5)+(3)(-0.5)+(1)(1.5)=0\), so \(T_2=0/10=0\). For variant 3, the numerator is \((1)(-1.5)+(-1)(0.5)+(-1)(-0.5)+(1)(1.5)=0\), so \(T_3=0/\sqrt{4\cdot5}=0\). The observed maximum is therefore \(\max_j|T_j^{\mathrm{obs}}|=0.800\).
For a resampled phenotype, recompute all three statistics and take their maximum absolute value. The maximum is not the largest statistic for one fixed variant across independently generated resamples; each row of the joint reference distribution contains all variant statistics from the same assignment.
Worked Example: Monte Carlo Max-T Adjusted p-Values
Suppose a Monte Carlo procedure uniformly samples four distinct phenotype assignments from the remaining permissible assignments, and the resulting maximum statistics are \(0.447\), \(0.800\), \(1.000\), and \(0.894\). These values can be checked by calculating all three correlations for each assignment. For example, the assignment \(y=(1,4,2,3)\) gives correlations \(0.400\), \(-0.400\), and \(-0.447\), so its maximum absolute statistic is \(0.447\). The assignment \(y=(2,4,1,3)\) gives \(0\), \(-0.800\), and \(0\), so its maximum is \(0.800\).
For a Monte Carlo estimate, include the observed assignment in the tail count and divide by \(B+1\), where \(B\) is the number of sampled assignments. Here \(B=4\). Three sampled maxima are at least \(0.800\), so the adjusted p-value for variant 1 is \((1+3)/(4+1)=0.800\). The added 1 accounts for the observed assignment. For variants 2 and 3, the observed absolute statistics are zero; all four maxima, as well as the observed maximum, are at least zero, giving adjusted p-values of \((1+4)/5=1.000\).
This tiny Monte Carlo calculation is illustrative, not precise: with only four draws, its p-value resolution is \(1/5\). In practice, use enough resamples to estimate the relevant tail with useful precision, and report Monte Carlo uncertainty when it matters. For exact enumeration of an equally likely assignment set, the p-value is the fraction of all permitted assignments whose maximum is at least the observed threshold, including the observed assignment. If assignments have unequal probabilities, the exact p-value is instead the sum of the probabilities of all permitted assignments in that inclusive tail.
Why the Maximum Gives an Adjustment
For a given resampled assignment, \(M^{(b)}\geq |T_j^{(b)}|\) for every variant \(j\). Consequently, the maximum-statistic tail is at least as inclusive as the corresponding single-variant tail. This is the source of the multiplicity adjustment: the reference distribution asks how extreme the largest result across the family could be, rather than how extreme one preselected result could be.
The formula is a Monte Carlo estimate under the conditions just described: resampled assignments are drawn according to the null resampling scheme, and the observed assignment is included through the added 1. If the entire assignment distribution is enumerated, use its exact tail probability instead. Under unequal assignment probabilities \(w_a\), where the probabilities sum to one, the exact tail probability at threshold \(c\) is \(\sum_{a:M^{(a)}\geq c}w_a\), not the unweighted fraction of assignments.
Worked Example: Unequal Assignment Probabilities
Consider a hypothetical randomization distribution with three permitted assignments having probabilities \(0.50\), \(0.30\), and \(0.20\), and maximum statistics \(2.4\), \(1.6\), and \(2.1\), respectively. For an observed threshold \(c=2.0\), assignments 1 and 3 are in the inclusive tail. The exact tail probability is \(0.50+0.20=0.70\).
Counting the two tail assignments and dividing by three would give \(2/3\), which is not the design-based p-value because the assignments are not equally likely. The probability-weighted sum, including any observed assignment that falls in the tail, is the correct calculation.
Familywise Error and What It Does—and Does Not—Guarantee
The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in the family. When every tested null is true, the maximum statistic directly calibrates the probability of at least one rejection: rejecting any variant at a common threshold \(c\) is equivalent to the event \(\max_j|T_j|\geq c\). A valid global-null resampling distribution therefore provides weak FWER control at level \(\alpha\): under the global null, the probability of at least one false rejection is at most \(\alpha\), allowing for the discreteness of a finite permutation distribution.
Strong FWER control requires that bound to hold for every configuration of true and false null hypotheses, not just when all nulls are true. A single-step global max-T procedure does not obtain strong control automatically. Its justification requires additional conditions, such as subset pivotality: informally, the joint distribution of statistics for any subset of true nulls must not depend on which hypotheses outside that subset are false. Alternatively, a procedure can calibrate the relevant subsets under assumptions appropriate to the design. The earlier material on strong familywise control and subset pivotality and on Westfall–Young step-down testing develops these distinctions. Do not infer strong control merely because a global-null permutation test has a valid p-value.
A max-T adjusted p-value is not the probability that a particular null is true, nor is it the probability that the observed association arose “by chance.” It is a tail probability under the specified null reference distribution. In the small example, the Monte Carlo adjusted p-value \(0.800\) for variant 1 means that the resampling procedure produced a maximum statistic at least \(0.800\) as often as estimated by the calculation; it does not say there is an 80% probability that the variant has no association.
Diagnostics, Alternatives, and Limitations
Before trusting a result, check that the scheme matches the data-generating design, that every resample repeats the same analysis for all variants, and that the maximum is calculated across the intended family. Confirm that the test statistic is defined in every resample; for example, correlation is undefined if a resampled phenotype has zero variance. Check the number of distinct permissible assignments and whether Monte Carlo sampling adequately explores the relevant tail. The earlier material on Monte Carlo permutation p-values and their uncertainty explains why a small number of exceedances makes a tail estimate imprecise.
Common warning signs include family or ancestry structure ignored by unrestricted shuffling, unequal treatment or assignment probabilities handled by unweighted counting, and variant-specific permutations that fail to preserve dependence. A large observed correlation at one variant is not itself evidence that the familywise calibration is sound. Diagnostics can identify an implausible scheme, but they cannot prove exchangeability when the study design or scientific knowledge contradicts it.
| Approach | What it uses | Main consideration |
|---|---|---|
| Single-step max-T | The maximum statistic over the full tested family | Uses joint dependence; strong control needs appropriate additional conditions |
| Bonferroni adjustment | The number of tests and single-test p-values | Does not need the tests to be independent, but can be conservative |
| Independent-test approximation | A model treating test statistics as independent | Can miscalibrate when correlated variants produce dependent tests |
Max-T gains information from the joint resampling distribution, but that is also its limitation: a wrong resampling model leads to a wrong reference distribution. It can be computationally intensive for large families and many resamples, and finite permutation distributions have discrete p-values. Its adjustment controls false rejections under its stated error-rate conditions; it does not correct confounding, repair poor phenotype measurement, or establish that an association is causal.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Why must the same phenotype permutation be used for all variants in a max-T resample?
- A maximum-statistic resample has \(M=1.25\), and a variant’s observed absolute statistic is \(1.10\). Does this resample enter that variant’s adjusted-p-value tail?
- In an exact distribution, three assignments have probabilities \(0.20\), \(0.50\), and \(0.30\), with maximum statistics \(1.4\), \(0.8\), and \(1.7\). Find the tail probability at threshold \(1.2\).
- What type of FWER control follows from valid calibration under the global null alone?
Solutions. (1) It preserves dependence among the variant statistics, including their shared use of the phenotype. Independent permutations by variant would not represent their joint null distribution. (2) Yes, because \(1.25\geq1.10\), so the resample is in the inclusive tail. (3) The first and third assignments meet the threshold, giving \(0.20+0.30=0.50\); do not count assignments equally because their probabilities differ. (4) Weak FWER control: the probability of at least one false rejection is controlled when all tested nulls are true. Strong control requires further justification.