Why a Pairwise Interaction Search Needs Multiplicity Control
A test of one prespecified variant pair asks whether that pair shows evidence of interaction under a specified model. A search across many pairs asks a different inferential question: how should evidence be interpreted after many opportunities to find a small p-value? Even if every null hypothesis is true and each test is individually calibrated, a large collection of tests can produce one or more small p-values by chance. The number of opportunities grows quickly with the marker set, so the testing family must be defined before interpreting results.
For pairwise interaction scans, tests are rarely independent. Variants in linkage disequilibrium can make their main effects and interaction statistics correlated, and two pairs that share a variant can also have related test statistics. Multiplicity procedures differ in what error rate they control and in what assumptions they make about this dependence. The appropriate procedure depends on the scientific aim: limiting any false discovery, or controlling the expected fraction of false discoveries among reported findings.
Counting the Pairwise Tests
The count assumes that each pair contributes exactly one test, such as one prespecified omnibus test of the interaction terms for that pair. If a pair is tested under several trait definitions, genetic codings, phenotypes, or interaction contrasts, those additional tests also belong in the multiplicity family if their results are searched and reported. Conversely, if only a defined subset of pairs is tested, count that actual family rather than all possible pairs. The family should match the set of results from which findings may be selected.
Worked Example: Counting Tests in a Marker Set
A hypothetical scan includes \(K=1{,}000\) variants and performs one omnibus interaction test for every unordered pair. The number of tests is
Thus, the scan contains 499,500 tests, not 999,000: each pair is counted only once. If the analysis instead tested two traits for every pair, the family would contain \(2(499{,}500)=999{,}000\) tests, assuming both sets of tests were used to select findings. A marker pair omitted in advance would not contribute a test, but omitting it after inspecting its result would not reduce the relevant search family.
Family-Wise Error Control
The family-wise error rate (FWER) is the probability of making at least one false rejection in a specified family of tests. For interaction scans, a false rejection is a rejected null interaction hypothesis that is in fact true. If the goal is to make a strong claim that no reported pair is a false positive, FWER control is a natural target.
The simplest procedure is Bonferroni: reject a hypothesis only when its p-value is at most \(\alpha/m\). Equivalently, multiply each p-value by \(m\), capping the adjusted value at 1. This procedure does not require independent tests. If each true-null p-value is valid, then for the set of true nulls,
The first inequality is the union bound: the chance that at least one event occurs is no larger than the sum of their individual chances. Bonferroni can be conservative, particularly when tests are strongly correlated, but dependence does not invalidate the bound. Holm’s step-down procedure also controls FWER under arbitrary dependence and is never less powerful than the single-step Bonferroni procedure. It orders the p-values \(p_{(1)}\leq\cdots\leq p_{(m)}\), then compares them in order with \(\alpha/m,\alpha/(m-1),\ldots,\alpha\), stopping at the first failure to reject.
Worked Example: Bonferroni and Holm Decisions
Suppose six hypothetical variant-pair interaction tests have ordered p-values \(0.001,\ 0.006,\ 0.012,\ 0.040,\ 0.200,\ 0.600\), and the target FWER is \(\alpha=0.05\). With Bonferroni, the common cutoff is
Only \(0.001\) and \(0.006\) are below 0.00833, so Bonferroni rejects the first two hypotheses. Under Holm, the ordered p-values are compared successively with \(0.05/6=0.00833\), \(0.05/5=0.0100\), and \(0.05/4=0.0125\). The first three comparisons pass: \(0.001\leq0.00833\), \(0.006\leq0.0100\), and \(0.012\leq0.0125\). The fourth comparison fails because \(0.040>0.05/3=0.0167\), so Holm rejects the first three and stops. At the stated level, Holm provides FWER control while retaining one additional rejection for these p-values.
False-Discovery-Rate Control
In a large exploratory scan, the objective may instead be to limit the expected proportion of false rejections among all rejected hypotheses. The false-discovery rate (FDR) is
where \(R\) is the total number of rejections. If there are no rejections, the fraction is defined as zero. FDR control does not promise that the chance of any false positive is at most the target level; it controls an expected proportion. This can be a useful trade-off when a scan aims to produce a list of candidates for follow-up rather than a set of individually definitive claims.
The Benjamini–Hochberg (BH) procedure at target \(q\) orders the p-values and finds the largest rank \(i\) such that \(p_{(i)}\leq iq/m\). It rejects the hypotheses ranked 1 through \(i\). BH controls FDR at level \(q\) when tests are independent and under certain positive dependence conditions, commonly stated as positive regression dependence on a subset of true nulls (PRDS). It is not a general guarantee for arbitrary dependence. A more conservative Benjamini–Yekutieli adjustment uses thresholds \(iq/(mH_m)\), where \(H_m=\sum_{j=1}^{m}1/j\), and controls FDR under arbitrary dependence for valid marginal p-values.
Worked Example: Applying Benjamini–Hochberg
Use the six ordered p-values from the preceding example, now with an FDR target \(q=0.05\). The BH cutoffs \(iq/m\) for ranks 1 through 6 are approximately \(0.00833,\ 0.0167,\ 0.0250,\ 0.0333,\ 0.0417,\ 0.0500\). At rank 3, \(p_{(3)}=0.012\leq0.0250\). At rank 4, however, \(p_{(4)}=0.040>0.0333\); ranks 5 and 6 also fail their comparisons. Therefore the largest passing rank is 3, and BH rejects the first three hypotheses.
This decision means the procedure controls the expected false-discovery proportion at the target under its dependence conditions. It does not mean that each of these three findings has a 95% probability of being true, nor that the probability of at least one false finding is 5%. In an interaction scan, BH’s guarantee should be qualified if the dependence structure does not meet its stated conditions; linkage disequilibrium alone is not a proof that a particular dependence condition holds.
Permutation Calibration for Correlated Tests
Permutation calibration can use the joint behavior of interaction statistics to account for their dependence. Instead of adjusting each test using only its marginal null distribution, a maximum-statistic procedure calibrates the largest statistic in the entire scan. Because each permutation repeats all tests together, correlations induced by shared markers and linkage disequilibrium can be reflected in the distribution of that maximum.
A permutation is valid only if the rearrangement is justified by the null model. For example, shuffling outcomes across all participants is appropriate only when outcomes are exchangeable under the null in that scheme. Covariates, relatedness, ancestry, or other structure may invalidate unrestricted shuffling. One must choose a design-appropriate permutation or other null-generating procedure, preserve relevant structure, and rerun the same complete set of tests for every replicate. Keeping the genotypes fixed while permuting outcomes preserves the observed linkage disequilibrium among markers.
Define the variant pairs, the interaction test for each pair, and whether larger values of the chosen statistic indicate stronger evidence.
Use a permutation scheme justified by exchangeability under the null, including any required strata or covariate handling.
Calculate the statistic for every pair and record the largest statistic from that replicate.
Compare each observed statistic with the replicate distribution of the maximum to obtain a family-wise adjusted p-value.
With \(B\) null replicates, an adjusted permutation p-value for observed statistic \(T_j\) can be estimated as \((1+b_j)/(B+1)\), where \(b_j\) is the number of replicate maximum statistics at least as large as \(T_j\). The added 1 prevents a reported p-value of zero and gives the usual finite-simulation correction. The statistic and tail direction must be consistent between observed data and replicates.
Worked Example: A Maximum-Statistic Permutation P-Value
Suppose a hypothetical scan has 100 pairs. Its analysis uses a permutation scheme justified by exchangeability of the outcome within prespecified blocks under the global null; genotype data remain fixed. For each of \(B=9{,}999\) replicates, the full scan is rerun and the largest interaction statistic is recorded. For one observed pair, its statistic is at least as large as the replicate maximum in \(b_j=173\) replicates. Its family-wise adjusted p-value is
At \(\alpha=0.05\), this pair passes the maximum-statistic family-wise threshold. The interpretation is that, under the specified null-generating scheme, a scan-wide maximum at least this large occurred in an estimated 1.74% of replicates. The result is not evidence that the interaction is causal, and the validity of the calibration depends on the stated exchangeability assumption and the adequacy of the permutation design.
A maximum-statistic procedure controls FWER under the complete global null when the permutation distribution is valid for that null. Strong FWER control, which applies when some interaction nulls are false and others true, requires additional conditions; a common sufficient condition for maxT procedures is subset pivotality, meaning that the joint distribution of statistics for any subset of true null hypotheses does not depend on which other hypotheses are false. Without such a condition or a suitable null-specific method, do not claim strong control solely because a global-null permutation was performed. Permutation calibration can also be computationally demanding, and its p-value resolution is limited by the number of replicates.
Choosing and Reporting a Procedure
| Procedure | Target | Dependence and interpretation |
|---|---|---|
| Bonferroni | FWER | Valid for arbitrary dependence with valid marginal p-values; can be conservative. |
| Holm | FWER | Valid for arbitrary dependence with valid marginal p-values; step-down and at least as powerful as Bonferroni. |
| Benjamini–Hochberg | FDR | Controls FDR under independence and certain positive dependence conditions. |
| Benjamini–Yekutieli | FDR | Controls FDR under arbitrary dependence, generally with more conservative thresholds. |
| Permutation max statistic | Typically FWER | Uses joint null behavior; requires a valid null-generating scheme, and strong control needs additional justification. |
For a confirmatory claim about a small number of pairs, an FWER procedure is often aligned with the goal of limiting any false rejection. For a broad discovery list, FDR may be a more suitable target, provided its dependence assumptions are credible or an appropriate dependence-robust procedure is used. A permutation maximum can exploit the correlation structure, but it is not automatically valid merely because permutations were computed.
Common Pitfalls
- Counting ordered pairs twice. If each unordered pair is tested once, \(K(K-1)\) is not the test count; divide by 2 to obtain \(\binom{K}{2}\).
- Assuming LD makes Bonferroni invalid. Correlation can make Bonferroni conservative, but the union-bound guarantee does not require independent tests.
- Calling FDR control a guarantee of no false positives. FDR limits an expected proportion, whereas FWER concerns the probability of at least one false rejection.
- Permuting without checking exchangeability. A permutation distribution is only meaningful when the chosen rearrangement is justified by the null model and study structure.
- Reporting only the nominal p-value. State the family size, multiplicity target, procedure, and adjusted result so readers can understand the evidence after the search.
Check Your Understanding
Answer these questions using the definitions and procedures in this tutorial.
- How many unordered pairwise tests are there among 250 variants if exactly one interaction test is run for every pair?
- State the difference between FWER and FDR in terms of false rejections.
- Why can Bonferroni control FWER even when interaction-test statistics are correlated?
- What dependence conditions support the usual Benjamini–Hochberg FDR guarantee, and what adjustment is designed for arbitrary dependence?
- Why must the permutation scheme be justified under the null, and what does a maximum-statistic permutation procedure record for each replicate?
- What additional condition may be needed to extend global-null maxT control to strong FWER control?