Permutation minP Control and Adjusted P Values

Why calibrate the smallest p-value?

In a genomic analysis, many hypotheses may be tested on the same individuals. Their test statistics are often dependent: genes can be co-expressed, variants can be in linkage disequilibrium, and tests can share covariates. A single-step permutation procedure can use this dependence rather than replacing it with an independence assumption. The earlier material on single-step permutation maxT uses the largest test statistic as its joint extremeness measure. Here, the corresponding measure is the smallest marginal p-value, called minP.

The idea is to compare each observed marginal p-value with the most extreme p-value that appears across the entire family in a joint permutation. A small observed p-value is compelling only to the extent that it is unusual relative to this family-wide minimum distribution. The resulting values are adjusted for the fact that the family contains multiple tests.

Definition: For a family of \(m\) hypotheses, let \(p_j\) be the observed marginal p-value for \(H_j\), \(j=1,\ldots,m\). For permutation replicate \(b\), calculate the marginal p-value \(p_j^{(b)}\) for every test using the same permuted data assignment. The minP statistic for that replicate is \(M^{(b)}=\min_{1\leq j\leq m}p_j^{(b)}\). The single-step minP-adjusted p-value for \(H_j\) is the permutation-tail probability \(P(M\leq p_j)\), estimated from the joint permutation distribution.

“Joint” matters: each replicate permutes the data in a way that produces a complete vector of test results, preserving the dependence among tests. Permuting each gene or test independently would generally destroy that dependence and would not estimate the intended minimum distribution. In a two-sided analysis, the marginal p-values must be computed from two-sided extremeness, such as the absolute value of the relevant test statistic.

Computing minP-adjusted p-values

With \(B\) sampled permutations, let \(r_j\) be the number of permutation minima no larger than the observed \(p_j\). A common Monte Carlo estimate is

\widehat p_{j,\mathrm{minP}}=\frac{r_j+1}{B+1}, \qquad r_j=\sum_{b=1}^{B} I\!\left(M^{(b)}\leq p_j\right).

The added one in numerator and denominator prevents a reported value of zero when none of the sampled permutation minima is as small as the observed p-value. It is a finite Monte Carlo correction; it does not make a poorly chosen permutation scheme valid. When the complete set of permitted randomizations is enumerated, use the corresponding exact randomization tail probability, with the observed assignment included in the reference set.

Because a smaller observed \(p_j\) gives a smaller event \(\{M\leq p_j\}\), the minP-adjusted p-values preserve the ordering of the raw p-values (allowing ties). A test is rejected at level \(\alpha\) when its adjusted p-value is at most \(\alpha\). Equivalently, the procedure uses one family-wide cutoff on the raw p-values, chosen from the permutation distribution of \(M\).

Worked Example: Calculating Two minP-Adjusted Values

Suppose two correlated genomic tests have observed marginal p-values \(p_1=0.018\) and \(p_2=0.041\). In \(B=19\) joint permutations, two permutation minima are at most 0.018, and five are at most 0.041. These counts are consistent: every minimum at most 0.018 is also at most 0.041.

For the first test, \(r_1=2\), so

\widehat p_{1,\mathrm{minP}}=\frac{2+1}{19+1}=\frac{3}{20}=0.150.

For the second test, \(r_2=5\), so

\widehat p_{2,\mathrm{minP}}=\frac{5+1}{19+1}=\frac{6}{20}=0.300.

At \(\alpha=0.05\), neither hypothesis is rejected by minP. The raw p-values alone might look noteworthy, but the permutation reference indicates that family-wide minima this small are not rare enough to pass the chosen familywise threshold. With only 19 sampled permutations, the estimates are coarse; a larger permutation sample would give finer resolution.

Control of the familywise error rate

The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in the family. If every hypothesis is null, the minP procedure has a direct calibration interpretation: it rejects at least one hypothesis exactly when the observed family minimum crosses the chosen minP cutoff. Calibrating that minimum distribution to have tail probability at most \(\alpha\) therefore controls FWER under the complete null. This is weak control.

Strong control means FWER is at most \(\alpha\) for every configuration of true and false null hypotheses. Complete-null calibration alone does not guarantee that stronger property. A standard sufficient condition for a single-step permutation procedure is subset pivotality: the joint distribution of the test statistics for any subset of true null hypotheses is the same whether or not the other hypotheses are null. Together with valid permutation calibration for the null distribution, this lets the complete-null reference distribution bound the chance of a false rejection among the true nulls.

Justification: Let \(I_0\) be the indices of the true null hypotheses and \(c_\alpha\) the minP cutoff calibrated from the complete-null permutation distribution. A false rejection requires at least one true-null p-value to be at most \(c_\alpha\), so its probability is \(P(\min_{j\in I_0}p_j\leq c_\alpha)\). Under subset pivotality, this has the same distribution as the corresponding true-null subset under the complete null. Since \(\min_{j\in I_0}p_j\geq\min_{1\leq j\leq m}p_j\), the event for the subset is contained in the event that the full-family minimum is at most \(c_\alpha\). The latter probability is calibrated to be at most \(\alpha\), giving strong FWER control.

This argument depends on the assumptions behind the permutation scheme and on subset pivotality for strong control. Valid permutation inference requires the permitted rearrangements to be justified under the null, such as by random assignment or an appropriate exchangeability condition. In observational genomic data, unrestricted shuffling of case labels may be invalid if it breaks ancestry, kinship, batch structure, matching, or other design features. The permutation must respect the design and the null being tested. If the subset-pivotality condition is not defensible, complete-null calibration should not be described as strong control.

Comparison with single-step maxT

Single-step maxT uses the maximum of the test statistics in each joint permutation. For two-sided tests, this is often the maximum absolute statistic. minP instead uses the minimum marginal p-value. These procedures use the same joint permutations but can have different calibrations and decisions.

When every test uses the same strictly decreasing transformation \(g\) from statistic magnitude to marginal p-value, the two extremeness measures contain the same ordering information:

\min_j p_j^{(b)} =\min_j g(|T_j^{(b)}|) =g\!\left(\max_j |T_j^{(b)}|\right).

In that setting, the minP and maxT permutation tails correspond, so their single-step rejection decisions agree, apart from differences caused by ties or Monte Carlo details. If tests have different marginal null distributions, their p-values may be test-specific transformations of their statistics. Then the smallest p-value need not correspond to the largest untransformed statistic, and the procedures can differ. The dependence among tests matters to both calibrations, but it does not force their decisions to match.

Worked Example: Different Decisions When Test Scales Differ

Consider two correlated tests with observed results \((|T_1|,p_1)=(3.0,0.030)\) and \((|T_2|,p_2)=(2.0,0.040)\). The marginal p-values use test-specific null scales, so a statistic of 2.2 for test 2 can correspond to a smaller p-value than a statistic of 3.0 for test 1. In \(B=1000\) joint permutations, suppose 30 replicates have \(\max_j|T_j^{(b)}|\geq3.0\). Of those and the other replicates, suppose 80 in total have \(M^{(b)}\leq0.030\).

For \(H_1\), the maxT-adjusted estimate is

\widehat p_{1,\mathrm{maxT}}=\frac{30+1}{1000+1} =\frac{31}{1001}\approx0.0310.

The minP-adjusted estimate for the same hypothesis is

\widehat p_{1,\mathrm{minP}}=\frac{80+1}{1000+1} =\frac{81}{1001}\approx0.0809.

At \(\alpha=0.05\), maxT rejects \(H_1\), while minP does not. The difference is possible because test 2’s marginal scale allows some permutation statistics below 3.0 to produce p-values at most 0.030. Those replicates contribute to the minP tail for \(H_1\), but not to the maxT tail at the threshold 3.0. These are hypothetical permutation summaries; the example illustrates a possible scale-driven difference, not a universal power ranking.

Worked Example: When minP and maxT Agree

Now suppose two tests use the same marginal calibration \(p_j=g(|T_j|)\), with \(g\) strictly decreasing. The observed statistics are \(|T_1|=2.5\) and \(|T_2|=2.1\). In \(B=19\) joint permutations, two replicates have \(\max_j|T_j^{(b)}|\geq2.5\), while five have \(\max_j|T_j^{(b)}|\geq2.1\).

For test 1, \(p_1=g(2.5)\). The event \(M^{(b)}\leq p_1\) is equivalent to \(\max_j|T_j^{(b)}|\geq2.5\), so both methods count two replicates and yield \(3/20=0.150\). For test 2, the corresponding events are both \(\max_j|T_j^{(b)}|\geq2.1\), so both yield \(6/20=0.300\). Thus the adjusted values agree test by test in this example, not merely the rejection decisions.

Diagnostics, limitations, and a common mistake

Before interpreting minP results, check that each permutation replicate applies a permitted rearrangement to the data and recalculates the entire test family. In genomic applications, the same sample-level label permutation should generally be applied across all genes or variants so their dependence is retained. Also verify that the marginal p-value calculation is consistent across observed and permuted data, including whether tests are one- or two-sided and whether nuisance covariates or restricted blocks are handled as required by the design.

A useful diagnostic is to inspect the permutation distribution of \(M\), its smallest attainable Monte Carlo p-value \(1/(B+1)\), and the number of exceedances used for each adjusted value. Very few exceedances imply substantial Monte Carlo uncertainty; adding permutations can improve resolution. A suspiciously concentrated or unexpectedly extreme minimum distribution can indicate a coding problem, an unsuitable permutation scheme, or an incorrectly computed marginal p-value. Diagnostics cannot establish exchangeability or subset pivotality by themselves: those are design and modeling assumptions.

minP is a single-step procedure. Every hypothesis is adjusted against the minimum over the full family, regardless of the other observed p-values. It can exploit dependence through the joint permutation distribution, but it does not automatically gain power over maxT or another FWER method. The comparison depends on the marginal scales, the joint distribution, and the validity of the procedure’s assumptions.

Mistake: “The minimum p-value was calibrated under the complete null, so the procedure necessarily has strong FWER control.”
Correction: Complete-null calibration establishes weak control. Strong control needs additional justification, such as subset pivotality, as well as a valid permutation scheme.
Mistake: “The largest raw statistic and the smallest p-value are always the same family-wide evidence.”
Correction: They correspond under a common decreasing marginal calibration, but test-specific scales can make minP and maxT produce different permutation tails.

Key takeaway: minP adjusts each marginal p-value using the joint permutation distribution of the family minimum. Its dependence-aware calibration is useful only when the permutation design is valid; strong FWER control also requires suitable conditions such as subset pivotality.

Check Your Understanding

Answer each question, then compare with the solutions below.

In 99 sampled joint permutations, 8 minima are at most an observed p-value of 0.025. Calculate its plus-one minP-adjusted estimate.
Why must the tests be recomputed together in each permutation rather than permuted independently?
What additional condition commonly supports strong, rather than only weak, FWER control for a single-step permutation procedure?
When will minP and maxT have matching permutation extremeness orderings for two-sided tests?
Give one reason unrestricted label permutation could be invalid in a genomic analysis.

Solutions. (1) With \(r=8\) and \(B=99\), the estimate is \((8+1)/(99+1)=9/100=0.0900\). (2) Joint permutation preserves the dependence among tests, which is part of the distribution of the family minimum; independent permutations generally replace it with a different joint distribution. (3) Subset pivotality is a standard sufficient condition, alongside valid permutation calibration. (4) They match when the marginal p-values are the same strictly decreasing function of the absolute test statistics across tests, subject to ties and Monte Carlo details. (5) For example, unrestricted shuffling may break ancestry, kinship, matching, batch structure, or another feature that makes the labels nonexchangeable under the null.