Step-Down MaxT and Subset Pivotality

Why Step-Down MaxT?

In genomic testing, many hypotheses can be assessed using related measurements or the same subjects. Their test statistics may therefore be dependent. A single-step maxT procedure uses the largest statistic across the entire family to calibrate every test. Step-down maxT can use a smaller set of hypotheses at later steps, which can improve power while still aiming to control the probability of at least one false rejection.

The key guarantee is strong familywise error rate (FWER) control: the probability of rejecting one or more true null hypotheses is at most the chosen significance level \(\alpha\), no matter which other hypotheses are false. Step-down maxT obtains this guarantee under conditions that include valid permutation calibration and subset pivotality. It is not enough to run permutations; the design and the joint null distributions must support the inference.

Algorithm and Assumptions

Write the ordered observed statistics as \(T_{(1)}\geq T_{(2)}\geq\cdots\geq T_{(m)}\), and let \(H_{(k)}\) identify the hypothesis at rank \(k\). For permutation \(b\), use the same permissible permutation of the data across all tests and write \(T_j^{(b)}\) for the resulting statistic. The remaining set at rank \(k\) is \(R_k=\{(k),(k+1),\ldots,(m)\}\). Its permutation maximum is \(M_k^{(b)}=\max_{j\in R_k} T_j^{(b)}\).

Here \(B\) is the number of sampled permutations, and \(I(\cdot)\) is 1 when its condition is true and 0 otherwise. The added 1 includes the observed arrangement and avoids reporting a zero Monte Carlo p-value. With an exact enumeration, the corresponding tail proportion is calculated over the permitted arrangements, with ties included. Adjusted p-values are the running maximum of the sequential p-values, so they cannot decrease as the observed evidence becomes weaker down the ordered list. At level \(\alpha\), reject in order while the sequential p-value is at most \(\alpha\), and stop at the first value above \(\alpha\). The adjusted-p-value decisions give the same rejection set.

The procedure requires a statistic whose direction is consistent across tests; for two-sided tests, use a statistic such as \(|T_j|\) in both the observed and permuted calculations. Permutations must respect the data collection and assignment structure. For example, if observations are paired, the permutation scheme must preserve pairing; if exchangeability holds only within blocks, permutations must remain within those blocks. The null hypothesis being tested must justify the chosen permutation scheme, and the statistics across tests must be permuted jointly rather than independently.

Subset Pivotality and Strong Control

The step-down calculation uses permutation distributions to calibrate maxima over sets of hypotheses. Strong FWER control requires those calibrations to remain valid when some hypotheses are false. Under subset pivotality, the joint distribution of the statistics for any subset of true nulls is unchanged by the status of the other hypotheses. In a valid permutation design, the maximum over a set containing all true nulls is at least the maximum over the true nulls alone, permutation by permutation. Thus calibrating a step against the larger remaining set can be conservative for the true nulls it contains.

The logic is especially clear at the first point in the ordered procedure where a true null could be rejected. All earlier rejections, if any, are false rejections, so every true null remains in the set being tested at that step. A valid max test for that remaining set therefore protects against rejecting a true null there. If each such intersection test has level at most \(\alpha\), the probability of reaching and rejecting a true null is at most \(\alpha\). This is the basis for the strong-control result; validity of the permutation tests and subset pivotality are essential, not optional technical details.

Justification, Diagnostics, and Alternatives

The adjustment \(p_{(k)}^{\mathrm{adj}}=\max_{\ell\leq k}p_{(\ell)}^{\mathrm{step}}\) follows from the rejection rule. To reject the hypothesis at rank \(k\), the procedure must have passed every earlier step. Therefore its adjusted p-value must be no smaller than any earlier sequential p-value. Taking the largest of them gives the smallest threshold at which rank \(k\) and all preceding ranks would be rejected. This also prevents the implausible result of a later, weaker statistic having a smaller adjusted p-value than an earlier, stronger one.

Diagnostics should focus first on whether the permutations are legitimate. Check that the permuted units and any pairing, blocks, or covariates reflect the actual design and null hypothesis. Confirm that every test uses the same permutation of the units, so dependence among statistics is retained. Also check the direction and scale of the statistics: mixing signed and absolute statistics, or using incompatible scales in the maximum, changes the test. With few permutations, p-values are coarse; report \(B\), the add-one convention, and the attainable resolution.

A common mistake is to say that subset pivotality means the hypotheses are independent. It does not: the statistics may be strongly dependent. The condition instead says that the joint distribution for any subset of true nulls is unaffected by which other hypotheses are false. Another mistake is to conclude that any permutation maxT procedure must strongly control FWER. Without a valid permutation scheme and suitable joint-null conditions, the calibration may fail.

Single-step maxT uses the maximum over the full family for every hypothesis. It is simpler and often more conservative because it does not shrink the set as hypotheses are considered. Step-down maxT recalibrates over successively smaller sets. Holm’s procedure instead uses ordered marginal p-values and Bonferroni-type thresholds; it provides strong FWER control under arbitrary dependence when the marginal p-values are valid, but it does not use the joint permutation distribution to exploit dependence. These methods address the same error rate through different assumptions and calibrations; none removes the need to define the testing family appropriately.

Step-down maxT is limited by the validity of its permutation reference distribution and, for the strong-control argument described here, subset pivotality. It may be inappropriate when the null distributions for true hypotheses change with effects at other hypotheses, or when the design does not justify the permutations. In such settings, a method with weaker assumptions may be preferable, although it may be less powerful. The procedure controls the chance of one or more false rejections; it does not estimate the proportion of false findings among the rejected tests.