Strong Familywise Control and Subset Pivotality

Why Complete-Null Calibration Is Not Enough

In a family of genomic hypotheses, a single-step max-T procedure compares each test statistic with a cutoff chosen from the distribution of the largest statistic under the complete null, the configuration in which every hypothesis is true. This calibration can limit the probability of any rejection when all hypotheses are true. That property is called weak familywise error control. Strong control is more demanding: it limits the probability of rejecting at least one true null for every possible mix of true and false hypotheses.

The distinction matters because an alternative can change the joint distribution of test statistics that correspond to true nulls. Even if each true-null statistic still has a valid marginal null distribution, its dependence with other statistics may change. A maximum calibrated using the complete-null joint distribution may then be too small to protect the familywise error rate under partial nulls.

Definition: For hypotheses $H_1,\ldots,H_m$, let $I_0$ be the set of indices of the true null hypotheses at the parameter configuration under consideration. The familywise error rate (FWER) at that configuration is $$ \operatorname{FWER}=P(\text{reject at least one }H_i\text{ with }i\in I_0). $$ A procedure has weak FWER control at level $\alpha$ if this probability is at most $\alpha$ when all $m$ null hypotheses are true. It has strong FWER control at level $\alpha$ if the probability is at most $\alpha$ at every configuration of true and false nulls.

For two-sided tests, write $T_i$ for the statistic for $H_i$. A single-step max-T procedure rejects $H_i$ when $|T_i|>c_\alpha$, where $c_\alpha$ is selected so that, under the complete null, $$ P\left(\max_{1\leq i\leq m}|T_i|>c_\alpha\right)\leq\alpha. $$ If the complete-null distribution is continuous and the cutoff is an exact upper quantile, this probability is $\alpha$. If a discrete calibration requires a conservative quantile, it can be smaller. The weak-control result follows directly from this calibration; strong control needs an additional argument about the distributions under partial nulls.

Worked Example: Complete-Null Calibration Fails Under Partial Nulls

Consider three two-sided tests and a nominal familywise level $\alpha=0.05$. Suppose that when all three nulls are true, their statistics have the hypothetical joint distribution $$ (T_1,T_2,T_3)=(Z,Z,Z),\qquad Z\sim N(0,1). $$ Thus the complete-null maximum is $|Z|$, and an exact cutoff is $c_\alpha=\Phi^{-1}(0.975)\approx1.959964$, since $P(|Z|>c_\alpha)=0.05$. The max-T procedure therefore has 0.05 FWER when all three nulls are true.

Now consider a configuration in which $H_1$ and $H_2$ are true and $H_3$ is false. Suppose, hypothetically, that in this configuration $T_1=Z_1$ and $T_2=Z_2$, where $Z_1,Z_2$ are independent standard normal variables, while $T_3$ has a nonzero mean. Each true-null statistic still has the standard normal marginal distribution, but their dependence has changed from perfect correlation to independence.

The FWER is now $$ P(|Z_1|>c_\alpha\text{ or }|Z_2|>c_\alpha) =1-P(|Z_1|\leq c_\alpha)P(|Z_2|\leq c_\alpha) =1-(0.95)^2=0.0975. $$ This exceeds 0.05. The cutoff calibrated under the complete null therefore has weak control but not strong control in this example. The issue is not a failure of either true-null test's marginal calibration; it is the changed joint distribution of the true-null statistics.

Subset Pivotality

The condition that rules out the failure in the example is called subset pivotality. It compares the joint distribution of any selected set of true-null statistics under a partial-null configuration with the distribution of those same statistics under the complete null. It is a property of the statistical model and the test statistics, not a consequence of using a maximum by itself.

Definition: The test statistics are subset pivotal if, for every subset $I\subseteq\{1,\ldots,m\}$, the joint distribution of $(T_i:i\in I)$ is the same under every parameter configuration at which all hypotheses in $I$ are true. In particular, for any configuration with true-null index set $I_0$, the distribution of $(T_i:i\in I_0)$ is the same as its distribution under the complete null.

The definition concerns the joint distribution. Checking that each statistic separately has a valid null distribution is insufficient: the example above satisfies that marginal condition for $T_1$ and $T_2$ but violates subset pivotality for the subset $\{1,2\}$. Subset pivotality also does not require all test statistics to be independent. Their dependence may be arbitrary, provided the joint distribution of each true-null subset is unchanged when other hypotheses are false.

Exact equality of distributions is a convenient sufficient condition. More generally, strong control also follows if, for every possible true subset and every relevant cutoff, the tail probability of the maximum over that subset under its partial-null configuration is no greater than the corresponding complete-null maximum tail probability. This is a stochastic domination condition. It must be justified for the setting at hand; it cannot be assumed merely because each individual test is valid.

Why Subset Pivotality Gives Strong Control

The argument is short but depends on identifying the correct event. Suppose the complete-null cutoff satisfies $P_0(\max_{1\leq i\leq m}|T_i|>c_\alpha)\leq\alpha$, where $P_0$ denotes probability under the complete null. Fix any parameter configuration and let $I_0$ be its set of true nulls. If $I_0$ is empty, no true null can be rejected, so the FWER is zero. Otherwise, at least one true null is rejected exactly when the maximum absolute statistic among the true nulls exceeds the cutoff.

\begin{aligned} \operatorname{FWER} &=P\left(\max_{i\in I_0}|T_i|>c_\alpha\right)\\ &=P_0\left(\max_{i\in I_0}|T_i|>c_\alpha\right)\\ &\leq P_0\left(\max_{1\leq i\leq m}|T_i|>c_\alpha\right)\\ &\leq\alpha. \end{aligned}

The second line uses subset pivotality: the true-null statistics have the same joint distribution at the partial-null configuration as under the complete null. The third line holds because $I_0$ is a subset of all indices, so the event that a true-null statistic exceeds the cutoff is contained in the event that some statistic exceeds it. This proves strong control for every configuration with at least one true null; the empty-true-set case was handled separately. If the subset distributions are stochastically dominated rather than equal, the same inequality follows by replacing the equality in the second line with an upper bound.

Worked Example: Independent Normal Tests

Suppose there are two hypotheses $H_i:\theta_i=0$, and the statistics satisfy $T_i\sim N(\theta_i,1)$ independently for $i=1,2$. The model assumptions here are that the statistics have these normal distributions, their variances are known and equal to one, and the two statistics are independent. Under any configuration where $H_i$ is true, $T_i\sim N(0,1)$, regardless of the value of the other parameter. Thus every true-null subset has the same distribution as under the complete null, and subset pivotality holds.

Under the complete null, independence gives $$ P_0\left(\max(|T_1|,|T_2|)\leq c_\alpha\right) =\{1-2\Phi(-c_\alpha)\}^2=0.95, $$ where $\Phi$ is the standard normal cumulative distribution function. Consequently, $$ \Phi(-c_\alpha)=\frac{1-\sqrt{0.95}}{2}\approx0.0126603, \qquad c_\alpha\approx2.237. $$ If both nulls are true, the FWER is 0.05 by construction. If only $H_1$ is true, for example with $\theta_1=0$ and $\theta_2=3$, the FWER is $$ P(|T_1|>2.237)\approx 2\Phi(-2.237)\approx0.0253. $$ If neither null is true, the FWER is zero because there is no true null to reject. Thus strong control holds, although in the one-true-null configuration the procedure is conservative.

Worked Example: Perfectly Correlated Statistics

Consider two tests with statistics $T_i=\theta_i+Z$, where $Z\sim N(0,1)$ is the same random error in both statistics. Assume the parameters are $\theta_1,\theta_2$, and $H_i$ is true exactly when $\theta_i=0$. This is a stylized model with perfectly correlated test statistics; its purpose is to show that subset pivotality does not require independence.

Under the complete null, $T_1=T_2=Z$, so the maximum absolute statistic is $|Z|$ and the 0.05 cutoff is $c_\alpha=\Phi^{-1}(0.975)\approx1.959964$. If, for example, $\theta_1=0$ and $\theta_2=2$, then $H_1$ is true and $H_2$ is false. The true-null statistic is still $T_1=Z$, exactly as under the complete null, so the FWER is $P(|Z|>c_\alpha)=0.05$. The true-null subset has the same distribution under both configurations. The full vector's distribution changes, but the subset pivotality condition needed for the proof remains true.

When the Condition Is Plausible—and How to Check

A useful setting is a correctly specified multivariate normal model with fixed covariance and a parameter vector that changes only the means. If the statistics are linear transformations of the data with their null means fixed at zero, the joint distribution of the statistics for true-null coordinates can be independent of the means attached to other coordinates. In such a setting, alternatives may shift other statistics without changing the joint law of the true-null subset. The precise model and the construction of the statistics still matter: estimated variance components, data-dependent filtering, or other shared estimation steps can make a true-null statistic depend on parameters outside its own null.

For permutation procedures, a complete-null permutation distribution is not automatically a valid reference distribution under partial alternatives. The permutations must preserve the relevant null distribution for the true-null statistics in the presence of the other effects. Exchangeability under a global null, by itself, does not establish subset pivotality. The permutation unit and restrictions must also match the design and the null being tested.

Before relying on strong control, check the following:

Write the joint model for the test statistics, including how its distribution changes when some effects are nonzero.
For each possible true-null subset, ask whether its joint statistic distribution is unchanged by parameters attached to false nulls.
Check shared nuisance estimates, variance estimation, filtering, and transformations; these can alter dependence or marginal distributions under partial nulls.
For a resampling calibration, verify that the resampling scheme represents the joint law needed for the true-null subset, not only the complete-null law.
Use simulations under representative partial-null configurations as a diagnostic. Simulation can reveal a violation, but cannot prove the condition for all configurations.

A common mistake is to argue, “Every true-null statistic has a valid marginal $p$-value, so the max-T threshold must strongly control the FWER.” The correction is that max-T calibration uses a joint maximum distribution. Strong control needs suitable control of the joint distribution of the statistics for the true nulls; valid marginal distributions alone do not provide it.

Comparison and Limitations

Approach	What supports its FWER guarantee?	Main trade-off
Complete-null single-step max-T	Strong control follows when subset pivotality, or a suitable maximum-tail domination condition, holds.	Can use dependence to improve power, but a complete-null calibration alone may only give weak control.
Bonferroni single-step testing	Valid marginal $p$-values for every true null; no particular dependence structure is required.	Can be conservative, especially when tests are strongly dependent.
Calibration under partial-null configurations	A valid upper bound on the true-null maximum tail probability for the configurations of interest.	May require modeling or resampling many configurations and can be difficult to justify uniformly.

Subset pivotality is a sufficient condition, not a universal necessity for strong control. Strong control may hold through a different valid bound on the true-null maximum, even when the distributions are not exactly equal. Conversely, a plausible-looking dependence structure or a reassuring simulation is not a proof of subset pivotality. Its practical value is that, when justified, it converts a complete-null max-T calibration into a strong familywise guarantee without needing a separate calibration for every pattern of alternatives.

Check Your Understanding

Answer each question, then compare with the solutions below.

What is the difference between weak and strong FWER control?
Why do valid marginal null distributions for every $T_i$ fail to guarantee strong control by a max-T procedure?
Suppose subset pivotality holds and $I_0$ is nonempty. State the event whose probability is the FWER and explain why it is bounded by the complete-null maximum tail probability.
In the independent two-test normal example, what is the approximate FWER if only one null is true at the cutoff $c_\alpha\approx2.237$?
Does subset pivotality require independent test statistics? Give a reason for your answer.

Solutions. (1) Weak control bounds the FWER only when all nulls are true; strong control bounds it under every configuration. (2) The FWER depends on the joint distribution of the true-null statistics through their maximum, and their dependence can change under alternatives even when each marginal distribution stays valid. (3) The event is $\{\max_{i\in I_0}|T_i|>c_\alpha\}$. Subset pivotality equates its probability to the corresponding complete-null subset probability, and that event is contained in $\{\max_{1\leq i\leq m}|T_i|>c_\alpha\}$, whose probability is at most $\alpha$. (4) It is $2\Phi(-2.237)\approx0.0253$. (5) No. It requires the joint distribution of each true-null subset to be unchanged under the relevant configurations; that joint distribution may include arbitrary dependence, as the perfectly correlated example shows.