Strong Control Under Partial Nulls

Why Partial Nulls Need Their Own Error Argument

A procedure can control error when every null hypothesis is true and still fail when some are false. The latter setting matters in a clinical trial: a treatment may have an effect on one endpoint while the null remains true for another. Strong familywise error control requires protection in every such configuration, not just under the global null.

Closed testing gives a useful way to verify that protection. The central argument is compact: if the procedure rejects even one true elementary null, it must also reject the intersection of all the true nulls. Consequently, the probability of any false rejection is bounded by the error probability of a local test of that intersection. The argument works only if that local test remains valid for the actual data-generating configuration, including any data-dependent changes.

Definition: Suppose a family contains \(m\) elementary null hypotheses \(H_1,\ldots,H_m\). The familywise error rate (FWER) is the probability of rejecting at least one true elementary null. Strong FWER control at level \(\alpha\) means that this probability is at most \(\alpha\) for every configuration of true and false null hypotheses, over the parameter values in that configuration.

Closure and the True Intersection

For each nonempty index set \(I\subseteq\{1,\ldots,m\}\), write \(H_I=\bigcap_{i\in I}H_i\) for the intersection hypothesis that all nulls indexed by \(I\) hold. A closed procedure specifies a local test for every such intersection. It rejects an elementary null \(H_i\) only if it rejects every \(H_I\) for which \(i\in I\). The local tests need not all have the same form, but each must have level at most \(\alpha\) whenever its intersection null is true.

Let \(T\) be the set of indices of the true elementary nulls in a particular configuration. If \(T\) is nonempty, \(H_T\) is the intersection of all true nulls. It is itself true under that configuration. If the procedure rejects any true \(H_i\), then \(i\in T\), so closure requires rejection of every intersection containing \(i\), including \(H_T\). This inclusion of events is the key to strong control.

Theorem: If every local test of \(H_I\) has rejection probability at most \(\alpha\) for every parameter value satisfying \(H_I\), then the closed procedure controls the FWER strongly at level \(\alpha\). For any configuration with at least one true null, the event of rejecting at least one true elementary null is a subset of the event of rejecting \(H_T\), the intersection of all true nulls.

Justification of the Error Bound

The proof does not require independence among endpoints, local test statistics, or decisions. It does require that the local test for the true intersection be valid under the actual parameter values, including nuisance parameters and any adaptation. For a fixed configuration with \(T\ne\varnothing\), let \(E\) denote the event of at least one false rejection. By the closure rule, \(E\) implies rejection of \(H_T\). Therefore:

P(E)\leq P(\text{reject }H_T)\leq\alpha.

The second inequality follows from the assumed level of the local test because \(H_T\) is true in this configuration. If \(T\) is empty, there are no true elementary nulls to reject, so \(P(E)=0\). Since the argument applies to every configuration, it establishes strong rather than only global-null control.

This is a configuration-by-configuration proof. It does not average error probabilities across different configurations, nor does it require identifying which configuration is most difficult. For each possible set of true nulls, the relevant local test is the one for their joint intersection.

Worked Example: Locating the Error-Bounding Test

Suppose a trial has four null hypotheses, and in one possible configuration \(H_1\) and \(H_3\) are true while \(H_2\) and \(H_4\) are false. Identify the intersection that bounds the FWER in that configuration.

Here \(T=\{1,3\}\), so the true intersection is \(H_T=H_{\{1,3\}}=H_1\cap H_3\). If \(H_1\) is rejected, closure requires rejection of every intersection containing index 1, including \(H_1\cap H_3\). If \(H_3\) is rejected, closure also requires rejection of \(H_1\cap H_3\). Thus:

\{\text{reject }H_1\text{ or }H_3\} \subseteq \{\text{reject }H_1\cap H_3\}.

If the local test of \(H_1\cap H_3\) has rejection probability at most 0.05 whenever both nulls are true, then the probability of rejecting either true elementary null in this configuration is at most 0.05. Rejections of \(H_2\) or \(H_4\) do not themselves count as familywise errors here because those nulls are false.

A Numerical Closed-Testing Calculation

For a concrete calculation, consider three elementary nulls and local Bonferroni tests at \(\alpha=0.05\). Let \(p_i\) be a valid marginal p-value for \(H_i\). For an intersection \(H_I\), the Bonferroni local test rejects when at least one component p-value is at most \(0.05/|I|\), where \(|I|\) is the number of hypotheses in \(I\). Equivalently, its local p-value is \(\min(1,|I|\min_{i\in I}p_i)\). These tests are valid under arbitrary dependence: when \(H_I\) is true, the probability that any of its \(|I|\) component p-values is at most \(0.05/|I|\) is at most \(0.05\), by the union bound.

Worked Example: A False Null Rejected Under Partial Nulls

Suppose the observed marginal p-values are \(p_1=0.012\), \(p_2=0.031\), and \(p_3=0.200\). The true configuration is that \(H_1\) is false and \(H_2,H_3\) are true. Calculate the local decisions and apply closure.

For singleton intersections, the local p-values are 0.012, 0.031, and 0.200. For the two-element intersections they are:

p_{\{1,2\}}=2\min(0.012,0.031)=0.024,\quad p_{\{1,3\}}=2\min(0.012,0.200)=0.024,\quad p_{\{2,3\}}=2\min(0.031,0.200)=0.062.

For the full intersection, \(p_{\{1,2,3\}}=3(0.012)=0.036\). At level 0.05, all local tests reject except those for \(H_3\) and \(H_{\{2,3\}}\). Closure rejects \(H_1\): the tests for \(H_1\), \(H_{\{1,2\}}\), \(H_{\{1,3\}}\), and \(H_{\{1,2,3\}}\) all reject. Closure does not reject \(H_2\), because the test for \(H_{\{2,3\}}\) does not reject; nor does it reject \(H_3\), whose own local test does not reject.

In this partial-null configuration the procedure rejects only \(H_1\), which is false, so this observed decision makes no familywise error. The relevant error bound is the local test of the true intersection \(H_2\cap H_3\). Its local p-value is 0.062, so it is not rejected in these data. The calculation illustrates the gate imposed by closure; the formal guarantee comes from the level of that local test over repeated samples, not from this one observed p-value.

What Adaptation Must Preserve

In an adaptive trial, local tests may use interim data to select a population, change an allocation, or choose a final analysis. The closure argument itself does not prohibit these changes. It requires that every local test used in the closure have a valid level-\(\alpha\) rejection rule under its intersection null after accounting for the full data-dependent procedure. In particular, validity under a fixed design is not enough if the analysis or critical value is selected using the data.

One way to check a local test is to condition on the selection information, when the procedure's justification provides a valid conditional error bound. If, for every possible selection outcome \(s\) and every parameter value satisfying \(H_I\), the conditional rejection probability is at most \(\alpha\), then averaging over the selection outcomes preserves the bound:

P(\text{reject }H_I) =\sum_s P(\text{reject }H_I\mid S=s)P(S=s) \leq\sum_s \alpha P(S=s)=\alpha.

Here \(S\) denotes the selection outcome. The conditional inequality is an assumption to verify, not a consequence of having performed a selection. Alternatively, a procedure may establish unconditional level-\(\alpha\) validity directly under the adaptive design. Either route is sufficient for the closure proof if it holds for each intersection null, including the intersection of the true nulls in every configuration.

Worked Example: Averaging a Conditional Bound

Suppose an adaptive local test of a particular intersection has two possible selection outcomes, \(S=A\) and \(S=B\). Under every parameter value satisfying that intersection, assume \(P(S=A)=0.70\), \(P(S=B)=0.30\), and the conditional rejection probabilities are 0.040 and 0.050, respectively. Check its unconditional level.

By the law of total probability:

P(\text{reject}) =0.040(0.70)+0.050(0.30) =0.028+0.015 =0.043.

The unconditional rejection probability is 0.043, below 0.05. More generally, if each conditional rejection probability is at most 0.05, their probability-weighted average is also at most 0.05. This calculation supports a local level claim only if the conditional bounds really hold under the intersection null for every relevant selection outcome and parameter value.

Diagnostics, Alternatives, and Limitations

A practical verification should examine both the logical closure and the statistical validity of its local tests. For each nonempty \(I\), check that the test of \(H_I\) has size at most \(\alpha\) under the whole intersection null—not merely under a convenient special case. For an adaptive procedure, include the selection rule, stopping rule, data-dependent critical values, and nuisance-parameter treatment in that check. Then confirm that an elementary null is declared rejected only after every intersection containing it passes its local test.

Common warning signs include a local test justified only when all elementary nulls are true, a marginal p-value treated as valid after data-dependent selection without justification, and an error bound averaged over different parameter configurations. Such arguments do not establish the needed bound for \(H_T\) under each fixed partial-null configuration.

Wrong reasoning: “The procedure has FWER at most 0.05 when all nulls are true, so it has strong control.” This establishes, at most, control under the global null. Correction: For every possible true set \(T\), verify that the local test of \(H_T\) is level 0.05.
Wrong reasoning: “The selected analysis has a 0.05 error rate when viewed on average across selection outcomes.” This may hide a selection outcome with inflated conditional error, and the average may not apply to every parameter configuration. Correction: establish a valid unconditional bound for the local test, or justify the conditional bounds and average them under the null being tested.
Wrong reasoning: “A rejection of one true null is unlikely because each marginal test uses 0.05.” Marginal levels alone do not show that the closed procedure's intersection tests are valid after adaptation. Correction: verify the local tests and the closure rule together.

A single-step Bonferroni procedure can offer a simpler route to strong control when valid marginal p-values are available: it protects against any rejection using a familywise adjustment. Closed testing is more flexible because local tests can be chosen for particular intersections and may exploit structure or dependence, but it can require checking many intersections—\(2^m-1\) for \(m\) hypotheses. Its guarantee is only as sound as the validity of those local tests. The proof does not promise high power, nor does it repair a local test whose assumptions fail.

Key takeaway: To verify strong control under a partial-null configuration, identify the set \(T\) of true elementary nulls. Closure makes any false rejection imply rejection of \(H_T\). A level-\(\alpha\) local test of that true intersection therefore bounds the probability of any false rejection by \(\alpha\).

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In a family of five hypotheses, \(H_2,H_4,H_5\) are true. Which local test gives the key bound for the FWER in this configuration?
Why does control under the global null alone fail to establish strong FWER control?
A data-dependent local test has conditional rejection probability at most 0.04 for every selection outcome under its intersection null. What can be said about its unconditional rejection probability?
Suppose a procedure rejects a true \(H_i\), but does not reject the intersection of all true nulls. What does this indicate about the procedure or its claimed closure?

Solutions. (1) The test of \(H_2\cap H_4\cap H_5\), because this is the intersection of all true elementary nulls. (2) Strong control must hold when some nulls are false as well as when all are true; the local test of the true intersection can behave differently in a partial-null configuration. (3) Its unconditional rejection probability is at most 0.04, by averaging the conditional probabilities over selection outcomes. This conclusion requires the conditional bound to hold under the intersection null for every relevant outcome and parameter value. (4) The decisions do not satisfy the defining closure requirement, or the intersection test was not correctly included. A genuine closed procedure cannot reject a true elementary null without rejecting that intersection.