Closure with Data-Dependent Local Tests

Why Local-Test Validity Must Survive Adaptation

In closed testing, conclusions about individual hypotheses depend on tests of intersections. When those local tests are selected or configured using interim data, the closure argument still applies only if each resulting local procedure controls its own type I error. It is not enough that the procedure looked valid before selection, or that its error rate is small after averaging over a convenient subset of interim outcomes.

The key requirement is attached to the entire data-dependent rule: for each intersection that may be tested, under every parameter value in that intersection, its probability of rejection must remain bounded by the chosen level. A conditional bound given the information used for adaptation is a transparent sufficient condition. The main result below explains how these local bounds produce strong familywise error control.

Definition: Let \(H_1,\ldots,H_m\) be a fixed finite family of hypotheses. For every nonempty index set \(I\subseteq\{1,\ldots,m\}\), write \(H_I=\bigcap_{i\in I}H_i\). A closed testing procedure tests each \(H_I\) with a local test and rejects an elementary hypothesis \(H_i\) only if it rejects every \(H_I\) for which \(i\in I\).

The local tests need not use the same statistic or be based on the same analysis. But the hypotheses being tested must have stable meanings: the local test labelled \(H_I\) must genuinely test the intersection \(H_I\). The focus here is the level validity of these local tests when interim information determines which test, statistic, or test configuration is used.

A Conditional Level Requirement

Let \(\mathcal F\) denote the information available when an adaptation is made, such as interim outcomes and the resulting test-selection decision. For each nonempty \(I\), let \(R_I\) be the event that the complete, possibly data-dependent local procedure rejects \(H_I\). A useful validity requirement is

Conditional local validity: For each nonempty \(I\) and each parameter value \(\theta\in H_I\), \(P_\theta(R_I\mid\mathcal F)\leq\alpha\) almost surely. The bound must hold for every allowed interim outcome and every parameter value in the intersection. A conditional p-value version requires \(P_\theta(p_I\leq u\mid\mathcal F)\leq u\) for every \(u\in[0,1]\), under the same conditions.

This requirement concerns the selected procedure, not merely each candidate test considered in isolation. If a rule chooses candidate \(A_I\) after observing \(\mathcal F\), the conditional bound must hold for the test actually chosen, at each possible adaptation outcome. For a composite intersection, the condition is uniform over its null parameter values; a bound that holds only at one convenient null value is insufficient.

Conditional validity is sufficient, not the only possible route. The closure argument needs each complete local procedure to have unconditional rejection probability at most \(\alpha\) for every parameter value in its null. A separately established unconditional bound can therefore suffice. The conditional requirement is especially useful when adaptation is explicit: integrating a conditional bound gives the required unconditional bound without relying on a particular distribution of interim outcomes.

Why the Local Bounds Give Strong Control

The familywise error rate is the probability of rejecting at least one true elementary hypothesis. Strong control means this probability is at most \(\alpha\) for every configuration of true and false hypotheses. The argument uses the intersection of all true hypotheses, not an assumption that every hypothesis is true.

Theorem: Suppose each local procedure in a closed testing procedure satisfies \(P_\theta(R_I)\leq\alpha\) for every nonempty \(I\) and every \(\theta\in H_I\). Then the procedure strongly controls the familywise error rate at level \(\alpha\). Conditional local validity implies the required unconditional local bounds.

Here is the justification. Fix a parameter value \(\theta\), and let \(T=\{i:\theta\in H_i\}\) be the set of true elementary hypotheses. If \(T\) is empty, there can be no false rejection. If \(T\) is nonempty and the procedure falsely rejects any true \(H_i\), it must have rejected every intersection containing that \(H_i\). In particular, it must have rejected \(H_T\), because \(i\in T\) implies \(H_T\) is one of those intersections. Since \(\theta\in H_T\), the local level bound for \(H_T\) applies. Thus the probability of any false rejection is bounded by the probability of rejecting \(H_T\), which is at most \(\alpha\).

For the conditional version, take expectations of the conditional rejection probability:

P_\theta(R_T) = E_\theta\!\left[P_\theta(R_T\mid\mathcal F)\right] \leq E_\theta[\alpha] = \alpha.

This is the central reason the requirement applies to every intersection: whichever hypotheses are true at the actual parameter value, their intersection supplies the local test that bounds the probability of any false rejection.

Worked Example: Finding the Intersection That Controls the Error

Suppose three hypotheses \(H_1,H_2,H_3\) are tested by closure, and the true parameter makes \(H_1\) and \(H_3\) true but \(H_2\) false. Assume every local test, including tests selected using interim data, has rejection probability at most \(0.05\) under its intersection null. What bounds the probability of falsely rejecting at least one true hypothesis?

The true index set is \(T=\{1,3\}\), so the relevant true intersection is \(H_T=H_1\cap H_3\). If \(H_1\) is falsely rejected, closure requires rejection of every intersection containing index 1, including \(H_{\{1,3\}}\). If \(H_3\) is falsely rejected, closure also requires rejection of \(H_{\{1,3\}}\). Therefore, either false rejection implies rejection of this same true intersection.

Consequently,

P_\theta(\text{at least one false rejection}) \leq P_\theta(R_{\{1,3\}}) \leq 0.05.

The claim is strong control: it holds for this partial-null configuration as well as for other configurations, provided each corresponding true intersection test satisfies its bound. The false hypothesis \(H_2\) does not need to be accepted or rejected for this error argument.

Checking a Data-Dependent Choice

A practical audit separates the adaptation rule from the test's null calibration. First identify exactly what interim information determines the choice. Then, for each possible choice and each relevant intersection null, check the rejection probability conditional on that information. If the local test uses a p-value, check conditional super-uniformity rather than relying only on its behavior in one simulated or averaged scenario.

Worked Example: Two Interim States, One Valid Selected Test

For a particular intersection \(H_I\), an interim review records either state \(F=a\) or \(F=b\). The design chooses test A in state \(a\) and test B in state \(b\). Under every parameter value in \(H_I\), suppose the conditional rejection probabilities are \(0.03\) for test A given \(F=a\), and \(0.04\) for test B given \(F=b\). These are hypothetical operating characteristics. Show that the selected local test meets a \(0.05\) bound.

The selection rule is \(A_I=A\) when \(F=a\), and \(A_I=B\) when \(F=b\). The conditional rejection probability of the selected procedure is therefore \(0.03\) in state \(a\) and \(0.04\) in state \(b\). Both are at most \(0.05\), so the conditional local validity requirement holds.

For illustration, if \(P_\theta(F=a)=0.60\) and \(P_\theta(F=b)=0.40\), the unconditional rejection probability is

P_\theta(R_I) =(0.60)(0.03)+(0.40)(0.04) =0.018+0.016 =0.034.

The unconditional value is \(0.034\), but the validity argument does not depend on these particular state probabilities: the largest conditional rejection probability is \(0.04\), below the target \(0.05\). The same conditional check must be made for every parameter value in \(H_I\), not just one.

A selection among individually valid tests does not automatically preserve validity. Validity must be checked after accounting for the selection rule. The next example isolates this issue with a simple null model.

Worked Example: Choosing the Smaller of Two Null P-Values

Under a particular intersection null, suppose two candidate p-values \(P_1\) and \(P_2\) are independent and each is uniformly distributed on \([0,1]\). Each candidate used alone is a valid level-\(0.05\) test. An interim rule selects the smaller p-value and rejects the intersection if that selected value is at most \(0.05\). Calculate the actual rejection probability.

The selected test rejects when \(P_1\leq0.05\) or \(P_2\leq0.05\). By independence, the probability of not rejecting is the probability that both p-values exceed \(0.05\):

P(\min(P_1,P_2)\leq0.05) =1-P(P_1>0.05,\ P_2>0.05) =1-(0.95)^2 =0.0975.

Thus the selected local test rejects with probability \(0.0975\), not \(0.05\). Although either candidate is valid when used on its own, choosing between them using their observed values creates a different local procedure. Its rejection probability exceeds the intended bound, so it cannot be used as a level-\(0.05\) local test in the closure argument without a valid adjustment or a different calibration.

Composite Intersections and Diagnostics

For a composite null, the local bound must cover nuisance-parameter values as well as possible interim outcomes. A useful finite audit is to tabulate conditional rejection probabilities for each interim state and each null configuration considered. For a continuous nuisance parameter, the corresponding requirement is a uniform bound over its allowed values; checking a few values is not itself a proof of that uniform claim.

Worked Example: Checking a Bound Across Nuisance Values

Suppose a local test of \(H_I\) is adapted to interim state \(a\) or \(b\), and its null contains a nuisance parameter \(\eta\). In a hypothetical finite model, all allowed values are \(\eta_1\) and \(\eta_2\). Exact calculations give the following conditional rejection probabilities:

Nuisance value	Given interim state \(a\)	Given interim state \(b\)
\(\eta_1\)	0.04	0.05
\(\eta_2\)	0.03	0.06

For \(\eta_1\), both conditional probabilities are at most \(0.05\). For \(\eta_2\), the state-\(b\) probability is \(0.06\), so the conditional validity requirement fails. For example, if \(P_{\eta_2}(F=a)=0.90\) and \(P_{\eta_2}(F=b)=0.10\), the unconditional rejection probability at \(\eta_2\) is

$$ (0.90)(0.03)+(0.10)(0.06)=0.027+0.006=0.033. $$

That particular unconditional probability is below \(0.05\), but it does not establish the stated conditional requirement: the local procedure rejects with probability \(0.06\) in state \(b\) at \(\eta_2\). If a separate argument established an unconditional bound for every allowed \(\eta\), it could support the closure proof. The conditional audit, however, has identified a state and nuisance value needing justification or recalibration.

A useful comparison is with Bonferroni testing, which controls multiplicity by allocating error bounds across hypotheses rather than requiring a valid test for every intersection in a closure. Closure can exploit tailored local tests, but that flexibility does not remove the local validity obligation. Regardless of the multiplicity strategy, an unverified data-dependent choice can change the error rate of the procedure being used.

Common Mistakes and Practical Checks

Checking candidates but not the selection rule. Each candidate may be level \(\alpha\) when fixed in advance, while the rule that selects among them is not. Calibrate or bound the complete selected procedure.
Checking only an average over interim outcomes. An average under one interim-state distribution does not prove a conditional bound, nor does it establish an unconditional bound uniformly over all null parameter values. State which validity claim is being made and justify it.
Checking only one point in a composite null. The local rejection probability must be bounded throughout the intersection null, including nuisance-parameter values.
Proving bounds for only some intersections. The set of true hypotheses can vary with the parameter. The closure argument needs the bound for whichever true intersection is relevant, so every nonempty intersection must have a valid local procedure.
Confusing a local decision with an elementary rejection. Rejecting one intersection alone does not reject an elementary hypothesis under closure; every intersection containing that hypothesis must be rejected.

Key takeaway: Closure yields strong familywise error control only when each intersection's complete, data-dependent local procedure retains its level bound under that intersection null. Conditional validity given adaptation information is a clear sufficient condition; the bound must also cover all null parameter values.

Check Your Understanding

Answer each question, then compare with the solutions below.

In a closed procedure, \(H_1\) and \(H_2\) are true and \(H_3\) is false. Which local rejection is necessary for a false rejection of either true elementary hypothesis?
A selected local test has conditional rejection probabilities \(0.02\), \(0.04\), and \(0.05\) across all possible interim states under a fixed null parameter. Does it satisfy a \(0.05\) conditional bound?
Two independent uniform null p-values are inspected, and the smaller is compared with \(0.01\). What is the rejection probability?
Why does a conditional bound at one nuisance-parameter value not establish validity for a composite intersection?
For a fixed parameter value, suppose the set of true elementary hypotheses is empty. What does the familywise error event contain?

Solutions. (1) Rejection of \(H_{\{1,2\}}=H_1\cap H_2\) is necessary: any false rejection of \(H_1\) or \(H_2\) under closure requires rejecting every intersection containing it, including \(H_{\{1,2\}}\). (2) Yes. Every listed conditional probability is at most \(0.05\), so the conditional bound holds for those states and that null parameter; validity still requires checking all null parameter values. (3) The probability is \(1-(0.99)^2=0.0199\), since rejection occurs if either p-value is at most \(0.01\). (4) A composite intersection contains multiple parameter values, and rejection probabilities can differ across them; the claimed bound must cover every value in the null. (5) No true elementary hypothesis can be falsely rejected, so the familywise error event is empty and its probability is zero.