Why an Error-Control Proof Needs an Audit
An adaptive test can have a plausible design, a prespecified rule, and a reassuring calculation under one null scenario, yet still lack a valid argument for its claimed error rate. The central audit question is not whether a calculation looks familiar. It is whether each probability in the argument is evaluated under the right null configuration and conditioning information, and whether the result covers every rejection claim the procedure can make.
For a family of hypotheses, controlling error under the configuration in which all null hypotheses are true is not automatically enough. Adaptation may use interim data to select hypotheses, change critical values, or alter the continuation rule. An argument must account for those choices and for configurations in which some hypotheses are true while others are false.
A gap in a proof is not itself a counterexample to the procedure. It means the stated reasoning has not established the claim. To show that the procedure actually fails, one needs a parameter configuration and a probability calculation (or other valid argument) showing that its error exceeds the target.
A Four-Part Audit
For an adaptive error-control claim, write down the object being controlled, the null configurations covered, the conditioning information used, and the event whose probability is bounded. This makes it easier to see whether a proof silently changes its target partway through.
For familywise error, identify the event that at least one true null is rejected. If the procedure makes several rejection claims, specify which claims count as errors under each configuration.
Check the all-null case and every relevant partial-null case. A proof for one configuration does not establish a supremum over all configurations unless a justified least-favorable argument covers the rest.
When a selection or interim event is conditioned on, use the conditional distribution under the null configuration at issue. An unconditional calibration cannot simply be carried over after conditioning.
For a closed procedure, verify the local test for each intersection that can support a rejection. For a conditional-error argument, compare the adapted continuation probability with the relevant conditional error at the same interim history.
Worked Example: A Complete-Null Check Misses a Partial Null
Worked Example: A Complete-Null Check Misses a Partial Null
Suppose a procedure makes a rejection claim only about \(H_1\), using two statistics \(P_1\) and \(P_2\). When \(P_2\leq 0.5\), it rejects \(H_1\) if \(P_1\leq 0.10\); when \(P_2>0.5\), it never rejects \(H_1\). Assume that under \(H_1\), \(P_1\) is Uniform\((0,1)\) and independent of \(P_2\). Under the complete null, also assume \(P_2\) is Uniform\((0,1)\). Under an alternative to \(H_2\), suppose \(P_2=0.1\) with probability one. What does the complete-null calculation show, and what does it miss?
Under the complete null, the rejection probability is $$ P(P_2\leq 0.5,\ P_1\leq 0.10) =P(P_2\leq 0.5)P(P_1\leq 0.10) =0.5(0.10)=0.05. $$ So this calculation is compatible with a level \(0.05\) claim for the complete-null configuration.
Now consider the partial-null configuration in which \(H_1\) is true and \(H_2\) is false. Since \(P_2=0.1\), the rule always enters the branch that tests \(H_1\) at \(0.10\). Thus $$ P(\text{reject true }H_1)=P(P_1\leq 0.10)=0.10. $$ This exceeds \(0.05\). The complete-null check did not establish strong control, and here the partial-null calculation shows actual failure. The issue is that the rule's threshold for the true hypothesis depends on evidence about the other hypothesis.
This example illustrates why “we checked the global null” is not a sufficient conclusion for a multiple-hypothesis procedure. It does not imply every adaptive rule fails under partial nulls. Rather, a proof needs either calculations covering the partial-null configurations or a valid argument that reduces them to configurations already checked.
Conditioning Must Match the Event
A conditional probability is a probability under a changed information set. If \(S\) is a selection event and \(R\) is a rejection event, the conditional rejection probability is \(P_{\theta}(R\mid S)\), not \(P_{\theta}(R)\). It may be larger or smaller than the unconditional probability. Therefore, an unconditional level statement cannot be used as a conditional level statement without justification.
Worked Example: Unconditional Calibration Is Not Conditional Calibration
Let \(U\sim\text{Uniform}(0,1)\) under a null hypothesis. A report is selected for follow-up when \(S=\{U\leq 0.20\}\). A proposed argument says that testing at the threshold \(U\leq 0.05\) has error probability \(0.05\), so its conditional error after selection is also \(0.05\).
Unconditionally, \(P(U\leq 0.05)=0.05\). But after selection, $$ P(U\leq 0.05\mid U\leq 0.20) =\frac{P(U\leq 0.05)}{P(U\leq 0.20)} =\frac{0.05}{0.20}=0.25. $$ Thus the conditional probability is \(0.25\), not \(0.05\). The proposed argument uses the unconditional null distribution after restricting attention to selected cases.
This calculation does not say that every adaptive procedure must condition on selection. It says that if a proof does condition on selection, the conditional law must be derived or otherwise justified. A valid alternative may calculate the unconditional probability of the full selection-and-rejection event directly.
Intersections Cannot Be Skipped in a Closure Argument
In a closed testing procedure, a hypothesis can be rejected only when the local tests for all intersection hypotheses containing it reject. Thus a proof that checks individual hypotheses but omits a necessary intersection has not verified the closure requirement. A local level bound for each individual test does not by itself control the probability of at least one false rejection.
Worked Example: Individual Tests Do Not Replace the Intersection Test
Consider two hypotheses \(H_1\) and \(H_2\). Under their joint null, suppose their p-values are independent and Uniform\((0,1)\). A proposed procedure rejects \(H_1\) when \(p_1\leq 0.05\) and rejects \(H_2\) when \(p_2\leq 0.05\). Each individual test has size \(0.05\). Is the familywise error rate \(0.05\) under the joint null?
Under the joint null, any rejection is an error. Independence gives $$ P(\text{at least one rejection}) =1-P(p_1>0.05,\ p_2>0.05) =1-(0.95)(0.95)=0.0975. $$ The familywise error rate is \(0.0975\), not \(0.05\). The individual local tests do not control the joint error event. In a closed-testing justification, the intersection \(H_1\cap H_2\) needs a valid local test, and the rejection logic must require it where closure demands it.
An audit should ask which intersections are logically required for each reported rejection, not merely whether the proof mentions “closure.” If a subset of hypotheses is selected adaptively, the argument must still account for the relevant intersections under the procedure's actual rejection rule. An untested intersection is a missing step unless another stated result validly supplies its bound.
Conditional Error: Check the Bound at Each History
Let \(D\) denote the interim information available when an adaptation is made, and let \(R\) denote rejection after continuation. For a null configuration \(\theta\), define the conditional continuation error at history \(D=d\) as $$ e_{\theta}(d)=P_{\theta}(R\mid D=d). $$ A conditional-error argument compares this quantity with the rejection probability of the adapted continuation rule, under the same null configuration and the same interim history.
Suppose a reference design has conditional error \(e_{\theta}(D)\), and an adapted rule has conditional rejection probability \(q_{\theta}(D)\). If \(q_{\theta}(D)\leq e_{\theta}(D)\) almost surely under each relevant null configuration, then taking expectations gives $$ P_{\theta}(R_{\mathrm{adapt}}) =E_{\theta}\{q_{\theta}(D)\} \leq E_{\theta}\{e_{\theta}(D)\} =P_{\theta}(R_{\mathrm{reference}}). $$ This is the core justification: the bound must hold conditionally, and the expectation is taken under the correct interim-data distribution. If the reference procedure controls the relevant error event at \(\alpha\), the inequality transfers that bound to the adapted rule, subject to the stated setup and null configuration.
Averaging can hide a violation at particular histories. Conversely, violating a proposed pointwise bound does not by itself prove that the adapted procedure exceeds \(\alpha\); another full probability calculation might still show control.
Worked Example: A Conditional Bound Is Exceeded, but the Overall Error Needs Calculation
Suppose the reference conditional error is \(0.05\) at both interim histories \(A\) and \(B\), and \(P(A)=0.75\), \(P(B)=0.25\) under a specified null configuration. An adapted rule has conditional rejection probabilities \(q(A)=0.08\) and \(q(B)=0\).
At history \(A\), \(q(A)=0.08>0.05\), so the claimed pointwise conditional-error condition fails. The overall rejection probability is $$ P(R_{\mathrm{adapt}}) =P(A)q(A)+P(B)q(B) =(0.75)(0.08)+(0.25)(0)=0.06. $$ It exceeds \(0.05\), so this configuration demonstrates actual error inflation.
If instead \(P(A)=0.25\) and \(P(B)=0.75\), with the same conditional rejection probabilities, the overall probability is $$ (0.25)(0.08)+(0.75)(0)=0.02. $$ The pointwise argument would still be unavailable because the bound fails at \(A\), but this particular unconditional calculation would not show error inflation. A sound audit distinguishes “the proposed conditional proof does not work” from “the procedure is proven invalid.”
What a Complete Justification Must Establish
For a strong familywise error claim, an argument must cover all parameter configurations under which one or more null hypotheses are true. If it checks only a subset, it needs a valid reduction. A least-favorable configuration can provide such a reduction only when the proof establishes that the error probability for every omitted configuration is bounded by the one actually analyzed; merely calling a configuration “least favorable” does not establish that property.
For adaptive tests, record the conditioning event and the distribution used at every step. If a selection event changes the law of the statistic, use that conditional law or calculate the joint event without conditioning. If a proof uses closed testing, trace every rejection back through the intersection tests required by the closure rule. If it uses conditional error, compare probabilities at the same interim history and under each relevant null configuration before integrating.
These approaches answer related but distinct questions. Direct enumeration of null configurations targets the strong-control definition explicitly. Closed testing organizes control through intersection hypotheses. Conditional error offers a way to compare an adapted rule with a reference rule history by history. None makes the others unnecessary in every design: the appropriate argument depends on the procedure, and each has its own conditions to verify.
Useful warning signs include a proof that names only the complete null; a conditional probability copied from an unconditional calculation; a closure argument with no local test for a required intersection; or an average conditional-error bound presented as though it were a pointwise bound. Simulation can help locate troublesome configurations and histories, but a finite set of simulated scenarios does not establish control over all configurations unless supplemented by a valid argument about what the simulations cover.
Check Your Understanding
For each question, identify the proof issue and state what additional calculation or justification would resolve it.
- A procedure controls error under the configuration where all three null hypotheses are true. What additional issue must be addressed before claiming strong familywise error control?
- Under a null, \(U\sim\text{Uniform}(0,1)\). A statistic is selected when \(U\leq 0.10\). Calculate \(P(U\leq 0.02\mid U\leq 0.10)\).
- Two independent p-values are Uniform\((0,1)\) under the joint null. Each hypothesis is rejected at \(0.05\). Calculate the familywise error rate.
- An adapted continuation rule exceeds the reference conditional error on a set of interim histories. Does that alone prove overall type I error exceeds \(\alpha\)? Explain.
Solutions. (1) The proof must address every partial-null configuration, or establish a valid reduction showing that the checked configuration bounds the omitted ones. Strong control is a supremum over configurations, not just the complete null. (2) The conditional probability is \(0.02/0.10=0.20\). The unconditional probability \(0.02\) cannot be reused after selection. (3) The probability of at least one rejection is \(1-(0.95)^2=0.0975\), so the familywise error rate exceeds \(0.05\). (4) No. It invalidates that pointwise conditional-error justification, but overall error requires integration over histories or another valid calculation. The resulting overall probability may or may not exceed \(\alpha\).