Why an Added Hypothesis Needs Its Own Error Argument
An interim review may suggest a new endpoint, subgroup, or treatment comparison worth investigating. The statistical issue is not whether the new question is scientifically reasonable. It is whether its eventual rejection has been protected against a Type I error after the data-dependent decision to add it. The original design's error control does not automatically extend to a claim that was not part of its hypothesis family.
A confirmatory hypothesis is a claim for which the study intends to make a formal rejection decision. Write \(H_1,\ldots,H_m\) for the hypotheses in the original family and \(H_*\) for a newly added claim. The family-wise Type I error rate (FWER) is the probability of rejecting at least one true null hypothesis in the family. Strong control at level \(\alpha\) means that this probability is at most \(\alpha\) for every configuration of true and false null hypotheses.
The key distinction is between adapting a procedure while preserving its error bound and creating an additional rejection claim. Earlier material on conditional error explains how an original test can sometimes be adapted under a suitable conditional bound. But that bound concerns the original rejection event. It does not, by itself, provide a bound for rejecting \(H_*\), or for rejecting an intersection null that includes \(H_*\).
How Selection Creates an Unprotected Opportunity
An unprotected opportunity arises when the interim data affect which claim is tested, while the analysis treats the selected claim as though it had been chosen in advance. Reusing a favorable interim statistic is a direct example. Searching among multiple possible endpoints or subgroups and then reporting the most favorable one can create the same problem. The size of the error depends on the selection rule, the dependence among statistics, and whether the final evidence is independent of the interim evidence; it cannot be inferred merely from the nominal level of the final test.
Worked Example: Reusing the Most Favorable Interim Result
Suppose two candidate endpoints have standardized interim statistics \(Z_1\) and \(Z_2\). Under the global null, assume for this illustration that they are independent and each has a standard normal distribution. After seeing both, the team adds the endpoint with the larger statistic and declares that endpoint positive if its statistic exceeds the usual one-sided 5% critical value.
Let \(c\) be the exact 95th percentile of the standard normal distribution, so that \(P(Z_i\leq c)=0.95\); numerically, \(c\) is approximately 1.644854. The probability of at least one false rejection is $$ P\{\max(Z_1,Z_2)>c\} =1-P(Z_1\leq c,\ Z_2\leq c) =1-(0.95)^2 =0.0975. $$
Thus the procedure has a 9.75% probability of a false rejection under the global null, not 5%. The calculation uses the exact quantile definition \(P(Z_i\leq c)=0.95\); the displayed decimal for \(c\) is only an approximation. This example represents reuse of the interim statistics as the confirmatory evidence. It does not imply that every data-dependent addition has a 9.75% error rate: with independent new validation data, the calculation can be different, as discussed below.
A nominal p-value for a selected hypothesis is not automatically a valid p-value for the selection procedure. A useful diagnostic question is: Would this same hypothesis, and this same test, have been specified if the interim results had looked different? If not, the analysis must account for that selection or use evidence with a valid conditional distribution after selection.
The Intersection Nulls Required by Closure
Closed testing provides a general route to strong FWER control. For every nonempty subset \(I\) of hypothesis indices, define the intersection null \(H_I=\bigcap_{i\in I}H_i\). This null says that all elementary null hypotheses indexed by \(I\) are true. A closed-testing procedure specifies a local test for every such intersection. It rejects an elementary \(H_i\) only if it rejects every intersection \(H_I\) for which \(i\in I\).
When \(H_*\) is added, closure must include intersections containing it. For example, if the family has just \(H_A\) and \(H_*\), the relevant tests are for \(H_A\), \(H_*\), and \(H_A\cap H_*\). With more original hypotheses, intersections such as \(H_A\cap H_B\cap H_*\) are also part of the closure. The tests must be appropriate for their respective nulls; simply testing each elementary null at level \(\alpha\) is not enough.
The justification is short but central. Let \(I_0\) be the set of all indices whose elementary nulls are true. If any true elementary null is rejected, say \(H_i\) with \(i\in I_0\), the closure rule requires rejection of \(H_{I_0}\), because \(i\in I_0\). But \(H_{I_0}\) is true: all its component nulls are true. Therefore $$ P(\text{reject at least one true } H_i) \leq P(\text{reject }H_{I_0}) \leq \alpha. $$ The second inequality follows from the assumed level-\(\alpha\) validity of the local test for that true intersection. This argument also shows why an untested intersection involving \(H_*\) matters: without a valid test for the relevant true intersection, the proof does not apply.
If each constituent p-value is valid for its own null, a simple local test for an intersection of \(r\) hypotheses is Bonferroni: reject that intersection when at least one constituent p-value is at most \(\alpha/r\). Under the intersection null, the probability of this event is at most the sum of \(r\) probabilities, each no greater than \(\alpha/r\), hence at most \(\alpha\). This local test does not require independence. For a data-selected claim, however, a constituent p-value must itself be valid after the selection; Bonferroni cannot repair an invalid p-value.
Worked Closed-Testing Decision
Worked Example: Adding One Claim to a Two-Hypothesis Family
Suppose an original claim \(H_A\) and a newly added claim \(H_*\) have p-values \(p_A=0.018\) and \(p_*=0.031\). For this illustration, assume each p-value is valid for its elementary null under the analysis actually used, including any interim selection. Test the singleton nulls at \(\alpha=0.05\), and use the Bonferroni local test for \(H_A\cap H_*\).
For the intersection, there are \(r=2\) constituent hypotheses. The Bonferroni local test rejects if either p-value is at most \(0.05/2=0.025\). Equivalently, its adjusted p-value is $$ p_{A,*}=\min\{1,\ 2\min(p_A,p_*)\} =\min\{1,\ 2(0.018)\} =0.036. $$
Both singleton tests reject because \(0.018<0.05\) and \(0.031<0.05\). The intersection test also rejects because \(0.036<0.05\). Thus the closed procedure rejects both elementary nulls. This conclusion depends on the stated validity assumption for \(p_*\): if \(0.031\) is an unadjusted p-value from a hypothesis selected by looking at the same data, closure does not make it valid.
When Independent Validation Can Support an Added Claim
An interim-selected hypothesis is not necessarily unusable for confirmatory testing. A strong option is to choose the claim using interim data and test it using genuinely independent validation data, with the analysis and null defined clearly. Under the null for each candidate, if its validation p-value is valid conditional on all interim information, then selection using only that interim information does not change the conditional null distribution of the validation p-value.
Worked Example: Selection Followed by Independent Validation
At interim, a team chooses one of two candidate endpoints using interim results. It then collects independent validation data and obtains a one-sided validation statistic \(Z_*=1.80\) for the selected endpoint. Assume the endpoint's null is true, the validation statistic is standard normal under that null, and the validation data are independent of the interim information. The one-sided p-value is $$ p_*=P(Z\geq 1.80)=1-\Phi(1.80)\approx 0.0359. $$
Conditional on the interim data, the selected endpoint is fixed, and the assumed independent validation data still give a valid null p-value. Since \(0.0359<0.05\), the selected claim would pass a single level-0.05 test. If the claim is added to a family that also contains \(H_A\), it must still pass the required closure tests. For example, if \(p_A=0.018\) is also valid, the Bonferroni p-value for the intersection is $$ \min\{1,\ 2\min(0.018,0.0359)\}=0.036. $$
This example does not mean that any future data automatically validate a selected claim. Independence must be credible, the endpoint and null must be sufficiently well-defined for testing, and the test must be valid under its null. If the validation analysis is altered in response to its own results, or if interim observations are reused as validation evidence, the stated conditional validity argument no longer follows.
Which Error-Control Route Is Appropriate?
The principal alternatives differ in what is prespecified and what evidence supports the new claim. They are not interchangeable shortcuts.
| Approach | What it addresses | Key requirement |
|---|---|---|
| Include the candidate hypotheses in the original closed family | Protects claims that may be selected later | Local tests for all relevant intersections are valid |
| Use a selection-aware local test | Accounts for the rule that chose the new claim | The selection rule and candidate set are represented in the test's calibration |
| Use independent validation data | Can make the selected claim's validation p-value valid conditional on interim selection | Validation evidence is genuinely independent and its test is specified and valid |
| Use the original test's conditional error bound | Can protect an adapted version of an original rejection event | The bound must cover the relevant event; it does not automatically cover a new hypothesis or its intersections |
A simple multiplicity adjustment such as Bonferroni is useful when the candidate family is known and the component p-values are valid. Closed testing gives a general structure for strong control, while more powerful local tests may be possible under additional assumptions. Neither method can compensate for a post-selection p-value whose null distribution is wrong. If a new claim is generated from the observed results without a prespecified candidate universe or valid selection-aware analysis, it is more defensible as exploratory than as confirmatory.
Diagnostics and Common Pitfalls
Before treating an added hypothesis as confirmatory, document the hypothesis family, the interim selection rule, the data used for each test, and the local test for every intersection required by closure. Check that the scientific meaning of each null is stable: changing the population, endpoint definition, or treatment contrast may create a different hypothesis, not merely a different test of the same one.
- Wrong reasoning: “The original trial controlled Type I error at 5%, so the added endpoint is also covered.” Correction: The original guarantee covers only its specified rejection events unless a valid argument extends it to the added claim and relevant intersections.
- Wrong reasoning: “The selected endpoint's ordinary p-value is below 0.05, so its null is rejected.” Correction: Establish that the p-value is valid after the selection, or adjust for the selection rule or use independent validation evidence.
- Wrong reasoning: “Testing both elementary hypotheses at 0.05 is enough.” Correction: Strong closed testing also requires valid tests of intersections, including those containing the new hypothesis.
- Wrong reasoning: “A conditional error bound for the old test covers every later claim.” Correction: A bound for the old rejection event does not create an error allowance for a new rejection event. The new claim and its intersections need their own validity argument.
A practical warning sign is an analysis plan that describes how to calculate the final p-value but does not say how the endpoint was chosen or which intersection nulls must be rejected. Another is a claim of conditional validity without stating what is conditioned on and why the test remains valid given that information. These omissions make the error-control argument impossible to verify.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Two independent one-sided tests each use level 0.05 under the global null. What is the probability of at least one rejection if either may be reported?
- For a family containing \(H_A\) and \(H_*\), name all intersection nulls required for closed testing.
- If \(p_A=0.012\) and \(p_*=0.040\), what is the Bonferroni p-value for their intersection?
- Why can an independent validation p-value remain valid after choosing an endpoint at interim?
- Does a conditional error bound for an original hypothesis automatically protect a newly added hypothesis? Explain.
Solutions. (1) The probability is \(1-(1-0.05)^2=1-0.95^2=0.0975\), or 9.75%. (2) Test \(H_A\), \(H_*\), and \(H_A\cap H_*\); reject an elementary null only if all intersections containing it are rejected. (3) The intersection p-value is \(\min\{1,2\min(0.012,0.040)\}=0.024\). (4) If validation data are independent of the interim information and the validation test is valid under the null, then conditional on interim selection the selected validation statistic still has its stated null distribution. (5) No. The original bound concerns the original rejection event; the new claim and every relevant intersection involving it need valid local tests or another justified error-control argument.