Why Two Endpoints Require Three Tests
Suppose a clinical trial has two endpoints, each representing a separate claim of treatment benefit. Testing each endpoint at significance level \(\alpha\) and rejecting either null when its unadjusted p-value is below \(\alpha\) does not, in general, control the probability of making at least one false rejection. Closed testing addresses this by requiring a claim about an endpoint to pass not only its own test, but also the test of every intersection hypothesis that contains it.
With two endpoints, the procedure is small enough to write out completely. There are two elementary null hypotheses, one for each endpoint, and one joint intersection null. The key is to distinguish the tests from the claims: a test of the joint intersection is required even when the final goal is to make separate claims about endpoints.
For two endpoints, the rule has a simple form. To reject \(H_1\), reject both \(H_1\) and \(H_{12}\); to reject \(H_2\), reject both \(H_2\) and \(H_{12}\). Here, “reject” means that the relevant local test rejects at level \(\alpha\). If the tests are expressed as p-values, the corresponding local p-value must be at most \(\alpha\).
Choosing the Local Tests
Each local test must be valid for the null hypothesis it tests. In particular, the test of \(H_{12}\) must control its Type I error probability at \(\alpha\) when both endpoint nulls are true. The joint test is not automatically supplied by the two endpoint tests; its validity must be justified.
One simple option is a Bonferroni test of \(H_{12}\). Suppose \(p_1\) and \(p_2\) are valid marginal p-values under \(H_1\) and \(H_2\), respectively. Under \(H_{12}\), both nulls are true. Reject \(H_{12}\) if either marginal p-value is at most \(\alpha/2\). The resulting intersection p-value is
This test is valid without requiring independence between the endpoint statistics. Under \(H_{12}\), the probability of rejecting is bounded by
The final inequality follows because each marginal p-value is valid under its own null, and under \(H_{12}\) both nulls hold. Other level-\(\alpha\) tests of \(H_{12}\) can also be used. The procedure’s decisions depend on the specified local tests; a different valid intersection test can therefore produce different decisions.
Worked Example: Computing the Three Tests
Consider two hypothetical one-sided tests of treatment benefit, with \(H_i:\theta_i\leq0\) versus \(H_{a,i}:\theta_i>0\), where each \(\theta_i\) is the treatment benefit on its endpoint’s measurement scale. Suppose the estimated benefits are 4.20 units and 3.48 units, respectively, and each has standard error 2.00 units. Assume the prespecified analysis supports the normal reference distribution for the test statistics and yields valid one-sided p-values. Let \(\alpha=0.05\), and use the Bonferroni test for \(H_{12}\).
The endpoint test statistics are \(z_1=4.20/2.00=2.10\) and \(z_2=3.48/2.00=1.74\). For one-sided standard normal tests, the corresponding p-values are approximately \(p_1=0.0179\) and \(p_2=0.0409\).
The intersection p-value is
All three local p-values are below 0.05. Therefore, reject \(H_{12}\), \(H_1\), and \(H_2\) under the closed-testing rule. In context, the analysis supports rejecting each endpoint’s null of no positive benefit at the familywise level controlled by this procedure. The estimated benefits are in endpoint-specific units; the tests do not make those units interchangeable.
Why the Rule Controls Familywise Error
The familywise Type I error rate is the probability of rejecting at least one true elementary null. Closure controls this rate if every local test is level \(\alpha\) for its own null. The short argument below shows why; it also explains why the intersection test matters even when only endpoint conclusions are reported.
Let \(T\) be the nonempty set of indices whose elementary nulls are true. For two endpoints, \(T\) is one of \(\{1\}\), \(\{2\}\), or \(\{1,2\}\).
If the procedure falsely rejects any elementary null \(H_i\) with \(i\in T\), the closure rule requires it also to reject every intersection containing \(H_i\), including \(H_T=\bigcap_{j\in T}H_j\).
The event of at least one false elementary rejection is therefore contained in the event that the local test of the true intersection \(H_T\) rejects. Since that local test is level \(\alpha\), the probability of a familywise Type I error is at most \(\alpha\).
In symbols, if \(E\) is the event of at least one false elementary rejection, then \(E\subseteq\{\text{reject }H_T\}\), and hence
This argument applies to each possible configuration of true and false endpoint nulls. It is a strong-control guarantee: the familywise error bound holds not only when both nulls are true, but also when exactly one is true.
Worked Example: One Endpoint Claim Survives
Suppose a hypothetical analysis has valid marginal p-values \(p_1=0.012\) and \(p_2=0.070\), with \(\alpha=0.05\). Use the same Bonferroni local test for \(H_{12}\).
First calculate the intersection p-value:
The local tests of \(H_1\) and \(H_{12}\) both reject because \(0.012\leq0.05\) and \(0.024\leq0.05\). Thus, the closed procedure rejects \(H_1\). Although the test of \(H_{12}\) rejects, \(p_2=0.070>0.05\), so the procedure does not reject \(H_2\). The supported endpoint claim is limited to endpoint 1; rejection of the joint null alone does not establish that both endpoints show benefit.
Adjusted P-Values and Interpretation
For a fixed family of two tests and fixed local tests, the closed-testing adjusted p-value for an elementary null is the largest local p-value among the hypotheses that must reject for that elementary null to be rejected. Thus, with the Bonferroni intersection test,
An elementary null is rejected at level \(\alpha\) exactly when its closed-testing adjusted p-value is at most \(\alpha\). These adjusted p-values summarize the closure decisions; they are not new raw endpoint p-values.
Worked Example: Neither Endpoint Claim Can Be Rejected
Suppose another hypothetical analysis yields \(p_1=0.032\) and \(p_2=0.039\), with \(\alpha=0.05\). Each marginal test, considered alone, would reject its null at 0.05. For the Bonferroni intersection test,
Since \(p_{12}=0.064>0.05\), the test of the intersection does not reject. The closed-adjusted p-values are \(p_{1,\mathrm{closed}}=\max(0.032,0.064)=0.064\) and \(p_{2,\mathrm{closed}}=\max(0.039,0.064)=0.064\). Neither elementary null is rejected. The marginal results alone are not enough to support either endpoint claim under this closed procedure.
Diagnostics, Alternatives, and Limitations
A useful diagnostic is to write down the three local decisions before writing any endpoint conclusion. Check that the marginal tests are valid for their respective nulls, that the intersection test is valid under \(H_{12}\), and that all tests use the prespecified significance level and compatible directions of benefit. If a p-value has been selected after examining data or endpoints, its validity cannot simply be assumed from its numerical value.
The Bonferroni intersection test is easy to apply and does not require independence, but it can be conservative, particularly when endpoint statistics are positively correlated. A more tailored joint test may gain power if its assumptions and calibration are justified. Whatever local test is chosen, it must control Type I error for the intersection it tests; choosing a more powerful test does not remove the closure requirement.
For comparison, testing each endpoint unadjusted at \(\alpha\) omits the intersection test and does not generally provide strong familywise control. A Bonferroni procedure that tests each endpoint at \(\alpha/2\) is another valid way to control familywise error, but it is a distinct procedure and can give different decisions from the closed procedure described here. With the Bonferroni intersection test, two-endpoint closure gives the same adjusted p-values as the Holm procedure for two tests. That equivalence depends on this choice of local intersection test; it is not a reason to omit the intersection test when implementing closure.
In an adaptive trial, these calculations are only as reliable as the local tests’ calibration for the design and analysis actually used. A p-value that is valid under a fixed design does not become valid for a data-dependent procedure merely because it is inserted into the closure formulas. This tutorial’s two-endpoint decision rule does not by itself establish the validity of a particular adaptive test.
Check Your Understanding
For each question, use \(\alpha=0.05\) and the Bonferroni intersection test unless stated otherwise. The solutions follow the questions.
- If \(p_1=0.020\) and \(p_2=0.080\), calculate \(p_{12}\) and state which elementary nulls the closed procedure rejects.
- If \(p_1=0.030\) and \(p_2=0.040\), calculate both closed-adjusted p-values and state the decision.
- Why is the Bonferroni p-value \(p_{12}=\min\{1,2\min(p_1,p_2)\}\) valid under \(H_{12}\) without an independence assumption?
- If only \(H_1\) is true, which event must occur for a false rejection of \(H_1\) under closure, and how does this fact support strong control?
Solutions. (1) \(p_{12}=2(0.020)=0.040\). The tests of \(H_1\) and \(H_{12}\) reject, so reject \(H_1\). The test of \(H_2\) does not reject because \(0.080>0.05\), so do not reject \(H_2\). (2) \(p_{12}=2(0.030)=0.060\); therefore \(p_{1,\mathrm{closed}}=\max(0.030,0.060)=0.060\) and \(p_{2,\mathrm{closed}}=\max(0.040,0.060)=0.060\). Neither null is rejected. (3) Under \(H_{12}\), both marginal nulls are true and each marginal p-value is valid. The union bound gives \(P(p_1\leq0.025\text{ or }p_2\leq0.025)\leq0.025+0.025=0.05\), regardless of dependence. (4) A false rejection of \(H_1\) requires rejecting the intersection \(H_1\), which is the true intersection when only \(H_1\) is true. The probability of that local rejection is at most \(\alpha\), bounding the familywise error for this configuration.