Why Population Selection Requires Error Control
A trial may collect outcomes in several biomarker-defined populations, then report the population in which the treatment result looks most favorable. That choice may be clinically sensible, but it is also data-dependent: the same results are being used both to select a population and to support a claim about it. If each population is tested at level \(\alpha\) and only the most favorable result is reported, the chance of at least one false claim can exceed \(\alpha\).
Closed testing addresses this issue by linking each subgroup claim to tests of the relevant intersections of claims. The population may be selected after examining the data, but a claim is permitted only if its own elementary hypothesis and every intersection hypothesis containing it are rejected. The key requirement is that each local test used in the procedure controls its Type I error at level \(\alpha\) under its corresponding null hypothesis.
For two populations, such as biomarker-positive and biomarker-negative patients, the family consists of \(H_1\), \(H_2\), and \(H_{12}=H_1\cap H_2\). Rejecting \(H_1\) supports a treatment-benefit claim in population 1. Rejecting \(H_{12}\), by itself, supports only the conclusion that at least one of the two elementary nulls is false; it does not identify which one.
How the Closed Procedure Works
For every nonempty \(I\), specify a local test of \(H_I\) at level \(\alpha\). The closed testing rule rejects \(H_j\) only if the local tests reject every intersection \(H_I\) for which \(j\in I\). With two populations, rejecting \(H_1\) therefore requires rejecting both \(H_1\) and \(H_{12}\). Rejecting \(H_2\) requires rejecting both \(H_2\) and \(H_{12}\).
Write an elementary null for each population that could be reported as benefiting.
For two claims, test \(H_1\), \(H_2\), and \(H_1\cap H_2\).
Each local test must have Type I error no greater than \(\alpha\) under its own null.
Reject an elementary null only when all intersections containing it have also been rejected.
A useful local test of \(H_{12}\) can be built from valid one-sided p-values \(p_1\) and \(p_2\) for the two elementary hypotheses. The Bonferroni intersection p-value is
Under \(H_{12}\), both elementary nulls are true. The probability that either valid elementary p-value is at most \(\alpha/2\) is no greater than the sum of the two probabilities, hence no greater than \(\alpha\). Thus rejecting \(H_{12}\) when \(p_{12}\leq\alpha\) is a level-\(\alpha\) test, without requiring independence between the subgroup tests. This local test is often conservative, but it is straightforward and valid.
Worked Example: A Subgroup Looks Favorable
Worked Example: A Subgroup Looks Favorable
Consider a hypothetical randomized trial with a binary response and a biomarker status determined before randomization. Among 80 biomarker-positive patients per arm, 35 of 80 treatment patients respond and 20 of 80 control patients respond. Assume independent patient outcomes within arms and use the usual large-sample, one-sided pooled two-proportion z test for \(H_1:\Delta_1\leq0\). The pooled estimate under the equality boundary is \(55/160=0.34375\). The expected response and nonresponse counts under that boundary are 27.5 and 52.5 per arm, so the normal approximation is reasonable for this illustration.
The estimated difference is \(35/80-20/80=0.1875\), or 18.75 percentage points. The pooled standard error and test statistic are
The one-sided p-value is \(P(Z\geq2.497)\approx0.0063\), where \(Z\) is standard normal. This local test rejects \(H_1\) at the 0.05 level. In context, the data provide evidence of a higher response probability with treatment in the prespecified biomarker-positive population; the estimated difference is 18.75 percentage points. This result alone does not yet justify rejecting \(H_1\) under closed testing, because \(H_{12}\) must also be rejected.
Worked Example: Testing the Intersection and Closing the Claim
Worked Example: Testing the Intersection and Closing the Claim
Continue the hypothetical trial. Among 120 biomarker-negative patients per arm, 36 of 120 treatment patients respond and 39 of 120 control patients respond. The one-sided pooled z test for \(H_2:\Delta_2\leq0\) has pooled response proportion \(75/240=0.3125\), estimated difference \(0.3000-0.3250=-0.0250\), and standard error
The one-sided p-value is approximately \(P(Z\geq-0.4178)=0.6618\). The pooled expected counts under equality are 37.5 responses and 82.5 nonresponses per arm, so the large-sample test is reasonable here too. There is no basis to reject \(H_2\) at 0.05.
Using the Bonferroni local test, \(p_{12}=\min\{1,2\min(0.0063,0.6618)\}=0.0126\), which is below 0.05. Thus \(H_{12}\) is rejected, as is \(H_1\), so closed testing permits the biomarker-positive benefit claim. It does not permit a biomarker-negative benefit claim, because the local test of \(H_2\) is not rejected. The Bonferroni adjustment has accounted for the fact that either subgroup could have been the favorable one.
For two hypotheses, the closed-test adjusted p-value for \(H_1\) is the larger of its own local p-value and the intersection local p-value: \(\max(0.0063,0.0126)=0.0126\). For \(H_2\), the corresponding value is \(\max(0.6618,0.0126)=0.6618\). These values summarize the requirement to pass both relevant tests.
Why the Overall-Population Test Is Not the Intersection Test
The intersection \(H_{12}\) says that neither biomarker subgroup has a treatment benefit. It is not generally equivalent to a null hypothesis about the average treatment effect in all patients. A pooled overall comparison may dilute a large subgroup effect, or obscure differences in baseline response rates and effects across populations. Therefore, an overall-population p-value cannot simply replace a valid local test of \(H_{12}\).
Worked Example: The Overall Contrast Can Tell a Different Story
In the hypothetical data above, the treatment arm has 71 responses among 200 patients: 35 of 80 biomarker-positive and 36 of 120 biomarker-negative. The control arm has 59 among 200: 20 of 80 positive and 39 of 120 negative. The overall response proportions are 0.355 and 0.295, a difference of 0.0600. For comparison, an unstratified pooled two-proportion calculation gives pooled proportion \(130/400=0.325\) and
The one-sided p-value is approximately 0.100. This calculation does not reject an overall superiority null at 0.05. It also does not test \(H_{12}\) in general: that intersection concerns both subgroup effects, not only a marginal contrast. The valid Bonferroni intersection test above is the one used for the closed subgroup claim. The arithmetic illustrates why one must define the hypothesis before selecting a test.
Why Closure Controls Familywise Error
The familywise Type I error rate is the probability of rejecting at least one true elementary null. The protection follows directly from the closure rule. Suppose the set \(T\) of elementary nulls that are true is nonempty. If the closed procedure falsely rejects any \(H_j\) with \(j\in T\), it must also reject every intersection containing \(H_j\), including \(H_T=\bigcap_{j\in T}H_j\). But \(H_T\) is true. Therefore, the event of at least one false elementary rejection is contained in the event that the local test of the true intersection \(H_T\) rejects.
The last inequality holds because the local test of each intersection is assumed to have Type I error at most \(\alpha\) under that intersection. This argument does not require the tests to be independent. It does require that the family of claims and the local tests be specified so that their validity applies to the actual analysis.
Assumptions, Diagnostics, and Limitations
For the response-rate examples, the subgroup tests rely on randomized treatment assignment, a binary outcome defined consistently, independent patient outcomes, and biomarker status measured before treatment assignment. The pooled z calculation additionally uses a large-sample normal approximation; expected response and nonresponse counts should not be too small. If these conditions are doubtful, an exact or otherwise valid subgroup test can supply the elementary p-values instead. The intersection procedure is only as valid as the local tests it uses.
- Check the subgroup definition. The biomarker threshold and populations should be prespecified or included in the family of hypotheses. Choosing a cutoff after examining outcomes creates additional selection that these two subgroup tests do not account for.
- Check test direction and population. A one-sided superiority test addresses benefit in the specified population; it does not establish benefit for every patient or on every endpoint.
- Check that the intersection test matches its null. An overall test or a test of an interaction is not automatically a valid test of the intersection of subgroup no-benefit hypotheses.
- Distinguish benefit from effect modification. Finding evidence of benefit in one subgroup but not another does not by itself establish that treatment effects differ between subgroups. That is a separate comparison of effects.
Closure is a general error-control structure, not a method for repairing invalid p-values or unplanned claims. Bonferroni provides a simple intersection test, but it may be conservative, especially with many populations or correlated tests. More powerful local tests are possible, provided their Type I error control under the relevant intersection is justified. The core principle remains unchanged: a subgroup claim selected because it looks favorable is confirmatory only when it passes the closed testing requirements.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For three biomarker populations, how many nonempty intersection hypotheses are in the closed family?
- If \(p_1=0.018\) and \(p_2=0.40\), what is the Bonferroni local p-value for \(H_1\cap H_2\)? At \(\alpha=0.05\), can \(H_1\) be rejected by closure?
- Why does rejecting the intersection \(H_1\cap H_2\) not identify which population benefits?
- What additional issue arises if investigators choose the biomarker cutoff after looking at the treatment outcomes?
Solutions. (1) There are \(2^3-1=7\) nonempty subsets, hence seven intersection hypotheses. (2) The intersection p-value is \(\min\{1,2(0.018)\}=0.036\), so it is rejected at 0.05. Since \(H_1\)'s own p-value, 0.018, is also below 0.05, \(H_1\) passes both required tests and may be rejected. (3) The intersection null says both elementary nulls hold; rejecting it shows that at least one is false, but the intersection test alone does not determine which. The elementary tests and closure identify permissible specific claims. (4) The chosen cutoff adds data-dependent candidate populations beyond those represented by the prespecified family. Unless that search is included in valid error control, the stated closed procedure does not account for the extra selection.