Closed Testing For Enrichment Claims

Why Close the Tests?

A trial may support a treatment-effect claim for all eligible patients, for a biomarker-positive subgroup, or for both. When several claims are tested, using the nominal significance level for each test separately does not generally control the probability of making at least one false claim. The closure principle addresses this multiplicity by requiring each claim to pass tests of the relevant intersection hypotheses as well as its own hypothesis.

Let \(A\) denote the all-comer population and \(B\) a prespecified biomarker-positive subgroup nested within it, so that every member of \(B\) is also in \(A\). Let \(\Delta_A\) and \(\Delta_B\) be the treatment-minus-control effects on a prespecified endpoint in those populations. For a superiority objective, define the one-sided elementary hypotheses as follows.

Nesting of the populations does not, by itself, make either elementary hypothesis imply the other. For example, an average effect can be positive in \(B\) while the all-comer average is not positive if the effect in the rest of \(A\) is sufficiently unfavorable. Conversely, a positive all-comer average does not require a positive average in every subgroup. The null hypotheses therefore need multiplicity control unless a stronger, justified model imposes a logical relationship between them.

The procedure requires valid p-values for each elementary null: under the null, the probability that its p-value is at most any threshold \(t\) must be no greater than \(t\). In the examples below, assume a prespecified one-sided analysis gives a test statistic that is approximately standard normal under its null. The trial should use a prespecified endpoint and analysis, an appropriate variance estimate, and a design and missing-data approach that support the reported p-values. If observations are clustered, for example, the standard error must account for that dependence. Closed testing adjusts multiplicity; it does not make invalid marginal tests valid.

The Closed Testing Rule for Two Claims

With two elementary hypotheses, the closed family consists of three hypotheses: \(H_A\), \(H_B\), and their intersection \(H_{AB}\). A local test is a test of one member of this family at level \(\alpha\). To reject \(H_A\), the procedure must reject \(H_A\) and \(H_{AB}\); to reject \(H_B\), it must reject \(H_B\) and \(H_{AB}\). There are no other intersection hypotheses in this two-claim family.

One simple local test of \(H_{AB}\) uses Bonferroni. Write \(p_A\) and \(p_B\) for valid one-sided p-values for the elementary tests. Reject \(H_{AB}\) if either elementary p-value is at most \(\alpha/2\). This test is valid without requiring independence between the two test statistics.

Here \(p_{AB}\) is a valid p-value for the intersection test using this Bonferroni rule. The closed-testing adjusted p-values for the elementary claims are therefore

Reject an elementary null at level \(\alpha\) when its adjusted p-value is at most \(\alpha\). This rule captures both requirements: the claim must pass its own local test, and the intersection hypothesis containing it must also be rejected.

Why the Closure Controls Familywise Error

The familywise error rate (FWER) is the probability of rejecting at least one true elementary null. For the Bonferroni intersection test, if \(H_{AB}\) is true, both elementary nulls are true. Each marginal test then has probability at most \(\alpha/2\) of producing a p-value no greater than \(\alpha/2\). The probability of the union of those two events is bounded by the sum of their probabilities:

This is the union bound; it does not assume independence. So the local test of \(H_{AB}\) has type I error no greater than \(\alpha\). Each elementary hypothesis is also tested at level \(\alpha\), and closure requires both its own test and the relevant intersection test to reject.

This argument is the closure principle: valid level-\(\alpha\) tests for all required intersection hypotheses yield strong FWER control, meaning control at level \(\alpha\) for every configuration of true and false elementary nulls. It also explains why a locally valid test of the intersection is essential.

Worked Decisions and Interpretation

Choice of Local Test, Diagnostics, and Limits

Bonferroni is transparent and remains valid under arbitrary dependence when the marginal p-values are valid. It can be conservative, particularly when the test statistics are strongly dependent, as may occur when one population contains the other. A different local test of \(H_{AB}\) can be used if its type I error is valid at level \(\alpha\) under the entire intersection null. For example, a prespecified joint test that accounts for the test-statistic covariance may improve power, but its null calibration and assumptions must be justified. The adjusted p-values must be computed from the chosen local test, not from a formula intended for Bonferroni.

A useful diagnostic is to check the logic of every reported claim: a rejected elementary null must have a rejected intersection test, and every reported adjusted p-value must be at least its corresponding marginal p-value. Also verify that the populations, endpoint, direction of testing, and local intersection procedure were specified consistently. Unexpected discrepancies between subgroup and all-comer estimates may be scientifically important, but closed testing alone does not explain heterogeneity or establish why the effects differ.

Closed testing is more general than a fixed gatekeeping sequence: it evaluates the intersection hypotheses required by the claims rather than simply allowing one test to proceed after another passes. Its protection is conditional on valid local tests. It does not repair a p-value made invalid by unaccounted selection, outcome switching, inappropriate variance estimates, or a misspecified analysis. Nor does statistical rejection establish clinical importance; effect estimates and their uncertainty still need interpretation in endpoint units.