Closed Testing Procedures Explained

Introduction

Clinical trials frequently evaluate more than one hypothesis. A study may have multiple primary endpoints, several doses, multiple treatment comparisons, or a combination of efficacy and key secondary objectives.

When several hypotheses are tested, the probability of obtaining at least one false-positive result can exceed the nominal significance level.

This is the multiple testing problem.

A central goal of multiplicity control is therefore to ensure that the family-wise error rate (FWER) remains at or below a prespecified level such as 5%.

The closed testing procedure provides one of the most general and powerful frameworks for achieving strong FWER control.

The Multiple Testing Problem

Suppose a clinical trial has two hypotheses:

and:

If each is tested at the 5% level independently, the probability of making at least one false rejection can exceed 5%.

If the hypotheses are independent and both are true, for example, the probability of at least one false rejection is:

or approximately 9.75%.

This is much larger than the desired 5% family-wise error rate.

What Is the Family-Wise Error Rate?

Suppose the set of hypotheses is:

The family-wise error rate is:

A multiple testing procedure has strong FWER control at level \(\alpha\) if:

for every possible configuration of true and false hypotheses.

This distinction is important. A method that controls FWER only when all null hypotheses are true provides weak control, not strong control.

The Closure Principle

The closed testing principle was introduced by Marcus, Peritz, and Gabriel (1976) as a general framework for strong control of the family-wise error rate.

The fundamental idea is to construct a family of hypotheses that is closed under intersection.

Suppose there are \(m\) elementary null hypotheses:

The closed family contains:

• Every elementary hypothesis
• Every nonempty intersection of elementary hypotheses

For two hypotheses, the closed family is:

For three hypotheses, the family becomes:

Elementary Hypotheses

The original scientific questions are usually represented by the elementary hypotheses.

For example, suppose a clinical trial compares an experimental treatment with control for two endpoints. Let:

and:

These are the hypotheses investigators ultimately want to reject.

Intersection Hypotheses

The closure principle adds hypotheses representing intersections of the elementary hypotheses.

For two hypotheses:

The intersection hypothesis is true when both elementary null hypotheses are true.

For three hypotheses:

This represents the global null that all three elementary hypotheses are true.

Why Is the Family Called "Closed"?

The family is called closed because if a collection of hypotheses belongs to the family, the relevant intersection hypothesis also belongs to the family.

For example, with three elementary hypotheses:

the family also contains:

The resulting structure allows the individual hypotheses to be tested while accounting for all relevant intersections.

Local Tests

Every hypothesis in a closed testing procedure is tested using a local test.

A local test is simply the statistical test assigned to a particular intersection hypothesis.

For example:

The local test does not necessarily have to use the same test statistic or procedure for every intersection.

What matters is that each local test has the required Type I error control.

The Closed Testing Decision Rule

For an elementary hypothesis \(H_i\), the closed testing procedure rejects \(H_i\) only if all intersection hypotheses containing \(H_i\) are rejected.

For two hypotheses:

Similarly:

A Two-Hypothesis Example

Suppose a trial has two confirmatory hypotheses:

and:

The closed testing family is:

Suppose all three local tests are performed at \(\alpha=0.05\).

To reject \(H_1\), we need:

To reject \(H_2\), we need:

Worked Example: Two Endpoints

Consider a randomized Phase III trial evaluating an experimental treatment against placebo. The study has two hypotheses:

Suppose the unadjusted p-values are:

Using a closed testing procedure with level \(\alpha=0.05\):

• \(H_1\) is rejected because both 0.018 and 0.011 are below 0.05.
• \(H_2\) is rejected because both 0.031 and 0.011 are below 0.05.

Therefore, both endpoints can be declared statistically significant while maintaining strong FWER control at 5%.

What If the Intersection Hypothesis Is Not Significant?

Now suppose instead that:

Although both elementary p-values are below 0.05, the intersection hypothesis is not rejected.

Therefore:

and:

under the closed testing procedure.

Why This Controls FWER

The logic behind strong FWER control is elegant. Suppose \(H_i\) is a true elementary null hypothesis. If the procedure incorrectly rejects \(H_i\), then the closure principle requires the rejection of every intersection hypothesis containing \(H_i\). In particular, the intersection of all true elementary hypotheses must also be rejected.

The local test for that intersection is performed at level \(\alpha\). Therefore, the probability of incorrectly rejecting a true elementary hypothesis cannot exceed the probability of incorrectly rejecting the relevant true intersection hypothesis.

Consequently:

under every configuration of true and false elementary hypotheses.

Strong FWER Control

The phrase strong control is important. Consider three possible configurations:

• All elementary null hypotheses are true.
• Some null hypotheses are true and some are false.
• Only one elementary null hypothesis is true.

A properly constructed closed testing procedure controls FWER under all of these configurations.

Closed Testing With Three Hypotheses

With three elementary hypotheses:

the closed family contains seven hypotheses:

where:

To reject \(H_1\), the following four hypotheses must all be rejected:

To reject \(H_2\):

To reject \(H_3\):

The Number of Hypotheses Grows Quickly

For \(m\) elementary hypotheses, the complete closed family contains:

nonempty hypotheses.

This exponential growth is one reason that practical closed testing procedures often exploit special structure to avoid explicitly performing every possible test.

Local Tests Can Be More Powerful Than Bonferroni

One of the major attractions of closed testing is that it is a framework, rather than a single fixed adjustment.

The local tests can be selected to reflect the dependence structure and scientific meaning of the hypotheses.

For example, an intersection hypothesis might be tested using:

• Bonferroni
• Sidak
• Fisher's combination test
• Parametric multiple comparison procedures
• Likelihood-ratio tests
• Specialized tests based on the endpoint structure

As long as the local tests satisfy the required Type I error properties, the resulting closed testing procedure can provide strong FWER control.

Closed Testing and Adjusted p-values

Closed testing can also be expressed using adjusted p-values.

For an elementary hypothesis \(H_i\), its adjusted p-value is essentially the largest local p-value among all intersection hypotheses containing \(H_i\).

Formally:

where \(p_I\) is the p-value for the corresponding intersection hypothesis.

For two hypotheses:

and:

The hypothesis can then be rejected at level \(\alpha\) when:

Adjusted p-values for the Worked Example

Using:

The adjusted p-value for \(H_1\) is:

The adjusted p-value for \(H_2\) is:

Both are below 0.05, so both elementary hypotheses can be rejected.

A Three-Hypothesis Adjusted p-value Example

Suppose the local p-values are:

For \(H_1\), the containing intersections are:

Therefore:

For \(H_2\):

For \(H_3\):

All three adjusted p-values are below 0.05. Therefore, all three elementary hypotheses are rejected.

Closed Testing vs. Bonferroni

Bonferroni is one of the simplest multiplicity procedures. For \(m\) hypotheses, each hypothesis can be tested at:

Closed testing can sometimes be more powerful because it can use less conservative local tests and exploit the logical relationships among hypotheses.

Closed Testing and Holm's Procedure

Holm's step-down procedure is particularly important because it can be viewed as a shortcut to a closed testing procedure under appropriate local testing choices.

Holm orders the p-values:

and compares them sequentially with:

The resulting procedure strongly controls FWER.

Why Closed Testing Can Be More General Than Holm

Holm imposes a specific step-down structure. Closed testing does not require a single universal ordering.

This allows closed testing to represent:

• Hierarchical hypotheses
• Multiple endpoints
• Multiple doses
• Composite testing strategies
• Gatekeeping structures
• Fallback procedures
• Specialized intersection tests

This flexibility makes closed testing particularly useful for complex confirmatory clinical trial strategies.

Closed Testing and Gatekeeping

Gatekeeping procedures are often used when hypotheses are organized into families.

For example, a trial might have:

• Family 1: primary endpoints
• Family 2: key secondary endpoints
• Family 3: exploratory secondary endpoints

A strict gatekeeping strategy may require success in Family 1 before testing Family 2.

Closed testing provides a mathematical framework for understanding many such procedures.

Hierarchical Testing as a Special Structure

Suppose the trial has:

where the arrow indicates a testing hierarchy.

The strategy might require:

1. Test \(H_1\).
2. If \(H_1\) is rejected, test \(H_2\).
3. If \(H_2\) is rejected, test \(H_3\).

This can be represented within broader closed-testing and logical-testing frameworks.

Closed Testing With Multiple Endpoints

Multiple endpoints are one of the most common clinical applications. Suppose a Phase III trial has:

• Overall survival
• Progression-free survival
• Objective response rate

If all three are confirmatory, the multiplicity strategy must specify how Type I error is controlled across them.

A closed testing framework could include:

with a prespecified local test for each intersection.

Closed Testing With Multiple Treatment Comparisons

The same principle can be applied to multiple treatment comparisons. Suppose a trial compares:

• Dose 1 vs. placebo
• Dose 2 vs. placebo
• Dose 3 vs. placebo

The elementary hypotheses could be:

The closure includes all nonempty intersections.

This can allow the multiplicity strategy to incorporate the scientific structure of the dose-response problem.

Local Test Choice Matters

The power of a closed testing procedure depends heavily on the local tests.

Suppose an intersection hypothesis is tested using a highly conservative method. The overall procedure may remain valid but have low power.

Conversely, an appropriately chosen local test can make better use of the available information.

Therefore, closed testing should be viewed as:

Closed Testing Does Not Mean "Test Everything at 0.05"

A common misunderstanding is that because each local test is performed at level 0.05, the overall procedure somehow has a 5% error rate automatically.

That is not the reason closed testing works.

The key is the logical relationship among the local tests and the intersection hypotheses.

The closure principle ensures that a false elementary rejection implies a corresponding false rejection of a true intersection hypothesis.

That logical implication is what establishes strong FWER control.

A More Formal Statement of the Closure Principle

Let \(H_1,\ldots,H_m\) be elementary hypotheses. For every nonempty subset \(I\subseteq\{1,\ldots,m\}\), define:

The closed family is:

Each \(H_I\) receives a level-\(\alpha\) local test. An elementary hypothesis \(H_i\) is rejected only if every \(H_I\) satisfying \(i\in I\) is rejected.

If all local tests are valid at level \(\alpha\), the resulting closed testing procedure strongly controls FWER at level \(\alpha\).

A Three-Endpoint Worked Example

Consider a Phase III trial with three confirmatory endpoints:

Suppose the local intersection p-values are:

For \(H_1\), the relevant p-values are:

Therefore:

For \(H_2\):

For \(H_3\):

All three adjusted p-values are below 0.05. Thus all three hypotheses can be rejected under this particular closed testing procedure.

What Happens When One Endpoint Is Clearly Strong?

Suppose instead:

A closed testing procedure does not automatically prevent \(H_1\) from being rejected.

The key question is whether every intersection hypothesis containing \(H_1\) can also be rejected.

If those intersection tests are sufficiently strong, \(H_1\) may be rejected even though \(H_2\) and \(H_3\) are not.

This is one of the advantages of a properly structured multiple testing procedure: failure of one endpoint does not necessarily destroy every other hypothesis.

Closed Testing and Alpha Recycling

In more complex multiplicity strategies, Type I error can sometimes be reallocated after a hypothesis is successfully rejected.

This is often described as alpha recycling.

The idea is that rejection of one hypothesis can make some portion of the available Type I error usable elsewhere, provided the procedure is constructed to preserve strong FWER control.

Such procedures can often be represented using graphical or closed-testing frameworks.

Closed Testing and Graphical Approaches

Graphical approaches to multiplicity provide a convenient way to represent complex alpha allocation and recycling.

A graph may specify:

• Initial alpha weights
• Testing priorities
• Transition probabilities
• Alpha recycling after rejection

The underlying statistical logic can often be understood through the closure principle.

This is particularly useful for trials containing several families of primary and key secondary endpoints.

Closed Testing in a Clinical Trial Protocol

A protocol using a closed testing strategy should describe the multiplicity procedure in enough detail that the confirmatory analysis can be reproduced without relying on post hoc decisions.

At minimum, specify:

• The family of hypotheses
• The elementary hypotheses
• The intersection hypotheses
• The local test for each relevant intersection
• The overall Type I error level
• The rejection rule
• Any hierarchical relationships
• Any gatekeeping rules
• Any alpha recycling rules
• How adjusted p-values will be calculated

Closed Testing in the Statistical Analysis Plan

The SAP should make the testing algorithm operationally explicit.

For example, it might specify:

The actual SAP should, of course, provide the precise statistical tests, estimands, populations, and endpoint definitions relevant to the trial.

Closed Testing and One-Sided vs. Two-Sided Alpha

The closure principle itself does not determine whether tests are one-sided or two-sided.

That decision comes from the scientific question and the prespecified confirmatory testing strategy.

For example, a trial might control:

for a one-sided confirmatory test.

Alternatively, a two-sided 5% procedure may be used.

Closed Testing vs. Fixed-Sequence Testing

Fixed-sequence testing is another common multiplicity strategy. Suppose:

The procedure may test \(H_1\) first, then \(H_2\), then \(H_3\), with the full significance level passed forward only after successful rejection.

Fixed-sequence testing can be extremely powerful when the ordering is scientifically justified.

Closed Testing vs. Hochberg

Hochberg's procedure is a step-up multiple testing method. It orders the p-values from largest to smallest and uses progressively less stringent critical values.

Like Holm's method, Hochberg provides strong FWER control under its required assumptions.

Closed testing is more general because it defines a framework in terms of intersection hypotheses and local tests rather than a single p-value ordering algorithm.

Closed Testing vs. Hommel

Hommel's procedure is another powerful FWER-controlling procedure related to the closure principle.

In fact, Hommel's procedure can be interpreted through closed testing using Simes-type local tests.

This illustrates a broader point: many familiar multiple-testing procedures can be understood as specialized or computationally efficient forms of closed testing.

Why Shortcuts Are Important

The theoretical closed testing family can become enormous. For \(m=20\) hypotheses:

intersection hypotheses would exist.

It would be impractical to perform more than one million local tests explicitly in many applications.

Therefore, many practical procedures derive mathematical shortcuts that produce the same or equivalent rejection decisions without enumerating every intersection.

Closed Testing and Multiplicity Strategy Selection

Closed testing is not necessarily the best choice for every clinical trial. The appropriate strategy depends on the structure of the scientific questions.

Closed Testing vs. FDR Control

Closed testing is primarily used when the goal is to control the family-wise error rate. False discovery rate procedures instead control the expected proportion of false discoveries among rejected hypotheses.

These are different objectives.

where \(V\) is the number of false rejections.

FDR is:

where \(R\) is the total number of rejections.

Common Mistakes

1. Thinking closed testing means testing each endpoint at 0.05. The elementary test alone is not sufficient. The relevant intersection hypotheses must also be rejected.
2. Forgetting the intersection hypotheses. The intersection hypotheses are the core of the closure principle.
3. Assuming closed testing is the same as Bonferroni. Bonferroni is one possible local testing approach; closed testing is a broader framework.
4. Confusing strong and weak FWER control. Strong control requires validity regardless of which null hypotheses are true.
5. Ignoring the local test. The power and behavior of a closed testing procedure depend heavily on how the intersection hypotheses are tested.
6. Using a post hoc hypothesis hierarchy. The hypothesis structure should be prespecified.
7. Reporting only unadjusted p-values. Confirmatory reporting should make clear how multiplicity was controlled.
8. Assuming every closed testing procedure has identical power. Different local tests can produce substantially different power.
9. Confusing closure with a simple testing order. A hierarchy or sequence describes testing order; closure concerns the complete intersection structure.

A Practical Closed Testing Workflow

R Implementation: A Simple Two-Hypothesis Example

A basic closed testing calculation can be represented directly in R. Suppose the local p-values are:

p1  <- 0.018
p2  <- 0.031
p12 <- 0.011

alpha <- 0.05

The adjusted p-values are:

p1_adj <- max(p1, p12)
p2_adj <- max(p2, p12)

p1_adj
p2_adj

The result is:

p1_adj
# 0.018

p2_adj
# 0.031

Both are below 0.05.

Decision Rules in R

reject_h1 <- p1_adj <= alpha
reject_h2 <- p2_adj <= alpha

reject_h1
reject_h2

Both hypotheses are rejected:

# TRUE
# TRUE

A Generic Function for Two Hypotheses

closed_test_2 <- function(p1, p2, p12, alpha = 0.05) {

  p1_adj <- max(p1, p12)
  p2_adj <- max(p2, p12)

  data.frame(
    hypothesis = c("H1", "H2"),
    unadjusted_p = c(p1, p2),
    adjusted_p = c(p1_adj, p2_adj),
    reject = c(
      p1_adj <= alpha,
      p2_adj <= alpha
    )
  )
}

closed_test_2(
  p1 = 0.018,
  p2 = 0.031,
  p12 = 0.011
)

Understanding the R Logic

The function works because each elementary hypothesis is compared with the largest p-value among all relevant closed-family hypotheses. For \(H_1\):

For \(H_2\):

The maximum operation captures the requirement that all relevant intersection tests must be significant.

Implementing a Three-Hypothesis Closure

For three hypotheses, a simple adjusted-p-value calculation can be written as:

p1   <- 0.012
p2   <- 0.041
p3   <- 0.020

p12  <- 0.025
p13  <- 0.014
p23  <- 0.035
p123 <- 0.010

p1_adj <- max(p1, p12, p13, p123)
p2_adj <- max(p2, p12, p23, p123)
p3_adj <- max(p3, p13, p23, p123)

data.frame(
  hypothesis = c("H1", "H2", "H3"),
  adjusted_p = c(
    p1_adj,
    p2_adj,
    p3_adj
  ),
  reject = c(
    p1_adj <= 0.05,
    p2_adj <= 0.05,
    p3_adj <= 0.05
  )
)

The resulting adjusted p-values are:

Closed Testing and Software

In real clinical trial analyses, closed testing is often implemented through specialized statistical procedures rather than manually calculating every intersection.

Depending on the design, software may need to account for:

• Correlated endpoints
• Repeated measurements
• Time-to-event endpoints
• Multiple treatment comparisons
• Stratification
• Covariate adjustment
• Hierarchical testing
• Graphical alpha recycling

The statistical software should reproduce the prespecified multiplicity strategy rather than determine the strategy after the data are examined.

Closed Testing and Regulatory Submissions

For confirmatory clinical trials, multiplicity procedures should be defined before the primary analysis.

The regulatory documentation should make it possible to determine:

• Which hypotheses belong to the confirmatory family
• Which hypotheses are elementary
• Which intersection hypotheses are relevant
• Which local tests are used
• How Type I error is controlled
• Which hypotheses can be declared successful
• How adjusted p-values are obtained

Closed Testing and Estimands

Multiplicity and estimands are different statistical concepts, but they interact in confirmatory trial design.

Each elementary hypothesis should correspond to a clearly defined scientific question and estimand.

For example, the trial might define:

where \(\theta_1\) is the treatment effect under a prespecified estimand.

If the estimand changes after the multiplicity strategy has been designed, the interpretation of the hypotheses may also change.

Closed Testing and Missing Data

Closed testing does not solve missing-data problems. The local tests must still be based on appropriately specified analyses and prespecified estimands.

If an endpoint has missing observations, the analysis method used to obtain its local test statistic and p-value should follow the prespecified primary analysis.

Multiplicity adjustment should then be applied to the resulting family of valid hypothesis tests.

Closed Testing Does Not Rescue a Poor Analysis

A closed testing procedure can control multiplicity only if the component tests themselves are valid.

For example, closure does not compensate for:

• An incorrectly specified statistical model
• Invalid variance estimation
• Uncontrolled informative missingness
• Post hoc endpoint definitions
• Unplanned treatment comparisons
• Incorrect analysis populations

Multiplicity control and model validity are separate requirements.

When Closed Testing Is Particularly Attractive

Closed testing can be especially useful when:

• Several hypotheses are confirmatory.
• Strong FWER control is required.
• The hypotheses have meaningful logical relationships.
• Different intersection tests can exploit different statistical structures.
• The trial contains multiple endpoints or treatment comparisons.
• Hierarchical or gatekeeping relationships are important.
• A transparent adjusted p-value framework is desired.

When a Simpler Procedure May Be Better

Closed testing is powerful and flexible, but it can be unnecessarily complicated for simple multiplicity problems.

For example, if a study has two equally important hypotheses and no special structure, Holm's procedure may provide an easier-to-communicate solution.

Likewise, if there is a compelling fixed hierarchy, fixed-sequence testing may be more efficient and easier to explain.

Closed Testing Design Checklist

The Most Important Concept

The most important idea in closed testing is simple:

This requirement creates the logical structure that provides strong family-wise error control.

For two hypotheses:

For three:

And in general, for \(m\) elementary hypotheses:

nonempty intersection hypotheses form the complete closure.

References

Marcus, R., Peritz, E., & Gabriel, K.R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika, 63(3), 655–660.
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75(4), 800–802.
Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75(2), 383–386.
Westfall, P.H. & Young, S.S. (1993). Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment. Wiley.
Bretz, F., Maurer, W., Brannath, W., & Posch, M. (2009). A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine, 28, 586–604.
Dmitrienko, A., Tamhane, A.C., & Bretz, F. (2009). Multiple Testing Problems in Pharmaceutical Statistics. Chapman & Hall/CRC.