Gatekeeping Procedures for Multiple Endpoints

Introduction

Clinical trials frequently evaluate more than one important endpoint. A Phase III trial might have a primary endpoint such as progression-free survival, followed by overall survival and several key secondary endpoints.

A trial may also have multiple clinically meaningful objectives, such as:

• Demonstrating efficacy in the overall population
• Demonstrating efficacy on more than one primary endpoint
• Testing multiple doses
• Testing multiple disease populations
• Establishing a sequence of primary and key secondary claims

If every hypothesis is tested at the full nominal significance level without considering multiplicity, the probability of making at least one false positive claim can exceed the desired family-wise type I error rate.

The Multiplicity Problem

Suppose a trial tests two null hypotheses:

If both hypotheses are tested independently at:

then the probability of at least one false rejection can exceed 5%.

If the tests were independent and both null hypotheses were true, the family-wise error rate would be:

With two tests at 0.05:

or approximately 9.75%.

The actual FWER depends on the dependence structure and testing procedure, but the example illustrates why simply testing every endpoint at 0.05 is not generally acceptable when confirmatory claims are being made.

What Is Family-Wise Type I Error?

The family-wise error rate is the probability of rejecting at least one true null hypothesis within a predefined family of hypotheses.

A multiplicity procedure is designed so that:

for the relevant family of confirmatory hypotheses.

Why Gatekeeping?

Traditional multiplicity procedures such as Bonferroni treat hypotheses symmetrically.

Clinical trials often do not.

For example, an investigator may consider overall survival a more important endpoint than a quality-of-life endpoint.

Or the trial may have a primary endpoint that must be successful before secondary endpoints are considered confirmatory.

A gatekeeping strategy incorporates this clinical hierarchy directly into the statistical testing procedure.

Gatekeeping Families

A useful way to construct a gatekeeping strategy is to divide the hypotheses into ordered families.

For example:

The important point is that the gatekeeping structure is defined before the confirmatory analysis.

Hierarchical Testing

The simplest gatekeeping strategy is a strict hierarchy.

Suppose there are three hypotheses:

The procedure might specify:

If \(H_1\) fails, neither \(H_2\) nor \(H_3\) can generate a confirmatory claim under this strict hierarchy.

Worked Example: Three Key Endpoints

Suppose a randomized Phase III trial has the following confirmatory objectives:

The statistical hierarchy is:

The procedure is:

• Test \(H_1\) at 0.05.
• If \(H_1\) succeeds, test \(H_2\) at 0.05.
• If \(H_2\) succeeds, test \(H_3\) at 0.05.

Suppose the observed p-values are:

All three hypotheses can be declared successful because every preceding gate was passed.

Now consider:

Although the quality-of-life p-value is below 0.05, it cannot be used to make a confirmatory claim because the second gate failed.

Parallel Gatekeeping

Strict hierarchy is sometimes too restrictive.

Suppose Family 1 contains two hypotheses:

The investigator may require that at least one hypothesis in Family 1 succeeds before testing Family 2.

This creates a parallel gate.

The exact statistical procedure must account for the multiplicity within the gate itself.

Conjunctive vs. Disjunctive Success

This distinction is particularly important for multiple primary endpoints.

Conjunctive Requirement

Suppose both endpoints must be successful:

The treatment is declared successful only if both hypotheses are rejected.

This can be appropriate when both endpoints are required to establish the overall treatment benefit.

Disjunctive Requirement

Alternatively, success may require at least one endpoint:

This is more permissive and therefore requires careful multiplicity control.

Serial Gatekeeping

A serial gatekeeping strategy allows alpha to move through a sequence of families.

For example:

Family 2 becomes eligible only after a specified success criterion in Family 1.

Family 3 becomes eligible after the corresponding criterion in Family 2.

The gates can therefore reflect a clinical development hierarchy.

Parallel Gatekeeping With Multiple Secondary Endpoints

Consider a trial with:

• One primary endpoint
• Three key secondary endpoints
• Several exploratory endpoints

The protocol could define:

Alpha Recycling

More sophisticated gatekeeping procedures allow unused alpha to be transferred between hypotheses or families.

Suppose a family receives:

and another family receives:

so that:

If a hypothesis succeeds, its allocated testing opportunity may be passed forward according to a prespecified transition rule.

This idea leads naturally to graphical multiple-testing procedures.

Graphical Multiple-Testing Procedures

A graphical approach represents hypotheses as nodes and alpha transfer possibilities as directed edges.

For example:

The graph provides a visual representation of the multiplicity strategy.

A Simple Alpha-Transfer Example

Suppose:

If \(H_1\) is rejected, all of its alpha is transferred to \(H_2\):

Thus \(H_2\) can subsequently be tested at 0.05.

If \(H_1\) is not rejected, \(H_2\) remains at 0.02 under this simple strategy.

Gatekeeping vs. Bonferroni

Bonferroni and gatekeeping solve related multiplicity problems but have different structures.

Gatekeeping can therefore provide more power for clinically important hypotheses while preserving overall error control.

Gatekeeping vs. Holm

The Holm step-down procedure orders p-values statistically.

Gatekeeping generally orders hypotheses according to their clinical or strategic role.

For example, Holm might sort:

whereas a gatekeeping procedure might explicitly define:

The distinction is important because the clinical hierarchy is fixed before observing the results.

Gatekeeping and Multiple Primary Endpoints

Multiple primary endpoints require especially careful specification.

Suppose the trial has two primary endpoints:

• Endpoint A: progression-free survival
• Endpoint B: overall survival

The trial could require both to succeed.

This conjunctive requirement has a different statistical interpretation from requiring either endpoint to succeed.

Alternatively, the procedure might allow success if either primary endpoint is significant, in which case multiplicity must be addressed explicitly.

A Two-Endpoint Worked Example

Suppose a Phase III study has two confirmatory endpoints:

The protocol specifies strict hierarchical testing:

The rules are:

1. Test \(H_1\) at 0.05.
2. If \(H_1\) is rejected, test \(H_2\) at 0.05.
3. If \(H_1\) is not rejected, no confirmatory claim is made for \(H_2\).

Suppose:

Then:

so \(H_1\) is rejected and the gate is passed.

The second endpoint is then tested at 0.05:

so \(H_2\) is also rejected.

Both confirmatory claims can therefore be supported under the prespecified hierarchical strategy.

What Happens When the Gate Fails?

Suppose instead:

The second endpoint has an impressive nominal p-value.

However, under the strict hierarchy, \(H_2\) was never eligible for confirmatory testing because \(H_1\) failed.

Gatekeeping and Power

One reason gatekeeping can be attractive is that it can preserve more power than a simple Bonferroni split when the clinical hierarchy is strong.

Suppose two hypotheses receive equal Bonferroni allocation:

Each test must then meet the stricter threshold.

Under a hierarchical strategy, the first hypothesis can instead receive:

and the second hypothesis can receive the same 0.05 only after the first gate has been passed.

This can be substantially more powerful for the second hypothesis when the clinical hierarchy is scientifically justified.

The Cost of a Strict Gate

The advantage comes with a tradeoff.

If the first endpoint is frequently unsuccessful even when later endpoints would show treatment benefit, a strict hierarchy can reduce the probability of making downstream claims.

For example, if:

then only 70% of trials can potentially reach the second gate under a strict hierarchy.

Therefore, gatekeeping should reflect the intended clinical claim.

Gatekeeping Should Follow the Claim Structure

A useful design question is: What exactly must be demonstrated before the treatment can be said to have succeeded?

For example:

• If both endpoints are essential, a conjunctive strategy may be appropriate.
• If either endpoint can establish efficacy, a disjunctive strategy may be appropriate.
• If one endpoint is clearly more important, a hierarchical strategy may be appropriate.
• If several secondary claims are clinically important, a graphical or alpha-recycling strategy may provide greater flexibility.

Gatekeeping With Multiple Endpoint Families

Consider a more complex trial:

A possible gatekeeping structure is:

But Family 2 itself may contain several hypotheses requiring a separate multiplicity procedure.

Thus, a complete strategy often has two levels:

1. Multiplicity control between families.
2. Multiplicity control within each family.

Within-Family Multiplicity

Suppose Family 2 contains:

If all three hypotheses are tested at 0.05 after Family 1 succeeds, the within-family FWER must still be controlled.

Possible approaches include:

• Bonferroni
• Holm
• Hochberg
• Hommel
• Graphical procedures
• Other prespecified multiple-testing procedures

The gate controls access to the family; the within-family procedure controls multiplicity among hypotheses inside that family.

A Two-Level Gatekeeping Example

Suppose:

This is a very common conceptual structure:

Gatekeeping Is Not the Same as Endpoint Selection

Gatekeeping should not be confused with deciding which endpoints to analyze.

An endpoint can remain scientifically important even if it is not eligible for a formal confirmatory claim.

The statistical hierarchy determines the conditions under which the endpoint can support a controlled inference.

Exploratory vs. Confirmatory Endpoints

Not every endpoint in a clinical trial necessarily belongs to the same confirmatory family.

A protocol may distinguish:

The classification should be prespecified and consistent with the intended regulatory and clinical interpretation.

Gatekeeping and p-Value Interpretation

A nominal p-value such as:

The endpoint must also satisfy the rules of the multiplicity procedure.

A useful conceptual notation is:

Both conditions matter.

Gatekeeping and Confidence Intervals

Multiplicity adjustment is often discussed in terms of p-values, but the same issue applies to confidence intervals used for confirmatory inference.

If multiple endpoints are presented with ordinary unadjusted 95% confidence intervals, those intervals do not necessarily provide simultaneous 95% coverage across the entire family.

The analysis plan should therefore specify how confidence intervals will be reported when simultaneous or multiplicity-adjusted inference is required.

Gatekeeping With Treatment Doses

Gatekeeping can also be applied when multiple doses are tested.

For example:

The high dose may be tested first. If it succeeds, testing opportunity can be passed to the low dose according to the prespecified strategy.

The same framework can be extended to multiple dose-endpoint combinations.

Gatekeeping With Multiple Populations

A trial may also contain a primary analysis population and a subgroup or special population.

For example:

This can be useful when the subgroup claim is scientifically meaningful but should not automatically receive independent confirmatory status.

Gatekeeping and Regulatory Strategy

In a confirmatory clinical trial, the gatekeeping procedure should be aligned with the intended claims.

A statistical reviewer should be able to determine:

• Which hypotheses belong to the confirmatory family
• Which hypotheses are tested first
• What constitutes passing a gate
• How alpha is allocated
• Whether alpha can be recycled
• Which hypotheses become eligible after each success
• What happens if a gate fails
• How multiplicity is controlled within each family

Pre-Specification Is Essential

The gatekeeping structure should be finalized before the confirmatory results are known.

The protocol and statistical analysis plan should identify:

• All confirmatory hypotheses
• The hierarchy or gate structure
• Initial alpha allocations
• Transfer or recycling rules
• Within-family testing procedures
• Success criteria
• Handling of missing data
• Analysis populations
• Multiplicity-adjusted confidence interval procedures where applicable

Common Gatekeeping Structures

Common Mistake: Testing Everything at 0.05

Suppose a trial has five endpoints and reports:

It is tempting to label three or four endpoints statistically significant because their p-values are below 0.05.

That conclusion may not be valid for confirmatory inference if the endpoints belong to a multiplicity family.

A gatekeeping procedure asks a different question: Which hypotheses were eligible to be tested, and at what alpha level, given the outcomes of the preceding hypotheses?

Common Mistake: Treating Gatekeeping as a Post Hoc Decision Tree

Another error is creating the decision tree after seeing the data.

For example:

• Choose whichever endpoint has the smallest p-value as the first gate.
• Then use its success to test the next endpoint.
• Continue adapting the hierarchy according to observed significance.

That is not the same as a prespecified gatekeeping strategy.

Common Mistake: Ignoring Within-Family Multiplicity

Suppose the primary endpoint succeeds and three secondary endpoints become eligible.

The fact that the gate was passed does not automatically mean all three secondary endpoints can be tested independently at 0.05.

The family may still require a multiplicity adjustment.

Common Mistake: Confusing Nominal and Adjusted Significance

A downstream endpoint may have a nominal p-value below 0.05 but fail to meet the threshold assigned by the gatekeeping procedure.

For example:

is not necessarily sufficient if the endpoint was allocated:

The endpoint would then fail its allocated testing threshold.

Gatekeeping With Alpha Splitting

Suppose two families receive:

so that:

Family 1 receives greater initial testing opportunity because it is considered more important.

A prespecified gate may then allow some or all of Family 1's alpha to flow to Family 2 if Family 1 succeeds.

Graphical Representation

A simple conceptual graph could be represented as:

Initially:

If \(H_1\) succeeds:

If \(H_2\) subsequently succeeds:

This is the simplest possible graphical gatekeeping structure.

Gatekeeping and the Closure Principle

The formal theory of multiple testing can also be connected to closed testing procedures.

The closure principle considers intersection hypotheses formed from the elementary hypotheses.

For a family containing:

the closed family can include intersection hypotheses such as:

A closed testing procedure can provide strong FWER control under broad dependence structures.

Gatekeeping strategies can be constructed within this broader multiple-testing framework.

Strong vs. Weak Type I Error Control

A confirmatory gatekeeping procedure generally seeks strong control of the family-wise error rate.

Strong control means that:

for every configuration of true and false null hypotheses in the relevant family.

This is stronger than controlling error only when all null hypotheses are true.

Gatekeeping and Dependence Between Endpoints

Clinical endpoints are often correlated.

For example, progression-free survival and symptom measures may be related, while overall survival and progression-free survival are also statistically dependent.

Some multiplicity procedures exploit correlation to improve power, while others remain valid under arbitrary dependence.

A gatekeeping procedure should therefore be evaluated using assumptions appropriate to the chosen statistical method.

Gatekeeping Does Not Automatically Solve Every Multiplicity Problem

Multiplicity may arise from many sources:

• Multiple endpoints
• Multiple treatment arms
• Multiple doses
• Multiple populations
• Multiple time points
• Multiple estimands
• Interim analyses
• Multiple subgroup analyses

A gatekeeping strategy for endpoints does not automatically control multiplicity arising from every other source.

Gatekeeping and Interim Analyses

Suppose a trial has both:

• An interim efficacy analysis
• Multiple confirmatory endpoints

The interim analysis introduces an additional multiplicity issue.

The alpha spent at interim and final analyses must be incorporated into the overall testing strategy.

For example, an alpha-spending approach might divide the overall error budget between interim and final analyses, while the endpoint gatekeeping procedure controls multiplicity across endpoints.

These are related but distinct layers of the design.

Gatekeeping and Estimands

Modern clinical trial design should define the estimand before selecting the multiplicity strategy.

For each endpoint, the protocol should clearly specify:

• Population
• Treatment condition
• Endpoint variable
• Intercurrent event strategy
• Summary measure

The gatekeeping procedure should then be applied to the resulting confirmatory hypotheses.

Gatekeeping and Missing Data

The multiplicity procedure does not determine how missing data are handled.

The statistical analysis plan should separately specify the primary analysis method and relevant sensitivity analyses.

For example:

• Primary estimand analysis
• Multiple imputation
• Pattern-mixture sensitivity analysis
• Tipping-point analysis
• Reference-based sensitivity analysis

If multiple sensitivity analyses are themselves used for confirmatory claims, their multiplicity implications should also be considered.

R Implementation: Simple Hierarchical Gate

A strict hierarchy can be implemented conceptually in R with sequential conditional logic.

alpha <- 0.05

p1 <- 0.018
p2 <- 0.027
p3 <- 0.041

h1_success <- p1 <= alpha

if (h1_success) {

  h2_success <- p2 <= alpha

  if (h2_success) {

    h3_success <- p3 <= alpha

  } else {

    h3_success <- FALSE

  }

} else {

  h2_success <- FALSE
  h3_success <- FALSE

}

h1_success
h2_success
h3_success

For the example:

so \(H_1\) passes the first gate.

Then:

so \(H_2\) passes the second gate.

Finally:

so \(H_3\) also passes.

R Function for a Strict Hierarchy

gatekeeping_hierarchical <- function(p_values,
                                     alpha = 0.05) {

  success <- rep(FALSE, length(p_values))

  for (i in seq_along(p_values)) {

    if (i == 1) {

      success[i] <- p_values[i] <= alpha

    } else {

      success[i] <- success[i - 1] &&
        p_values[i] <= alpha

    }
  }

  success
}

gatekeeping_hierarchical(
  c(0.018, 0.027, 0.041)
)

The result is:

TRUE TRUE TRUE

Now consider:

gatekeeping_hierarchical(
  c(0.018, 0.083, 0.009)
)

The result is:

TRUE FALSE FALSE

The third endpoint cannot generate a confirmatory success because the second gate failed.

R Implementation of Alpha Allocation

Suppose two hypotheses initially receive:

alpha1 <- 0.03
alpha2 <- 0.02

p1 <- 0.018
p2 <- 0.031

If all of \(H_1\)'s alpha is transferred to \(H_2\) after \(H_1\) succeeds:

if (p1 <= alpha1) {

  alpha2 <- alpha2 + alpha1

}

h2_success <- p2 <= alpha2

alpha2
h2_success

The second hypothesis now has:

and because:

it succeeds under this prespecified strategy.

A More General Alpha-Transfer Function

transfer_alpha <- function(
  alpha,
  from,
  to,
  fraction = 1
) {

  amount <- alpha[from] * fraction

  alpha[to] <- alpha[to] + amount

  alpha[from] <- alpha[from] - amount

  alpha
}

alpha <- c(
  H1 = 0.03,
  H2 = 0.02,
  H3 = 0.00
)

alpha <- transfer_alpha(
  alpha,
  from = "H1",
  to = "H2",
  fraction = 1
)

alpha

This provides a simple computational representation of alpha movement.

Checking Alpha Conservation

A useful programming check is to verify that the total alpha budget remains constant.

sum(alpha)

stopifnot(
  abs(sum(alpha) - 0.05) < 1e-12
)

For a valid alpha-transfer implementation:

unless the particular procedure explicitly uses a different representation of the available testing budget.

Reporting a Gatekeeping Procedure

A statistical analysis plan should describe the procedure in enough detail that an independent statistician can reproduce every testing decision.

At minimum, report:

• The complete list of confirmatory hypotheses
• The ordering of the hypotheses or families
• The initial alpha assigned to each hypothesis or family
• The definition of each gate
• The conditions required to pass each gate
• Alpha-transfer rules
• Within-family multiplicity procedures
• Rules following a failed gate
• Rules following a successful gate
• Multiplicity-adjusted confidence interval procedures
• Any relationship to interim analyses

A Recommended Statistical Analysis Plan Table

How to Draw a Gatekeeping Strategy

For complex trials, a visual diagram is often much easier to understand than a paragraph of statistical prose.

A simple hierarchy can be represented as:

Decision Tree Example

Gatekeeping and Successive Claims

Gatekeeping is particularly useful when a trial seeks a sequence of claims.

For example:

The procedure converts this clinical hierarchy into a statistical hierarchy.

Gatekeeping Can Increase Power Where It Matters

Suppose investigators strongly prioritize Endpoint 1.

A Bonferroni procedure might divide alpha equally:

A hierarchical procedure can instead give Endpoint 1 the full:

while allowing Endpoint 2 to use 0.05 only after Endpoint 1 succeeds.

This can provide substantially greater power for the primary endpoint and preserve testing opportunity for the secondary endpoint when the primary claim is established.

When Gatekeeping Is Especially Useful

• There is a clear clinical hierarchy among endpoints.
• The trial has primary and key secondary endpoints.
• Different endpoint families support different claims.
• Some claims should only be made after a more important claim succeeds.
• The sponsor wants to preserve alpha for strategically important endpoints.
• Alpha recycling can increase power without exceeding the overall error budget.
• The trial has a complex but prespecified claim structure.

When Gatekeeping May Be Less Appropriate

• All endpoints are genuinely equal in importance.
• There is no scientifically defensible hierarchy.
• Success on any one endpoint is sufficient and no ordering is justified.
• The clinical claim does not depend on sequential success.
• The testing strategy would become unnecessarily complicated.

In such settings, a conventional multiple-testing procedure may be easier to justify and communicate.

Gatekeeping vs. Fixed-Sequence Testing

Fixed-sequence testing is a special case of hierarchical testing.

The sequence:

is fixed before observing the data.

Gatekeeping is broader because it can involve:

• Multiple hypotheses within gates
• Conjunctive conditions
• Disjunctive conditions
• Alpha recycling
• Graphical testing
• Multiple families

Gatekeeping vs. Fallback Procedures

Fallback procedures allow testing to continue when an earlier hypothesis does not succeed, usually with a different allocation of alpha.

For example, a fallback strategy might allow:

rather than completely stopping downstream testing.

This can provide greater power but requires carefully specified transition rules.

Gatekeeping and Graphical Testing

Graphical procedures are especially useful when the hierarchy is not strictly linear.

For example:

Alpha from \(H_1\) might be divided between \(H_2\) and \(H_3\).

Similarly:

can represent multiple pathways into a downstream hypothesis.

This is difficult to describe cleanly using ordinary prose but can be represented naturally using a graphical testing framework.

Example of Alpha Redistribution

Suppose:

and if \(H_1\) succeeds, the alpha is split 60:40 between \(H_2\) and \(H_3\).

Then:

The total remains:

This is an example of how a graphical procedure can encode a more flexible clinical strategy.

Operating Characteristics of a Gatekeeping Procedure

A complete evaluation should examine more than the overall type I error.

Useful operating characteristics include:

• Family-wise type I error
• Power for each hypothesis
• Probability of each possible claim pattern
• Probability of passing each gate
• Probability of reaching downstream families
• Power under clinically plausible effect configurations

For example, if \(G_1\) denotes passing Gate 1:

and if \(G_2\) denotes reaching the second family:

these probabilities can be important for understanding the practical behavior of the strategy.

Strong FWER Evaluation

When validating a gatekeeping procedure, investigators should consider different configurations of true and false hypotheses.

For example:

Strong control requires the relevant FWER guarantee across the configurations under which the procedure is intended to operate.

Simulation Can Be Useful

For complicated gatekeeping strategies, simulation can help validate the implementation.

A simulation can:

1. Generate data under specified effect configurations.
2. Calculate endpoint test statistics.
3. Apply the exact gatekeeping algorithm.
4. Record which hypotheses are rejected.
5. Repeat the process many times.
6. Estimate FWER and power.

For example, if \(M\) simulated trials are generated and \(E\) simulations produce at least one false rejection under a relevant null configuration, the estimated FWER is:

Simulation is especially useful as a validation tool for complex alpha recycling and graphical procedures.

Simple R Simulation Skeleton

set.seed(123)

B <- 10000
alpha <- 0.05

false_claims <- logical(B)

for (b in seq_len(B)) {

  # Simulated p-values under a selected
  # null configuration
  p1 <- runif(1)
  p2 <- runif(1)
  p3 <- runif(1)

  h1 <- p1 <= alpha

  h2 <- FALSE
  h3 <- FALSE

  if (h1) {

    h2 <- p2 <= alpha

    if (h2) {
      h3 <- p3 <= alpha
    }
  }

  false_claims[b] <-
    h1 || h2 || h3
}

mean(false_claims)

For a properly specified and implemented procedure, the simulation should produce an empirical FWER consistent with the theoretical guarantee, subject to Monte Carlo error.

Monte Carlo Error Matters

If the true FWER is approximately 0.05, a simulation with only a few hundred replications may be too noisy to provide useful validation.

A larger number of replications provides a more precise estimate.

The simulation should also record Monte Carlo uncertainty rather than treating the empirical result as an exact value.

Documentation for Production Programming

For clinical trial programming, a gatekeeping implementation should ideally have:

• Independent derivation of the testing rules
• Independent programming or code review
• Unit tests for every branch
• Test cases for successful and failed gates
• Boundary-value testing
• Checks that alpha is conserved
• Documentation of every transition rule
• Reproducible outputs

This is particularly important when the procedure includes multiple alpha transfers or complex graphical structures.

Boundary Testing

Suppose an endpoint has:

The implementation should explicitly test:

• \(p=0.024999\)
• \(p=0.025000\)
• \(p=0.025001\)

This ensures that the equality convention is implemented correctly.

A Practical Gatekeeping Workflow

What Should Be Included in the Protocol?

A protocol using gatekeeping should state the strategy clearly enough that the clinical and statistical teams understand how confirmatory claims will be generated.

At minimum, specify:

• Primary endpoint(s)
• Key secondary endpoint(s)
• Confirmatory hypothesis definitions
• Overall alpha level
• Testing order
• Gate conditions
• Alpha allocations
• Alpha recycling rules
• Within-family multiplicity method
• Handling of failed gates
• Multiplicity-adjusted confidence intervals
• Relationship to interim analyses

What Should Be Included in the Statistical Analysis Plan?

The SAP should provide the operational details needed to reproduce the testing sequence.

A useful SAP section might include a table such as:

Example of a Complete Hierarchy

Consider:

Interpretation:

• \(H_1\) is the primary endpoint.
• After \(H_1\) succeeds, \(H_2\) and \(H_3\) become eligible.
• \(H_2\) and \(H_3\) are handled using a prespecified within-family procedure.
• \(H_4\) becomes eligible only after the specified criterion for the secondary family is satisfied.

This structure is substantially more realistic for many modern clinical trials than a simple linear sequence.

The Most Important Statistical Distinction

The central distinction to remember is between:

and:

A hypothesis can have a small nominal p-value and still be ineligible for a confirmatory claim because its gate was not passed.

Conversely, a hypothesis may receive a larger testing opportunity after an earlier hypothesis succeeds, because the prespecified procedure permits alpha to be transferred.

Common Mistakes Summary

1. Testing every endpoint at 0.05. This can inflate the family-wise type I error rate.
2. Creating the hierarchy after seeing results. The gatekeeping structure should be prespecified.
3. Ignoring within-family multiplicity. Passing a gate does not automatically eliminate multiplicity within the next family.
4. Confusing nominal p-values with confirmatory significance. Eligibility and allocated alpha both matter.
5. Failing to define what happens when a gate fails. The procedure should explicitly specify whether downstream testing stops, continues with reduced alpha, or follows a recycling rule.
6. Failing to document alpha transfers. Every transfer should be reproducible.
7. Ignoring other multiplicity sources. Multiple endpoints are only one source of multiplicity.
8. Using a complicated procedure without clinical justification. The statistical strategy should correspond to the intended claims.
9. Not validating the implementation. Complex graphical or recycling procedures require careful independent testing.

Worked Example: Full Testing Path

Suppose a trial uses the following hierarchy:

with:

Consider three possible result patterns.

Scenario A: All Gates Pass

All three endpoints are successful.

Scenario B: Second Gate Fails

Only \(H_1\) can support a confirmatory claim.

Scenario C: First Gate Fails

Under strict hierarchy, only the first hypothesis is eligible for testing and the downstream nominal p-values do not create confirmatory claims.

Why Gatekeeping Can Be Powerful

Gatekeeping can avoid unnecessarily dividing the alpha budget when the clinical hierarchy is strong.

Instead of permanently assigning:

the procedure can initially give one hypothesis the full:

and make the second hypothesis eligible only after the first succeeds.

This uses the clinical relationship between the claims to improve efficiency.

Limitations of Gatekeeping

Gatekeeping is not universally more powerful.

Its efficiency depends on the true effect configuration.

If the first endpoint is unlikely to succeed but a downstream endpoint has a strong treatment effect, a strict gate can make the downstream claim difficult or impossible to establish.

Therefore, design simulations should consider clinically plausible scenarios, not only the scenario in which every endpoint has a treatment effect.

Design Evaluation Matrix

Final Checklist

The Most Important Concept

Gatekeeping procedures turn a clinical hierarchy into a statistical testing strategy.

Instead of treating every endpoint as an independent opportunity for a p-value below 0.05, the procedure defines which hypotheses are eligible for testing and how the available type I error can move through the hierarchy.

The basic structure is:

More sophisticated procedures can recycle alpha and allow multiple pathways through a graphical testing strategy.

References

Dmitrienko, A., Tamhane, A.C., & Bretz, F. (2009). Multiple Testing Problems in Pharmaceutical Statistics. Chapman & Hall/CRC.

Dmitrienko, A., Offen, W.W., & Westfall, P.H. (2003). Gatekeeping strategies for clinical trials that do not require all hypotheses to be tested. Statistics in Medicine.

Dmitrienko, A., Wiens, B.L., Tamhane, A.C., & Wang, X. (2007). Tree-structured gatekeeping tests in clinical trials with multiple objectives. Statistics in Medicine.

Bretz, F., Maurer, W., Brannath, W., & Posch, M. (2009). A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine, 28, 586–604.

Maurer, W., & Bretz, F. (2013). Multiple testing in group sequential trials using graphical approaches. Statistics in Biopharmaceutical Research.

Goeman, J.J., & Solari, A. (2014). Multiple testing for exploratory research. Statistical Science, 29, 584–597.

Hochberg, Y., & Benjamini, Y. (1990). More powerful procedures for multiple significance testing. Statistics in Medicine.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.