Hierarchical Testing Strategies in Clinical Trials

Introduction

Confirmatory clinical trials frequently have more than one scientific question that investigators would like to answer. A study may evaluate a treatment on a primary endpoint, several key secondary endpoints, multiple patient populations, or combinations of treatment effects and clinical outcomes.

The problem is that testing several null hypotheses can increase the probability of obtaining at least one statistically significant result purely by chance. This is the familiar multiplicity problem.

One solution is to test every hypothesis independently at the full significance level. That approach is usually inappropriate for a confirmatory claim when the hypotheses belong to the same family of claims.

A hierarchical testing strategy instead establishes an ordered or structured sequence for the hypotheses. Some hypotheses are tested first. Other hypotheses become eligible for testing only if earlier conditions are satisfied.

What Is a Hierarchical Testing Strategy?

Suppose a trial contains three hypotheses:

A simple hierarchy might require:

The hierarchy therefore creates a logical dependency between the hypotheses. If \(H_1\) fails, the procedure may prevent \(H_2\) and \(H_3\) from being formally tested for confirmatory purposes.

Why Multiple Testing Matters

Suppose five independent null hypotheses are each tested at:

If all five null hypotheses are true and each is tested independently, the probability of making at least one false-positive declaration is:

which is approximately:

Thus the probability of at least one false-positive result can be approximately 22.6%, rather than 5%.

The relevant error rate for a family of confirmatory hypotheses is commonly the familywise error rate:

A hierarchical strategy is one way to keep this probability at or below a prespecified level.

Familywise Error Rate

For a family of null hypotheses:

the familywise error rate is:

A confirmatory multiplicity strategy is designed so that:

for the prespecified family.

The Simplest Hierarchy: Fixed-Sequence Testing

The simplest hierarchical procedure is the fixed-sequence procedure. Suppose the hypotheses are ordered:

The rule is:

• Test \(H_1\) at level \(\alpha\).
• If \(H_1\) is rejected, test \(H_2\) at level \(\alpha\).
• If \(H_2\) is rejected, test \(H_3\) at level \(\alpha\).
• If \(H_3\) is rejected, test \(H_4\) at level \(\alpha\).
• Stop the confirmatory sequence when a hypothesis fails.

At first glance, it may appear that this procedure should inflate type I error because every hypothesis is tested at 0.05. The key is that later hypotheses are tested only after earlier hypotheses have been rejected.

Why Fixed-Sequence Testing Controls FWER

Consider the first true null hypothesis encountered in the sequence. Suppose:

are tested sequentially. If \(H_1\) is true, it can produce a false rejection with probability no greater than \(\alpha\). If \(H_1\) is false and rejected, but \(H_2\) is true, then \(H_2\) is the first true null hypothesis encountered. If \(H_2\) is rejected falsely, that occurs with probability no greater than \(\alpha\).

The same logic applies to the first true null hypothesis at any position.

Fixed-Sequence Example

Suppose a Phase III trial has one primary endpoint and three key secondary endpoints:

The testing strategy could be:

Each hypothesis is tested at the one-sided 2.5% level, for example, if that is the prespecified confirmatory alpha.

If \(H_1\) is significant, \(H_2\) becomes eligible. If \(H_2\) is significant, \(H_3\) becomes eligible. If \(H_3\) is significant, \(H_4\) becomes eligible.

If \(H_2\) fails, the sequence stops.

What Happens to a Hypothesis That Is Not Reached?

This is an important distinction. A hypothesis that is not reached by the hierarchical procedure is not necessarily "proven false."

Instead, it has not been granted a confirmatory rejection under that particular multiplicity strategy.

The statistical analysis may still report:

• Its estimated treatment effect
• Its confidence interval
• Its nominal p-value
• Descriptive statistics

But the SAP and clinical report should distinguish these nominal or descriptive results from multiplicity-controlled confirmatory claims.

Hierarchical Testing vs. Bonferroni

Bonferroni testing distributes the available alpha across hypotheses. For \(m\) hypotheses, the simplest Bonferroni threshold is:

For five hypotheses and an overall alpha of 0.05:

A strict hierarchy can instead allow each hypothesis to be tested at the full alpha because access to later hypotheses depends on earlier rejections.

Gatekeeping Procedures

A more flexible hierarchical strategy is called a gatekeeping procedure.

A gatekeeper is a hypothesis or family of hypotheses that must be successfully addressed before another family becomes eligible for formal testing.

For example, consider:

The gatekeeping structure reflects the scientific importance of the endpoint families.

Why Gatekeeping Is Useful in Clinical Trials

Many confirmatory trials have a natural hierarchy. For example:

• The primary population may need to succeed before a broader population can support a claim.
• The primary endpoint may need to succeed before secondary endpoints can support confirmatory claims.
• A clinical benefit endpoint may need to succeed before a surrogate endpoint is formally interpreted.
• One treatment comparison may need to succeed before another dose comparison becomes confirmatory.

Gatekeeping allows the statistical procedure to reflect those scientific relationships.

Simple Gatekeeping Example

Suppose a trial contains:

• \(H_1\): primary endpoint
• \(H_2\): key secondary endpoint 1
• \(H_3\): key secondary endpoint 2
• \(H_4\): additional secondary endpoint

A simple gatekeeping strategy might be:

The primary endpoint is the first gate. Only if \(H_1\) is significant do the secondary hypotheses become eligible.

If \(H_1\) succeeds, \(H_2\) and \(H_3\) may then be tested using a prespecified multiplicity procedure.

If the secondary family satisfies its gatekeeping rule, \(H_4\) can become eligible.

Serial Gatekeeping

In serial gatekeeping, one family must pass before the next family can be tested.

This is essentially a family-level extension of fixed-sequence testing.

For example:

Parallel Gatekeeping

In parallel gatekeeping, multiple hypotheses or families can be tested in parallel after an earlier gate is passed.

For example:

Once \(H_1\) is rejected, the three downstream hypotheses become available. They still require an appropriate multiplicity procedure if more than one downstream hypothesis is tested.

A common mistake is to assume that passing the first gate automatically permits every downstream hypothesis to be tested at unrestricted alpha. That is not generally true.

Fallback Procedures

A fallback procedure allows alpha that is not used successfully in one part of the hierarchy to be transferred to another hypothesis or family.

This is more flexible than a strict fixed sequence.

For example:

Fallback testing can preserve the logical hierarchy while allowing more opportunities to demonstrate treatment benefit.

Weighted Hypotheses

Hierarchical procedures can also use weights. Suppose three hypotheses receive weights:

with:

At a total alpha of 0.05, the initial alpha allocation could be:

The weights can then be transferred according to the prespecified testing graph or hierarchy.

Graphical Approaches

A powerful generalization is the graphical multiple-testing procedure. In a graphical procedure:

• Each hypothesis is represented as a node.
• Each node has an alpha weight.
• Directed edges specify how alpha is transferred after rejection.
• The procedure continues as hypotheses are rejected.

Conceptually:

The edge weights determine how testing level is redistributed.

Hierarchical Testing With Multiple Endpoints

Multiple endpoints are one of the most common applications. Suppose a trial has:

A strict hierarchy might be:

This means that each endpoint must wait for success of the preceding endpoint.

Co-Primary Endpoints Are Different

Co-primary endpoints should not automatically be treated as a simple sequential hierarchy.

Suppose success requires both:

This is fundamentally different from:

With co-primary endpoints, the clinical success criterion may require all components to demonstrate benefit.

Multiple Treatment Doses

Hierarchical testing can also organize multiple dose comparisons. Suppose a trial compares:

• High dose vs. placebo
• Medium dose vs. placebo
• Low dose vs. placebo

A prespecified hierarchy could prioritize the highest dose:

This can be clinically sensible when the high dose is considered the primary candidate for approval.

Alternatively, the design could use a graphical strategy in which alpha is distributed among doses and redistributed after successful rejection.

Multiple Populations

Another common situation involves a primary population and a broader population. For example:

This hierarchy may reflect the scientific claim that evidence in the target population should be established before extending the claim to a broader group.

The exact strategy depends on the estimand, population definitions, and intended regulatory claim.

Hierarchical Testing and the Primary Endpoint

The primary endpoint is frequently placed first because it represents the main confirmatory question.

For example:

versus:

If the primary hypothesis is rejected, the strategy may release alpha to secondary hypotheses.

This can make the hierarchy closely aligned with the clinical development strategy.

A Worked Example: Two Key Secondary Endpoints

Consider a randomized Phase III trial with:

• Primary endpoint: overall survival
• Key secondary endpoint 1: progression-free survival
• Key secondary endpoint 2: objective response rate

Suppose the total one-sided confirmatory alpha is:

Use the fixed sequence:

where:

Each hypothesis is tested at the full one-sided 0.025 level when it is reached.

Worked Example: Scenario A

Suppose the p-values are:

Because each p-value is below 0.025 at the point at which the hypothesis is tested, all three hypotheses can be declared significant under this strict fixed-sequence strategy.

Worked Example: Scenario B

Now suppose:

Overall survival is significant, so progression-free survival can be tested. Progression-free survival fails the prespecified 0.025 criterion. Therefore the strict sequence stops.

The objective response p-value may still be reported descriptively, but it does not receive a confirmatory rejection under this fixed-sequence procedure.

Worked Example: Gatekeeping Two Secondary Endpoints

Now consider a different strategy. After the primary endpoint succeeds, two key secondary endpoints are tested as a family.

Suppose the secondary family receives the full alpha after \(H_1\) succeeds, and the two secondary hypotheses are then handled using a multiplicity method. For example, a simple Bonferroni allocation could be:

The strategy is therefore:

Why Hierarchy Can Improve Power

The advantage of a hierarchical strategy is not that it magically creates statistical information. The advantage is that it can use the logical structure of the scientific questions to avoid unnecessarily splitting alpha among hypotheses that are not simultaneously eligible for confirmatory testing.

Suppose:

If \(H_1\) fails, there is no need to spend confirmatory alpha on \(H_2\) and \(H_3\). If \(H_1\) succeeds, \(H_2\) becomes relevant. If \(H_2\) succeeds, \(H_3\) becomes relevant.

This can be substantially more powerful than splitting alpha equally across all three hypotheses when the scientific hierarchy is strong.

Hierarchy Does Not Eliminate Multiplicity Automatically

This is one of the most important practical points. Consider:

If all three downstream hypotheses are tested at the full alpha after \(H_1\) succeeds, the downstream family may still have an inflated FWER.

Therefore, the downstream hypotheses need their own valid multiplicity strategy. Possible approaches include:

• Bonferroni or Holm adjustment
• Fixed sequences
• Fallback procedures
• Weighted testing
• Graphical procedures
• Other closed-testing-based methods

Hierarchical Testing and Closed Testing

Hierarchical procedures can be understood in relation to the broader framework of multiple-testing methods. A closed testing procedure considers intersection hypotheses and uses local tests to control the FWER.

For two hypotheses:

the intersection hypothesis is:

The closed-testing principle requires appropriate testing of the relevant intersection hypotheses before individual hypotheses can be rejected.

Not every practical hierarchical strategy needs to be implemented explicitly as a full closure in the SAP, but the underlying multiplicity logic should remain valid.

Hierarchical Testing and the Graphical Approach

Graphical procedures provide a convenient way to visualize alpha transfer. Imagine three hypotheses with initial alpha weights:

The total is:

If \(H_1\) is rejected, its alpha can be transferred according to prespecified transition weights. For example, 60% might be transferred to \(H_2\) and 40% to \(H_3\).

and:

The precise graph must be defined before the confirmatory analysis.

Alpha Recycling

Some hierarchical procedures allow rejected hypotheses to release their remaining alpha to other hypotheses. This is sometimes called alpha recycling.

The basic idea is:

Alpha recycling is particularly useful when the clinical development strategy does not fit a simple one-directional sequence.

Primary, Key Secondary, and Exploratory Endpoints

A useful way to organize a clinical trial is to distinguish:

Not every secondary or exploratory endpoint needs to belong to the same multiplicity family. The key question is: Which statistical claims are intended to be confirmatory?

Multiplicity Families

A clinical trial may contain several distinct multiplicity families. For example:

• Family A: confirmatory efficacy endpoints
• Family B: dose comparisons
• Family C: subgroup analyses
• Family D: exploratory biomarkers

The statistical analysis plan should explain which hypotheses belong to which family and why.

Hierarchical Testing for Multiple Indications

Suppose a development program evaluates a treatment in:

• Indication A
• Indication B
• Indication C

A hierarchy could prioritize:

Alternatively, indications could be tested in parallel with weighted alpha. The appropriate strategy depends on the intended claims and regulatory context.

Hierarchical Testing for Multiple Doses and Endpoints

A more complicated trial might simultaneously contain:

• Two treatment doses
• Two populations
• Three efficacy endpoints

This creates many possible hypotheses. A naive approach might test all combinations independently. That can create a substantial multiplicity burden.

A hierarchical strategy can instead prioritize the most important clinical path. For example:

More sophisticated graphical methods may be preferable if several branches need to remain active.

Hierarchical Testing vs. Sequential Testing Over Time

The word "sequential" can cause confusion. Hierarchical testing is sequential in the sense that hypothesis eligibility depends on previous results. This is different from interim monitoring of the same hypothesis over time.

For example:

is an interim-monitoring structure. By contrast:

is a hierarchical hypothesis-testing structure.

A clinical trial can contain both. The multiplicity strategy should account for both sources of repeated testing.

Hierarchical Testing and Interim Analyses

Suppose a trial has:

• One interim efficacy analysis
• One final efficacy analysis
• Three confirmatory endpoints

There are now potentially two layers of multiplicity:

1. Repeated looks at each endpoint over time
2. Multiple endpoints within the trial

The interim monitoring boundaries and endpoint multiplicity strategy must be coordinated.

Testing Direction Matters

A hierarchical procedure should specify whether hypotheses are one-sided or two-sided. For example:

may use a one-sided alpha of 0.025. A two-sided hypothesis might instead use:

with a two-sided alpha of 0.05.

The testing direction should be prespecified and aligned with the clinical question.

Confidence Intervals and Hierarchical Testing

A hierarchy primarily determines which hypotheses can receive confirmatory rejection. It does not necessarily mean that confidence intervals for downstream endpoints cannot be calculated.

However, interpretation should distinguish:

• Nominal confidence intervals
• Multiplicity-adjusted confidence intervals
• Confidence intervals associated with confirmatory claims

If formal simultaneous inference is required, the confidence-interval procedure should be aligned with the multiplicity method.

Multiplicity-Adjusted P-Values

Some hierarchical procedures are naturally described through sequential testing levels. Others can be represented using adjusted p-values.

For example, under a strict fixed sequence, a hypothesis is rejected when its raw p-value is below the prespecified alpha and all required predecessors have already been rejected.

The SAP should clearly state whether the results will be presented as:

• Sequential testing decisions
• Adjusted p-values
• Both adjusted and nominal p-values

Why the Testing Order Must Be Clinically Defensible

A hierarchy should not be chosen solely because it produces the greatest statistical power.

Suppose two endpoints are:

• Endpoint A: clinically more important but statistically less sensitive
• Endpoint B: statistically more sensitive but clinically secondary

Putting B first merely because it is easier to reject may produce a hierarchy that does not reflect the intended clinical claim.

Common Hierarchical Structures

A Practical Decision Tree

Hierarchical Testing in the Statistical Analysis Plan

The SAP should contain enough information for an independent statistician to reproduce every testing decision.

At minimum, specify:

• The complete set of confirmatory hypotheses
• The definition of each hypothesis
• The testing order
• The significance level for each hypothesis
• The gatekeeping conditions
• Alpha-transfer or recycling rules
• Whether tests are one-sided or two-sided
• The multiplicity family
• Handling of untested hypotheses
• Relationship to interim analyses
• Relationship to estimands and analysis populations
• Presentation of nominal versus multiplicity-controlled p-values

Example SAP Language

A simplified hierarchical strategy could be described as follows:

The actual SAP should provide the exact hypotheses, estimands, analysis methods, and decision rules rather than relying on generic language.

Handling Nominal P-Values

Suppose the hierarchy stops at \(H_2\), but \(H_3\) has a p-value of 0.001. It may be tempting to write:

"\(H_3\) was statistically significant."

That statement can be misleading if \(H_3\) was not eligible for confirmatory testing.

A more appropriate interpretation is that the nominal p-value was small, but the hypothesis did not achieve a confirmatory rejection under the prespecified hierarchical procedure.

Hierarchical Testing and Estimands

Multiplicity strategies operate on statistical hypotheses, but those hypotheses should be connected to clearly defined estimands.

For example, a primary hypothesis may concern a treatment effect under a specific treatment-policy or hypothetical strategy for an intercurrent event.

The hierarchy should not obscure the estimand. Each hypothesis should correspond to a clearly defined scientific question.

Hierarchical Testing and Subgroups

Subgroup analyses require particular caution. Suppose the primary analysis is performed in the overall population, followed by a subgroup. A hierarchy might be:

But this does not automatically make the subgroup claim clinically or statistically valid. The subgroup hypothesis must be explicitly defined, and interaction effects may be more appropriate than simply testing the treatment effect within a subgroup.

Common Mistakes

1. Testing every endpoint at the full alpha without a valid hierarchy. This can inflate the familywise error rate.
2. Assuming that putting hypotheses in an order automatically controls FWER. The testing rule and stopping conditions must actually define a valid multiplicity procedure.
3. Ignoring multiplicity within a gate. If three hypotheses become eligible simultaneously, those three tests may still require multiplicity adjustment.
4. Choosing the hierarchy after looking at the results. The hierarchy should be prespecified for confirmatory inference.
5. Confusing nominal significance with confirmatory significance. An unadjusted p-value below 0.05 does not necessarily support a formal claim.
6. Confusing hierarchical testing with interim monitoring. These are different sources of multiplicity and may need to be coordinated.
7. Ignoring co-primary endpoint logic. Co-primary endpoints can require joint success rather than sequential success.
8. Using a statistically convenient hierarchy that does not reflect clinical priorities. The testing order should have a scientific rationale.
9. Failing to specify one-sided versus two-sided testing. The alpha level and hypothesis direction must be clear.
10. Not documenting alpha transfer. Graphical and fallback procedures require explicit transition rules.

Hierarchical Testing Workflow

Power Considerations

Multiplicity affects power as well as type I error. If alpha is divided across many hypotheses, each individual hypothesis may have less power.

A hierarchical strategy can preserve more alpha for the most important hypotheses.

However, the power of a downstream hypothesis depends on the probability that the trial reaches it. For a simple fixed sequence:

Therefore, downstream power is constrained by the success probability of the preceding hypotheses.

Conditional Power Within a Hierarchy

Suppose \(H_2\) can only be tested if \(H_1\) succeeds. The probability of ultimately rejecting \(H_2\) is not simply the conditional power of the \(H_2\) test. Instead:

This distinction is important during trial planning. A highly powered test of a downstream endpoint may still have modest overall probability of producing a confirmatory claim if the gatekeeper frequently fails.

Planning a Fixed-Sequence Trial

Suppose a trial wants to establish three claims. The investigators estimate the following probabilities under the planned alternative scenario:

The approximate probability of reaching the third confirmatory claim is:

or approximately 61.2%.

This illustrates why downstream endpoints can have lower overall probability of successful confirmatory demonstration even when their conditional tests are reasonably well powered.

When Fixed Sequence Is Especially Attractive

Fixed-sequence testing works particularly well when:

• The clinical questions have an obvious priority order.
• The first endpoint is clearly the most important.
• Later claims are meaningful only after earlier claims succeed.
• The sponsor wants a simple and transparent multiplicity strategy.
• The number of hypotheses is relatively small.

When Fixed Sequence May Be Too Restrictive

A strict sequence may be unnecessarily conservative when several hypotheses are scientifically important and should remain eligible even if one particular hypothesis fails.

For example:

may be inappropriate if failure of \(H_2\) should not logically prevent assessment of \(H_3\).

A graphical or fallback procedure may better represent such a situation.

Choosing Between Common Strategies

Regulatory and Confirmatory Considerations

For a pivotal trial, the multiplicity strategy should be finalized before the confirmatory results are known. The protocol and SAP should make the intended claims and their statistical dependencies explicit.

Important documentation includes:

• Definition of the multiplicity family
• Clinical rationale for the hierarchy
• Alpha level
• Testing direction
• Endpoint order
• Gatekeeping rules
• Alpha-transfer rules
• Interim-analysis integration
• Decision rules for untested hypotheses
• Presentation of adjusted and nominal results

A Compact Worked Example

Consider a trial with an overall two-sided alpha of:

The hierarchy is:

Suppose the observed p-values are:

Because the first two hypotheses are significant at 0.05, the testing sequence reaches \(H_3\). \(H_3\) is not rejected. Therefore, the first two hypotheses support confirmatory claims under the hierarchical procedure, while the third does not.

Another Example: Failure of the First Hypothesis

Suppose instead:

Under a strict fixed sequence, \(H_2\) and \(H_3\) do not receive confirmatory rejections because the first hypothesis failed.

Their small nominal p-values do not change the prespecified hierarchy.

Implementation in R

A simple fixed-sequence procedure can be implemented directly in R. Suppose the p-values are:

p_values <- c(
  H1 = 0.018,
  H2 = 0.032,
  H3 = 0.071
)

alpha <- 0.05

The sequential decision rule can be represented as:

reject <- logical(length(p_values))

for (i in seq_along(p_values)) {

  if (i == 1) {

    reject[i] <- p_values[i] <= alpha

  } else {

    reject[i] <- reject[i - 1] &&
      p_values[i] <= alpha
  }
}

reject

The result indicates which hypotheses are rejected under the strict fixed-sequence rule.

Implementing the Sequence More Explicitly

h1 <- p_values["H1"] <= alpha

h2 <- h1 &&
  p_values["H2"] <= alpha

h3 <- h2 &&
  p_values["H3"] <= alpha

c(
  H1 = h1,
  H2 = h2,
  H3 = h3
)

This makes the dependency structure explicit.

A Simple Gatekeeping Implementation

Suppose \(H_1\) is the gatekeeper and \(H_2\) and \(H_3\) form a secondary family tested using Bonferroni.

alpha <- 0.05

p1 <- 0.018
p2 <- 0.021
p3 <- 0.031

gate_passed <- p1 <= alpha

if (gate_passed) {

  reject_h2 <- p2 <= alpha / 2
  reject_h3 <- p3 <= alpha / 2

} else {

  reject_h2 <- FALSE
  reject_h3 <- FALSE
}

c(
  H1 = gate_passed,
  H2 = reject_h2,
  H3 = reject_h3
)

This illustrates the key distinction between:

• Passing the gate, and
• Controlling multiplicity within the downstream family.

Testing Strategy Should Be Evaluated Before the Trial

The hierarchy should be evaluated during protocol and statistical design. Useful questions include:

• Does the hierarchy reflect the clinical development strategy?
• Are the most important claims tested first?
• Is the resulting power adequate?
• What happens if the first hypothesis fails?
• Which secondary claims remain possible?
• Is the procedure understandable to clinical stakeholders?
• Can the analysis be implemented reproducibly?
• Does the strategy remain appropriate if interim analyses are added?

What to Include in the Protocol

A protocol should identify the confirmatory testing strategy clearly enough that the statistical analysis is not dependent on post hoc decisions.

A useful description includes:

What to Include in the Statistical Analysis Plan

The SAP should be more detailed than the protocol. It should specify the exact analysis associated with each hypothesis. For example:

• Analysis population
• Estimand
• Endpoint definition
• Statistical model
• Contrast of interest
• Testing direction
• Significance threshold
• Multiplicity hierarchy
• Gatekeeping conditions
• Alpha transfer rules
• Missing-data strategy
• Interim-analysis rules
• Confidence-interval interpretation

Documentation of the Testing Graph

For a complex graphical strategy, a table is often useful.

The corresponding alpha levels are obtained by multiplying the total alpha by the current weights. The complete transition rules should be reproducible from the SAP.

Hierarchy and Multiplicity Are Not the Same Thing

It is useful to distinguish the concepts. Multiplicity is the statistical problem created by making multiple inferential claims. Hierarchical testing is one family of strategies for organizing those claims so that FWER can be controlled.

Other multiplicity procedures include:

• Bonferroni
• Holm
• Hochberg
• Hommel
• Fixed-sequence procedures
• Gatekeeping procedures
• Graphical procedures
• Closed testing

The appropriate choice depends on the scientific structure of the trial.

Hierarchy vs. Holm-Bonferroni

Holm's step-down procedure does not require a clinical hierarchy. It orders observed p-values from smallest to largest.

A fixed-sequence procedure, by contrast, uses a prespecified hypothesis order.

The distinction is fundamental. One ordering is determined by the hypothesis structure; the other is determined by the observed p-values.

Hierarchy vs. Hochberg

Hochberg's procedure is a step-up multiplicity adjustment. It is generally useful when hypotheses do not have a strong clinical hierarchy and the investigator wants to exploit the distribution of the observed p-values.

A hierarchical strategy is preferable when the scientific relationship among hypotheses itself provides useful structure.

Hierarchical Testing and Clinical Interpretation

Statistical significance should always be interpreted in the context of the clinical question. A successful hierarchical procedure can establish statistical evidence for a claim, but it does not by itself establish:

• Clinical relevance
• Benefit-risk acceptability
• Generalizability
• Durability of effect
• Meaningful patient benefit

Those considerations remain essential to the overall interpretation of a clinical trial.

Common Questions

Does every hypothesis need to be tested?

No. A hierarchical strategy may intentionally make later hypotheses eligible only after earlier hypotheses succeed.

Can an untested hypothesis still have a small p-value?

Yes. Its nominal p-value can be small, but that does not necessarily provide a multiplicity-controlled confirmatory claim.

Does hierarchical testing always use the full alpha?

Not necessarily. A strict fixed sequence can use the full alpha at each stage, while gatekeeping, weighted, fallback, and graphical strategies may allocate or transfer alpha in more complex ways.

Does gatekeeping eliminate all multiplicity?

No. If multiple hypotheses become eligible simultaneously, multiplicity within that family must still be addressed.

Can hierarchical testing be used with multiple endpoints?

Yes. Multiple endpoints are among the most common applications.

Can hierarchical testing be combined with interim analyses?

Yes. The endpoint hierarchy and interim-monitoring strategy must be coordinated so that overall error control remains valid.

Best Practices

• Define the confirmatory family before the trial begins.
• Use a clinically defensible hierarchy.
• Distinguish primary, key secondary, and exploratory claims.
• Specify whether testing is one-sided or two-sided.
• Define every gate and transition rule prospectively.
• Account for multiplicity within parallel branches.
• Coordinate hierarchical testing with interim analyses.
• Distinguish nominal p-values from multiplicity-controlled claims.
• Evaluate power for downstream hypotheses.
• Document the strategy clearly in the protocol and SAP.

Hierarchical Testing Checklist

The Most Important Concept

The most important idea is that hierarchical testing turns the clinical logic of a trial into a statistical decision pathway.

Instead of asking:

"How can we test every endpoint at 0.05?"

the investigator asks:

"Which claims must succeed before other claims become meaningful?"

That distinction allows the statistical procedure to reflect the clinical development strategy.

A strict sequence may look like:

while a more flexible strategy may look like:

or:

The more complex the hierarchy becomes, the more important it is to specify the alpha allocation and transition rules precisely.

References

Dmitrienko, A., Tamhane, A.C., & Bretz, F. (2009). Multiple Testing Problems in Pharmaceutical Statistics. Chapman & Hall/CRC.

Bretz, F., Maurer, W., Brannath, W., & Posch, M. (2009). A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine, 28, 586–604.

Dmitrienko, A., Offen, W.W., & Westfall, P.H. (2003). Gatekeeping strategies for clinical trials that do not require all hypotheses to be tested. Statistics in Medicine, 22, 2387–2400.

Maurer, W., Bretz, F., & Hochberg, Y. (2011). Multiple testing and gatekeeping procedures. In Design and Analysis of Clinical Trials.

Bretz, F., Posch, M., Glimm, E., Klinglmueller, F., Maurer, W., & Rohmeyer, K. (2011). Graphical approaches for multiple comparison procedures using weighted Bonferroni, Simes, or parametric tests. Biometrical Journal, 53, 894–913.

Hochberg, Y., & Benjamini, Y. (1990). More powerful procedures for multiple significance testing. Statistics in Medicine, 9, 811–818.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.

ICH. ICH E9: Statistical Principles for Clinical Trials.

ICH. ICH E9(R1): Addendum on Estimands and Sensitivity Analysis in Clinical Trials.