Multiplicity Guidance From FDA and EMA

Introduction

Multiplicity is one of the most important statistical issues in confirmatory clinical trials because a development program rarely asks only one question.

A pivotal trial may compare several treatments, evaluate several primary or key secondary endpoints, examine several doses, perform analyses at multiple time points, investigate important subgroups, and conduct one or more interim analyses.

If every hypothesis is tested at the conventional two-sided 5% significance level without an appropriate strategy, the probability of obtaining at least one apparently positive result under the global null hypothesis can exceed 5%.

Regulatory agencies therefore expect sponsors to distinguish between exploratory analyses and analyses intended to support confirmatory claims, and to prospectively specify how multiplicity will be handled for the latter.

FDA and EMA Regulatory Context

The FDA's Multiple Endpoints in Clinical Trials guidance, finalized in October 2022, addresses the problems created by multiple endpoints and describes strategies for grouping and ordering endpoints and managing multiplicity in clinical trials for human drugs and biologics.

EMA's current effective document is the Points to consider on multiplicity issues in clinical trials, adopted under reference CPMP/EWP/908/99 with an effective date of April 1, 2003.

EMA has also published a later draft guideline that was developed as an update to the 2002/2003 document. Because the EMA page identifies the 2003 document as the current effective version, sponsors should distinguish between the effective guidance and later consultation material when describing the regulatory basis for a statistical strategy.

What Is Multiplicity?

Suppose a trial tests one null hypothesis:

at a two-sided significance level of 0.05.

If the null hypothesis is true and the test is properly calibrated, the probability of a false positive is approximately:

Now suppose the same trial tests several independent hypotheses, each at 0.05.

The probability of obtaining at least one statistically significant result under the global null is:

where \(m\) is the number of tests.

For five independent tests:

or approximately 22.6%.

This does not mean that every collection of five clinical-trial tests has a 22.6% false-positive probability. Correlations, hypotheses, testing relationships, and the definition of the error criterion all matter.

But it illustrates the central problem: the nominal alpha level of each individual test does not automatically describe the error probability for a family of claims.

Family-Wise Type I Error

A common regulatory objective is control of the family-wise Type I error rate, or FWER.

Conceptually:

For a confirmatory family, the target may be:

for a two-sided confirmatory testing framework.

Multiplicity Is Broader Than Multiple Endpoints

The word multiplicity is sometimes used as if it means only "multiple endpoints."

In practice, multiple-testing problems can arise from many sources.

Multiplicity Begins With the Claim

One of the most useful ways to approach multiplicity is to start with the regulatory claim rather than the statistical test.

Suppose the proposed label claim is:

That is different from a trial in which the sponsor wants to claim:

The statistical strategy should reflect the logical structure of the claim.

Intersection Versus Union Claims

Suppose there are two hypotheses:

If success requires both endpoints to demonstrate efficacy, the claim is an intersection-type claim.

If success requires at least one endpoint to demonstrate efficacy, the claim has a different logical structure.

This distinction matters because multiplicity procedures should match the claim being supported.

Multiple Primary Endpoints

Suppose a trial has two primary endpoints:

• Overall survival
• Progression-free survival

There are several possible strategies.

Both Endpoints Required

If both endpoints must be statistically significant to declare success, a conservative intersection strategy may avoid the need for a conventional Bonferroni split in the simplest case.

However, the complete statistical argument must consider the precise testing procedure and the regulatory claim.

Either Endpoint Can Support Success

If either endpoint can independently support the primary claim, then the family needs explicit Type I error control.

A simple Bonferroni allocation could divide:

for a total two-sided alpha of 0.05.

Bonferroni Adjustment

The Bonferroni method is conceptually simple.

For \(m\) hypotheses and family-wise alpha \(\alpha\), one can test each hypothesis at:

For four hypotheses and \(\alpha=0.05\):

The procedure is easy to explain and broadly applicable.

Its limitation is that it can be conservative, particularly when endpoints are correlated or when only some hypotheses are relevant after earlier successes.

Holm's Step-Down Procedure

Holm's procedure improves on simple Bonferroni while maintaining strong FWER control under general dependence.

Order the p-values:

Then compare:

The testing stops when the first comparison fails.

Holm is useful when hypotheses form a family but do not have a clinically necessary testing hierarchy.

Hierarchical Testing

A hierarchical strategy orders hypotheses according to clinical importance.

For example:

This approach is often called a fixed-sequence procedure.

It can be highly efficient because alpha is not necessarily divided across all hypotheses at the beginning.

Fixed-Sequence Testing

Consider:

with a two-sided family alpha of 0.05.

The strategy might be:

The procedure can support a sequence of confirmatory claims while preserving the specified error-control logic.

Gatekeeping Procedures

Gatekeeping is useful when hypotheses are organized into families.

For example:

Gatekeeping is especially useful when the clinical interpretation naturally requires an earlier success before later claims become meaningful.

Alpha Recycling

More flexible procedures can allow alpha to move between hypotheses after success or failure.

Suppose:

is initially assigned to a primary endpoint.

If the primary hypothesis succeeds, some or all of its alpha may be passed to a subsequent hypothesis according to a prespecified graphical or gatekeeping procedure.

This is often called alpha recycling.

Graphical Testing Procedures

Graphical approaches provide a useful visual representation of alpha allocation and recycling.

Why Prespecification Matters

Multiplicity is not only a programming issue.

The strategy should be defined before confirmatory results are known.

A robust statistical plan should specify:

• The confirmatory hypothesis family
• The alpha level
• The order of testing
• Any alpha allocation
• Any alpha recycling
• Decision rules after each test
• Handling of interim analyses
• Handling of subgroup claims
• Handling of additional endpoints
• The relationship between primary and key secondary endpoints

Multiplicity and the Statistical Analysis Plan

The SAP should make the testing strategy sufficiently explicit that an independent statistician could reconstruct the confirmatory decision process.

A vague statement such as:

is generally inadequate.

Instead, the SAP should identify the hypotheses, their order, their alpha allocations, and the conditions under which later hypotheses can be tested.

A Practical SAP Example

Suppose the trial has:

• Primary endpoint: progression-free survival
• Key secondary endpoint 1: overall survival
• Key secondary endpoint 2: objective response rate
• Key secondary endpoint 3: duration of response

A fixed sequence could be:

with each subsequent hypothesis tested only if its predecessor succeeds.

This is a simple example. In a real trial, the scientific and regulatory justification for the sequence should be explicit.

Multiple Treatment Comparisons

Multiplicity can arise even when there is only one endpoint.

Suppose a dose-ranging trial has three active doses and placebo:

• Placebo
• Low dose
• Medium dose
• High dose

If all three doses are compared independently with placebo at 0.05, the family of treatment comparisons may require an appropriate multiplicity strategy if the comparisons are intended to support confirmatory claims.

Dose Selection and Multiplicity

A common clinical development structure is:

The multiplicity strategy depends on whether dose comparisons are exploratory, confirmatory, adaptive, or part of a broader selection procedure.

Multiple Secondary Endpoints

Secondary endpoints can range from supportive measurements to endpoints that are themselves intended to support important regulatory claims.

The statistical plan should therefore distinguish:

The label "secondary" does not automatically mean "free from multiplicity."

Multiplicity and Subgroup Analyses

Subgroup analyses can produce a large number of potential comparisons.

Examples include:

• Age groups
• Sex
• Geographic region
• Race or ethnicity
• Disease stage
• Biomarker status
• Prior treatment
• Baseline disease severity

A trial may have a primary treatment effect and then investigate whether the effect differs across several subgroups.

These subgroup analyses are valuable for understanding treatment consistency and heterogeneity, but they should not automatically be treated as independent confirmatory claims.

Interaction Testing Versus Within-Subgroup Testing

Suppose treatment effects are estimated separately for two subgroups.

Finding:

in subgroup A and:

in subgroup B does not establish that the treatment effect differs between the subgroups.

A treatment-by-subgroup interaction is the appropriate statistical concept when the scientific question is whether treatment effects differ.

Biomarker-Defined Populations

Biomarker programs can create particularly complex multiplicity structures.

For example, a trial might consider:

• All randomized patients
• Biomarker-positive patients
• Biomarker-negative patients
• Several biomarker thresholds

If more than one population is intended to support a confirmatory claim, the strategy should address the resulting multiplicity.

Multiplicity Across Time Points

Repeatedly testing an endpoint at many time points can also create multiplicity.

Suppose a continuous endpoint is analyzed at:

If every time point is independently declared confirmatory at alpha 0.05, the family-level error rate may exceed the desired level.

Potential strategies include:

• Prespecifying one primary time point
• Using a longitudinal model with a prespecified estimand
• Using a hierarchical sequence of time points
• Using an appropriate multiplicity adjustment
• Treating additional time points as supportive or exploratory

Multiplicity and Repeated Measures

Not every repeated measurement automatically requires a simple Bonferroni correction.

A longitudinal model may estimate a treatment effect over time according to a single prespecified estimand.

The important question is not simply:

but rather:

Multiplicity and Interim Analyses

Interim efficacy analyses create another potential source of Type I error.

For example, a trial might conduct:

• One interim efficacy analysis
• One final analysis

Testing at 0.05 at both looks without accounting for the repeated opportunity to reject the null can inflate the Type I error.

Group sequential methods can control the overall Type I error while allowing the trial to look at the data before the final analysis.

Alpha-Spending Concept

Instead of treating each analysis as a completely independent 0.05 test, a group sequential design can distribute the overall alpha across interim and final analyses.

For example:

where the allocation depends on the chosen boundary or alpha-spending function.

Multiplicity Across Endpoint and Interim Dimensions

Suppose a trial has two key endpoints and two efficacy looks.

The testing problem has at least two dimensions:

The statistical strategy must control the relevant family of confirmatory decisions across both dimensions.

Multiplicity and Confidence Intervals

Multiplicity is not only a p-value issue.

When multiple confirmatory confidence intervals are reported, their coverage properties should be considered in relation to the inferential claims.

For example, if a family of several treatment effects is used to support simultaneous claims, reporting separate unadjusted 95% confidence intervals may not provide simultaneous 95% coverage across the entire family.

Adjusted or simultaneous confidence intervals may therefore be appropriate depending on the intended interpretation.

Point Estimates Still Matter

Multiplicity adjustment should not obscure the estimated treatment effect.

A complete presentation often includes:

• Estimated treatment effect
• Confidence interval
• Multiplicity-adjusted inferential result where applicable
• Prespecified hypothesis decision

The purpose of multiplicity control is not to make the effect estimate disappear.

Multiplicity and Estimands

The ICH E9(R1) estimand framework emphasizes defining the treatment effect question before selecting the estimator and analysis method.

Multiplicity planning should be aligned with this framework.

For example, a trial could theoretically consider different treatment-effect questions involving:

• Treatment policy
• Hypothetical strategy
• Composite strategy
• While-on-treatment strategy
• Principal stratum strategy

If multiple estimands are each intended to support separate confirmatory claims, multiplicity may become relevant.

A Simple Multiplicity Decision Framework

Common Multiplicity Strategies

Choosing Between Bonferroni and Hierarchical Testing

Consider two endpoints:

• Primary endpoint A
• Key secondary endpoint B

If the scientific claim is:

a different strategy may be appropriate than if the claim is:

Likewise, if B is clinically meaningful only after A succeeds, a hierarchical strategy may be more efficient and more interpretable than mechanically dividing alpha.

Multiplicity and Non-Inferiority

Multiplicity can also matter when a trial uses non-inferiority and superiority objectives.

For example, a study may seek to demonstrate:

1. Non-inferiority first
2. Then superiority

The statistical decision sequence should be prespecified.

Switching between hypotheses without a clearly defined strategy can complicate interpretation and error control.

Multiplicity and Composite Endpoints

A composite endpoint can reduce some multiplicity problems by defining one primary endpoint that contains multiple components.

However, composite endpoints introduce other interpretational issues.

For example, a composite might include:

• Death
• Hospitalization
• Symptom worsening

A significant treatment effect on the composite does not necessarily mean that every component is significantly improved.

Therefore, component analyses should be clearly characterized as primary, secondary, supportive, or exploratory.

Multiplicity and Missing Data

Multiplicity adjustment does not solve missing-data problems.

A trial can have perfect Type I error control and still have a biased or poorly defined estimator because of inappropriate handling of missing outcomes.

The following are separate but related issues:

Multiplicity and Sensitivity Analyses

Sensitivity analyses are not automatically additional confirmatory hypotheses.

A sensitivity analysis may be designed to assess robustness of the primary conclusion under alternative assumptions.

However, if a sensitivity analysis is itself presented as an alternative route to establish efficacy, its role in the confirmatory testing strategy should be considered carefully.

Multiplicity and Exploratory Analyses

Exploratory analyses can be scientifically valuable without being subjected to the same confirmatory claim structure.

The key is transparent labeling.

Examples include:

• Exploratory biomarkers
• Additional subgroup analyses
• Unplanned time points
• Additional endpoint transformations
• Post hoc responder definitions

The problem arises when an exploratory finding is subsequently presented as though it had been a prespecified confirmatory hypothesis.

Multiplicity and Protocol Amendments

Changes to endpoint hierarchy, analysis populations, or testing strategies after trial initiation require careful documentation.

The later the change occurs relative to unblinding or accumulation of unblinded evidence, the greater the concern that the change could influence the statistical properties or interpretation of the trial.

Multiplicity and Blinded Sample Size Re-Estimation

A blinded sample-size reassessment is generally a different issue from multiplicity.

The important distinction is whether the reassessment uses information that could alter the Type I error properties of the confirmatory testing procedure.

Any adaptive design should be evaluated as a complete statistical system, rather than assuming that multiplicity can be considered independently from adaptation.

Multiplicity in Platform and Master Protocols

Platform trials can create unusually complex multiplicity structures because there may be:

• Multiple experimental treatments
• Multiple control comparisons
• Multiple populations
• Multiple endpoints
• Shared control groups
• Sequential treatment additions or removals

A platform-specific multiplicity strategy should therefore define the relevant family of confirmatory claims and account for the design's dependence structure.

Interactive Example: Family-Wise Error Inflation

The following figure illustrates how the probability of at least one false positive can increase as the number of independent unadjusted tests increases.

Understanding the 5% Level

A common misconception is:

That statement is not generally correct when multiple confirmatory decisions are made.

The 5% level needs to be associated with the appropriate family and testing procedure.

Multiplicity Does Not Mean "Make Every p-Value Larger"

A useful conceptual correction is that multiplicity procedures are about controlling a defined error rate, not simply penalizing p-values.

For example, a fixed-sequence procedure can allow a later hypothesis to be tested at the full alpha after an earlier hypothesis succeeds.

Similarly, a graphical procedure can recycle alpha according to a prespecified transition matrix.

A Practical Hypothesis Tree

Multiplicity and Statistical Programming

Statistical programmers must implement the prespecified strategy exactly as defined in the SAP.

Programming requirements may include:

• Raw p-values
• Adjusted p-values
• Sequential decision flags
• Alpha allocation
• Gatekeeping rules
• Alpha recycling
• Confidence interval adjustment
• Interim boundary calculations
• Multiplicity-adjusted tables

Simple Bonferroni Calculation in R

p_values <- c(
  PFS = 0.008,
  OS  = 0.021,
  ORR = 0.041,
  DoR = 0.012
)

alpha <- 0.05

adjusted_alpha <- alpha / length(p_values)

decision <- p_values <= adjusted_alpha

data.frame(
  Endpoint = names(p_values),
  P_Value = p_values,
  Adjusted_Alpha = adjusted_alpha,
  Reject = decision
)

For four equally weighted hypotheses, the Bonferroni threshold is:

Holm Adjustment in R

p_values <- c(
  PFS = 0.008,
  OS  = 0.021,
  ORR = 0.041,
  DoR = 0.012
)

holm_p <- p.adjust(
  p_values,
  method = "holm"
)

data.frame(
  Endpoint = names(p_values),
  Raw_P = p_values,
  Holm_Adjusted_P = holm_p
)

The p.adjust() function provides a convenient implementation of several standard p-value adjustment procedures.

Fixed-Sequence Testing in R

A fixed-sequence procedure can be implemented explicitly rather than relying on a generic adjustment function.

p_pfs <- 0.008
p_os  <- 0.021
p_orr <- 0.041
p_dor <- 0.012

alpha <- 0.05

reject_pfs <- p_pfs <= alpha

reject_os <- reject_pfs &&
  p_os <= alpha

reject_orr <- reject_os &&
  p_orr <= alpha

reject_dor <- reject_orr &&
  p_dor <= alpha

data.frame(
  Endpoint = c(
    "PFS",
    "OS",
    "ORR",
    "DoR"
  ),
  Reject = c(
    reject_pfs,
    reject_os,
    reject_orr,
    reject_dor
  )
)

Why Explicit Decision Logic Matters

For a complex multiplicity strategy, a table of p-values is not enough.

The program should encode the actual decision tree.

For example:

if (PFS_success) {

    test_OS()

    if (OS_success) {

        test_ORR()

        if (ORR_success) {
            test_DoR()
        }
    }
}

The production implementation may be more sophisticated, but the underlying logic should remain transparent.

Graphical Alpha Allocation

Suppose the initial allocation is:

with:

If hypothesis 1 succeeds, a prespecified portion of its alpha may be transferred to hypothesis 2.

The exact transition structure must be documented before confirmatory results are available.

Multiplicity QC

Multiplicity programming requires specialized quality control.

At minimum, verify:

• The correct hypothesis family is included.
• Every hypothesis has the correct alpha allocation.
• The testing sequence matches the SAP.
• Gatekeeping rules are implemented correctly.
• Alpha recycling occurs only when permitted.
• Failed gates prevent prohibited downstream claims.
• Interim and final analyses use the correct boundaries.
• Adjusted confidence intervals correspond to the inferential procedure.
• Rounding does not change decision logic.
• All confirmatory claims can be traced to the prespecified procedure.

Rounding and Boundary Decisions

Suppose the true p-value is:

and the decision threshold is:

The result is statistically significant at the threshold.

If the reporting table displays only 0.025, however, an independent reader cannot determine whether the unrounded value was above or below the boundary.

Multiplicity and P-Value Reporting

A useful reporting table may contain:

The exact columns should reflect the prespecified inferential method.

What Regulators Generally Want to Understand

For a confirmatory analysis, reviewers need to be able to determine:

• Which hypotheses were confirmatory?
• Which hypotheses belonged to the same family?
• What was the overall Type I error objective?
• How was alpha allocated?
• What was the testing sequence?
• What happened after success?
• What happened after failure?
• How were interim analyses incorporated?
• Which claims were actually supported?

Multiplicity in the Protocol

The protocol should describe the broad confirmatory testing strategy.

The SAP should then provide the operational details.

This separation is useful because the protocol describes the trial's prospective design, while the SAP provides the statistical implementation.

Multiplicity in the Statistical Analysis Plan

A strong SAP section should include:

Example SAP Language

This is an illustration rather than a universal template. The actual wording must match the study design.

Multiplicity and Regulatory Claims

The final interpretation should distinguish between:

Common Mistake: "The Smallest P-Value Wins"

Suppose ten endpoints are analyzed and the smallest p-value is 0.008.

It is tempting to write:

That statement is incomplete.

The correct question is:

Common Mistake: Adjusting Only the Secondary Endpoints

Suppose a trial has one primary endpoint and five key secondary endpoints.

It is not sufficient to say:

The primary endpoint and the secondary endpoints may form one broader family of claims depending on the planned inference.

The family definition should be based on the intended confirmatory claims.

Common Mistake: Treating Exploratory Analyses as Confirmatory

A trial may produce an impressive finding in an unplanned subgroup.

The result may be clinically interesting.

But statistical significance alone does not convert an exploratory analysis into a prespecified confirmatory claim.

Common Mistake: Ignoring Endpoint Hierarchy

If overall survival is clinically more important than a symptom endpoint, the testing hierarchy may reasonably reflect that importance.

A mechanically equal split of alpha is not always the most informative design.

Common Mistake: Assuming All Multiplicity Is Independent

Clinical-trial endpoints are often correlated.

For example:

• PFS and OS are related.
• Repeated measures within a patient are correlated.
• Biomarker-defined populations may overlap.
• Multiple doses share the same control group.

Therefore, simplistic independent-test calculations should be treated as illustrations rather than universal error-rate calculations.

Common Mistake: Confusing Statistical Significance With Regulatory Success

Even when a hypothesis passes its multiplicity threshold, regulatory success requires more than a p-value.

Review also considers:

• Magnitude of treatment effect
• Precision
• Clinical relevance
• Safety
• Consistency
• Study conduct
• Estimand interpretation
• Data quality

FDA Versus EMA: Practical Comparison

The agencies share the central objective of preventing multiplicity from producing unsupported confirmatory conclusions.

The practical emphasis can be summarized as follows.

Does FDA Require One Specific Multiplicity Method?

No single multiplicity procedure is universally appropriate for every trial.

The FDA guidance discusses several approaches to managing multiplicity, including grouping and ordering endpoints and established statistical methods.

The sponsor should select a method that is scientifically justified, statistically valid, and aligned with the intended claims.

Does EMA Require Bonferroni?

No.

Bonferroni is one possible method for controlling multiplicity, but it is not the only valid approach.

Hierarchical and other structured procedures can be more appropriate when the clinical claims have a natural ordering.

Multiplicity and Statistical Power

Multiplicity control can reduce power because the available Type I error must be distributed across a family of claims.

This creates an important design trade-off:

Therefore, trial designers should avoid declaring an unnecessarily large number of endpoints as co-equal confirmatory objectives.

Clinical Importance and Endpoint Prioritization

One practical strategy is to prioritize endpoints according to their clinical importance.

For example:

Multiplicity and the Number of Claims

A useful planning exercise is to count not only endpoints but claims.

For example, one endpoint could potentially generate several claims:

• Overall population
• Biomarker-positive population
• Biomarker-negative population
• Older patients
• Younger patients

If all are intended to support regulatory conclusions, the multiplicity structure may be substantially more complex than the endpoint list suggests.

Multiplicity Planning Checklist

End-to-End Example

Consider a hypothetical randomized trial with:

• Primary endpoint: progression-free survival
• Secondary endpoint: overall survival
• Secondary endpoint: objective response rate
• Secondary endpoint: patient-reported outcome

Suppose the scientific hierarchy is:

The sponsor proposes a fixed-sequence procedure.

The interpretation is:

1. If PFS succeeds, the OS hypothesis can be tested confirmatorily.
2. If OS succeeds, ORR can be tested confirmatorily.
3. If ORR succeeds, the PRO endpoint can be tested confirmatorily.
4. If any gate fails, downstream claims are not established by this sequence.

This design aligns the inferential hierarchy with the clinical importance of the endpoints.

What Happens If the Primary Endpoint Fails?

Suppose:

The primary endpoint fails at the prespecified threshold.

If OS was specified as a downstream hypothesis in a fixed sequence, the OS result may still be scientifically informative.

But it may not support the same confirmatory claim under the prespecified hierarchical procedure.

What Happens If the Primary Endpoint Succeeds?

Suppose:

If the strategy permits the next test at the full allocated alpha, OS may then be tested according to the prespecified sequence.

This illustrates why hierarchical procedures can be more powerful than automatically dividing alpha equally among every endpoint.

Multiplicity and Clinical Interpretation

The final report should not simply state:

A stronger interpretation states:

What to Put in Tables, Listings, and Figures

Multiplicity affects not only the primary efficacy table but the entire presentation strategy.

Useful reporting elements include:

• Raw p-values
• Adjusted p-values when appropriate
• Alpha thresholds
• Hypothesis sequence
• Decision flags
• Adjusted confidence intervals where required
• Footnotes describing the multiplicity procedure

Recommended Clinical Study Report Footnote

Multiplicity Documentation Table

Regulatory Review Perspective

From a reviewer perspective, a multiplicity strategy should be understandable without reconstructing the sponsor's intentions from the results.

The reviewer should be able to see:

• What was planned?
• What was confirmatory?
• What was exploratory?
• What was the error-control family?
• What alpha was available?
• How was alpha allocated?
• Which hypotheses passed?
• Which claims therefore remain supported?

The Difference Between a Statistical Finding and a Confirmatory Claim

This distinction is central to regulatory statistics.

A statistical finding can be interesting.

A confirmatory claim requires that the finding arise within an appropriate prespecified inferential framework.

Therefore:

How Multiplicity Fits Into the Overall Trial Design

Practical Decision Tree

When deciding how to handle multiplicity, ask the following questions in order.

1. Are there multiple confirmatory claims?
2. Do they belong to one inferential family?
3. Does success require all claims or at least one?
4. Is there a clinically meaningful hierarchy?
5. Can later hypotheses be tested only after earlier success?
6. Are there multiple treatment comparisons?
7. Are there multiple doses?
8. Are there multiple interim looks?
9. Are subgroup findings intended to support regulatory claims?
10. Are confidence intervals intended to support simultaneous claims?

The answers determine the appropriate statistical architecture.

Multiplicity and Study Power

Multiplicity planning should occur before final sample-size determination when the multiplicity strategy affects the critical values used for the primary analysis.

For example, if alpha is split across multiple co-primary hypotheses, the required sample size may differ from a design using a single primary endpoint.

Similarly, a complex hierarchical strategy may affect the probability that specific secondary claims can be established.

Multiplicity and Simulation

Complex procedures should often be evaluated by simulation.

Simulation can examine:

• Family-wise Type I error
• Power for each hypothesis
• Probability of reaching downstream gates
• Impact of endpoint correlation
• Impact of interim analyses
• Operating characteristics of adaptive procedures

A simulation framework is especially valuable when the multiplicity procedure is more complicated than a simple closed-form adjustment.

Illustrative Simulation in R

set.seed(123)

B <- 100000
m <- 5
alpha <- 0.05

false_positive <- replicate(
  B,
  any(
    runif(m) < alpha
  )
)

mean(false_positive)

The simple simulation above illustrates the unadjusted probability of at least one false-positive result under independent uniform p-values.

A more realistic clinical-trial simulation should generate correlated test statistics according to the planned endpoint structure.

Simulation of Bonferroni Control

set.seed(123)

B <- 100000
m <- 5
alpha <- 0.05

alpha_bonferroni <- alpha / m

false_positive <- replicate(
  B,
  any(
    runif(m) < alpha_bonferroni
  )
)

mean(false_positive)

This illustrates the basic principle of distributing alpha across five tests.

Multiplicity and Correlated Endpoints

Clinical endpoints are often correlated.

If two endpoints are strongly positively correlated, the actual family-wise error behavior may differ substantially from the independent-test example.

This is one reason regulatory multiplicity planning should consider the specific testing procedure rather than relying on simplistic formulas.

Multiplicity and Missing Endpoints

Suppose one of three key secondary endpoints cannot be analyzed because of a data-quality issue.

The SAP should define how the testing strategy handles the unavailable endpoint.

Possible approaches depend on the prespecified procedure.

The key principle is:

Multiplicity and Early Trial Termination

If a trial terminates early for efficacy, futility, safety, or operational reasons, the statistical interpretation should follow the prespecified interim and stopping framework.

An early stopping decision can interact with:

• Interim Type I error
• Multiplicity across endpoints
• Conditional power
• Alpha spending
• Confidence intervals

Multiplicity and Futility

Futility rules are not automatically equivalent to efficacy multiplicity rules.

A nonbinding futility boundary, for example, may have a different role in the operating characteristics of the trial than an efficacy boundary.

The statistical plan should distinguish:

• Efficacy stopping
• Futility stopping
• Safety stopping
• Operational stopping

Multiplicity and Adaptive Designs

Adaptive designs can modify aspects of the trial based on accumulating information.

Examples include:

• Sample-size reassessment
• Dropping treatment arms
• Population enrichment
• Changing randomization probabilities
• Selecting a dose

When adaptive modifications interact with confirmatory hypotheses, the multiplicity and Type I error properties should be evaluated jointly.

Multiplicity and Master Protocols

Master protocols can include basket, umbrella, and platform designs.

The multiplicity problem may involve:

• Multiple diseases
• Multiple biomarkers
• Multiple treatment arms
• Multiple shared controls
• Multiple endpoint families

The definition of the error-control family becomes especially important.

Multiplicity and Bayesian Methods

Bayesian analyses require a somewhat different inferential framework.

The classical FWER concept does not automatically translate directly to every Bayesian decision rule.

However, multiplicity concerns do not disappear simply because a Bayesian model is used.

The decision criterion, prior structure, posterior probabilities, and intended claims should be prespecified and justified.

Multiplicity and External Data

External controls, historical controls, and real-world data can create additional sources of uncertainty and potentially multiple analytical questions.

If multiple external data sources or matching strategies are explored, the distinction between exploratory and confirmatory inference remains important.

Multiplicity and Real-World Evidence

Real-world evidence analyses can involve particularly large numbers of covariates, subgroups, endpoints, and sensitivity analyses.

Not every descriptive analysis requires classical FWER control.

But a prespecified confirmatory claim should have an inferential strategy appropriate to the number and relationship of hypotheses used to establish that claim.

How to Explain Multiplicity to a Clinical Team

A concise explanation is:

How to Explain Multiplicity to a Programmer

A concise programming explanation is:

How to Explain Multiplicity to a Reviewer

A concise regulatory explanation is:

Quality-Control Checklist for a Multiplicity Program

Final Practical Workflow

Key Takeaways

The central lesson of FDA and EMA multiplicity guidance is not simply "adjust your p-values."

The more important principle is:

Several practical conclusions follow.

• Multiplicity arises from multiple confirmatory questions, not merely from the number of endpoints.
• Family-wise Type I error is a property of the complete testing procedure.
• Bonferroni is useful but is not the only regulatory approach.
• Hierarchical and fixed-sequence procedures can efficiently reflect clinical priorities.
• Gatekeeping can organize multiple endpoint families.
• Alpha recycling can improve efficiency when properly prespecified.
• Interim analyses introduce another dimension of multiplicity.
• Subgroup analyses require careful distinction between interaction evidence and within-subgroup significance.
• Confidence intervals should be interpreted consistently with the inferential procedure.
• Exploratory findings should not be presented as though they were prespecified confirmatory claims.
• The SAP should contain an explicit, reproducible decision algorithm.
• Complex procedures should be validated analytically or by simulation as appropriate.

References

U.S. Food and Drug Administration. (2022). Multiple Endpoints in Clinical Trials: Guidance for Industry. Final Guidance, October 2022.

European Medicines Agency. (2002/2003). Points to consider on multiplicity issues in clinical trials. CPMP/EWP/908/99. Current effective version identified by EMA with legal effective date 1 April 2003.

European Medicines Agency. (2017). Guideline on multiplicity issues in clinical trials. Draft guideline developed as an update of the earlier Points to Consider document; consultation closed in 2017.

International Council for Harmonisation. ICH E9: Statistical Principles for Clinical Trials.

International Council for Harmonisation. ICH E9(R1): Addendum on Estimands and Sensitivity Analysis in Clinical Trials.

Westfall, P.H. and Krishen, A. Multiple testing and clinical-trial inference methods provide a broader statistical framework for understanding family-wise error control and structured hypothesis testing.