Multiplicity Across Primary Endpoints

Why Endpoint Multiplicity Needs Its Own Plan

A seamless phase 2/3 trial may select a treatment at the stage transition and then make confirmatory claims on more than one primary endpoint. These are two distinct sources of multiplicity. The treatment-selection procedure must protect the validity of evidence for the treatment that continues; the endpoint procedure must protect the set of endpoint claims made from that evidence. Solving one problem does not automatically solve the other.

First decide what “success” means. If success means that at least one of several endpoints shows benefit, each endpoint can become a separate opportunity to declare success, so the family of endpoint claims needs multiplicity control. If success requires benefit on every co-primary endpoint, the overall claim is a conjunction. Testing each endpoint at level $\alpha$ and declaring overall success only when all tests reject controls the type I error for that conjunction; no division of $\alpha$ is needed for that one all-endpoints claim. Separate claims about individual endpoints still require multiplicity control.

Definition: For endpoint $j$, let $H_j$ denote the null hypothesis of no treatment benefit on that endpoint for the treatment that continues after selection. The endpoint family is the set of confirmatory claims the trial permits investigators to make. Strong family-wise error control at level $\alpha$ means that, for every configuration of true and false null hypotheses in this family, the probability of rejecting at least one true null is no greater than $\alpha$.

The hypotheses, endpoint definitions, direction of benefit, and claim structure should be specified in advance. For a superiority trial, a one-sided endpoint test may be written as $H_j:\theta_j\leq 0$ against $H_{aj}:\theta_j>0$, where larger $\theta_j$ represents greater benefit. If benefit corresponds to a lower outcome, orient the effect or define the alternative accordingly. The multiplicity procedure applies to valid p-values for these prespecified hypotheses; it cannot fix an ambiguous estimand or an endpoint whose meaning changes after seeing the data.

Separate Stage Combination From Endpoint Adjustment

For each endpoint, first construct a valid confirmatory p-value that accounts for the seamless design and treatment selection. One common construction combines evidence from the stages using the inverse-normal method. The endpoint p-value produced by that construction is then adjusted across the endpoint family. The order matters conceptually: an ordinary p-value calculated after selecting the most promising treatment is not made valid merely by applying Holm or Bonferroni to it.

Definition: For endpoint $j$, let $p_{1j}$ and $p_{2j}$ be the stagewise p-values used in the confirmatory combination test. With prespecified nonnegative weights $w_1,w_2$ satisfying $w_1^2+w_2^2=1$, the inverse-normal statistic and its one-sided p-value are $$ Z_j=w_1\Phi^{-1}(1-p_{1j})+w_2\Phi^{-1}(1-p_{2j}),\qquad p_j=1-\Phi(Z_j). $$ Here $\Phi$ is the standard normal cumulative distribution function. The stagewise p-values must be valid for the selected treatment under $H_j$, and the combination procedure must have its stated null distribution under the design’s stagewise assumptions.

In particular, the stagewise p-values cannot simply be treated as ordinary unselected p-values if treatment selection used the same phase 2 outcomes. The selection-aware validity methods from the earlier material on valid stagewise p-values and combination tests are needed. The usual inverse-normal justification also relies on the prespecified weights and appropriate stagewise null behavior, including the independent-increments structure or another justified calibration. If the selection rule or information structure changes, the p-value calculation must be revalidated rather than assumed to remain valid.

Worked Example: Combining Evidence for a Selected Treatment

Suppose a hypothetical seamless trial selected one treatment using a prespecified phase 2 rule. For endpoint B, the selection-valid stagewise p-values for the selected treatment are $p_{1B}=0.08$ and $p_{2B}=0.03$. The trial prespecified inverse-normal weights $w_1=0.6$ and $w_2=0.8$, so $0.6^2+0.8^2=1$. Assume the required stagewise validity and null calibration conditions hold.

The one-sided normal quantiles are $\Phi^{-1}(0.92)\approx1.4051$ and $\Phi^{-1}(0.97)\approx1.8808$. Thus $$ Z_B=0.6(1.4051)+0.8(1.8808) =0.8431+1.5046 =2.3477. $$ The corresponding upper-tail probability is $$ p_B=1-\Phi(2.3477)\approx0.00944. $$

This is the combined p-value for endpoint B, not yet an endpoint-adjusted p-value. Its interpretation is that, under the endpoint B null and the stated design assumptions, the combined statistic is at least this extreme with probability about 0.00944. It is not the probability that the null is true.

Controlling Error Across Endpoint Claims

If any one endpoint can support a positive confirmatory conclusion, a simple strategy is Bonferroni: for $m$ endpoint hypotheses, test each valid endpoint p-value at $\alpha/m$. The argument does not require the endpoints to be independent. If $I_0$ is the set of true endpoint nulls, then $$ P(\text{reject at least one true }H_j) \leq \sum_{j\in I_0}P(p_j\leq\alpha/m) \leq |I_0|\frac{\alpha}{m} \leq\alpha. $$ The first inequality is the union bound; the second follows because each p-value is valid under its null. This also shows why endpoint-level validity after treatment selection is essential.

Holm’s step-down procedure can be more powerful while retaining strong family-wise error control under arbitrary dependence. Sort the $m$ endpoint p-values as $p_{(1)}\leq\cdots\leq p_{(m)}$. Starting with the smallest, compare $p_{(i)}$ with $\alpha/(m-i+1)$. Reject in order until the first comparison fails; do not reject that hypothesis or any larger p-value. For two endpoints, compare the smaller p-value with $\alpha/2$, then compare the larger with $\alpha$ only if the first comparison passes.

Worked Example: Applying Holm Across Three Endpoints

In a hypothetical trial, the selection-valid combined p-values for three prespecified endpoint claims are 0.012, 0.031, and 0.20. Suppose the family-wise level is $\alpha=0.05$. Order them as shown: $p_{(1)}=0.012$, $p_{(2)}=0.031$, and $p_{(3)}=0.20$.

The first Holm threshold is $0.05/3=0.01667$. Since $0.012\leq0.01667$, reject the first endpoint null. The second threshold is $0.05/2=0.025$, but $0.031>0.025$. The step-down procedure stops there, so neither the second nor the third endpoint null is rejected. The conclusion is one confirmatory endpoint claim, not two. The p-values are hypothetical and already assumed valid for the selected treatment before endpoint multiplicity adjustment.

Holm’s guarantee can be understood through closed testing. For every nonempty subset of endpoint hypotheses, test the intersection null that all hypotheses in that subset are true at level $\alpha$. An individual $H_j$ is rejected only if every intersection containing $H_j$ is rejected. Using Bonferroni local tests for those intersections yields Holm’s step-down procedure. This construction explains why the procedure controls error under any dependence among endpoints, provided its local tests are valid.

Worked Example: Closed Testing for Two Endpoint Claims

Return to a hypothetical two-endpoint family with combined p-values $p_A=0.000756$ and $p_B=0.00944$, at family-wise level 0.05. To reject $H_A$, closed testing requires rejection of both $H_A$ alone and the intersection $H_A\cap H_B$. To reject $H_B$, it requires rejection of $H_B$ alone and that same intersection.

Each singleton hypothesis has p-value below 0.05. For the intersection, a Bonferroni local test has p-value no greater than $$ \min\{1,2\min(p_A,p_B)\} =2(0.000756) =0.001512. $$ This is below 0.05, so the intersection is rejected and both endpoint null hypotheses can be rejected. This example illustrates the logical structure of closed testing; the reported intersection p-value is a valid Bonferroni bound, not necessarily the smallest possible p-value obtainable from information about endpoint dependence.

Co-Primary Endpoints and the Claim Being Tested

When the protocol defines overall success as demonstrating benefit on every co-primary endpoint, the null for that single overall claim is that at least one endpoint fails to show benefit. Test each endpoint at level $\alpha$, and declare overall success only if every endpoint rejects its own null. If the overall null is true, at least one component null is true. The probability of declaring overall success is therefore no greater than the probability of rejecting that true component null, which is at most $\alpha$. This is an intersection-union test.

This rule does not authorize separate, unadjusted claims that whichever endpoint succeeds establishes efficacy. Nor does it cover multiple populations, doses, or other confirmatory families unless those claims are included in the error-control plan. A protocol should make clear whether endpoints are co-primary, whether any one can establish success, and which endpoint-specific claims may be reported as confirmatory.

Worked Example: Distinguishing “Any” From “All” Success

Suppose two selection-valid endpoint tests in a hypothetical trial give p-values 0.018 and 0.041, with $\alpha=0.05$. If the trial claims success when either endpoint is positive, Bonferroni requires each p-value to be at most $0.05/2=0.025$. The first passes, but the second does not, so the multiplicity-controlled conclusion is a claim on the first endpoint only.

If instead the prespecified overall success criterion requires both co-primary endpoints to pass, each is compared with 0.05 and both pass: $0.018\leq0.05$ and $0.041\leq0.05$. The conjunction is successful at level 0.05. The two conclusions differ because they answer different confirmatory questions, not because the p-values changed.

Diagnostics, Limitations, and Common Mistakes

Before interpreting adjusted results, check the whole path from selection to claims. Verify that the selected treatment was chosen by the prespecified rule; the endpoint definitions, directions, and weights were fixed as planned; and each combined endpoint p-value is valid under its own null after selection. Check that the number of endpoints in the adjustment matches the family of claims. If a reported analysis adds endpoints or omits an endpoint after seeing the results, the stated family-wise guarantee may no longer apply.

Dependence among endpoints is not, by itself, a failure for Bonferroni or Holm. It may mean those methods are conservative, because they do not fully exploit the correlation structure. A prespecified closed-testing strategy with valid, dependence-aware local tests can sometimes gain power, but its local tests must be justified for every intersection null. Conversely, adjusting ordinary p-values does not repair selection bias, invalid stagewise p-values, unplanned changes to the endpoint family, or an unsuitable estimand.

Mistake: “Holm corrects for selecting the treatment.” It corrects across the endpoint family only when its input p-values are already valid for the selected treatment. Use selection-aware stagewise inference first.
Mistake: “Two primary endpoints always mean divide alpha by two.” The right rule depends on the claim. “Either endpoint” requires multiplicity control; an all-endpoints conjunction can use level-$\alpha$ component tests and requires every test to pass.
Mistake: “Endpoint p-values must be independent for Holm.” Holm controls family-wise error under arbitrary endpoint dependence when each null p-value is valid.
Mistake: “A successful adjusted test proves clinical importance.” A p-value addresses evidence against a null, not the magnitude or clinical relevance of benefit. Report effect estimates, uncertainty, and units alongside confirmatory conclusions.

Key takeaway: Build the confirmatory strategy in two layers: obtain a valid p-value for each endpoint after the adaptive treatment selection and stage combination, then control error across the prespecified endpoint claims. State separately whether success means “any endpoint” or “all co-primary endpoints.”

Check Your Understanding

Answer each question, then compare with the brief solutions below.

A trial has four endpoint claims and valid endpoint p-values. What Bonferroni threshold controls family-wise error at 0.05?
For two co-primary endpoints, the success rule requires both to reject. Why can each be tested at level 0.05 for the single overall claim?
Two valid endpoint p-values are 0.01 and 0.04. Apply Holm at level 0.05.
Why does applying Holm to unadjusted p-values after data-dependent treatment selection not necessarily provide valid confirmatory inference?
With inverse-normal weights $w_1=0.6$ and $w_2=0.8$, what condition must the weights satisfy, and why must they be prespecified?

Solutions. (1) The threshold is $0.05/4=0.0125$ for each endpoint. (2) If the overall null is true, at least one component null is true; success requires rejecting every component, so the chance of overall success is no greater than the rejection probability for that true component, at most 0.05. (3) The ordered p-values are 0.01 and 0.04. Compare 0.01 with $0.05/2=0.025$, so reject the first; then compare 0.04 with 0.05, so reject the second as well. (4) The p-values may not be valid under the null after selection; endpoint adjustment does not undo treatment-selection bias. (5) They must be nonnegative and satisfy $w_1^2+w_2^2=1$; prespecification supports the stated null calibration and prevents choosing weights to favor observed results.