Why Two Kinds of Multiplicity Arise
A monitored trial can examine more than one endpoint and examine accumulating data at more than one interim look. These create two distinct ways to make a false claim. A trial might falsely declare an effect on one of several endpoints, or it might falsely declare an effect on one endpoint at one of several opportunities to look at the data. Controlling only one of these risks does not automatically control the other.
The key is to define the set of claims before assessing error control. An interim look is an occasion to apply a decision rule; it is not necessarily a new scientific hypothesis. Conversely, if efficacy on any of several endpoints can support a confirmatory conclusion, those endpoint claims all matter to the familywise error rate, even if each endpoint is monitored using a valid group sequential procedure.
For example, let \(H_i\) denote the null hypothesis for endpoint \(i\), and let \(R_i\) be the event that the trial rejects \(H_i\) at any allowed look. If the family contains \(m\) endpoint hypotheses, its FWER is
The probability is evaluated under the trial's data-generating model and decision rules. Strong FWER control at level \(\alpha\) means that this probability is at most \(\alpha\) for every configuration of true and false null hypotheses in the family. It is not enough to control error only in the special case where all null hypotheses are true.
Endpoint Families and Repeated Looks
Endpoint multiplicity concerns which hypotheses can lead to a claim. If a trial plans to claim efficacy on either its primary endpoint or a key secondary endpoint, the family must reflect both claims if either can independently support a confirmatory conclusion. If secondary endpoints are purely descriptive and cannot support such a claim, their role may differ; the intended claims and decision rules determine the family.
Sequential error concerns when a hypothesis may be rejected. For a single endpoint \(i\), define its sequential rejection event as the union of rejection events over all scheduled looks. A group sequential test calibrated so that \(P(R_i\mid H_i\text{ true})\leq \alpha_i\) controls the probability of a false rejection for that endpoint across its looks. This does not, by itself, ensure that the probability of a false rejection on at least one of several endpoints is at most \(\alpha_i\), or at most any other desired family level.
The same distinction applies when endpoints have different schedules or information levels. Each endpoint may have its own valid rule across its own looks, while the trial-level error probability still depends on the union of the endpoint rejection events. Dependence between endpoints can affect that union probability. Dependence among interim statistics also matters when assessing the validity of a sequential rule; repeated looks at cumulative data are generally not independent tests.
Worked Example: Two Endpoints, Each Monitored Sequentially
Suppose a hypothetical trial has two confirmatory endpoints, \(A\) and \(B\). Each endpoint is monitored at several planned looks. Assume the sequential procedure for each endpoint has been validly calibrated so that, if its own null hypothesis is true, its probability of rejection at any look is \(0.05\). The endpoint-level rejection events are assumed independent under the global null for this example.
The family contains \(H_A\) and \(H_B\). Under the global null, both are true. The probability of no false rejection on endpoint \(A\) is \(1-0.05=0.95\), and likewise for endpoint \(B\). By the stated independence assumption, the probability of no rejection on either is \(0.95^2=0.9025\). Therefore,
Thus, despite each endpoint having sequential error control at \(0.05\) across its looks, the probability of at least one false endpoint claim is \(0.0975\) under these assumptions. This calculation does not establish the FWER for correlated endpoints; it illustrates why endpoint-level control alone is not the same as familywise control.
A General Bound and Its Meaning
A useful general result does not require independence. If endpoint \(i\)'s sequential rejection event has probability at most \(\alpha_i\) when \(H_i\) is true, then the probability of at least one false rejection is at most the sum of these endpoint-specific bounds over the true null hypotheses. This follows from the union bound.
where \(I_0\) is the set of endpoints whose null hypotheses are true. To see the first inequality, the indicator of a union of events is never larger than the sum of their individual indicators. Taking expectations gives \(P(\bigcup_i R_i)\leq\sum_i P(R_i)\). The second inequality uses the assumed sequential error bound for each true endpoint.
Consequently, if the endpoint-specific bounds sum to no more than a chosen family level \(\alpha\), the union bound guarantees FWER no greater than \(\alpha\), regardless of dependence between endpoint rejection events. This bound can be conservative. A smaller actual FWER under some dependence pattern does not justify assuming that pattern in advance or claiming universal control from it.
Worked Example: A Dependence-Free Upper Bound
Consider a hypothetical trial with three endpoints. Suppose the sequential procedures for the endpoints have been calibrated, respectively, to \(0.02\), \(0.015\), and \(0.01\) under their own true null hypotheses. Assume these bounds apply over all looks for each endpoint. No independence assumption is made.
If all three null hypotheses are true, the union bound gives
Thus the FWER is no greater than \(0.045\), which is below \(0.05\), without specifying how the endpoints are correlated. If only the first and third null hypotheses are true, the same reasoning bounds the probability of any false rejection by \(0.02+0.01=0.03\). The calculation uses the endpoint-level bounds for the true nulls; it does not add error probabilities for false nulls as though rejecting them were a type I error.
Why Repeated Looks Need Their Own Control
Endpoint multiplicity is not the only issue. A procedure that tests the same null hypothesis repeatedly at an unadjusted nominal level can have a larger chance of at least one false rejection than a single test at that level. The size of the increase depends on the joint distribution of the test statistics and on the stopping rule. Group sequential methods address this repeated-opportunity problem by calibrating decisions over the planned looks.
Worked Example: Naive Repeated Testing at One Endpoint
For a simple illustration, imagine a hypothetical single-endpoint trial with two tests under a true null. Suppose each test has a false-positive probability of \(0.025\), and, just for this illustration, the two test outcomes are independent. If either test can trigger rejection, the probability of at least one false rejection is
The result is \(0.049375\), or about \(4.94\%\), rather than \(2.5\%\). This is an illustration of repeated testing, not a model for the dependence of ordinary interim statistics based on accumulating trial data. In a real sequential design, the test statistics across looks are typically dependent, so the probability must be evaluated using the design's joint model and decision rule rather than by multiplying probabilities as if the looks were independent.
The two error dimensions can coexist. A sequential procedure addresses the repeated looks for an endpoint; endpoint-level control addresses the chance of false claims across endpoints. A complete trial-level statement must specify both the family of claims and the monitoring rules under which rejections can occur.
Assumptions, Diagnostics, and Limitations
Before using an error-control statement, check that its assumptions match the planned trial:
- Complete family: list every null hypothesis whose rejection could support a confirmatory claim. Omitting an endpoint that can trigger a claim makes the reported familywise guarantee incomplete.
- Complete monitoring rule: identify the interim and final looks, the rejection rules at each look, and whether early rejection on any endpoint is permitted. Error control for one schedule does not automatically cover an unplanned schedule or altered stopping rule.
- Valid endpoint-level calibration: verify that the stated \(\alpha_i\) bound applies over all looks for endpoint \(i\), not merely at one selected look. A nominal per-look threshold is not itself evidence of sequential control.
- Dependence assumptions: use an independence calculation only when independence of the relevant endpoint-level rejection events is justified. Otherwise, a dependence-robust bound such as the union bound may be used, or the joint distribution must be modeled appropriately.
- Strong rather than only global-null control: ensure the claim covers configurations where some endpoints have real effects and others remain null, if strong FWER control is required.
A frequent diagnostic warning is a protocol or report that gives one significance level for each endpoint and separately describes several looks, but never states the probability of a false claim across the full set of endpoint-look decisions. Another warning is an error calculation that treats every look as an independent test without a model-based justification. Finally, an FWER guarantee covers only the prespecified family and decision process; it does not automatically cover unplanned endpoints, post hoc subgroup claims, or changes to monitoring rules.
The union bound is transparent and robust to dependence, but it may be conservative. An exact calculation can be less conservative when the joint distribution of the endpoint procedures is known and correctly modeled. Neither approach removes the need to define the family first. The introductory material on group sequential testing concerns repeated looks for a hypothesis; the additional issue here is the trial-wide collection of endpoint claims and the probability of any false claim across that collection.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A trial has two endpoints, and either can independently support an efficacy claim. Both are monitored at interim looks. What determines whether both belong to the hypothesis family?
- If each of four true endpoint nulls has sequential rejection probability at most \(0.01\), what upper bound does the union bound give for FWER?
- Why is it generally inappropriate to compute the probability of rejection across interim looks by treating cumulative-data tests as independent?
- Under the global null, two endpoint-level rejection events are independent and each has probability \(0.04\). What is the probability of at least one rejection?
Solutions. (1) Both belong if rejection of either can support a confirmatory claim; the planned claims and decision rules define the family. (2) The bound is \(4(0.01)=0.04\), so FWER is at most \(0.04\). (3) Cumulative data at different looks generally produce dependent test statistics, so independence-based multiplication need not represent the joint rejection probability. (4) The probability of no rejection is \((1-0.04)^2=0.96^2=0.9216\); therefore the probability of at least one is \(1-0.9216=0.0784\), or \(7.84\%\).