Multiplicity Across Endpoints, Doses, and Assessment Times

Why Multiplicity Is a Planning Problem

A trial may compare several doses with a control, assess more than one endpoint, or measure an outcome at several times. Each comparison can create an opportunity for a false-positive claim. Choosing the analysis after seeing which dose, endpoint, or time looks most favorable does not remove that opportunity; it can make the reported error rate substantially larger than the nominal level of a single test.

Multiplicity planning starts with the intended claims, not simply with a count of variables in the data set. If a confirmatory conclusion may be based on any one of several comparisons, those hypotheses generally belong to a common testing family. Conversely, collecting several outcomes does not automatically mean every outcome must be included in one family if only a prespecified primary claim is confirmatory and the others are clearly exploratory. The family and the procedure should be specified before examining outcome results.

Definition: For a family containing hypotheses $H_1,\ldots,H_m$, the family-wise error rate (FWER) is the probability of rejecting at least one null hypothesis that is true: $$ \mathrm{FWER}=P(\text{at least one true null hypothesis is rejected}). $$ A procedure has strong FWER control at level $\alpha$ if this probability is no greater than $\alpha$ for every configuration of true and false null hypotheses in the family.

The word “family” is therefore a scientific and decision-making choice. It should include the opportunities for making the claims that matter. A family might contain all active-dose-versus-control tests for one primary endpoint, all confirmatory endpoints, or every prespecified treatment contrast at each confirmatory assessment time. Testing more hypotheses can increase the adjustment required, but including unrelated exploratory analyses indiscriminately can obscure which conclusions are actually confirmatory.

Error Control Is Not the Same as Power

FWER control limits false-positive risk across a family. It does not say that every treatment effect has the same probability of detection. Marginal power is the probability of rejecting a particular null hypothesis when its alternative is true. Disjunctive power is the probability of rejecting at least one relevant false null, while conjunctive power is the probability of rejecting all specified false nulls. These answer different planning questions: detecting any effective dose is not the same objective as establishing efficacy for every dose or endpoint.

The tests may be dependent. Outcomes measured on the same participants, treatment-versus-control contrasts that share a control group, and measurements at nearby times often have correlated test statistics. Dependence affects the joint probability of one or more rejections and therefore can affect power. Some procedures remain valid without specifying that dependence; procedures that use it can sometimes be more powerful, but require a suitable joint model or justified correlation estimates.

Bonferroni procedure: For $m$ hypotheses, test each two-sided hypothesis at level $\alpha/m$, or equivalently compare its unadjusted $p$-value with $\alpha/m$. This controls FWER at no more than $\alpha$ under any dependence among the tests.

The justification is the union bound. Let $A_i$ be the event of rejecting a true $H_i$, and let $I_0$ be the index set of true null hypotheses. If every individual test has type I error no greater than $\alpha/m$, then

\begin{aligned} \mathrm{FWER} &=P\left(\bigcup_{i\in I_0} A_i\right)\\ &\leq \sum_{i\in I_0}P(A_i)\\ &\leq |I_0|\frac{\alpha}{m} \leq \alpha. \end{aligned}

This argument explains both the advantage and limitation of Bonferroni: it does not require independent tests, but it can be conservative, especially when tests are strongly correlated or when many hypotheses are included.

Procedures for Prespecified Families

Bonferroni is simple and useful for planning, but it is not the only way to control FWER. Holm’s step-down procedure also provides strong FWER control under arbitrary dependence and is at least as powerful as applying the same Bonferroni threshold to every test. Sort the $m$ unadjusted $p$-values from smallest to largest, $p_{(1)}\leq\cdots\leq p_{(m)}$. Compare $p_{(1)}$ with $\alpha/m$; if it passes, compare $p_{(2)}$ with $\alpha/(m-1)$, continuing in order. Stop at the first failure; that hypothesis and all remaining hypotheses are not rejected. Holm is useful when a set of claims is intended, but the likely ordering of their evidence is not known in advance.

A fixed-sequence or hierarchical procedure is appropriate when the claims have a prespecified priority. For example, test a primary endpoint first at level $\alpha$, and test a secondary endpoint at level $\alpha$ only if the primary test rejects. The protected conclusion is limited by the sequence: a later claim cannot be declared confirmatory if an earlier required test fails. The order must be specified in advance rather than chosen to follow the observed $p$-values.

For multiple doses compared with a shared control, a Dunnett procedure can use the known structure and correlation of those contrasts to control FWER, often with better power than Bonferroni. For correlated endpoints or assessment times, a prespecified maximum-statistic or other joint-model procedure may also use their correlation. Such methods are not automatically preferable: their validity and operating characteristics depend on the model, covariance assumptions, and planned analysis being appropriate. Bonferroni or Holm can be more robust when that structure is uncertain.

Worked Example: Planning for Three Confirmatory Endpoints

Suppose a hypothetical two-arm trial will support a confirmatory claim if it shows benefit on any of three prespecified endpoints. For planning, assume each endpoint has a treatment difference of 4 units, a common standard deviation of 10 units, equal group sizes, and a two-sided test. The target marginal power is 90% for each endpoint, and the family-wise level is $\alpha=0.05$. Use Bonferroni planning.

The family has $m=3$ tests, so the per-test two-sided level is $\alpha^*=0.05/3=0.01667$. Under the planning assumptions, the estimated difference has standard error $\sigma\sqrt{2/n}$, where $n$ is the number of participants per group. For a positive target difference $\delta$, the usual normal-approximation sample-size formula is

n=\frac{2\sigma^2\left(z_{1-\alpha^*/2}+z_{1-\beta}\right)^2}{\delta^2}.

Here $z_q$ is the $q$-quantile of the standard normal distribution, and $1-\beta=0.90$. Using $z_{1-0.01667/2}\approx2.394$, $z_{0.90}\approx1.282$, $\sigma=10$, and $\delta=4$:

n= \frac{2(10)^2(2.394+1.282)^2}{4^2} = \frac{200(3.676)^2}{16} \approx168.9.

Rounding up gives 169 participants per group, or 338 in total, before any allowance for attrition. This design targets about 90% marginal power for each endpoint at the Bonferroni threshold; it does not, by that calculation alone, guarantee 90% probability of rejecting all three hypotheses. The endpoints may be correlated, but that correlation is not needed for Bonferroni FWER control.

Planning Across Doses and Assessment Times

For doses, the family may contain every planned active-dose-versus-control contrast, or a smaller set if only particular dose comparisons can support a confirmatory claim. Shared-control contrasts are correlated, which makes a correlation-aware procedure such as Dunnett a possible alternative. A conservative Bonferroni sample-size calculation applies the threshold $\alpha/m$ to every planned contrast. A design using Holm can also be assessed conservatively at that threshold, since its first comparison uses $\alpha/m$ and later comparisons can be less stringent. Exact power for a step-down procedure depends on the joint distribution of the test statistics and the pattern of true effects.

For assessment times, distinguish a prespecified primary time from selecting the most favorable observed time. If the primary time is fixed in advance and other times are descriptive, the primary test need not automatically share its confirmatory error budget with every descriptive analysis. If any one of several times can establish the efficacy claim, those tests create a multiplicity problem. The same principle applies to several endpoints or endpoint-by-time contrasts. Repeated measurements can be correlated, but correlation does not justify treating the tests as independent or selecting a time after seeing the results without adjustment.

Worked Example: Holm Testing for Three Dose Comparisons

In a hypothetical dose-ranging trial, three prespecified active-dose-versus-control tests give unadjusted two-sided $p$-values of 0.008, 0.021, and 0.041. Apply Holm’s procedure at family-wise $\alpha=0.05$. Assume each $p$-value comes from a valid test of its corresponding null hypothesis.

The ordered values are $p_{(1)}=0.008$, $p_{(2)}=0.021$, and $p_{(3)}=0.041$. At the first step, $0.008<0.05/3=0.01667$, so reject the first null. At the second step, $0.021<0.05/2=0.025$, so reject the second. At the third step, $0.041<0.05$, so reject the third. Thus, all three comparisons are rejected by Holm’s procedure.

The corresponding Holm-adjusted $p$-values, in sorted order, are 0.024, 0.042, and 0.042. For example, the first is $3(0.008)=0.024$; the second is the larger of 0.024 and $2(0.021)=0.042$; the third is the larger of 0.042 and $1(0.041)=0.041$, which is 0.042. Each is below 0.05. These conclusions concern the three prespecified dose comparisons as a family; they do not establish that every possible dose or endpoint would show benefit.

Worked Example: Four Confirmatory Endpoint-Time Comparisons

Suppose a hypothetical trial plans confirmatory treatment contrasts for two outcomes at two fixed assessment times, giving four two-sided tests. Each contrast is expected to be 5 score units, with a marginal standard deviation of 12 score units. With equal allocation and a target 90% marginal power for each contrast, use Bonferroni planning at family-wise $\alpha=0.05$. Assume the model-based contrast at each time has standard error $12\sqrt{2/n}$, with $n$ participants per group.

The per-test level is $\alpha^*=0.05/4=0.0125$, so $z_{1-\alpha^*/2}=z_{0.99375}\approx2.498$. The sample-size calculation is

\begin{aligned} n &=\frac{2(12)^2(2.498+1.282)^2}{5^2}\\ &=\frac{288(3.780)^2}{25}\\ &\approx164.6. \end{aligned}

Round up to 165 participants per group, or 330 total. This calculation protects the FWER even if the tests at the two times or on the two outcomes are correlated, provided the individual tests are valid. It targets 90% marginal power for each contrast under the stated effect and variance assumptions; it does not target a 90% chance of success on all four contrasts. A joint model and a prespecified correlation-aware procedure could yield a different sample size, but that plan would need its dependence assumptions and joint power objective stated explicitly.

Assumptions, Diagnostics, and Limitations

The normal-approximation sample-size formula is appropriate when the planned treatment contrast is approximately normal, its standard error is adequately represented by the assumed variance and allocation, and the target effect and variance are reasonable planning inputs. It assumes the stated number of participants is analyzable. If the final analysis uses a small-sample $t$ distribution, a complex longitudinal model, or another estimator with a different standard error, use the corresponding model-based calculation or simulation rather than treating the normal formula as exact. The formula also does not account for losses to follow-up; any inflation for expected attrition is a separate planning step.

Several warning signs merit review before locking the plan. A family defined only after examining results, an unplanned choice of the best dose or time, or a change in the endpoint used for the primary claim can invalidate the intended error control. For correlation-aware procedures, compare assumed correlations with plausible values and check whether the planned covariance model matches the analysis. For power, inspect the effect size, variability, allocation, and definition of success. A high marginal power for each test does not imply equally high conjunctive power, and a stringent correction can make power much lower than a single-test calculation suggests.

Common pitfall: “Each of four tests uses $\alpha=0.05$, so the family has a 5% false-positive rate.” This is not generally true: the chance of at least one false rejection can exceed 5%. If the four tests define one confirmatory family, use a prespecified procedure that controls its FWER, such as Bonferroni, Holm, or a justified correlation-aware method. Do not assume independence unless it is justified and used by the procedure.

No multiplicity procedure can repair an unclear claim strategy. Bonferroni is transparent and robust but may cost power; Holm usually improves on Bonferroni without requiring dependence assumptions; hierarchical testing can prioritize claims but makes later claims conditional on earlier success; and correlation-aware methods can gain power when their joint model is credible. The choice should follow the intended clinical conclusions and be reflected in the sample-size target, not selected after observing which approach gives the most favorable result.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A trial has five confirmatory tests in one family and uses Bonferroni at $\alpha=0.05$. What is the two-sided per-test significance level?
Why does Bonferroni control FWER when the tests are correlated?
For three tests with ordered $p$-values 0.012, 0.030, and 0.060, which hypotheses does Holm reject at $\alpha=0.05$?
Does 90% marginal power for each of four outcomes mean there is a 90% probability of rejecting all four null hypotheses?

Solutions: (1) $0.05/5=0.010$ per test. (2) The union bound gives $P(\text{one or more false rejections})\leq\sum_i P(A_i)\leq m(\alpha/m)=\alpha$, without requiring independence. (3) The first value passes $0.05/3\approx0.01667$, but the second value, 0.030, does not pass $0.05/2=0.025$. Holm stops there, so only the first hypothesis is rejected. (4) No. Marginal power describes each test separately; the probability of rejecting all four is conjunctive power and depends on their joint distribution.