The Pitfall of Testing Every Arm at Nominal Alpha

Why Nominal Alpha Is Not a Familywise Guarantee

In a multi-arm trial, investigators often compare several treatments with a control. Each comparison may be tested at a nominal significance level such as $\alpha=0.05$. That level describes the type I error rate for an individual test under its null hypothesis; it does not, by itself, describe the chance of making one or more false rejections across all comparisons. When several treatment arms are tested, the distinction matters: a trial may report success if any treatment comparison is significant, even though the family contains multiple opportunities for a false positive.

The relevant family must be specified before interpreting the results. For example, it might consist of the $m$ treatment-versus-control hypotheses for the arms included in a primary efficacy analysis. The familywise error rate (FWER) is the probability of rejecting at least one null hypothesis that is true in that family. Under the global null, every hypothesis in the family is true, so any rejection is a familywise error. Strong FWER control requires the probability of at least one false rejection to be at most $\alpha$ for every configuration of true and false hypotheses.

Definition: For a family of hypotheses $H_1,\ldots,H_m$, let $I_0$ be the indices of the true null hypotheses. The familywise error rate is $$ \mathrm{FWER}=P(\text{reject at least one }H_i\text{ with }i\in I_0). $$ A procedure has strong FWER control at level $\alpha$ if this probability is at most $\alpha$ for every possible configuration of true and false null hypotheses. Testing each hypothesis at nominal level $\alpha$ only requires that its own type I error probability be at most $\alpha$; it does not imply strong FWER control.

This is a probability statement about the repeated use of a testing procedure, not a statement that a particular set of reported results contains a false positive with a known probability. It also does not mean that every family of nominal-level tests must have an inflated FWER. Dependence can make the familywise probability smaller or larger than the independent-test calculation. The point is that nominal levels alone do not guarantee the desired familywise bound.

Independent Tests: A Direct Calculation

Suppose that all $m$ null hypotheses are true, each test has type I error probability $\alpha$, and the test outcomes are mutually independent. For each test, the probability of not rejecting its true null is $1-\alpha$. Independence makes the probability of no rejections across the family equal to the product of those probabilities. Taking the complement gives the probability of at least one false rejection:

\mathrm{FWER} =1-P(\text{no rejections}) =1-(1-\alpha)^m.

For $m>1$ and $0<\alpha<1$, this is greater than $\alpha$, because $(1-\alpha)^m<1-\alpha$. Thus the nominal level for each test is not the familywise level. This calculation applies exactly when the stated independence and all-null assumptions hold; it should not be transferred without checking those assumptions.

Worked Example: Four Independent Treatment Comparisons

Consider a hypothetical trial with four treatment-versus-control hypotheses. Assume all four null hypotheses are true, each test is conducted at $\alpha=0.05$, and the four test outcomes are independent. These assumptions make this an illustrative calculation, not a claim about the dependence structure in every multi-arm trial.

The probability of not rejecting any one true null is $1-0.05=0.95$. By independence, the probability of no false rejection in all four tests is $$ P(\text{no rejections})=0.95^4=0.81450625. $$ Therefore, $$ \mathrm{FWER}=1-0.95^4=1-0.81450625=0.18549375\approx 0.1855. $$

The familywise error rate is about $18.55\%$, not $5\%$, under these assumptions. Each individual test still has a $5\%$ type I error rate. The difference arises because the family has four chances to reject a true null.

What Dependence Changes—and What It Does Not

In a trial with a shared control group, treatment-control comparisons are generally dependent: each contrast contains the same control-group estimate. Consequently, the independent-test formula $1-(1-\alpha)^m$ is not generally the correct FWER calculation for such comparisons. Dependence changes the probability of the joint event that no true null is rejected. It does not, however, turn marginal level-$\alpha$ testing into a general FWER guarantee.

A simple normal-model calculation shows the source of dependence. Suppose three independent groups have sample means $\bar{X}_0,\bar{X}_1,\bar{X}_2$, each based on $n$ observations with common variance $\sigma^2$. Under the global null, the treatment-control contrasts are $D_1=\bar{X}_1-\bar{X}_0$ and $D_2=\bar{X}_2-\bar{X}_0$. The independent group means have variance $\sigma^2/n$, so $$ \operatorname{Var}(D_1)=\operatorname{Var}(D_2)=\frac{2\sigma^2}{n}, \qquad \operatorname{Cov}(D_1,D_2)=\operatorname{Var}(\bar{X}_0)=\frac{\sigma^2}{n}. $$ Their correlation is therefore $ (\sigma^2/n)/(2\sigma^2/n)=1/2$. A shared control induces positive dependence between these contrasts in this model. The exact FWER then depends on the joint distribution of the test statistics and the rejection rule, not just on their marginal type I error rates.

Worked Example: Checking Dependence From a Shared Control

Suppose, hypothetically, that two treatment groups and one control group each have $n=100$ independent observations, with a common outcome variance of $\sigma^2=25$. Define $D_1=\bar{X}_1-\bar{X}_0$ and $D_2=\bar{X}_2-\bar{X}_0$. Under the normal equal-variance model, calculate the variance of each contrast, their covariance, and their correlation.

Each group mean has variance $25/100=0.25$. Thus $$ \operatorname{Var}(D_1)=0.25+0.25=0.50,\qquad \operatorname{Var}(D_2)=0.50. $$ Both contrasts contain $-\bar{X}_0$, so their covariance is the variance of the shared control mean: $$ \operatorname{Cov}(D_1,D_2)=0.25. $$ It follows that $$ \operatorname{Corr}(D_1,D_2) =\frac{0.25}{\sqrt{0.50\cdot 0.50}} =0.50. $$

The two contrast estimates are not independent. Hence it would be unjustified to calculate their exact joint rejection probability by multiplying two non-rejection probabilities. This covariance calculation diagnoses dependence; to find an exact FWER for a specified test, one would also need the joint distribution of the test statistics and the precise rejection rule.

A useful result that does not require independence is the union bound. Let $A_i$ be the event that a true null $H_i$ is rejected. Then a familywise error occurs if at least one of the events $A_i$, $i\in I_0$, occurs. The probability of a union is no greater than the sum of the event probabilities:

\mathrm{FWER} =P\left(\bigcup_{i\in I_0} A_i\right) \leq \sum_{i\in I_0}P(A_i) \leq |I_0|\alpha \leq m\alpha.

The first inequality is the union bound; the second uses the assumption that each true null is tested at level at most $\alpha$. This bound can exceed $\alpha$, so it does not establish FWER control at $\alpha$ for unadjusted tests. It does show why the number of hypotheses matters even without an independence assumption.

Worked Example: Comparing Family Sizes and a General Bound

Suppose every test is conducted at nominal level $0.05$, and all null hypotheses are true. If the tests are independent, the exact FWER is $1-(1-0.05)^m$. If their dependence is unknown, the union bound gives $\mathrm{FWER}\leq m(0.05)$, capped at 1.

For two independent tests, the calculation is $$ 1-0.95^2=1-0.9025=0.0975. $$ For three independent tests, it is $$ 1-0.95^3=1-0.857375=0.142625\approx 0.1426. $$ For ten independent tests, it is $$ 1-0.95^{10}\approx 1-0.598737=0.4013. $$ These values illustrate how the independent-test FWER increases with family size.

With four tests and dependence left unspecified, the union bound is $$ \mathrm{FWER}\leq 4(0.05)=0.20. $$ This is a valid upper bound under the marginal level assumptions, not an assertion that the actual FWER equals $0.20$. It is also not a guarantee of an upper bound of $0.05$. The exact familywise probability depends on the joint behavior of the tests.

How to Avoid the Pitfall

The first practical step is to identify the family of hypotheses relevant to the trial’s success claim. If success means that at least one of several treatment-versus-control comparisons is significant, those comparisons form a family for that claim. The decision rule and the intended error-rate guarantee should be specified before results are examined. A test being individually valid does not answer whether the combined decision rule has the desired error control.

For a familywise claim, use a procedure whose conditions and error guarantee match the analysis. Bonferroni and Holm procedures, as well as closed testing and prespecified graphical or gatekeeping procedures, are examples of approaches that can control FWER when applied according to their definitions and assumptions. For example, Bonferroni testing of $m$ hypotheses uses a per-hypothesis threshold $\alpha/m$; the union bound then gives FWER no greater than $\alpha$, regardless of dependence. More structured procedures can use the hypotheses’ ordering or logical relationships, but their validity depends on following the specified procedure.

The main alternative is not simply “adjust” versus “do not adjust.” Investigators may instead define a different inferential objective, such as reporting estimates without making a familywise success claim, or may prespecify a testing strategy that allocates the error budget across hypotheses. The interpretation must match that objective. If the trial’s stated conclusion depends on finding any effective arm, reporting unadjusted $p$-values as though they jointly carried a $5\%$ familywise guarantee is misleading.

Common pitfall: “Each comparison has a type I error rate of $0.05$, so the chance of any false positive across the trial is $0.05$.” This confuses a per-test guarantee with a familywise guarantee. The correction is to evaluate the joint rejection rule for the prespecified family or use a procedure that controls the FWER under its stated conditions.

There are two opposite errors to avoid. First, assuming independence when comparisons share observations can give the wrong exact familywise probability. Second, treating dependence as a reason that multiplicity no longer matters is also incorrect: the marginal levels alone still do not establish FWER control. The exact FWER may equal $\alpha$ in special dependence structures, or exceed it under others. A sound analysis states the family, the error criterion, the assumptions about dependence, and the procedure used to support the conclusion.

Check Your Understanding

Answer each question, then compare with the brief solutions.

Four independent tests are each conducted at level $0.05$, and all four null hypotheses are true. What is the probability of at least one rejection?
In one sentence, distinguish a test’s nominal level from the familywise error rate.
Why are treatment-control contrasts generally dependent when they share the same control group?
For $m$ tests, each with type I error probability at most $\alpha$, what upper bound on FWER follows from the union bound, without assuming independence?
Does a positive correlation between test statistics make nominal-$\alpha$ testing a general FWER control procedure? Explain.

Solutions. (1) $1-(1-0.05)^4=1-0.81450625=0.1855$ to four significant figures. (2) Nominal level bounds the type I error probability of an individual test under its null, whereas FWER is the probability of at least one false rejection in the family. (3) Each contrast includes the same control estimate, so their covariance includes the variance of that shared estimate. (4) $\mathrm{FWER}\leq m\alpha$, with the probability also bounded above by 1. (5) No. Correlation changes the joint rejection probability, but nominal marginal levels alone do not guarantee that the familywise probability is at most $\alpha$.

The Pitfall of Testing Every Arm at Nominal Alpha

What You'll Learn