Familywise Error Control for Contrast Families

Why a Family of Contrasts Needs a Joint Error Criterion

A single test asks whether one null hypothesis is rejected incorrectly. When an analysis tests several contrasts, a different question matters as well: what is the chance of making at least one false rejection anywhere in the family? That probability is not generally the same as the error probability for any one test. Even if each comparison has a small type I error rate, opportunities for a false rejection can accumulate across the family.

The distinction depends on which hypotheses are actually true. A procedure may control the chance of any false rejection when every null hypothesis is true, yet fail to control that chance when only some are true. These are called weak and strong familywise error control, respectively.

Contrasts and Their Hypotheses

Consider the full-rank normal linear model with a fixed design matrix $X$:

Y\sim N(X\beta,\sigma^2 I_n), \qquad \operatorname{rank}(X)=p.

Here $Y$ is the response vector, $\beta$ is the vector of regression parameters, and $\sigma^2$ is the error variance. A contrast is a specified linear combination of the parameters. For a vector $c_j$, the contrast value is $c_j^T\beta$. Choose a finite family of $m$ contrasts, fixed before examining the data, and consider the two-sided null hypotheses

H_j: c_j^T\beta=0, \qquad j=1,\ldots,m.

The vectors $c_j$ need not be linearly independent: two contrast hypotheses can be related, and the test statistics can consequently be dependent. For example, with three group means $\mu_A,\mu_B,\mu_C$, the contrasts $\mu_A-\mu_B$, $\mu_A-\mu_C$, and $\mu_B-\mu_C$ satisfy a linear relation. The family is the collection of hypotheses treated as one inferential set, not merely a list of unrelated calculations.

Definition: For a specified family of hypotheses $H_1,\ldots,H_m$ and a joint testing procedure, let $R_j$ denote the event that the procedure rejects $H_j$. At parameter value $\beta$, the true-null index set is $I_0(\beta)=\{j:c_j^T\beta=0\}$. The familywise error rate (FWER) is the probability of at least one false rejection: $$ \operatorname{FWER}(\beta)=P_\beta\!\left(\bigcup_{j\in I_0(\beta)}R_j\right). $$ If no null hypotheses are true, this probability is zero.

The definition is about the union of false-rejection events among the hypotheses that are true. It is not the expected number of rejections, and it is not the probability of rejecting any hypothesis at all: rejecting a false null is not a familywise error.

Per-Test Error and Familywise Error

For a particular hypothesis $H_j$, its type I error probability at a parameter value satisfying that null is $P_\beta(R_j)$. A test has level $\alpha$ if this probability is at most $\alpha$ for every parameter value under $H_j$. This is a per-test statement. FWER instead concerns all true nulls together. In particular, whenever $H_j$ is true, the event $R_j$ is contained in the event of at least one false rejection, so its probability cannot exceed the FWER at that parameter value.

Definition: A procedure has strong FWER control at level $\alpha$ if $\operatorname{FWER}(\beta)\leq\alpha$ for every parameter value $\beta$, including configurations where only some family nulls are true. It has weak FWER control at level $\alpha$ if this inequality is guaranteed only under the complete null, where all $m$ hypotheses are true.

Strong control includes the complete-null configuration, so it implies weak control. The converse does not follow from the definitions: control in one configuration gives no general guarantee in the others. The following results make these distinctions precise.

Result: At any parameter value, $P_\beta(R_j)\leq\operatorname{FWER}(\beta)$ for every true $H_j$. Also, strong control at level $\alpha$ implies weak control at level $\alpha$.

Proof. If $H_j$ is true, then $j\in I_0(\beta)$, and therefore $R_j\subseteq\bigcup_{k\in I_0(\beta)}R_k$. Taking probabilities gives $P_\beta(R_j)\leq\operatorname{FWER}(\beta)$. Strong control bounds the FWER for every $\beta$; it therefore bounds it in particular at every complete-null parameter value. That is weak control. $\square$

Worked Example: Three Contrasts Among Group Means

Suppose the parameter vector is $\beta=(\mu_A,\mu_B,\mu_C)^T$, and the family consists of the three pairwise differences. Write each contrast as a coefficient vector:

c_1=(1,-1,0)^T,\qquad c_2=(1,0,-1)^T,\qquad c_3=(0,1,-1)^T.

The corresponding nulls are $H_1:\mu_A-\mu_B=0$, $H_2:\mu_A-\mu_C=0$, and $H_3:\mu_B-\mu_C=0$. At $(\mu_A,\mu_B,\mu_C)=(10,10,14)$, the contrast values are $0,-4,-4$. Thus $I_0(\beta)=\{1\}$: only $H_1$ is true.

At this parameter value, a false rejection occurs precisely if the procedure rejects $H_1$. Rejecting $H_2$ or $H_3$, if it occurs, is not a familywise error because those nulls are false here. At $(10,10,10)$, by contrast, all three hypotheses are true, and rejecting any one of them is a familywise error. The same rejection decision can therefore count as an error in one parameter configuration and not in another.

Worked Example: Individual Error Rates Do Not Equal FWER

Suppose two tests each have type I error probability $0.05$ under their nulls, and suppose their rejection events are independent when both nulls are true. Under the complete null, a familywise error occurs if the first test rejects or the second test rejects. The probability of no rejection is

P(\text{neither rejects})=(1-0.05)(1-0.05)=0.95^2=0.9025.

Consequently,

\operatorname{FWER}=1-0.9025=0.0975.

Each test still has a $0.05$ type I error probability, but the probability of at least one false rejection is $0.0975$, not $0.05$. Independence is essential to this particular calculation; with dependent rejection events, the joint probability can differ. The example shows why a per-test error rate alone does not specify the familywise error rate.

Weak Control Does Not Guarantee Strong Control

A weakly controlling procedure can behave acceptably when all nulls are true but differently when an alternative is present. The example below constructs such a procedure in a simple normal linear model. Its unusual joint rejection rule is useful because it makes the logical distinction explicit; it is not offered as a recommended testing method.

Worked Example: A Procedure With Weak but Not Strong Control

Let $Z=(Z_1,Z_2)^T\sim N((\theta_1,\theta_2)^T,I_2)$, a normal linear model with design matrix $I_2$, known variance $1$, and contrasts $\theta_1$ and $\theta_2$. The family is $H_1:\theta_1=0$ and $H_2:\theta_2=0$. Set $\alpha=0.05$, let $d=z_{0.95}$ and $c=z_{0.90}$, where $z_q$ is the $q$-quantile of the standard normal distribution. Thus $P(N(0,1)>d)=0.05$ and $P(N(0,1)>c)=0.10$.

Define the joint decisions as follows: reject $H_1$ if $Z_1>c$ and $Z_2>d$; reject $H_2$ if $Z_1\leq c$ and $Z_2>d$. Under the complete null, these two rejection events partition the event $Z_2>d$, apart from a boundary of probability zero. Hence

\operatorname{FWER}(0,0) =P_{(0,0)}(Z_2>d) =0.05.

The procedure therefore has weak FWER control at $0.05$. Now set $\theta_1=0$ and $\theta_2=3$. Here $H_1$ is true and $H_2$ is false. Rejecting $H_2$ is not an error; only rejecting $H_1$ counts. By independence of $Z_1$ and $Z_2$,

\begin{aligned} \operatorname{FWER}(0,3) &=P_{(0,3)}(Z_1>c,\ Z_2>d)\\ &=P_{(0,3)}(Z_1>c)P_{(0,3)}(Z_2>d)\\ &=0.10\,\Phi(3-d)\\ &>0.05. \end{aligned}

For the last inequality, $d\approx1.645$, so $3-d\approx1.355>0$, and therefore $\Phi(3-d)>0.5$. The FWER is approximately $0.10(0.912)=0.091$, exceeding $0.05$. Thus control under the complete null does not establish control when only part of the family is true. $\square$

Interpreting the Error Criterion

The family and the set of true nulls determine which rejections count as errors. This makes it important to specify the family in advance and to state whether a claim concerns a single test or the whole family. For a contrast family, the question is not only whether each $c_j^T\beta$ is tested, but also which collection of those tests is being treated as one inferential task.

A frequent mistake is to say, “Every test has a type I error rate of $0.05$, so the probability of any false rejection is $0.05$.” That conclusion does not follow. The individual probabilities describe separate events; the familywise probability is the probability of their union over true nulls and depends on their joint behavior. Another mistake is to check only the all-null case and call that strong control. Strong control requires the stated bound at every parameter value, including configurations with a mixture of true and false nulls.

Key takeaway: Per-test type I error is attached to an individual true null. FWER is the probability of at least one false rejection among all true nulls in the specified family. Weak control checks the complete null; strong control checks every configuration of true and false nulls.

Check Your Understanding

Answer each question by identifying the relevant hypotheses and error event.

In a family of four contrasts, only $H_2$ and $H_4$ are true. Which rejection events contribute to FWER?
Explain why strong FWER control implies weak FWER control.
Two independent tests each have type I error probability $0.02$ under the complete null. What is the complete-null FWER?
Can a procedure have weak control without strong control? What feature must a counterexample demonstrate?
Why does a false rejection of a false null not count toward FWER?