Familywise Error Rate as a Probability

From Individual Errors to a Familywise Event

In a multi-arm trial, investigators may test several treatment comparisons using the same control group. Each comparison has its own null hypothesis, but the trial also raises a family-level question: what is the probability that at least one true null hypothesis is rejected? This probability depends on the set of hypotheses being considered and on how their test decisions behave jointly.

The familywise error rate is not obtained simply by naming a nominal significance level. It is a probability under a specified configuration of true and false hypotheses. To calculate it, identify the hypotheses that are true at the parameter value under consideration, identify the rejection events for those hypotheses, and find the probability of their union.

Definition: Let a prespecified family contain null hypotheses $H_1,\ldots,H_m$. For a parameter value $\theta$, let $I_0(\theta)$ be the set of indices $i$ for which $H_i$ is true, and let $R_i$ be the event that the test of $H_i$ rejects. The familywise error rate at $\theta$ is $$ \operatorname{FWER}(\theta)=P_\theta\left(\bigcup_{i\in I_0(\theta)}R_i\right). $$ It is the probability of at least one false rejection among the true null hypotheses in the family.

The probability is calculated under the data-generating distribution specified by $\theta$, including the sampling design and any dependence among test statistics. If no null hypothesis in the family is true, then $I_0(\theta)$ is empty and the familywise error probability is zero: there is no true null hypothesis to reject falsely. This does not mean the tests have no errors of other kinds, such as failing to reject a false null hypothesis.

Calculating the Probability of at Least One Error

For a given configuration, let $V$ denote the number of true null hypotheses rejected. Then the familywise error event is $V\geq 1$, so an equivalent definition is

\operatorname{FWER}(\theta)=P_\theta(V\geq 1).

This expression makes clear why a familywise rate is different from a per-test rate. A per-test type I error probability refers to one particular true null hypothesis. The familywise rate concerns whether any of the true null hypotheses is rejected.

When the rejection events for the true null hypotheses are independent, the probability of no false rejections is the product of the probabilities of not rejecting each true null. Taking the complement gives an exact calculation. Independence is essential for this product calculation; in a multi-arm trial, shared participants or a shared control group can make test decisions dependent.

\operatorname{FWER}(\theta) =1-P_\theta\left(\bigcap_{i\in I_0(\theta)}R_i^c\right) =1-\prod_{i\in I_0(\theta)}\left[1-P_\theta(R_i)\right], \qquad\text{if the }R_i\text{ are independent.}

Worked Example: Four Independent Tests Under the Complete Null

Suppose a hypothetical trial tests four null hypotheses, all of which are true. Each test rejects with probability $0.025$ under its null, and assume the four rejection events are independent. Calculate the familywise error rate.

The familywise error event is at least one rejection. Each test has probability $1-0.025=0.975$ of not rejecting. Under independence, the probability that none of the four rejects is

P(\text{no false rejections})=(0.975)^4=0.9036879.

Therefore,

\operatorname{FWER}=1-(0.975)^4=0.0963121\approx 0.0963.

Thus, under the stated complete-null configuration and independence assumption, the probability of at least one false rejection is about 9.63%. It is larger than the 2.5% error probability of any one test. The calculation is exact for these assumptions; it is not a claim that every set of four tests has this familywise rate.

Dependence Changes the Joint Probability

Without independence, the marginal rejection probabilities do not determine the familywise error rate. The probability of the union depends also on how often rejection events occur together. For two true null hypotheses, the exact relation is

P_\theta(R_1\cup R_2) =P_\theta(R_1)+P_\theta(R_2)-P_\theta(R_1\cap R_2).

The intersection is subtracted because outcomes in which both tests reject are counted twice when the two marginal probabilities are added. With more hypotheses, the same principle applies: calculate the probability of the union from the joint distribution, or from a complete partition of the possible rejection patterns.

Worked Example: Two Dependent Tests

Suppose two true null hypotheses are tested in a hypothetical multi-arm trial. Their rejection events have the following joint probabilities: both tests reject with probability $0.025$; only the first rejects with probability $0.015$; only the second rejects with probability $0.015$; and neither rejects with probability $0.945$. These probabilities sum to 1. Calculate the familywise error rate and the marginal rejection probability for each test.

The first test rejects either when both reject or when only it rejects:

$$ P(R_1)=0.025+0.015=0.040. $$

Likewise, $P(R_2)=0.025+0.015=0.040$. The probability of at least one rejection is the sum of the probabilities for the three patterns containing a rejection:

\operatorname{FWER}=0.025+0.015+0.015=0.055.

Equivalently, $P(R_1\cup R_2)=0.040+0.040-0.025=0.055$. The independence calculation based only on the two marginal probabilities would instead give $1-(0.96)^2=0.0784$, which is not correct for this joint distribution. The overlap probability $0.025$ is needed for the exact answer.

Mixed Configurations: Count Only True Null Rejections

A family can contain both true and false null hypotheses. In that case, rejecting a false null is not a type I error and does not count toward the familywise error event. The definition therefore uses $I_0(\theta)$, not the full set of hypotheses. This distinction is particularly important when describing a specific parameter configuration: first determine which null hypotheses are true, then form the union of their rejection events.

Worked Example: Two True Nulls and One False Null

Consider a hypothetical family of three treatment comparisons. At the parameter value being considered, $H_1$ and $H_2$ are true, while $H_3$ is false. Suppose the joint probabilities for the two true-null tests are: both reject, $0.012$; only $H_1$ rejects, $0.018$; only $H_2$ rejects, $0.008$; neither rejects, $0.962$. Calculate the familywise error rate.

The true-null rejection probabilities are $P(R_1)=0.012+0.018=0.030$ and $P(R_2)=0.012+0.008=0.020$. The familywise error event is $R_1\cup R_2$; rejection of $H_3$ is not a false rejection because $H_3$ is false. Thus,

\operatorname{FWER}=0.012+0.018+0.008=0.038.

The familywise error rate for this configuration is 3.8%. The probability that the test of $H_3$ rejects, whether large or small, does not enter this calculation. It may matter for other questions, but it is outside the event defining familywise type I error here.

What the Number Does—and Does Not—Say

A familywise error rate is a repeated-sampling probability. For a fixed parameter configuration and a fully specified testing procedure, it describes the long-run proportion of repetitions in which at least one true null hypothesis in the family is rejected. It is not the probability that a particular rejection is false, nor the probability that any null hypothesis is true.

It is also distinct from the expected number of false rejections. If $V$ is the number of true null hypotheses rejected, then

E_\theta(V)=\sum_{i\in I_0(\theta)}P_\theta(R_i), \qquad \operatorname{FWER}(\theta)=P_\theta(V\geq 1).

The first expression adds marginal probabilities and measures an expected count; the second measures the chance that the count is nonzero. They are not generally equal. For instance, in the four-test independent example, the expected number of false rejections is $4(0.025)=0.100$, while the familywise probability is approximately $0.0963$.

Exact calculation is useful when the joint rejection distribution is known or can be derived. When it is not available analytically, a simulation can estimate the probability: for each simulated data set, record whether at least one true null is rejected, then average those zero-or-one indicators. The simulation must reproduce the relevant design, the testing rules, and the dependence among test statistics. A simulation that generates each test independently would not answer the question for a design in which the tests share data.

Key takeaway: Familywise error is the probability of a union of rejection events for true null hypotheses. Marginal type I error probabilities alone are sufficient only when the required dependence assumptions are supplied; otherwise, the joint rejection probabilities matter.

Common Mistakes and Checks

Multiplying non-rejection probabilities without checking independence. The product formula requires independent rejection events. With dependent tests, use the joint distribution or a valid calculation of the union.
Counting rejection of a false null as a type I error. Only rejections of hypotheses that are true at the parameter configuration enter the familywise error event.
Equating the familywise rate with a single-test rate. The single-test probability concerns one null; the familywise rate concerns at least one false rejection across the true nulls in the family.
Using the expected number of false rejections as the probability of any error. The expected count and the probability that the count is at least one are different quantities.
Leaving the family or configuration unspecified. State which hypotheses are included and which are true before reporting a familywise error probability.

Check Your Understanding

Try each question by identifying the relevant true-null rejection events before calculating.

Three independent tests are applied to three true null hypotheses. Each has rejection probability $0.01$. What is the familywise error rate?
For two true null hypotheses, $P(R_1)=0.06$, $P(R_2)=0.04$, and $P(R_1\cap R_2)=0.02$. Find the familywise error rate.
A family contains one true null and two false nulls. The true-null test rejects with probability $0.05$. What additional information about the two false-null tests is needed to calculate the familywise error rate?
For two independent true-null tests, each with rejection probability $0.05$, calculate both the familywise error rate and the expected number of false rejections. Explain why they differ.

Solutions. (1) Independence gives $1-(1-0.01)^3=1-0.99^3=0.029701$, or about 2.97%. (2) The union probability is $0.06+0.04-0.02=0.08$, so the familywise error rate is 8%. (3) None is needed: only the true-null rejection event counts, so the familywise error rate is 0.05. (4) The familywise rate is $1-(0.95)^2=0.0975$, or 9.75%. The expected number of false rejections is $0.05+0.05=0.10$. They differ because the expected count gives outcomes with two rejections a count of two, whereas the familywise event counts that outcome only once.