Choosing All-Control or All-Pairs Comparisons

Why the Comparison Set Matters

A multi-arm trial can answer different questions with the same set of treatment groups. Investigators might want to know which treatments improve on control, which active treatments differ from each other, or both. Those are different inferential goals, so they lead to different collections of hypotheses. The collection matters because familywise error control applies to a specified family: changing the comparisons changes the multiplicity problem.

The choice is about the planned scientific claims, not just which pairwise calculations are convenient after the data are collected. An all-control family supports claims about each treatment relative to control. It does not, by itself, establish whether two treatments differ. An all-pairs family among active treatments addresses those active-versus-active differences, but it does not automatically include treatment-versus-control claims.

Definition: A hypothesis family is the prespecified set of null hypotheses for which a joint error guarantee is sought. The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in that family. A comparison belongs in the family when its conclusion is among the claims the analysis is intended to support.

Defining the Two Main Families

Label the control group $0$ and the $K$ treatment groups $1,\ldots,K$. Let $\theta_g$ denote the prespecified parameter for group $g$, such as the mean of a continuous endpoint. The hypotheses below compare that same parameter across groups. They can be formulated with a chosen alternative direction; the family counts do not depend on that choice.

All-control family: The family contains one null hypothesis for each treatment-versus-control comparison: $$ H_{0j}: \theta_j-\theta_0=0,\qquad j=1,\ldots,K. $$ It contains $K$ hypotheses.

All-pairs treatment family: If the goal is to compare every pair of active treatments, the family contains $$ H_{0ij}: \theta_i-\theta_j=0,\qquad 1\leq i<j\leq K. $$ It contains $\binom{K}{2}=K(K-1)/2$ hypotheses. This definition excludes the control group.

The term “all pairs” can be ambiguous unless the groups are named. If the intended family includes every pair among all $K+1$ groups, including control, then it contains $\binom{K+1}{2}=K(K+1)/2$ hypotheses. That is a combined family: it includes all treatment-versus-control comparisons as well as all active-treatment pairs.

Planned claims	Family	Number of hypotheses
Each treatment compared with control	All-control	$K$
Every active treatment compared with every other active treatment	All-pairs among treatments	$K(K-1)/2$
Every pair among control and all treatments	All pairs among all groups	$K(K+1)/2$

Why Family Size Changes Multiplicity

Suppose each of $m$ hypotheses is tested at level $\alpha/m$, with valid marginal $p$-values. By the union bound, the probability of at least one false rejection is no greater than the sum of the probabilities of false rejection for the individual tests. For true null hypotheses, each such probability is at most $\alpha/m$. Therefore,

P(\text{at least one false rejection}) \leq \sum_{i\ \text{true}} P(\text{reject }H_{0i}) \leq m\frac{\alpha}{m} =\alpha.

This is the Bonferroni justification: testing each hypothesis at $\alpha/m$ controls the FWER at no more than $\alpha$, without requiring the tests to be independent. It also shows why defining the family matters: a larger $m$ gives a smaller Bonferroni threshold. Other FWER procedures can use different rules and may take dependence or logical relationships into account, but they still need a clearly specified family and valid tests.

The number of hypotheses does not tell us the exact FWER for a particular design. Comparisons can be dependent, especially when they share an arm. For instance, several treatment-versus-control estimates use data from the same control group. Dependence is not a reason to ignore multiplicity; it is a feature a chosen procedure may account for. Earlier material on closed testing and multiplicity procedures provides tools for controlling error once the family is defined.

Worked Example: Three Treatments Compared With Control

Consider a hypothetical trial with one control group and three treatment groups. The primary question is whether each treatment differs from control on the prespecified endpoint. The observed group means are 10.2, 12.1, 11.4, and 13.0 points for groups 0, 1, 2, and 3, respectively. The all-control family is $H_{01},H_{02},H_{03}$, so $m=K=3$.

The estimated treatment-minus-control differences are $12.1-10.2=1.9$ points for treatment 1, $11.4-10.2=1.2$ points for treatment 2, and $13.0-10.2=2.8$ points for treatment 3. These are descriptive estimates; they do not alone establish which differences are statistically significant.

For illustration, suppose valid marginal $p$-values from the prespecified tests are 0.012, 0.028, and 0.004. With Bonferroni control at $\alpha=0.05$, the per-hypothesis threshold is $$ \frac{\alpha}{m}=\frac{0.05}{3}=0.0167. $$ Thus the first and third hypotheses are rejected by this rule, while the second is not. The FWER guarantee concerns the three treatment-versus-control claims in this family. It says nothing about differences between the active treatments, which are not included here.

Worked Example: Comparing Every Pair of Active Treatments

Now suppose a hypothetical trial has four active treatments, with endpoint means of 8.1, 9.0, 10.4, and 8.8 points. The stated goal is to compare every active treatment with every other active treatment; control comparisons are not part of this particular family.

There are $$ m=\binom{4}{2}=\frac{4(4-1)}{2}=6 $$ active-treatment hypotheses. Their estimated differences, in points, are $9.0-8.1=0.9$, $10.4-8.1=2.3$, $8.8-8.1=0.7$, $10.4-9.0=1.4$, $8.8-9.0=-0.2$, and $8.8-10.4=-1.6$. Each estimate corresponds to one of the six unordered pairs.

For a Bonferroni FWER target of 0.05, the threshold is $$ \frac{0.05}{6}=0.00833. $$ If the scientific goal also includes comparisons of all four treatments with control, those four additional hypotheses must be included in the relevant family. The total then becomes $6+4=10$, and the Bonferroni threshold for that combined family is $0.05/10=0.00500$. The change is not caused by different observed means; it follows from the additional planned claims.

Choose the Family From the Decision Goal

A family should match the conclusions the trial is designed to support. All-control comparisons are a natural choice when the decision is which treatments meet a criterion relative to standard care or placebo. All-pairs comparisons among active treatments are relevant when the intended claims concern differences between active options. A study seeking both kinds of conclusions needs to account for both sets of hypotheses under its prespecified multiplicity strategy.

The family is not necessarily every contrast that can be computed from the data. Nor is it automatically all possible contrasts across all endpoints and objectives. The protocol and analysis plan should state which hypotheses belong to the family and how error control will be handled across any additional families. A separate family label does not, on its own, guarantee an overall error rate across the whole trial; any relationship between families must be addressed by the planned strategy.

Worked Example: When Both Kinds of Comparison Matter

Suppose a hypothetical study has three active treatments and control, with group means of 20, 24, 22, and 25 units for groups 0, 1, 2, and 3. Its stated goals are both to assess each treatment against control and to compare the active treatments with one another.

The all-control set has $K=3$ hypotheses. The active-treatment set has $$ \binom{3}{2}=\frac{3(2)}{2}=3 $$ hypotheses. If the plan requires FWER control across the union of these two sets, the combined family has $3+3=6$ distinct comparisons. A Bonferroni illustration at $\alpha=0.05$ gives $0.05/6=0.00833$ per hypothesis.

For example, the estimated treatment-versus-control differences are $24-20=4$, $22-20=2$, and $25-20=5$ units. The active-treatment differences are $24-22=2$, $25-24=1$, and $25-22=3$ units. The family choice follows from the two stated goals, not from which observed differences happen to look largest. The largest sample mean is not, by itself, evidence that its treatment is superior to every other treatment.

Assumptions, Diagnostics, and Limitations

The family definitions are combinatorial, but valid inference also depends on the tests used for its hypotheses. The parameter, endpoint, population, analysis time, and comparison direction should be specified consistently with the trial’s estimand and analysis plan. Each marginal $p$-value supplied to an FWER procedure must be valid under its corresponding null hypothesis under the assumed model and analysis. For the Bonferroni argument above, validity of those marginal $p$-values is sufficient; independence is not required.

Before applying a procedure, check that the comparison list matches the decision question, that no intended contrast has been omitted, and that no comparison has been counted twice. Verify the number of arms and the resulting count: all-control has $K$; active-treatment all-pairs has $K(K-1)/2$; all pairs including control has $K(K+1)/2$. In analysis, also check that each test uses the intended endpoint and contrast, and that shared-control dependence or other design features have not been mistakenly treated as independence when a procedure requires stronger assumptions.

All-control and all-pairs families answer different questions; neither is universally preferable. Bonferroni gives a simple dependence-robust benchmark but can be conservative. Procedures tailored to a comparison structure can be more efficient, but their assumptions and scope must fit the design. No family definition can repair invalid marginal tests, and no multiplicity adjustment can turn an unplanned, data-selected question into a prespecified confirmatory claim.

Mistake: “We tested every treatment against control, so we have also tested whether the treatments differ.” Correction: A difference between treatment $i$ and control and a difference between treatments $i$ and $j$ are distinct hypotheses. Include active-treatment pairs if those comparisons support planned claims.
Mistake: “There are four treatments, so all-pairs means six tests.” Correction: Six is the number of active-treatment pairs when $K=4$. If all pairs among those four treatments and control are intended, there are $\binom{5}{2}=10$ comparisons.
Mistake: “The family can be narrowed to whichever comparisons look interesting after seeing the results.” Correction: Define the family from the planned claims before interpreting results; otherwise the stated error guarantee may not apply to the selection process.

Key takeaway: Write down the intended claims, translate them into a set of null hypotheses, and count that set before choosing an FWER procedure. All-control, active-treatment all-pairs, and all-pairs including control are different families.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A trial has five treatment arms and one control. How many hypotheses are in the all-control family?
For the same trial, how many hypotheses are in the all-pairs family among active treatments only?
If the family includes every pair among all six groups, how many hypotheses are there?
Why does an all-control family not establish whether two active treatments differ?
At $\alpha=0.05$, what is the Bonferroni threshold for an all-control family with five treatments?

Solutions: (1) $K=5$, so there are 5 treatment-versus-control hypotheses. (2) There are $\binom{5}{2}=5(4)/2=10$ active-treatment pairs. (3) There are $\binom{6}{2}=6(5)/2=15$ pairs when control is included. (4) The all-control hypotheses compare each treatment parameter with $\theta_0$; none directly tests $\theta_i-\theta_j=0$ for two active treatments. (5) The threshold is $0.05/5=0.0100$, assuming valid marginal $p$-values for the five prespecified tests.

Planned claims	Family	Number of hypotheses
Each treatment compared with control	All-control	\(K\)
Every active treatment compared with every other active treatment	All-pairs among treatments	\(K(K-1)/2\)
Every pair among control and all treatments	All pairs among all groups	\(K(K+1)/2\)