Why the Family Begins With the Trial’s Claims
A multi-arm trial can produce many treatment comparisons. Yet the number of experimental arms alone does not determine how many scientific claims the trial intends to make. A trial with three treatments might ask whether any treatment improves the primary outcome versus control, whether each treatment improves two outcomes, or whether treatments differ from one another. These are different objectives, and they imply different collections of hypotheses.
The family is defined by the claims the trial plans to evaluate together. A statistical test is a procedure for assessing one such claim using observed data. Keeping these ideas separate matters: a family is not simply “all the tests in the software,” and one cannot identify it just by counting arms, endpoints, or reported p-values.
Writing Claims as Parameter Contrasts
For a trial with one control group, label the control arm \(0\) and the experimental arms \(1,\ldots,K\). Let \(\mu_j\) denote the population mean outcome under arm \(j\). For a continuous outcome, a treatment-control contrast is
The scientific claim determines the hypotheses for this contrast. If lower values are better and the claim is that treatment \(j\) reduces the outcome, one possible formulation is \(H_{0j}:\delta_j\geq 0\) versus \(H_{aj}:\delta_j<0\). If the question is whether the treatment has any effect in either direction, the hypotheses would instead be two-sided. The direction is part of the claim, not a choice to make after seeing the observed results.
The contrast must also be tied to a defined estimand: for example, which participant population, outcome, follow-up time, and treatment comparison are meant. Changing one of these can change the scientific claim. For a binary outcome, the contrast might instead be a difference in event probabilities, such as \(p_j-p_0\). The parameter scale and direction should be clear enough that another analyst can tell exactly what proposition is being tested.
Worked Examples of Defining Families
Worked Example: Two Treatments, One Continuous Outcome
In a hypothetical randomized trial, two treatments are compared with a shared control for a symptom score measured at 12 weeks. Lower scores indicate fewer symptoms. The observed sample means are 10.2 points for control, 8.9 points for Treatment 1, and 9.4 points for Treatment 2. The prespecified objective is to assess whether each treatment reduces the mean score versus control.
For each treatment, define \(\delta_j=\mu_j-\mu_0\). The two observed differences are
Because lower scores are better, the claims are \(\delta_1<0\) and \(\delta_2<0\). Their null hypotheses are \(H_{01}:\delta_1\geq0\) and \(H_{02}:\delta_2\geq0\); their alternatives are \(H_{a1}:\delta_1<0\) and \(H_{a2}:\delta_2<0\). The family contains these two treatment-control claims.
The observed differences describe the hypothetical sample; they do not by themselves establish either claim about population means. A later testing procedure would use the trial data to assess each null. The family’s membership follows from the stated objective, not from whether either observed difference looks favourable.
Worked Example: Two Treatments and Two Outcomes
Consider a hypothetical trial with two experimental treatments and a shared control. The trial intends to make a separate confirmatory claim for each treatment on each of two outcomes: a symptom score at 12 weeks, where lower is better, and hospitalization by 12 weeks, where a lower event risk is better. The sample symptom means are 14 points for control, 12 for Treatment A, and 13 for Treatment B. The sample hospitalization risks are 0.10, 0.08, and 0.07, respectively.
For the symptom outcome, the treatment-control differences are \(12-14=-2\) points for A and \(13-14=-1\) point for B. For hospitalization, the risk differences are \(0.08-0.10=-0.02\) for A and \(0.07-0.10=-0.03\) for B, corresponding to 2 and 3 percentage-point lower observed risks. Since the objective makes each treatment-outcome statement a separate claim, the family has four hypotheses: A versus control on symptoms, B versus control on symptoms, A versus control on hospitalization, and B versus control on hospitalization.
The first two claims concern mean differences in points; the latter two concern risk differences. For each, the alternative is that its treatment-control contrast is below zero, and the null is that the contrast is at least zero. These four claims are not collapsed into one merely because the outcomes were measured in the same participants. If the trial instead defined hospitalization as descriptive only, the two hospitalization claims would not be part of this stated set of confirmatory claims.
Worked Example: Treatment-Control Comparisons or All Pairwise Comparisons?
A hypothetical dose-finding trial has one control group and three dose groups. A response score is measured at the end of follow-up, and higher scores are better. The sample means are 50 points for control, 54 for the low dose, 57 for the medium dose, and 59 for the high dose. The stated objective is to determine whether any dose improves the mean response versus control.
Write \(\delta_j=\mu_j-\mu_0\) for each dose. The observed contrasts are \(54-50=4\) points, \(57-50=7\) points, and \(59-50=9\) points. The objective defines three claims, one for each dose versus control, with alternatives \(\delta_j>0\) and nulls \(\delta_j\leq0\). Thus this family contains three hypotheses.
If the objective also included every pairwise dose comparison, the set would be different. There are four arms in total, so the number of distinct pairs would be \(4\times3/2=6\): the three dose-control pairs plus low versus medium, low versus high, and medium versus high. Those extra comparisons are not part of the stated objective simply because the data allow them to be calculated. The intended claims, rather than the available arithmetic, define the family.
A Short Justification for Counting Claims, Not Calculations
The distinction can be stated formally. Suppose the trial has a finite list of intended claims \(C=\{c_1,\ldots,c_m\}\). Each claim \(c_\ell\) refers to a parameter contrast \(g_\ell(\theta)\) and defines a null region for that contrast. The family size is \(m\), the number of listed claims. A test procedure for claim \(c_\ell\) uses observed data to decide whether the evidence is sufficient to reject its null.
This mapping explains why a computation does not automatically create a new scientific claim. Recalculating a test statistic, or using another valid test method for the same contrast, does not change the underlying parameter proposition. Conversely, if the trial intends to make distinct claims for multiple outcomes or treatment contrasts, those claims remain distinct even if one analysis program reports them in a single table.
For example, if \(K\) experimental arms are each compared with one control on one outcome, there are \(K\) treatment-control claims. If each arm is compared with control on \(L\) outcomes and every arm-outcome pair is a separate intended claim, there are \(K L\) such claims. If all pairwise comparisons among \(K+1\) arms on one outcome are intended, there are \((K+1)K/2\) claims. These counts follow from the stated comparison set; they are not universal rules for defining a family.
What Belongs in the Definition
A useful family definition makes the scope concrete. It identifies the treatment comparisons, outcomes, time points, populations, and directions of interest. It also makes clear whether the trial intends separate claims for each listed combination. This is particularly important when treatment arms share a control, because a common control does not merge the treatment-specific claims into one.
Stages in a multi-stage trial add another distinction. An interim analysis and a final analysis may examine the same treatment-outcome claim at different times; that does not necessarily create a new scientific claim. However, repeated opportunities to make a decision can affect the statistical procedure needed to support that claim. Likewise, alternative analysis methods may assess the same claim, while choosing among them based on results can create a selection issue. Defining the family clarifies the claims, but does not by itself specify how repeated looks or analytic choices will be handled.
The family also depends on the purpose of the conclusions. If each treatment is intended to support its own conclusion, each treatment-specific comparison is a claim. If the objective is only to compare doses with one another, treatment-control comparisons need not be included merely because a control arm exists. If outcomes are labelled primary, secondary, or exploratory, that hierarchy should be reflected in which conclusions are meant to be supported as a group.
Common Pitfalls, Alternatives, and Limits
- Counting arms and stopping there. Three treatment arms do not always mean three hypotheses: the trial might test only one selected dose, or it might assess several outcomes and many pairwise contrasts. List the claims before counting.
- Equating hypotheses with p-values. A p-value is an output from a test, not a scientific claim. Multiple analyses of one contrast do not automatically represent multiple distinct claims, though selecting whichever analysis looks best can affect the credibility of the conclusion.
- Adding comparisons because they are easy to calculate. A full pairwise table may contain claims beyond the trial’s objective. Distinguish treatment-versus-control objectives from all-pairs objectives.
- Leaving outcomes or time points vague. “Improves health” does not identify a single parameter contrast. Specify the outcome, measurement time, population, and comparison.
- Assuming the family definition solves every multiplicity issue. It identifies what conclusions are being considered together. It does not itself give a testing procedure or guarantee any particular error rate.
The main alternative to a single broad family is to define separate families for distinct objectives, such as a primary outcome and a clearly separate exploratory analysis. This can make the inferential aims easier to communicate, but the separation must reflect the planned scientific conclusions rather than a convenient grouping discovered after results are known. Whatever scope is chosen, its limitations should be stated: conclusions concern the listed contrasts and do not automatically extend to untested doses, outcomes, populations, or time points.
Check Your Understanding
Answer each question by identifying the intended claims, not just the number of available calculations.
- A trial has four experimental arms and a shared control. Its only objective is to compare each experimental arm with control on one endpoint. How many hypotheses are in this family?
- Two treatments are assessed on three outcomes, and the objective is a separate claim for every treatment-outcome combination. How many hypotheses are in the family?
- Why do the three treatment-control comparisons in a three-dose trial not automatically imply that every active-dose comparison belongs to the family?
- An interim and final analysis both assess the same treatment-control contrast. Are these necessarily two different scientific claims?
Solutions. (1) Four: one treatment-control claim for each experimental arm. (2) Six: \(2\times3=6\) treatment-outcome claims. (3) Because family membership follows the intended objective; active-dose comparisons are separate claims only if dose-versus-dose conclusions are intended. (4) No. They can be repeated assessments of the same claim at different stages, although the decision procedure must account for the repeated opportunities to act on the evidence.