Why the Family Must Be Chosen in Advance
A multi-arm trial can produce many potentially interesting comparisons: each treatment against control, treatment against treatment, and perhaps several endpoints or patient subgroups. The multiplicity family specifies which of these hypotheses will support confirmatory claims. It is not merely a list prepared for convenience after the analysis. Its membership affects the interpretation of the study’s error-control plan, so it needs to be decided before outcomes are observed.
The central practical question is: Which possible claims does the trial intend to treat as confirmatory? If all treatment-versus-control comparisons may lead to a claim of efficacy, then each such hypothesis belongs in the family, even if some arms later look unpromising. If only one comparison is designated as the confirmatory primary claim, other analyses may be exploratory—but they must not be relabelled as confirmatory because their results are attractive.
A family is defined by the scientific claims, not simply by the number of arms or the number of test statistics in a software output. For example, three treatment arms do not by themselves tell us whether the confirmatory family contains three treatment-versus-control hypotheses, one primary hypothesis, or a larger set that also includes endpoints or subgroup claims. The trial’s intended conclusions determine the relevant set.
What to Specify Before Outcomes Are Known
A usable specification should let a reader determine, without inspecting the results, what does and does not count as a confirmatory claim. For each planned claim, state the treatment comparison, endpoint, target population, and direction of the hypothesis. If the claim concerns a particular time point, definition of the outcome, or subgroup, specify that as well. State which claims are primary and whether any additional claims are also confirmatory.
For instance, let \(\theta_j\) be the mean difference in a specified outcome between treatment \(j\) and control, with positive values favoring treatment. A one-sided superiority claim for that comparison can be expressed as \(H_{0j}: \theta_j \leq 0\) against \(H_{aj}: \theta_j > 0\). A protocol must make clear which treatment indices \(j\) are included, and that the outcome, population, and direction are the intended ones. Choosing the direction after seeing which way the observed difference points is not prespecification.
The specification should also distinguish a planned confirmatory hypothesis from a measurement that happens to be collected. Collecting several outcomes does not automatically make every outcome a confirmatory claim. Conversely, describing a comparison as “secondary” does not make it exploratory if the study intends to use it to support a confirmatory efficacy claim. The intended role of the result, rather than its label alone, is decisive.
Write each treatment comparison and any endpoint, population, or subgroup condition that defines a distinct scientific claim.
Mark each as primary confirmatory, additional confirmatory, or exploratory. Do not leave this decision to the results.
Specify the contrast, outcome, target population, and direction or other criterion for each confirmatory claim.
Confirm that the planned error-control approach covers the full set of claims designated as confirmatory, rather than only a convenient subset.
This sequence is a design task, not an analysis-driven selection rule. It is useful to document the family in the protocol and analysis plan, including the rationale for any claims placed outside it. The specification should be fixed before outcome information can influence which claims are included.
Worked Specifications
Worked Example: Two Active Treatments and One Primary Claim
Hypothetical design. A trial compares two active treatments, A and B, with a shared control. Its protocol names the mean change in a symptom score at 12 weeks among all randomized participants as the primary endpoint. Treatment A versus control is the only primary confirmatory claim. The protocol also proposes comparing B with control and measuring a quality-of-life endpoint, both for exploratory purposes.
Step 1: Identify the primary hypothesis. Let \(\theta_A\) be the mean 12-week symptom-score change for A minus the mean change for control, with positive values defined as greater improvement. The primary superiority claim corresponds to \(H_{0A}: \theta_A \leq 0\) versus \(H_{aA}: \theta_A > 0\).
Step 2: Classify the other analyses. The B-versus-control symptom comparison and the quality-of-life comparison are explicitly exploratory. They are not additional members of this confirmatory family as specified. The confirmatory family therefore contains one hypothesis: \(F=\{H_{0A}\}\).
Step 3: State the boundary on interpretation. A favorable exploratory result for B or quality of life does not become a confirmatory finding merely because it is statistically compelling. If the trial intends to support confirmatory claims about those results, their hypotheses need to be included in a prespecified error-control plan. This example specifies family membership; it does not establish a method for adjusting tests.
Worked Example: Every Treatment-versus-Control Comparison Is Confirmatory
Hypothetical design. A three-arm study compares doses A, B, and C with a shared control on a single, prespecified binary primary outcome at 30 days. The investigators intend to make a confirmatory claim for any dose that demonstrates superiority to control. They specify the same target population and one-sided direction for all three comparisons.
Step 1: Write the claims. Let \(p_A\), \(p_B\), \(p_C\), and \(p_0\) denote the event probabilities under the treatments and control. Suppose a lower event probability is better. The three superiority hypotheses can be written \(H_{0A}: p_A \geq p_0\), \(H_{0B}: p_B \geq p_0\), and \(H_{0C}: p_C \geq p_0\), each against the corresponding alternative \(p_j < p_0\).
Step 2: Determine the family from the intended claims. Since any of the three dose-versus-control results may support a confirmatory claim, all three null hypotheses belong in the family: \(F=\{H_{0A},H_{0B},H_{0C}\}\). The shared control does not turn these into one hypothesis; the intended claims remain three distinct comparisons.
Step 3: Check consistency with the plan. If the investigators instead included only A in the confirmatory family but planned to describe B or C as proven effective whenever their results were favorable, the family would not match the intended claims. The correct design decision is to include every comparison that can support a confirmatory conclusion, or to limit confirmatory conclusions to the prespecified subset.
Worked Example: Primary and Additional Confirmatory Endpoints
Hypothetical design. A trial compares one active treatment with control on two outcomes: a primary 6-month clinical endpoint and a key secondary 6-month functional endpoint. The investigators state before enrollment that they intend to make confirmatory claims on both outcomes, in the full randomized population, with higher values representing improvement.
Step 1: Define the hypotheses. Let \(\theta_1\) be the treatment-minus-control mean difference on the clinical endpoint and \(\theta_2\) the corresponding difference on the functional endpoint. The planned one-sided superiority hypotheses are \(H_{01}: \theta_1 \leq 0\) versus \(H_{a1}: \theta_1 > 0\), and \(H_{02}: \theta_2 \leq 0\) versus \(H_{a2}: \theta_2 > 0\).
Step 2: Include both intended claims. Because each endpoint may support a confirmatory conclusion, the family is \(F=\{H_{01},H_{02}\}\). Calling the second endpoint “key secondary” does not remove it from the family when it is intended to support a confirmatory claim.
Step 3: Contrast an exploratory endpoint. Suppose a biomarker is also measured, but the protocol states that its results are hypothesis-generating and will not support a confirmatory claim. That biomarker analysis is outside this specified confirmatory family. If the investigators later wish to make a confirmatory biomarker claim, they cannot treat it as if that intention had been part of the original specification.
Why Post-Outcome Selection Changes the Question
The family defines the collection of chances to make a false confirmatory claim. If the possible confirmatory hypotheses are \(H_{01},\ldots,H_{0K}\), the event of at least one false rejection is the union of the false-rejection events for the true null hypotheses in that family. Changing which hypotheses are eligible for a claim changes that event. Therefore, an error-control statement applies to a defined set of hypotheses and claims; it cannot be interpreted independently of how that set was chosen.
This makes outcome-driven reclassification especially problematic. If investigators inspect results, keep only favorable comparisons, and then describe those comparisons as the family, the selected set is no longer the prespecified set to which the original error-control plan referred. The issue is not repaired by calling the remaining analyses “primary” in a report.
This does not mean every statistic computed in a trial must be placed in one confirmatory family. Exploratory analyses can be scientifically useful, including analyses motivated by unexpected patterns. The distinction is that exploratory findings should be presented as exploratory, with appropriate caution, rather than treated as if they had the same confirmatory status as claims covered by a prespecified plan.
Separate Families, Repeated Looks, and Scope
Sometimes a study has scientifically distinct claim domains, such as efficacy and safety, or separate populations with different intended interpretations. A protocol may define separate families, but merely splitting a long list does not automatically provide error control across all claims in the study. The rationale for the separation and the scope of any error-control statement should be explicit. If the intended overall conclusion can arise from a claim in either family, the plan must say what error-control meaning applies to that collection of conclusions.
A related distinction is between multiple hypotheses and repeated assessments of the same hypothesis. If the same treatment comparison is examined at interim stages and at the final analysis, that remains one scientific comparison, not a new treatment contrast at each stage. However, the possibility of making the claim at more than one stage still matters to the inferential procedure. The protocol should identify the claim and the planned stages; the method used for repeated looks must account for them. Counting stage-specific analyses as separate hypotheses, or ignoring repeated looks because there is only one named contrast, confuses two different design features.
Likewise, a subgroup can define a distinct claim if the trial intends to make a confirmatory conclusion for that subgroup. It is not enough to specify “treatment effect” if confirmatory claims may also be made for particular age groups, disease-severity categories, or other populations. The subgroup definition and intended role should be stated before outcomes are known.
Diagnostics and Common Pitfalls
- Only counting arms. Three active arms do not necessarily imply exactly three confirmatory hypotheses. Endpoints, populations, and treatment-to-treatment claims may add hypotheses; a prespecified single primary comparison may make the confirmatory family smaller.
- Assuming secondary means exploratory. A secondary result intended to support a confirmatory claim belongs in the planned scope. Labels should agree with the intended interpretation.
- Promoting a favorable result. The wrong reasoning is, “This comparison had the strongest result, so it can be the confirmatory one.” The correction is to preserve the prespecified family and describe any unplanned claim as exploratory.
- Ignoring the direction of a claim. A superiority hypothesis, a noninferiority hypothesis, and a claim in the opposite direction are not interchangeable. The target contrast and decision direction need to be stated in advance.
- Splitting a family without explaining the scope. Separate lists do not, by themselves, ensure that the chance of any false claim across those lists is controlled. State which conclusions the plan covers.
- Confusing interim stages with hypotheses. Repeated looks at one contrast do not create new scientific comparisons, but they still require an inferential plan that accounts for the repeated opportunities to decide.
A final consistency check is practical: show the proposed family to someone who has not seen the outcomes and ask them to list every claim the study could make confirmatorily. If their list includes a comparison or endpoint omitted from the plan, or excludes one that is included, the specification needs clarification.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A trial has four active treatments and a control. Only treatment A versus control on the primary endpoint is designated confirmatory; all other comparisons are explicitly exploratory. What is the specified confirmatory family?
- A protocol calls two endpoints “secondary,” but says that either can support a confirmatory efficacy conclusion. Should both belong to the confirmatory family? Explain.
- Investigators examine three treatment comparisons and, after seeing the outcomes, designate the one with the smallest p-value as the sole confirmatory test. What is wrong with this reasoning?
- A single treatment contrast is tested at two planned interim stages and at the final analysis. Does this automatically mean the family contains three hypotheses? What issue still needs to be addressed?
- A study defines separate confirmatory families for two populations but makes no statement about error control across conclusions from both populations. What limitation should be made explicit?
Solutions. (1) The family contains one null hypothesis: the prespecified A-versus-control hypothesis on the primary endpoint. The other analyses are exploratory under the stated plan. (2) Yes. Their intended confirmatory role, not the “secondary” label, determines membership; both hypotheses belong in the family. (3) Family membership has been made dependent on observed outcomes. The resulting selected claim was not the claim set specified in advance, so the original error-control plan cannot simply be invoked for it. (4) No. There is one scientific contrast, not automatically three distinct hypotheses. The repeated opportunities to make a decision must nevertheless be handled by an appropriate planned procedure. (5) Separate family specifications do not alone establish control over the union of conclusions across populations. The plan should state the scope of its error-control guarantee, and any broader claim needs a corresponding plan.