The Pitfall of Redefining the Family After Results

Why the Family Must Not Follow the Results

A multiplicity procedure controls error for a specified set of hypotheses. The set matters: it determines which claims are protected by the guarantee and, for procedures such as Bonferroni, how much of the error budget is assigned to each test. If investigators look at the results first and then remove hypotheses that seem unpromising, they change the question after seeing evidence. Applying a procedure as if the smaller set had been chosen in advance can make the probability of at least one false rejection larger than the stated familywise error level.

This is not a rule that every hypothesis ever considered must always be included in every analysis. The family is defined by the inferential claims and analysis plan, and a genuinely prespecified family can be narrower than the full list of possible comparisons. The problem addressed here is outcome-driven redefinition: using observed test results to decide which hypotheses count, and then claiming the error guarantee for that selected set.

Definition: A testing family is the set of hypotheses to which a specified multiplicity guarantee applies. For a family $\mathcal{F}$, the familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in $\mathcal{F}$: $$ \operatorname{FWER}(\mathcal{F})= P\left(\bigcup_{j\in I_0(\mathcal{F})}\{\text{reject }H_j\}\right), $$ where $I_0(\mathcal{F})$ indexes the true null hypotheses in that family. Strong FWER control at level $\alpha$ means this probability is no greater than $\alpha$ for every configuration of true and false null hypotheses.

A family is usually determined from the scientific objectives and the analysis plan before the outcomes are examined. In a multi-arm trial, for example, it may include all planned treatment-versus-control claims for an endpoint. The choice of family is a scientific and inferential choice; a multiplicity adjustment cannot repair a family that has been selected to favor the observed results.

Why Bonferroni Requires a Fixed Family

Suppose there are $m$ hypotheses in a fixed family, and each hypothesis is tested at level $\alpha/m$. For any true null hypothesis, its probability of rejection is at most $\alpha/m$. The union bound then gives a familywise error no greater than $\alpha$, without requiring independence among the tests.

\begin{aligned} \operatorname{FWER}(\mathcal{F}) &=P\left(\bigcup_{j\in I_0(\mathcal{F})} \{\text{reject }H_j\}\right)\\ &\leq \sum_{j\in I_0(\mathcal{F})} P(\text{reject }H_j)\\ &\leq |I_0(\mathcal{F})|\frac{\alpha}{m} \leq \alpha. \end{aligned}

The key quantity $m$ is the size of the family being protected, not the number of hypotheses that happen to look interesting after testing. If results determine which hypotheses remain in the family, the argument above cannot simply be applied with the selected number of hypotheses: selection and rejection are now linked to the same data.

Worked Example: Selecting the Smaller of Two Null p-Values

Consider two true null hypotheses. Assume, for this hypothetical example, that their p-values $P_1$ and $P_2$ are independent and each is uniformly distributed on $[0,1]$ under its null. The planned family contains both hypotheses, and the intended FWER level is $\alpha=0.05$.

With Bonferroni applied to the original family, each hypothesis is tested at $0.05/2=0.025$. The probability of at least one rejection is

P(\min(P_1,P_2)\leq 0.025) =1-P(P_1>0.025,\ P_2>0.025) =1-(0.975)^2 =0.049375.

This is below $0.05$, as the planned procedure requires. Now suppose the analyst first looks at both p-values, removes the hypothesis with the larger p-value, and tests the remaining one at $0.05$, treating it as a one-hypothesis family. A rejection occurs whenever the smaller p-value is at most $0.05$. Thus,

\operatorname{FWER} =P(\min(P_1,P_2)\leq 0.05) =1-(0.95)^2 =0.0975.

The resulting FWER is $9.75\%$, nearly twice the intended $5\%$. The p-values are valid under their respective null hypotheses; the error arises from selecting the test to retain after observing both results and then using a threshold for a family of size one.

What Changes in a Multi-Arm Trial

The same issue arises when several treatment arms are compared with a common control. The planned family could contain all treatment-control hypotheses for a particular endpoint. Dropping an arm's hypothesis because its observed p-value is large does not erase the fact that the trial generated evidence on that comparison, nor does it make the other comparisons equivalent to tests planned alone.

Worked Example: Shrinking a Four-Comparison Family

Suppose a hypothetical trial planned four treatment-versus-control tests for one endpoint, with FWER target $\alpha=0.05$. For illustration, the planned procedure is Bonferroni. The observed p-values are $0.008$, $0.018$, $0.20$, and $0.40$. Assume the four hypotheses were all part of the prespecified family.

The valid Bonferroni threshold for every comparison is

\frac{\alpha}{m}=\frac{0.05}{4}=0.0125.

Only the first p-value, $0.008$, is at most $0.0125$, so only that null hypothesis is rejected using the planned rule. If the analyst deletes the two hypotheses with p-values $0.20$ and $0.40$, then recalculates the threshold using $m=2$, the new threshold is $0.05/2=0.025$. The second p-value, $0.018$, is then also declared significant. That conclusion does not carry the original $5\%$ FWER guarantee: the family size was changed using the observed results.

The calculation illustrates the problem; it does not establish that the second null is true or false. A p-value of $0.018$ is evidence assessed under its null model, not the probability that the null is true. The error-control concern is about the long-run chance of making one or more false rejections across the claims in the planned family.

It is also possible to redefine a family without changing any particular p-value. For example, a report may present only comparisons that meet a chosen descriptive threshold, while omitting other planned tests. If those omitted tests are then excluded from the multiplicity calculation, the displayed familywise guarantee may no longer apply to the selection process. A reader needs to know both the original family and the rule used to select what is shown.

Prespecification, Reporting, and Alternatives

A smaller family can be legitimate when it is specified independently of the results and matches the claims for which error control is intended. For example, a protocol might identify two treatment arms as confirmatory for a particular endpoint and label another arm's analysis exploratory. If that distinction is made before outcomes are examined and followed consistently, the confirmatory family can be the two planned hypotheses. The fact that other analyses exist should still be disclosed; the confirmatory label does not turn exploratory findings into confirmatory evidence.

Worked Example: A Narrower Family Specified in Advance

In a hypothetical three-arm trial, the analysis plan identifies comparisons of treatments A and B with control as the confirmatory family for an endpoint. Treatment C is designated exploratory before results are available. The confirmatory family therefore has $m=2$, and Bonferroni uses a threshold of $0.05/2=0.025$. Suppose the p-values for A versus control, B versus control, and C versus control are $0.012$, $0.032$, and $0.004$, respectively.

For the stated confirmatory family, $0.012\leq0.025$, so the A comparison rejects its null hypothesis, while $0.032>0.025$, so the B comparison does not. The C result is not a confirmatory rejection under this plan, even though its p-value is numerically small; it belongs to the separately labelled exploratory analysis. This is different from seeing all three p-values and then excluding whichever comparisons do not support the desired conclusion.

When a full family has already been tested, the safest approach is to retain that family for interpretation. Report all planned comparisons, their raw p-values, the prespecified procedure, and the adjusted conclusions. If the analysis plan did not specify a multiplicity method, make that limitation explicit rather than retrofitting a smaller family and presenting its result as if it had been planned.

Several alternatives are available, depending on the scientific objective. Bonferroni and Holm procedures can provide FWER control for a fixed family, including under arbitrary dependence. Weighted procedures, fixed-sequence testing, gatekeeping, and graphical alpha-transfer procedures can encode prespecified priorities or decision structures. These methods change how error is allocated; they do not justify deleting hypotheses because of their observed results. If the scientific question genuinely concerns a data-selected subset, an appropriate selection-adjusted method or a separate confirmatory study may be needed. Simply applying a familiar multiplicity procedure to the selected subset is not a substitute.

Diagnostics and Common Failure Modes

The most useful diagnostic is a comparison between the analysis plan and the reported analysis. Ask whether the hypotheses, endpoint, direction of testing, and multiplicity procedure were specified before outcome data were examined. Then check whether every planned hypothesis appears in the results, including nonsignificant ones. A mismatch does not automatically prove that the results are invalid, but it signals that the stated guarantee must be reconsidered.

Removing large p-values. A large p-value does not make a hypothesis irrelevant to a family that was already tested. Keep it in the family for the planned error calculation.
Counting only reported rejections. The number of significant results is not the family size. The relevant size comes from the hypotheses covered by the inferential plan.
Using the word “exploratory” after seeing results. Relabelling an unexpected result does not retroactively make the selection rule prespecified. Report the timing and treat the finding accordingly.
Assuming a different procedure removes the selection problem. Holm or another valid fixed-family procedure still requires the family to be the one whose claims are being protected. Changing the procedure does not make an outcome-selected family fixed.
Confusing omission with non-testing. Leaving a comparison out of a table does not undo its role in a planned analysis. Transparent reporting makes the scope of the guarantee assessable.

The principal limitation is that a protocol cannot anticipate every scientific question. New questions may arise during analysis, and exploring them can be valuable. The appropriate response is not to suppress exploration, but to distinguish exploratory evidence from confirmatory error-controlled claims and to avoid overstating what the original FWER guarantee covers.

Key takeaway: Define the confirmatory family before examining results and preserve it when applying its multiplicity procedure. If hypotheses are removed or promoted based on observed outcomes, disclose the selection and do not claim the original familywise error guarantee for the resulting data-selected set without a method that accounts for that selection.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For three true null hypotheses with independent uniform p-values, an analyst selects the smallest p-value and tests it at $0.05$. What is the probability of at least one false rejection?
A planned family has five hypotheses and uses Bonferroni at $\alpha=0.05$. After seeing results, an analyst drops three hypotheses and uses $m=2$. Why is the new threshold not covered by the original guarantee?
When can a family of two hypotheses be appropriate even if a trial collected data for three treatment comparisons?
Does a p-value above the Bonferroni threshold mean the corresponding hypothesis should be removed from the family? Explain.

Solutions. 1. The selected p-value is the minimum, so the probability of a rejection is $1-(1-0.05)^3=1-0.95^3=0.142625$, or about $14.3\%$. 2. The threshold $0.05/2=0.025$ is based on a family size chosen using observed results; it does not preserve the original five-hypothesis procedure's guarantee. 3. It can be appropriate if the two-hypothesis confirmatory family was specified before outcomes were examined and the third comparison is clearly treated under a separate plan, such as an exploratory analysis. 4. No. The p-value affects the decision for that hypothesis, not whether it belonged to the prespecified family; removing it after observing the result changes the selection process and can invalidate the stated FWER guarantee.