Why Define Error Control After Selection?
In a multiarm trial, an interim rule may select some treatments for continued study, and the final analysis may test hypotheses only for those selected treatments. The selection rule changes which claims can be made, but it does not change the basic error-control question: what is the probability that the trial makes at least one false efficacy claim? A precise definition must account for both selection and final testing.
The key is to define the family of possible claims before observing which treatments are selected. The family includes all candidate treatment hypotheses eligible for selection and testing. A hypothesis that is not selected cannot be rejected under a rule that tests only selected hypotheses, but it remains part of the prespecified family because the data determined whether it became eligible for a claim.
The probability is taken over the trial’s randomness under the fixed parameter value \(\theta\): this can include random assignment, observed outcomes, and any randomized elements of the prespecified design. The definition concerns the full procedure, from selection through final rejection. If a candidate is not selected, its index is not in \(R\), so it contributes no rejection on that trial.
Strong Control Means Every Realizable Truth Configuration
A truth configuration records which candidate null hypotheses are true and which are false. Strong familywise error control at level \(\alpha\) requires the FWER to be at most \(\alpha\) for every parameter value in the model:
Here, \(\Theta\) is the parameter space for the trial. Equivalently, the bound must hold for every realizable configuration of true and false nulls. Some statistical models impose logical relationships between hypotheses, so not every imaginable pattern need correspond to any parameter value. Strong control covers every pattern that can occur under the specified model, including the global-null configuration in which all candidate nulls are true and partial-null configurations in which some are true and others false.
By contrast, weak familywise error control establishes the bound only when all null hypotheses are true. It does not, by itself, establish control when some treatments have effects and other nulls remain true. This distinction matters after selection: the probability of a false rejection must be evaluated against the true-null set in the configuration being considered, not just under the global null.
Worked Examples: Evaluating the Definition
Worked Example: False Rejections After Selecting Two Treatments
Suppose a hypothetical trial has three candidate null hypotheses, \(H_{01},H_{02},H_{03}\). Under one fixed parameter configuration, \(H_{01}\) and \(H_{03}\) are true, while \(H_{02}\) is false. The following table gives the complete hypothetical distribution of selection and rejection outcomes under this configuration.
| Outcome | Probability | Selected set \(S\) | Rejected set \(R\) |
|---|---|---|---|
| A | 0.40 | \(\{1,2\}\) | \(\{1\}\) |
| B | 0.25 | \(\{2,3\}\) | \(\{2\}\) |
| C | 0.20 | \(\{1,3\}\) | \(\{3\}\) |
| D | 0.15 | \(\{2\}\) | \(\varnothing\) |
The probabilities sum to \(0.40+0.25+0.20+0.15=1.00\), and in every outcome \(R\subseteq S\). Here \(I_0=\{1,3\}\). Outcomes A and C reject at least one true null; B rejects only the false null, and D makes no rejection. Therefore,
The FWER in this hypothetical configuration is 0.60. Counting every rejection would give the wrong answer: rejecting \(H_{02}\) in outcome B is not a familywise error because \(H_{02}\) is false in this configuration. The event of interest is at least one rejection among the true nulls.
Worked Example: Why Global-Null Control Is Not Enough
Consider a hypothetical selection-and-testing rule with two candidates. Suppose its FWER under the global null, where both nulls are true, is 0.04. Under another realizable configuration, \(H_{01}\) is true and \(H_{02}\) is false. Suppose that in this configuration the trial rejects \(H_{01}\) with probability 0.12, and never rejects a true null in the remaining outcomes.
Under the global null, the stated error probability is \(0.04\). But in the partial-null configuration, the only possible false rejection is \(H_{01}\), so
At a target level \(\alpha=0.05\), the global-null result is below the target, but the partial-null result exceeds it. Thus the stated evidence supports, at most, weak control; it does not establish strong control. To establish strong control, the required bound must hold in both configurations and in every other realizable configuration.
Worked Example: Unconditional and Selection-Conditional Error
Suppose that, under a fixed configuration with a true null in the candidate family, a particular selection outcome \(A\) occurs with probability 0.20. Conditional on \(A\), the procedure rejects a true null with probability 0.10. For all outcomes outside \(A\), suppose it never rejects a true null. Then the unconditional FWER is
The unconditional FWER is 0.020, which is below 0.05, even though the error probability conditional on selection outcome \(A\) is 0.10. This calculation illustrates that ordinary strong FWER control is an unconditional guarantee for each parameter configuration. A stronger requirement that controls error conditional on each selection outcome is a different criterion and does not follow from the unconditional definition.
Justifying the Error Event
Let \(I_i\) equal 1 when the procedure rejects \(H_{0i}\), and 0 otherwise. The indicator of a familywise error is 1 exactly when at least one true null is rejected. Thus, for a fixed \(\theta\),
Taking expectations under \(P_\theta\) gives the FWER definition because the expectation of an event indicator is the probability of that event:
If testing is allowed only for selected treatments, then \(I_i=0\) whenever \(i\notin S\). The same expression therefore includes the effect of selection without needing a separate error definition for selected and unselected candidates. Dependence among test statistics also does not alter this event identity; it affects the probability of the event and hence the value of the FWER.
Defining the Family and Checking a Guarantee
The mathematical definition is simple, but its application depends on specifying the family correctly. In a treatment-selection trial, the family is generally the set of candidate treatment claims that the prespecified procedure could make. It is not merely the set of treatments that happened to be selected in the observed trial. Defining the family after seeing the selection can omit opportunities for a false claim and misstate what the guarantee covers.
A clear error-control statement should identify:
- The claims: each candidate null hypothesis, including its endpoint, population, and estimand.
- The family: all candidate hypotheses to which the confirmatory guarantee applies.
- The full rule: how selection, testing eligibility, and final rejection are determined.
- The probability and scope: the target \(\alpha\), the parameter space, and whether control is claimed for every realizable truth configuration.
For example, if the hypotheses concern one-sided treatment effects, the protocol might define \(H_{0i}\) as “treatment \(i\) does not improve the specified endpoint relative to control,” with the alternative specifying improvement. The direction, endpoint, and target population must be fixed precisely enough that “true null” has an unambiguous meaning. Strong control for that family does not automatically cover additional endpoints, unlisted subgroups, or later analyses that were not included in the definition.
Common Mistakes, Diagnostics, and Limits
- Wrong reasoning: “The all-null case is controlled, so the procedure has strong control.” The all-null case checks only one configuration. Correction: evaluate the error criterion for every realizable pattern of true and false nulls.
- Wrong reasoning: “Only the selected hypotheses belong to the family.” Selection depends on trial data, so treating the observed selected set as the entire prespecified family can omit candidate claims. Correction: define the candidate family before selection and let the rule determine which members can be rejected.
- Wrong reasoning: “Control at level \(\alpha\) means the false-rejection probability is at most \(\alpha\) after every selection outcome.” That is a conditional guarantee, not the unconditional strong-FWER definition. Correction: state whether the intended criterion is unconditional or selection-conditional.
- Wrong reasoning: “A rejection is an error whenever the null is true, and every rejection is therefore counted.” Rejections of false nulls are not Type I errors. Correction: count only rejections in \(R\cap I_0(\theta)\), and count the trial once if that set is nonempty.
A practical diagnostic is to tabulate the realizable truth configurations and check the complete procedure’s FWER under each one, analytically or by appropriately designed simulation. Simulation estimates a probability and has its own Monte Carlo uncertainty; a small simulated estimate in a few configurations is not, by itself, a proof of a uniform bound over the parameter space. The definition also has limits: it controls the chance of at least one false rejection in the specified family, but does not guarantee high power, precise effect estimates, or protection for claims outside that family.
Different procedures can be used to seek strong control, and later material develops particular adjustments and testing strategies. Those are methods for satisfying an error criterion; they do not change the criterion itself. The defining question remains whether, under every realizable configuration, the probability of rejecting at least one true null in the prespecified family is no greater than \(\alpha\).
Check Your Understanding
Answer each question using the full selection-and-testing procedure and the prespecified family.
- There are four candidate nulls. In one configuration, \(H_{01}\) and \(H_{04}\) are true. The procedure rejects \(H_{02}\) and \(H_{04}\). Did a familywise error occur? Explain.
- What additional configurations must be considered to establish strong control if a trial has three candidate nulls and the all-null configuration has FWER 0.03?
- A procedure has unconditional FWER 0.04, but its false-rejection probability conditional on one selection outcome is 0.08. Does that contradict the unconditional result?
- Why should the family not be defined as only the treatments selected in the observed trial?
Solutions. (1) Yes. \(H_{04}\) is true and is rejected, so \(R\cap I_0\) is nonempty; rejecting the false \(H_{02}\) is not itself an error. (2) Check every other realizable truth configuration, including partial-null configurations; the all-null result alone cannot establish strong control. (3) No. Conditional error can exceed 0.04 while the unconditional probability, averaged over selection outcomes, is 0.04. These are distinct guarantees. (4) Because selection is data-dependent. Defining the family only after observing selection can exclude candidate claims that the procedure could have made and can misrepresent the scope of the error-control guarantee.