Familywise Error With Intermediate-Based Selection

Why Selection Changes the Error-Control Question

In a time-to-event MAMS design, an interim outcome may be used to retain some treatment arms and drop others before the final endpoint is fully observed. The final confirmatory comparisons are then made only for retained arms. This can save resources, but it changes the set of hypotheses that reaches final testing. The relevant error rate is not just the error rate for one comparison considered in isolation: it is the probability of making at least one false confirmatory claim across the treatment comparisons in the design.

Selection itself does not automatically invalidate final testing, and prespecifying a selection rule does not automatically guarantee control. What matters is how selection relates to the final test statistics, how many hypotheses can be tested, and how the testing thresholds are calibrated. In particular, a final p-value that is valid before selection may not remain valid conditional on the arm having been selected.

Definition: Let $H_1,\ldots,H_m$ be the confirmatory null hypotheses for the $m$ treatment-versus-control comparisons that could be made. Let $I_0$ be the indices of the null hypotheses that are true, and let $R$ be the set of hypotheses rejected at the end of the design. The familywise error rate (FWER) is $$ P(R\cap I_0\ne\varnothing). $$ Strong FWER control at level $\alpha$ means this probability is at most $\alpha$ for every configuration of true and false null hypotheses, not only when all null hypotheses are true.

Write $S$ for the set of treatment arms retained by the interim selection rule. For a retained arm $i$, let $p_i$ denote the p-value for its final comparison, with smaller values favoring the treatment under the specified one-sided alternative. A valid p-value for a true $H_i$ is super-uniform: $P(p_i\leq u)\leq u$ for every $u$ between 0 and 1. This marginal property alone does not imply that $p_i$, conditional on $i\in S$, is super-uniform.

A useful distinction is between marginal validity, which concerns the distribution of a p-value without conditioning on selection, and conditional validity, which concerns its distribution given the observed selection path. Selection can favor arms with unusually promising intermediate results. If those results are associated with the final test statistics, the final statistics among selected arms can also be unusually favorable under a null hypothesis.

When Selection and Final Testing Are Dependent

Suppose two arms are both null. Let $W_i$ be an intermediate score for arm $i$, and let $Z_i$ be its final test statistic, with larger values favoring treatment. For an illustrative limiting case, suppose $W_i=Z_i$, and the two arm-specific statistics are independent standard normal variables under the global null. Retain the arm with the larger intermediate score and test only that arm at one-sided level $0.025$. The critical value is approximately 1.96.

Worked Example: Selecting the Larger of Two Related Statistics

Under the stated global-null model, the selected arm is the one with the larger of $Z_1$ and $Z_2$. A false rejection occurs if that selected statistic exceeds 1.96, which is equivalent to at least one of the two statistics exceeding 1.96.

Since $Z_1$ and $Z_2$ are independent and each is standard normal under the null, $$ \begin{aligned} \mathrm{FWER} &=P\{\max(Z_1,Z_2)>1.96\}\\ &=1-P(Z_1\leq1.96,\ Z_2\leq1.96)\\ &=1-\{P(Z\leq1.96)\}^2\\ &\approx 1-(0.975)^2\\ &=0.04938. \end{aligned} $$ Here $Z\sim N(0,1)$. The FWER is about 4.94%, nearly twice the nominal 2.5% level for a single test. The rule selected one arm, but it selected the arm with the more favorable value of the very statistic used for testing. This example is an idealized extreme of dependence; less extreme association can still cause inflation, with the amount depending on the joint distribution and the selection rule.

Intermediate and final outcomes need not be identical for this issue to arise. They may share patients, events, or other information, or be associated because the intermediate measure predicts the final outcome. The degree and form of dependence matter. In a time-to-event design, the analysis must also use a final testing procedure that is valid for the actual event and information structure; treating a statistic as standard normal is an approximation that requires justification, such as an appropriate large-sample test model.

Prespecification and Error-Control Strategies

A prespecified selection rule makes the adaptive procedure explicit enough to assess and calibrate. It should identify the intermediate outcomes used, the assessment time, the ranking or threshold rule, the maximum number of arms retained, and how ties, missing information, and departures from the planned assessment are handled. The confirmatory family and final testing thresholds should also be stated. Prespecification is not a mathematical correction by itself: an unadjusted test can still be invalid if the selected statistic is favorably biased.

There are two useful strategies. One is to adjust over the original family of all potentially testable hypotheses. Another is to adjust over the selected family, provided the final p-values remain valid conditional on the selection path. The first can be more conservative, but its basic Bonferroni guarantee does not require selection to be independent of the final statistics.

Result: If every true-null p-value $p_i$ is marginally valid and each rejected hypothesis must satisfy $p_i\leq\alpha/m$, then strong FWER is at most $\alpha$, even if selection depends on the p-values or their underlying data. This guarantee applies when the tests use those same valid p-values and the adjustment covers all $m$ potentially confirmatory comparisons.

The justification is an application of the union bound. Any false rejection must be a rejection of at least one true null, so

\begin{aligned} P(R\cap I_0\ne\varnothing) &\leq P\left(\bigcup_{i\in I_0}\{p_i\leq\alpha/m\}\right)\\ &\leq \sum_{i\in I_0}P(p_i\leq\alpha/m)\\ &\leq |I_0|\frac{\alpha}{m}\\ &\leq \alpha. \end{aligned}

No independence assumption is used in this argument. Selection cannot make the probability of the union exceed the sum of the marginal probabilities. However, the result does depend on the final p-values being valid under their null hypotheses for the design actually used. If adaptive stopping, data reuse, or other design features invalidate those p-values, Bonferroni adjustment alone does not repair that problem.

If final data are independent of the selection information under each true null, the p-values can instead be valid conditional on the selected set. In that case, a selected-family adjustment can control error while using the known maximum number of final tests. For example, if at most $k$ arms are retained and each selected comparison is tested at level $\alpha/k$, then, conditional on any selection set, the union bound gives conditional FWER at most $k(\alpha/k)=\alpha$. Averaging over possible selection sets gives unconditional FWER at most $\alpha$.

Worked Example: One Arm Selected Using Independent Interim Data

Suppose three arms are all null, exactly one is selected using intermediate data, and the final test for the selected arm uses independent final data. Assume each final p-value is uniform under its null conditional on the selected arm. Let the one-sided final test level be $0.025$.

For any selected arm, the conditional probability of a false rejection is $0.025$. If the probabilities of selecting the three arms are $0.50$, $0.30$, and $0.20$, then $$ \begin{aligned} \mathrm{FWER} &=0.50(0.025)+0.30(0.025)+0.20(0.025)\\ &=(0.50+0.30+0.20)(0.025)\\ &=0.025. \end{aligned} $$ Thus, testing the single selected comparison at level $0.025$ controls the FWER in this setting. More generally, when exactly one arm is tested and its final p-value is conditionally valid given selection, its conditional false-rejection probability is at most $0.025$ for every selection outcome. This conclusion is not justified if multiple arms can be tested at that same level or if selection and final statistics are dependent in a way that compromises conditional validity.

Worked Example: At Most Two of Three Arms Retained

Suppose three arms are all null, intermediate data select at most two, and the final data are independent of the selection information. Assume each final p-value is valid conditional on selection. For a design with overall one-sided level $\alpha=0.025$, test each selected comparison at $\alpha/2=0.0125$. For two selected arms, the standard-normal critical value is approximately 2.241.

Conditional on selecting two arms, the union bound gives $$ \begin{aligned} P(\text{at least one false rejection}\mid S) &\leq 0.0125+0.0125\\ &=0.025. \end{aligned} $$ If, in addition, the two final test statistics are independent standard normal variables under the global null, the conditional FWER for two selected arms is $$ 1-(1-0.0125)^2=1-(0.9875)^2=0.02484. $$ If only one arm is selected and the same per-comparison threshold is used, the conditional FWER is at most $0.0125$. In either case, averaging the conditional error probabilities over the selection outcomes gives unconditional FWER no greater than $0.025$. The independence of the two final statistics is needed for the displayed exact calculation, but not for the union-bound guarantee. The conditional validity of each p-value is essential to this selected-family approach.

Choosing and Checking an Approach

A fixed adjustment over all possible comparisons is a straightforward choice when the selection rule may be related to the final data or when conditional validity is difficult to establish. It can spend error-control allowance on arms that are ultimately dropped. A selected-family adjustment can be less conservative when selection uses information independent of the final tests and the maximum number of tests is specified. If neither simple argument matches the design, calibration must account for the joint selection-and-testing procedure; a threshold cannot be justified merely because it is familiar from a nonadaptive analysis.

The same issue arises if the design uses intermediate information to change final testing thresholds, to choose among multiple final endpoints, or to make additional comparisons after observing which arms survive. Each such decision can expand the set of opportunities for a false rejection. Error control must cover the complete rule for selecting, testing, and making confirmatory claims, not just the final test considered in isolation.

Check what is being counted. Include every comparison that could support a confirmatory claim, not only the comparisons expected to be retained.
Check the conditioning argument. If using a selected-family adjustment, establish that final p-values are valid given each possible selection outcome under every true null.
Check the testing model. Confirm that the final test accounts for the time-to-event data, stopping, and information used in the analysis.
Check all selection paths. A rule with several thresholds, rankings, or permitted numbers of retained arms may create different testing families or thresholds.

A common mistake is to reason, “Only one arm will be selected, so testing it at the usual level must control the error.” The conclusion is justified only if the selected arm’s final test is valid conditional on selection, or if a procedure otherwise controls error across the full selection-and-testing mechanism. Selecting the arm with the strongest related intermediate signal can make its final statistic unusually favorable under the null. Conversely, when one arm is selected using independent interim data and only one conditionally valid final test is performed, testing at the planned single-comparison level can be valid. The number of selected arms alone does not settle the question; the relationship between selection and testing does.

Key takeaway: Prespecify both the selection rule and the final testing rule. Then justify FWER control for their combination: use an adjustment over the original family when marginal p-value validity is available, or a selected-family adjustment only when conditional validity and the maximum number of tests are established.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What probability does strong FWER control bound, and why is control under the global null alone not enough?
Three null arms are all eligible for testing. Each has a marginally valid p-value, and every rejection requires $p_i\leq0.05/3$. Explain why selection may depend on the data without invalidating the Bonferroni guarantee.
A design retains at most two arms using intermediate data and tests both retained arms at level $0.025$. What is the problem with this rule under independent valid final tests?
Why does specifying an intermediate threshold in advance not, by itself, prove that the final FWER is controlled?

Solutions. (1) Strong FWER bounds $P(R\cap I_0\ne\varnothing)$ by $\alpha$ for every configuration of true and false nulls. Global-null control checks only the configuration in which every hypothesis is true; it does not establish the bound when some hypotheses are false and the remaining true hypotheses have different selection probabilities. (2) A false rejection is contained in the union of the three events $\{p_i\leq0.05/3\}$. The union bound and marginal validity give FWER at most $3(0.05/3)=0.05$, regardless of dependence or selection. (3) With two selected null comparisons, testing each at $0.025$ can yield FWER as high as $1-(1-0.025)^2=0.04938$ when the tests are independent and both are tested. A selected-family adjustment such as $0.025/2$ per comparison would give the stated level under conditional validity. (4) Prespecification makes the rule explicit but does not change the joint distribution of selection and final statistics. The final thresholds still need a valid error-control argument for that joint procedure.

Familywise Error With Intermediate-Based Selection

What You'll Learn