Treatment Selection and the Null Intersection

Why Treatment Selection Changes the Error Question

Suppose a trial compares several experimental treatments with a control. An interim review ranks the treatments, and the trial continues with one selected treatment. The final claim may be only that the selected treatment is efficacious. This restriction to one claim does not, by itself, guarantee Type I error control: if interim results also influence the final test, the chance of a false claim can be larger than the nominal level.

The issue is not simply how many treatments appear in the final analysis. The key question is which null hypotheses remain true, and whether the procedure protects the intersection of those nulls that could support a false claim. The closure principle provides a precise way to state that protection. For a data-dependent selection, each local test used in the closure must also remain valid under the selection rule.

Definition: For treatment \(j\), let \(\theta_j\) be its effect relative to control, with positive values representing benefit. For a one-sided efficacy claim, the elementary null is \(H_j:\theta_j\leq 0\), against \(H_{a,j}:\theta_j>0\). For any nonempty set \(I\) of treatments, the intersection null is \(H_I=\bigcap_{j\in I}H_j\): every treatment in \(I\) has effect no greater than zero. The global null is the intersection over all treatments.

If the interim rule selects treatment \(J\), the procedure may claim efficacy only for \(H_J\). Under the closure principle, rejecting \(H_J\) requires rejecting every intersection \(H_I\) that contains \(H_J\). In other words, it must reject \(H_I\) for every \(I\) with \(J\in I\). The relevant intersections include \(H_J\) itself and the intersections of \(H_J\) with other treatments. In particular, the global intersection is relevant when it contains \(J\).

What a Selection-Valid Intersection Test Requires

Let the interim data be \(Z\), and let the selection rule be \(J=g(Z)\). A local test of \(H_I\) is selection-valid at level \(\alpha\) if its rejection probability is at most \(\alpha\) for every parameter configuration in \(H_I\), including when its decision depends on \(Z\) and on the selected treatment. Formally, if \(\phi_I\) is the indicator that the procedure rejects \(H_I\), validity requires

\sup_{\theta\in H_I} P_{\theta}(\phi_I=1)\leq \alpha.

It is not enough that a test would have level \(\alpha\) if the treatment had been named before the trial. The probability statement must cover the actual selection-and-testing procedure. A standard way to establish it is to condition on the interim data and bound the conditional rejection probability under \(H_I\), then average over possible interim data. This is the conditional-error perspective.

Selection-validity must be checked for each intersection used by closure, not just for the global null. If an intersection is true, its local test must control false rejection even when the interim rule favors a treatment from that intersection. A favorable result for a treatment outside the intersection does not make the intersection true or false; it may, however, affect which test the procedure applies.

Worked Example: Selecting the Larger Interim Statistic

Assume two treatments have independent interim statistics \(Z_1\) and \(Z_2\), each distributed as \(N(0,1)\) under the global null. The rule selects \(J=\arg\max(Z_1,Z_2)\), and the selected treatment is declared efficacious if its interim statistic exceeds the one-sided 5% cutoff, 1.645. Calculate the false-claim probability under the global null.

Under the global null, either treatment can produce a false efficacy signal. The procedure rejects if the larger statistic exceeds 1.645:

P\{\max(Z_1,Z_2)>1.645\} =1-P(Z_1\leq1.645,\ Z_2\leq1.645).

By independence and \(P(Z_j\leq1.645)\approx0.9500\),

P\{\max(Z_1,Z_2)>1.645\} =1-(0.9500)^2 =0.0975.

Thus, although each fixed-treatment test has a 5% one-sided Type I error rate, selecting the larger statistic and testing it at the same cutoff gives about a 9.75% false-claim probability when both treatments are ineffective. The global intersection null \(H_{\{1,2\}}\) is not protected at level 0.05 by this rule. This calculation depends on the stated independence and standard-normal model; with dependent statistics, the joint probability must instead use their joint distribution.

For these independent statistics, an exact level-0.05 test of the global intersection can reject when \(\max(Z_1,Z_2)>c\), where

1-\Phi(c)^2=0.05, \qquad c=\Phi^{-1}(\sqrt{0.95})\approx1.955.

A simpler Bonferroni threshold is 1.960 for each one-sided test: the probability of at least one false rejection is at most \(0.025+0.025=0.05\). These are possible global-intersection protections; closure still requires valid tests for the other relevant intersections as well.

Independent Confirmation and Conditional Error

Selection does not always require a multiplicity penalty. If interim data select a treatment and a genuinely independent confirmation sample is used for the final test, then under a true selected-treatment null the final statistic retains its null distribution conditional on the interim selection. This requires independence between the selection data and confirmation data under the model, a valid final statistic for each treatment null, and a rule that makes only one efficacy claim. It is the separation of selection from confirmation—not the fact that one treatment is reported—that supports the error control.

Worked Example: Independent Confirmation of the Selected Treatment

Suppose the interim data select one of two treatments, and the selected treatment is tested using an independent confirmation statistic \(Z_J^{\mathrm{conf}}\). Assume that under \(H_j\), \(Z_j^{\mathrm{conf}}\sim N(0,1)\) at the null boundary, with a distribution shifted toward smaller values when \(\theta_j<0\). The final rule claims efficacy if \(Z_J^{\mathrm{conf}}>1.645\). Check the false-claim probability when the selected treatment’s null is true.

Conditional on the interim data, \(J\) is fixed. Since the confirmation data are independent of those interim data, under \(H_J\) the conditional rejection probability is at most the one-sided normal tail at the boundary:

P_{\theta_J\leq0} \left(Z_J^{\mathrm{conf}}>1.645\mid Z\right) \leq 1-\Phi(1.645) \approx0.0500.

Averaging this conditional bound over the interim data leaves the rejection probability at most 0.05 for any true elementary null. Because only one treatment can be claimed efficacious, this also bounds the probability of at least one false claim at 0.05. The argument would fail if the confirmation statistic reused interim observations in a way that made its conditional null distribution different from the stated normal distribution.

When interim observations contribute to the final test, conditional error gives a different route. For a valid original test of \(H_I\), define its conditional error at interim data \(Z=z\) as the original test’s probability of rejecting \(H_I\), conditional on \(Z=z\), under the relevant null model. An adapted continuation test preserves protection if, for every relevant \(I\) and every possible interim outcome, its conditional probability of rejecting \(H_I\) is no greater than that original conditional error. The bound is intersection-specific: a budget calculated for the global null does not automatically establish validity for every other intersection.

Worked Example: A Conditional Error Budget After Interim Data

For one treatment, suppose an original one-sided test rejects when \( (Z_1+Z_2)/\sqrt{2}>1.645\), where under the null \(Z_1\) is the interim statistic, \(Z_2\sim N(0,1)\) is independent continuation data, and the combined statistic has a standard normal null distribution. The observed interim value is \(Z_1=1.000\). Calculate the original conditional error at the null boundary.

Given \(Z_1=1.000\), rejection requires

\frac{1.000+Z_2}{\sqrt{2}}>1.645 \quad\Longleftrightarrow\quad Z_2>1.645\sqrt{2}-1.000 \approx1.326.

Therefore, the conditional error is

P(Z_2>1.326)\approx0.0924.

This conditional error exceeds 0.05, but that does not contradict the original test’s 5% unconditional level: favorable interim values have larger conditional error, balanced by less favorable values with smaller conditional error. At this observed interim value, a new continuation rule for this same intersection must have conditional rejection probability no greater than about 0.0924 to satisfy this bound. For example, a continuation rule with conditional rejection probability 0.0800 meets the bound at this interim value. To establish overall validity, the comparison must hold at every interim value and for every relevant intersection. If a treatment is selected using interim results, these conditional comparisons must account for that selection.

Why Intersections Protect Against Partial Nulls

Control under the global null alone is not the same as strong control. Strong family-wise error control means that, under any configuration of true and false elementary nulls, the probability of making at least one false efficacy claim is at most \(\alpha\). Closure explains why testing the relevant intersections is sufficient. Let \(I_0\) be the set of all treatments whose elementary nulls are true under the actual parameter configuration. If any true \(H_j\), with \(j\in I_0\), is rejected by a closed procedure, then the procedure must also reject \(H_{I_0}\), because \(j\in I_0\). Consequently,

P(\text{at least one false claim}) \leq P(\text{reject }H_{I_0}) \leq \alpha,

provided the local test of \(H_{I_0}\) is valid under that intersection, including after selection. This argument also shows why a test of only the global null is insufficient: when some treatments are effective and others are not, the set of true nulls may be a proper subset of all treatments.

The main alternatives address different sources of dependence. Independent confirmation separates treatment selection from the final test. Bonferroni or a calibrated maximum-statistic test can protect a specified intersection by accounting for the joint behavior of treatment statistics. Conditional-error preservation allows a continuation rule to change while retaining a bound for each relevant intersection. These approaches are not interchangeable without their assumptions: independence must be genuine, joint calibration must match the statistic dependence, and conditional bounds must cover the actual adaptation.

Diagnostics and Common Mistakes

Before using a selected-treatment procedure, verify the following points:

State the claim rule. Identify the selection statistic, the selected treatment, and the exact event that produces an efficacy claim.
List the intersections. For every elementary hypothesis that could be rejected, identify all intersections containing it that the closed procedure requires.
Check selection under each null. Confirm that each local test remains valid after conditioning or otherwise accounting for how the treatment was selected.
Check the data dependence. Confirm whether confirmation data are independent of interim data, or whether a valid conditional-error comparison is available.
Match the model to the analysis. If the treatment statistics are correlated, use their joint distribution rather than an independence calculation. For composite nulls, verify that the stated bound holds throughout the null, not just at a convenient parameter value.

A common mistake is: “Only one treatment will be claimed, so testing that treatment at level 0.05 always controls the false-claim probability at 0.05.” The correction is that the final test must be valid after the data-dependent selection. Reusing the same interim evidence can make the selected statistic unusually large even when every treatment is ineffective. A second mistake is to test only the global intersection and assume this guarantees strong control. The correction is to protect the intersections required by closure, including proper subsets that may be the true null set when other treatments are effective.

Key takeaway: Selecting one treatment restricts the possible efficacy claim, but does not erase the selection mechanism. Protect the intersection nulls required by closure with local tests that remain valid under the actual selection rule, using independent confirmation, suitable joint calibration, or intersection-specific conditional-error bounds.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For three treatments with elementary nulls \(H_1,H_2,H_3\), write the intersection null for treatments 1 and 3.
Two independent standard normal interim statistics are compared, and the larger is tested using the one-sided 5% cutoff 1.645. What is the global-null false-claim probability?
Why can an independent confirmation statistic permit a level-0.05 test of the selected treatment without using the interim statistic’s null distribution again?
If the true elementary nulls are \(H_1\) and \(H_3\), which intersection must be rejected if a closed procedure falsely rejects \(H_1\)?

Solutions. (1) \(H_{\{1,3\}}=H_1\cap H_3\), meaning both treatments 1 and 3 have effects no greater than zero. (2) The probability is \(1-(0.95)^2=0.0975\), so testing the selected maximum at 1.645 does not control the global-null false-claim probability at 0.05. (3) Conditional on the interim data, the selected treatment is fixed, and independence leaves its confirmation statistic with the stated null distribution; the one-sided rejection probability is therefore at most 0.05 under its null. (4) It must also reject \(H_{\{1,3\}}\), the intersection of all the true elementary nulls. A valid level-\(\alpha\) test of that intersection bounds the probability of falsely rejecting any true elementary null by \(\alpha\).