Proving Error Control Under Data-Independent Selection

Why Independence Can Preserve Error Control

Selecting a treatment arm and testing its efficacy are two distinct operations. Selection changes which hypotheses will be tested; whether this also changes the false-positive risk depends on how selection was made. If the selection decision is independent of the confirmatory test statistics, conditioning on the selected set does not alter the distribution of those statistics. That fact can turn a procedure already known to control familywise error for a fixed set into a procedure that controls error for a randomly selected set.

The result is useful, but its scope is precise. “Independent selection” does not mean merely that the selection rule was written down in advance, nor that it uses a different variable name. It means that the random selected set and the full vector of confirmatory test statistics are independent under the parameter configurations for which error control is claimed. The fixed-set testing procedure must also have the required error control for every possible selected set.

Definition: Let \(J=\{1,\ldots,K\}\) index the candidate treatment hypotheses, and let \(S\subseteq J\) be the random set selected for confirmatory testing. For a parameter configuration \(\theta\), let \(I_0(\theta)\subseteq J\) be the indices of the true null hypotheses. The familywise error event is that at least one true null among the selected hypotheses is rejected. Strong familywise error control at level \(\alpha\) means that, for every configuration \(\theta\), the probability of this event is at most \(\alpha\).

A selected family can contain no true null hypotheses, in which case the familywise error probability is zero. Strong control is more demanding: the bound must also hold when any nonempty combination of the candidate null hypotheses is true. Let \(R(S,T)\) denote the set of rejected hypotheses under the rule applied to selected set \(S\) and confirmatory statistic vector \(T\). The number of false rejections is \(|R(S,T)\cap I_0(\theta)|\), so the error event is \(|R(S,T)\cap I_0(\theta)|\geq 1\).

Conditions and Preservation Result

The key assumption is joint independence of \(S\) and \(T\), not just independence of each individual test statistic from some summary of the selection decision. The assumption must hold under each parameter configuration considered. In addition, for every possible selected set \(s\) with positive probability, the testing rule used when \(S=s\) must control the familywise error rate at level \(\alpha\) for the true null hypotheses in that set.

Theorem: Suppose that for every relevant \(\theta\), \(S\) is independent of the full confirmatory statistic vector \(T\). Suppose also that, for each selected set \(s\) with \(P_\theta(S=s)>0\), the fixed-set testing rule has familywise error probability at most \(\alpha\) for the true nulls in \(s\), under the distribution of \(T\) for that \(\theta\). Then the procedure that selects \(S\) and applies that rule has strong familywise error control at level \(\alpha\).

The result does not require the confirmatory statistics for different arms to be independent. Their dependence may be substantial, for example because treatment comparisons share a control group. What matters is that the fixed-set procedure accounts for the dependence it has, and that selection does not change the joint distribution on which that procedure relies.

Condition on a selected set.
For a fixed \(s\), evaluate the probability of rejecting at least one true null in \(s\), conditional on \(S=s\).

Use independence.
Since \(S\) and \(T\) are independent, conditioning on \(S=s\) leaves the distribution of \(T\) unchanged. The conditional error probability is therefore the fixed-set error probability for \(s\).

Average over selected sets.
Multiply each conditional error probability by \(P_\theta(S=s)\) and sum over all possible selected sets. Each conditional probability is at most \(\alpha\), so their weighted average is at most \(\alpha\).

In equations, write \(E\) for the event of at least one false rejection. The law of total probability and the assumptions give

\begin{aligned} P_\theta(E) &=\sum_{s:P_\theta(S=s)>0}P_\theta(E\mid S=s)P_\theta(S=s)\\ &\leq\sum_{s:P_\theta(S=s)>0}\alpha P_\theta(S=s)\\ &=\alpha. \end{aligned}

The final equality follows because the probabilities of all possible selected sets sum to one. This argument applies separately to every \(\theta\), which establishes strong rather than only all-null control.

Worked Applications

Worked Example: Bonferroni Testing After Selecting Two Arms

Consider three treatment hypotheses. An independent selection rule chooses \(S=\{1,3\}\) for confirmation. Suppose the confirmatory \(p\)-values for these arms are \(0.018\) and \(0.041\), respectively, and the planned procedure is Bonferroni testing at \(\alpha=0.05\) across the two selected hypotheses. Determine the decisions and explain why the error control applies.

For \(m=|S|=2\) selected hypotheses, the Bonferroni critical value for each \(p\)-value is \( \alpha/m=0.05/2=0.025\). Thus arm 1 is rejected because \(0.018\leq0.025\), while arm 3 is not rejected because \(0.041>0.025\). These are hypothetical confirmatory results; the decision does not itself prove that arm 1 is efficacious.

To check error control, fix any configuration of true and false nulls among the selected arms. Let \(q\) of the two selected nulls be true. For each true null, assume its confirmatory \(p\)-value is valid: its probability of being at most \(u\) is no greater than \(u\) for \(0\leq u\leq1\). The probability of at least one false rejection is bounded by the sum of the individual probabilities:

P_\theta(E\mid S=\{1,3\}) \leq q\frac{0.05}{2} \leq 2\frac{0.05}{2} =0.05.

This union bound does not require the two \(p\)-values to be independent. Because the selected set is independent of the confirmatory statistics, the same fixed-set bound holds conditionally on each possible selection outcome. The theorem then gives unconditional strong familywise error control at 0.05.

Worked Example: Averaging Error Bounds Across Random Sets

Suppose two hypotheses are both true, and an independent selection rule chooses \(\{1\}\) with probability 0.50, \(\{2\}\) with probability 0.30, and \(\{1,2\}\) with probability 0.20. Assume their confirmatory \(p\)-values are independent Uniform(0,1) under this configuration. Use Bonferroni testing at level 0.05 within the selected set to calculate the unconditional familywise error probability.

If the selection is \(\{1\}\), the sole test uses threshold 0.05, so its error probability is 0.05. The same is true for selection \(\{2\}\). If both hypotheses are selected, each test uses threshold \(0.05/2=0.025\). Independence and uniformity give the conditional probability of no rejection as \((1-0.025)^2=0.975^2=0.950625\). Therefore, the conditional familywise error probability for that set is \(1-0.950625=0.049375\).

Weighting these conditional probabilities by the selection probabilities gives

\begin{aligned} P_\theta(E) &=0.50(0.05)+0.30(0.05)+0.20(0.049375)\\ &=0.025+0.015+0.009875\\ &=0.049875. \end{aligned}

The result is below 0.05. This calculation uses independence of the two \(p\)-values to obtain the exact probability for the two-test set. If their dependence were unknown, the Bonferroni union bound would still give a conditional error probability no greater than 0.05, and hence the same overall upper bound.

Worked Example: What Changes When Selection Uses the Test Statistics

As a contrast, suppose two all-null hypotheses have independent Uniform(0,1) confirmatory \(p\)-values, and the rule selects the arm with the smaller \(p\)-value. It then tests only that arm at the unadjusted 0.05 level. Find the familywise error probability.

A false rejection occurs exactly when the smaller \(p\)-value is at most 0.05, which is the event that at least one of the two \(p\)-values is at most 0.05. Under the stated independence and uniformity assumptions,

\begin{aligned} P_\theta(E) &=1-P_\theta(p_1>0.05,\ p_2>0.05)\\ &=1-(1-0.05)^2\\ &=1-0.9025\\ &=0.0975. \end{aligned}

The familywise error probability is 0.0975, not 0.05. The selected arm's \(p\)-value is not Uniform(0,1) conditional on winning; it is the minimum of two uniform \(p\)-values. Thus the independence condition in the theorem fails, and the fixed-set nominal test does not retain its stated error rate after this selection.

Diagnostics, Alternatives, and Limits

Before relying on the theorem, check both the independence claim and the fixed-set testing claim. A useful audit asks what random quantities determine the selected set, what data determine each confirmatory statistic, and whether those quantities are independent under the null configurations of interest. Independence should be justified from the design or a probability model, not inferred from a small observed correlation. A zero correlation alone does not establish independence.

Check the whole vector. Independence between selection and each statistic considered separately may not imply independence between selection and the joint vector. The theorem requires the latter, unless a direct conditional-error argument establishes the needed bound.
Check every possible selected set. The testing procedure must control error for each set that can actually be selected. A procedure justified only for one fixed number of arms may not automatically apply when the number selected varies.
Check conditional validity. If the distribution of a test statistic changes after conditioning on selection, then a \(p\)-value that was valid before selection may not be valid conditional on the selected set.
Check shared-control dependence. Correlated treatment comparisons do not by themselves defeat this result. They do require a fixed-set procedure whose control is justified for their joint distribution.

Bonferroni, Holm, Dunnett, closed-testing, and maxT procedures are alternative ways to control multiplicity for a fixed family, with different assumptions and power characteristics. Under the theorem's conditions, a procedure with valid fixed-set control can be applied to the selected family and retain its level. The theorem does not say that every method can be used without adjustment: testing each selected arm at level \(\alpha\) is not generally a fixed-set familywise procedure when more than one hypothesis can be tested.

When selection and confirmation are dependent, the proof's conditioning step no longer yields the fixed-set distribution. The nominal error rate may then be exceeded, as in the minimum-\(p\)-value example. Nor is the theorem a general result about reuse of data, adaptive allocation, or all interim decisions: each can create dependence that must be addressed with its own valid error-control argument. The limitation is also important in the other direction: independence preserves a stated fixed-set guarantee, but does not make an inefficient or incorrectly specified fixed-set procedure more powerful or more valid.

Common Mistake

Common mistake: “The selection rule was specified before the trial, so selection is independent of the confirmatory statistics and the usual test is safe.” Prespecification alone is not a probability statement. A rule fixed in advance can still select the largest observed confirmatory statistic, making the selected set dependent on the statistics. The correction is to establish independence under the relevant model—or prove the required conditional error bound directly—and then verify control for every possible selected family.

Check Your Understanding

Answer each question, then compare with the short solution immediately below it.

State the independence condition needed for the preservation theorem. Solution: For each relevant parameter configuration, the random selected set \(S\) must be independent of the full vector of confirmatory statistics \(T\), or an equivalent conditional argument must establish fixed-set control given each selected set.
Why does averaging conditional familywise error probabilities no greater than \(\alpha\) give an unconditional bound no greater than \(\alpha\)? Solution: The unconditional probability is their weighted sum, with nonnegative weights \(P_\theta(S=s)\) that sum to one. Hence the weighted average cannot exceed \(\alpha\).
With three selected hypotheses and Bonferroni testing at \(\alpha=0.06\), what is the per-test threshold? Solution: \(0.06/3=0.02\). Each selected \(p\)-value at or below 0.02 is rejected.
Does dependence among treatment-versus-control statistics invalidate the theorem? Solution: Not by itself. The fixed-set procedure must account for the dependence and control its familywise error, while the selected set must remain independent of the full statistic vector.
Two independent null \(p\)-values are Uniform(0,1). One is selected by choosing the smaller, then tested at 0.05. What is the familywise error probability? Solution: \(1-(1-0.05)^2=0.0975\), which exceeds 0.05 because selection depends on the confirmatory \(p\)-values.