Worked Selection Rule For Binary Outcomes

Why Specify the Rule Before Looking at Interim Results?

A trial with several candidate treatments may use an interim analysis to decide which arms continue to confirmatory evaluation. For a binary endpoint, such as response by a specified visit, a simple rule can rank treatments by their observed response rates and retain only those meeting a pre-specified criterion. The rule needs to be fixed in advance: choosing the ranking method, cutoff, or number of arms after seeing the interim data can change which treatments survive.

Selection and confirmation have different jobs. Interim response rates can guide which arms are retained; they do not, by themselves, establish efficacy. Confirmatory tests on a separate validation sample can assess retained treatments while avoiding reuse of the observations that determined selection. Because several treatments may be tested, the confirmatory analysis also needs a multiplicity procedure.

Assumptions and a Concrete Selection Rule

The illustration uses a binary response endpoint, defined consistently across treatment and control arms. Each participant contributes one outcome. Within each arm, outcomes are treated as independent Bernoulli observations with a common response probability. Randomization, consistent outcome ascertainment, and the planned analysis population are assumed. There is no differential missingness in the examples; in an actual trial, the handling of missing outcomes must be specified because excluding or classifying them differently can change response rates.

Suppose the interim rule ranks candidate arms by their observed response-rate difference from control. It retains no more than two arms, and an arm is eligible only if its observed difference is at least 0.05, or five percentage points. Ties are resolved by a fixed arm priority specified in the protocol. This threshold is a selection criterion, not a significance threshold. The rule is descriptive: it does not claim that an eligible arm has a true benefit of at least five percentage points.

Confirmatory Score Tests for Binary Outcomes

For each retained treatment, consider a one-sided test of whether its response probability exceeds the control response probability. Write the treatment and control probabilities as \(p_T\) and \(p_C\). The hypotheses are \(H_0:p_T\le p_C\) against \(H_a:p_T>p_C\). The two-sample score test evaluates the boundary \(p_T=p_C\), which is the least favorable null value for this upper-tail test.

For independent treatment and control samples of sizes \(n_T\) and \(n_C\), let \(\hat p_T\) and \(\hat p_C\) be their observed response proportions. Under the equality boundary, the pooled estimate is \(\hat p_{\text{pool}}=(x_T+x_C)/(n_T+n_C)\), where \(x_T\) and \(x_C\) are the responder counts. The score statistic and its large-sample one-sided p-value are:

This normal approximation relies on independent observations and sufficiently large expected counts of both responses and nonresponses under the null. It is not an exact finite-sample test. In particular, using a large-sample score p-value does not guarantee exact error-rate control for every possible sample size or data configuration.

Holm Adjustment and the Error-Control Argument

Holm’s step-down procedure adjusts for testing a family of hypotheses while allowing arbitrary dependence among p-values, provided each p-value is valid under its null. For \(m\) hypotheses, sort the p-values as \(p_{(1)}\le\cdots\le p_{(m)}\). Compare \(p_{(1)}\) with \(\alpha/m\). If it exceeds that threshold, reject none. Otherwise reject its hypothesis and compare \(p_{(2)}\) with \(\alpha/(m-1)\), continuing in order. Stop at the first failure; do not reject that hypothesis or any later one.

The reason Holm controls the familywise error rate under valid p-values can be seen directly. Suppose there are \(k\) true null hypotheses. If any true null is rejected, consider the first true null in the ordered list. At its position, at least the \(k\) true-null p-values remain, so the Holm threshold at that position is no larger than \(\alpha/k\). Thus a false rejection implies that at least one true-null p-value is at most \(\alpha/k\). By the union bound and validity of each true-null p-value, the probability of this event is at most \(k(\alpha/k)=\alpha\). This argument gives strong familywise error control, meaning control for any configuration of true and false null hypotheses.

When selection is based only on interim data and validation data are independent of those data, conditioning on the selected set does not alter the validation p-value distributions under their null hypotheses. Consequently, if the confirmatory p-values are valid, Holm controls the conditional familywise error rate at no more than \(\alpha\), and hence also controls the unconditional rate. With the large-sample score tests used here, however, this guarantee is approximate: their p-values are not guaranteed to be super-uniform at finite sample sizes. Exact control requires valid p-values for the design and null being tested.

Diagnostics, Alternatives, and Limitations

Before interpreting the result, check that the implemented rule matches the protocol: the endpoint and time point are consistent, the ranking uses the stated estimator, the cutoff is applied in the stated direction, and ties are resolved as planned. Reconstruct the counts and denominators for each arm. Unexpected denominator changes, missing outcomes, or inconsistent endpoint coding can alter both interim selection and validation inference.

For the score tests, inspect the responder and nonresponder counts in each group. Very small expected counts make the normal approximation questionable; extreme response rates can also make the approximation less reliable. Examine randomization and data-quality summaries for departures from independence, such as cluster assignment or repeated observations treated as separate participants. If these issues arise, the stated score-test calculation may not be appropriate; a design-appropriate model or valid small-sample test should be specified instead.

The independent validation sample is the central protection against selection affecting confirmatory inference. If the same outcomes are used both to rank arms and to test them, the selected arms tend to have favorable interim noise, and ordinary unadjusted confirmatory p-values need not retain their nominal null behavior. Selection-adjusted procedures and analyses that explicitly account for data reuse are alternatives, but they require methods designed for that dependence.

Holm is a practical alternative to a joint maxT procedure: it is valid under arbitrary dependence when the component p-values are valid, though it may be less powerful when the correlation structure can be used effectively. Bonferroni is simpler but generally no more powerful than Holm. A maxT approach instead uses the joint distribution of test statistics and can exploit known correlation, but its calibration must match the design. None of these multiplicity procedures repairs invalid component p-values, poor outcome measurement, or an unplanned selection rule.

A common mistake is to say, “The validation sample is independent, so Holm guarantees an exact 5% familywise error rate.” Independence addresses the effect of selection on the validation data; it does not turn a normal approximation into an exact p-value. The correction is to distinguish the claims: with valid null p-values, Holm gives the stated bound conditional on selection; with large-sample score p-values, the bound is approximate and depends on the adequacy of the approximation.