Implementing Selection-Adjusted Tests Reproducibly

From an Adaptive Rule to a Reproducible Test

A selection-adjusted test is not fully specified by naming a correction such as Bonferroni or a max-statistic test. The analysis also depends on which hypotheses are eligible, how arms are selected or dropped, whether the selection data are reused, and what covariance structure the calculation assumes. If any of these inputs are left implicit, two analysts can apply different tests to the same trial.

The goal is to define a sequence of decisions that another analyst can reproduce from the protocol and the analysis data. The examples below use one-sided superiority tests and independent confirmatory data after interim selection. That separation is important: it permits valid testing conditional on the selected set without requiring the confirmatory statistics to account for the interim measurements.

Specify the Family, Selection, and Data Roles

Begin by writing each clinical claim as a hypothesis. For example, for treatment \(i\) versus a shared control, let \(\theta_i\) be the treatment effect in the prespecified estimand and let larger values favor treatment. For a one-sided superiority objective, use \(H_{0i}:\theta_i\leq 0\) against \(H_{ai}:\theta_i>0\). State the estimand, endpoint, analysis population, direction of benefit, and analysis time point; otherwise, the same symbolic hypothesis could refer to different clinical questions.

Next, enumerate the original hypotheses and define the selection rule in operational terms. A rule should say what interim statistic is used, when it is calculated, how ties are resolved, which arms are retained, and whether there is a minimum or maximum number of arms that may continue. “Select the promising arms” is not reproducible. “Retain the two arms with the largest interim standardized treatment-control statistics, provided each is at least 0.5; break ties by lower prespecified treatment code” is reproducible.

Finally, identify which data contribute to selection and which contribute to confirmatory testing. If the confirmatory stage is newly accrued and independent of interim data, then, conditional on the selected set, its test statistics have their prespecified null distributions. If the same observations inform both selection and testing, ordinary post-selection p-values generally do not have that property. Reusing data requires a method whose null distribution explicitly accounts for the selection procedure; simply applying a correction to selected ordinary p-values does not automatically do so.

Multiplicity and Covariance Inputs

Suppose a selected set \(S\) contains \(m\) treatment hypotheses. Bonferroni testing assigns each hypothesis the level \(\alpha/m\). For a one-sided standard-normal statistic \(Z_i\), reject \(H_{0i}\) when \(Z_i\) exceeds the standard-normal quantile \(z_{1-\alpha/m}\). Equivalently, compare its one-sided p-value with \(\alpha/m\). Bonferroni does not require independent test statistics.

This selected-set calculation is valid when selection is independent of the confirmatory statistics and the claims are limited to hypotheses in \(S\). It controls the probability of one or more false rejections among those claims, conditional on the interim selection result. If the plan instead tests the entire original family, including arms not selected, specify a procedure for that full family; do not silently change the family after seeing results.

When treatment contrasts share a control, their confirmatory statistics are typically correlated. For independent treatment and control samples with common outcome variance \(\sigma^2\), equal sample sizes \(n\), and contrasts \(D_i=\bar{Y}_i-\bar{Y}_0\), each contrast has variance \(2\sigma^2/n\). Two different contrasts share the control mean, so their covariance is \(\sigma^2/n\), giving correlation \(1/2\). More generally, record the covariance matrix of the contrasts and standardize it to obtain their correlation matrix \(R\). A max-statistic procedure can use \(R\) to choose a critical value \(c\) satisfying \(P(\max_{i\in S} Z_i\leq c)=1-\alpha\) under the joint null model. The numerical integration method, software, and precision should also be recorded.

A covariance-based method can be less conservative than Bonferroni, but it depends on the validity of the joint distribution used to calculate the critical value. Bonferroni is a useful transparent alternative when a reliable covariance calculation is unavailable. Neither method repairs data reuse that has not been included in the inferential procedure.

Why the Selected-Set Bonferroni Test Controls Error

The justification is conditional on the selection data. Fix any interim result that produces a nonempty set \(S\) of size \(m\). For each true null hypothesis in \(S\), a valid confirmatory p-value satisfies \(P(p_i\leq\alpha/m\mid\text{interim data})\leq\alpha/m\). The union bound then gives

If \(S\) is empty, there are no confirmatory rejections and the error probability is zero. Averaging the conditional bound over possible interim results gives unconditional familywise error at most \(\alpha\). Independence matters because it supports the conditional validity of the confirmatory p-values after the interim selection result is fixed. The argument does not establish validity for tests that reuse the selection data without adjustment.

Covariance Calculation and Method Choice

Covariance inputs should be tied to the actual estimator, not copied from a generic template. Under the independent normal sampling model, with treatment \(i\) and control sample sizes \(n_i\) and \(n_0\), and variances \(\sigma_i^2\) and \(\sigma_0^2\), the contrast \(D_i=\bar{Y}_i-\bar{Y}_0\) has variance \(\sigma_i^2/n_i+\sigma_0^2/n_0\). For two treatment arms \(i\) and \(j\), independent of each other and sharing the same control sample, the covariance is \(\sigma_0^2/n_0\). Therefore the correlation used for standardized contrasts is

For stratified, repeated-measures, or otherwise more complex estimators, the covariance must follow the specified analysis model. A max-statistic implementation should state the ordering of hypotheses, the entries of \(R\), the tail direction, the target \(\alpha\), the method used to find \(c\), and its numerical tolerance. If simulation estimates the critical value or adjusted p-values, record the generator, seed, number of replicates, and Monte Carlo uncertainty.

Decision Records, Diagnostics, and Pitfalls

A reproducible result should preserve both the inputs and the decisions. A compact analysis record should include:

• the estimand, endpoint, analysis population, hypothesis direction, and original hypothesis family;
• the interim data cutoff and exact selection and tie-breaking rules;
• the selected set and the data source for each confirmatory statistic;
• the test statistic definition, standard error method, covariance or correlation matrix, and distributional assumptions;
• the multiplicity method, family size, \(\alpha\), critical value or adjusted p-values, and numerical settings; and
• the rejection decision for each hypothesis and the permitted clinical interpretation of each claim.

Before unblinding or finalizing decisions, check that the selected set is generated by the written rule, the family size matches the claimed procedure, the covariance matrix has unit diagonal and is positive semidefinite, and the tail direction matches the clinical hypothesis. Recompute at least one critical value or adjusted p-value independently, and verify that displayed decisions follow the stated inequality. If simulation is used, assess Monte Carlo stability rather than reporting only a point estimate.

A frequent mistake is to select one apparently best arm and then test it at the unadjusted level \(\alpha\), reasoning that only one hypothesis remains. That is not automatically valid when the same data drove selection: the winning statistic is selected for looking unusually favorable. The correction is to use an inferential method that accounts for selection and data reuse, or to test on independent confirmatory data with a valid multiplicity procedure for the selected claims.

Another mistake is to use the full-family cutoff in the written rule but compare results with a selected-family cutoff in the report, or to calculate a covariance matrix for one model and use it with statistics from another. The remedy is not merely better documentation: the implemented calculation must match the prespecified family, model, and data pathway. Selection-adjusted tests also have limits. Their guarantees do not validate a misspecified estimand, repair informative missingness, or protect claims that were not included in the defined testing family.