Data-Dependent Critical Values

Why a Critical Value Chosen at Interim Needs a Conditional Check

An interim review can lead investigators to choose a different final critical value depending on the observed data. The difficulty is that the chosen threshold is no longer fixed before the trial begins. To assess its type I error, it is not enough to look only at the nominal significance level attached to each possible threshold. Instead, condition on the interim data and compare the revised test’s rejection probability with the rejection probability the original test had remaining at that same interim result.

That remaining rejection probability is the conditional error. A revised critical value preserves the original test’s error control if, for every relevant interim result, the revised procedure rejects under the null with probability no greater than the original conditional error. This is a pointwise check on the rule for selecting the threshold, not a check performed only after averaging across interim outcomes.

A Two-Stage Normal Model

Use a simple one-sided test to make the calculation explicit. Under the null hypothesis, suppose the standardized interim and final-stage increments \(Z_1\) and \(Z_2\) are independent standard normal random variables. The combined statistic is \(Z=(Z_1+Z_2)/\sqrt{2}\), so \(Z\sim N(0,1)\) under the null. The prespecified test rejects when \(Z>1.96\), giving a one-sided significance level of approximately 0.025. This model assumes independent increments, known information weights of one half at each stage, and the stated null distribution. If those assumptions do not describe the trial, the conditional distribution and bound must be recalculated for its actual design.

After observing \(Z_1=x\), the original test rejects exactly when the final increment exceeds a threshold \(q(x)\). Rearranging the original rejection rule gives

Since \(Z_2\) remains standard normal conditional on \(Z_1=x\), the original conditional error is \(A(x)=1-\Phi(q(x))\), where \(\Phi\) is the standard normal cumulative distribution function. The interim value changes the amount of type I error remaining: a low interim statistic makes the original rejection harder to achieve at the final stage, while a high one makes it easier.

Why the Pointwise Bound Preserves Overall Error

The conditional check works because an unconditional rejection probability is the average of the conditional rejection probabilities over the interim data. Under the null, the law of total probability gives

The last equality is the same law of total probability applied to the original rejection event. Thus, if the original test has type I error at most \(\alpha\), the revised test also has type I error at most \(\alpha\). This short justification depends on the conditional inequality holding across the interim data, not merely on the revised probabilities averaging to a particular value in one calculation.

Checking a Candidate Threshold at a Given Interim Value

A practical assessment has three parts: identify the original rejection rule, condition on the observed interim statistic to find the remaining null rejection probability, and compute the revised rule’s conditional rejection probability using the same conditional distribution. In the normal model, both probabilities are upper-tail probabilities for \(Z_2\), so comparing their thresholds is sufficient. When the revised test uses a different statistic or distribution, compare the probabilities themselves rather than assuming that a numerical threshold alone determines which procedure is more conservative.

When the Adaptive Cutoff Is Written on the Combined Statistic

A protocol may specify a data-dependent critical value \(k(x)\) for the combined statistic \(Z\), rather than a threshold \(d(x)\) for \(Z_2\). These forms are equivalent in the two-stage model: conditional on \(Z_1=x\), the rule \(Z>k(x)\) requires \(Z_2>\sqrt{2}k(x)-x\). The original critical value is 1.96. Therefore, a combined-statistic cutoff no smaller than 1.96 gives a final-stage threshold no smaller than the original conditional threshold. A cutoff below 1.96 gives a lower final-stage threshold and exceeds the original conditional error in this model.

Assumptions, Diagnostics, and Alternatives

The calculations above are valid only under the stated probability model. In particular, the final increment must have the specified conditional distribution given the interim data. Independent standard normal increments are a convenient model, not a universal feature of adaptive trials. Unequal information at the two stages changes the weights in the combined statistic. Estimated variance, discrete outcomes, dependence, or nuisance parameters can also change the conditional distribution. In those settings, derive or bound the conditional rejection probability using the actual design assumptions; do not reuse \(q(x)=\sqrt{2}(1.96)-x\) automatically.

A useful diagnostic is to evaluate the proposed rule over the full range of interim values it may encounter, paying particular attention to values where the adaptive cutoff is lowest. Check that the rule is defined at boundary values and that any numerical approximations to tail probabilities are accurate enough to determine which side of the bound they fall on. If nuisance parameters are present, the bound must be valid for the relevant null parameter values; a calculation at only one convenient parameter value may not establish control for a composite null.

The main alternative is to keep the original, prespecified critical value. That avoids recalculating conditional bounds, but it may not permit the desired design changes. Another approach is to specify a valid adaptive procedure in advance and evaluate its unconditional type I error under the full null model. Conditional-error checking instead provides a direct way to judge whether a post-interim choice spends more rejection probability than the original test had remaining at each interim result. These approaches answer related but distinct questions: a nominal cutoff by itself does not show that a data-dependent selection rule is valid.

A common mistake is to argue, “The revised cutoff is stringent, and the average rejection rate appears to be at most \(\alpha\), so the change is safe.” The correction is to compute the original conditional error and compare it with the revised conditional rejection probability for each relevant interim result. A threshold can be numerically large yet still be too permissive at a particular interim value, and an average calculation alone does not establish the pointwise conditional bound required here.

The method also has limits. It does not make an incorrect conditional model correct, nor does it by itself validate every aspect of an adaptive design. Its conclusion is specifically about preserving the stated null rejection bound under the assumed conditional distribution. If a design changes the statistic, endpoint, or hypotheses, the revised rejection event must be expressed precisely before its conditional probability can be assessed.