Auditing an Adaptive Test by Conditional Error

Why Audit an Adaptation Conditionally?

An interim-driven change can alter the final test even when the trial continues as planned. For example, a design might change its final critical value after observing interim results. Comparing the original and adapted tests only by their overall rejection probabilities can conceal a problem: an adaptation may use more rejection probability for some interim outcomes and less for others. The relevant question is whether the change spends more Type I error than the original test allowed at the particular interim outcome that triggered it.

The conditional-error audit makes this comparison explicit. It identifies the information available when the decision is made, fixes the null hypothesis being protected, and compares two conditional probabilities: the original test’s chance of eventual rejection and the adapted procedure’s chance of rejection, both given that interim information. The audit is specific to a test and a null hypothesis; it is not a general guarantee independent of the assumptions used to calculate those probabilities.

Definition: Let $\mathcal{F}$ denote the information available at the interim decision, and let $R_0$ be the event that the original test rejects. Under a specified null hypothesis $H_0$, the original conditional error is $$ e_0(\mathcal{F})=P_{H_0}(R_0\mid\mathcal{F}). $$ For a proposed adapted procedure with final rejection event $R_A$, its adapted conditional rejection probability is $$ a(\mathcal{F})=P_{H_0}(R_A\mid\mathcal{F}). $$ The pointwise conditional-error criterion is $a(\mathcal{F})\le e_0(\mathcal{F})$ almost surely under $H_0$.

“Almost surely” means that the inequality may fail only on a set of interim outcomes having probability zero under the null. The conditioning information must include everything used to choose the adaptation: for instance, the interim statistic, any selected branch of a rule, and any randomization used at that stage. Leaving out a decision-driving variable can make the calculated conditional probabilities answer the wrong question.

A Two-Stage Gaussian Calculation

Consider a hypothetical one-sided test with a single interim look. Under the null hypothesis, suppose the standardized interim increment $Z_1$ and the independent standardized future increment $Z_2$ satisfy $Z_1,Z_2\sim N(0,1)$. The original final statistic is $Z=(Z_1+Z_2)/\sqrt{2}$, and the original test rejects when $Z>1.96$. The value 1.96 is used as the approximate one-sided 0.025 critical value.

For this calculation, the model assumptions are that the increments have the stated standard normal distributions, are independent under the null, and are combined in the specified way. The interim information is $\mathcal{F}=\sigma(Z_1)$: the observed value $z$ is known, but $Z_2$ remains random. Conditional on $Z_1=z$, the final statistic is normal with mean $z/\sqrt{2}$ and variance $1/2$. Therefore the original conditional error is

e_0(z)=P_{H_0}\!\left(\frac{z+Z_2}{\sqrt{2}}>1.96\mid Z_1=z\right) =1-\Phi\!\left(\frac{1.96-z/\sqrt{2}}{1/\sqrt{2}}\right).

Here $\Phi$ is the standard normal cumulative distribution function. The formula conditions on the observed interim value and calculates the probability, over the future increment alone, that the original test will reject.

Worked Example: Original Conditional Error at an Interim Value

Suppose the hypothetical trial observes $Z_1=1.50$. Calculate the original conditional error under the null.

Substitute $z=1.50$ into the conditional-error formula:

e_0(1.50) =1-\Phi\!\left(\frac{1.96-\sqrt{0.5}(1.50)}{\sqrt{0.5}}\right) =1-\Phi(1.2719) \approx 0.1017.

Thus, conditional on this interim result, the original test has about a 10.17% chance of eventual rejection under the null. This is a conditional probability at $Z_1=1.50$, not the overall Type I error rate of the design.

Why the Pointwise Comparison Controls Error

The reason for comparing conditional probabilities outcome by outcome is the law of total probability, expressed for a random event through conditional expectation. Under a simple null hypothesis, the adapted test’s unconditional rejection probability is the expectation of its conditional rejection probability:

P_{H_0}(R_A)=E_{H_0}\{P_{H_0}(R_A\mid\mathcal{F})\} =E_{H_0}\{a(\mathcal{F})\}.

If $a(\mathcal{F})\le e_0(\mathcal{F})$ almost surely, taking expectations preserves the inequality. Applying the same identity to the original test gives a complete justification:

P_{H_0}(R_A) =E_{H_0}\{a(\mathcal{F})\} \le E_{H_0}\{e_0(\mathcal{F})\} = P_{H_0}(R_0).

Consequently, if the original test has Type I error at most $\alpha$ under this null, the adapted procedure also has Type I error at most $\alpha$. This argument does not require the adapted test to match the original test’s rejection probability at every interim outcome; it may use less conditional error. It does require the inequality to hold conditionally, rather than only after averaging over interim outcomes.

Worked Example: Auditing a More Stringent Final Boundary

Suppose that after the interim look the proposed procedure keeps the same statistic $Z$ but raises the final rejection boundary from 1.96 to 2.20. Under the Gaussian model above, its conditional rejection probability is

a(z)=1-\Phi\!\left(\frac{2.20-z/\sqrt{2}}{1/\sqrt{2}}\right).

At $z=1.20$, the interim contribution is $\sqrt{0.5}(1.20)$. Using that value without rounding in the tail calculation gives:

e_0(1.20) =1-\Phi\!\left(\frac{1.96-\sqrt{0.5}(1.20)}{\sqrt{0.5}}\right) =1-\Phi(1.5719) \approx 0.0580, $$ $$ a(1.20) =1-\Phi\!\left(\frac{2.20-\sqrt{0.5}(1.20)}{\sqrt{0.5}}\right) =1-\Phi(1.9113) \approx 0.0280.

At this interim value the adapted probability is smaller than the original conditional error. More generally, for every $z$, the adapted threshold inside the upper-tail calculation is larger than the original one by $(2.20-1.96)/\sqrt{0.5}>0$. Since $1-\Phi(x)$ decreases as $x$ increases, $a(z)\le e_0(z)$ for all $z$. The proposed change therefore passes the pointwise audit under the stated null model. The argument verifies conditional error preservation; the unconditional Type I error is consequently no greater than that of the original test.

Audit the Rule That Was Actually Proposed

An adaptation is often a rule with different actions in different interim regions. It must be audited as a rule, not just at one convenient value. In the Gaussian example, suppose a proposed rule uses a final boundary of 1.80 whenever $Z_1\ge 1.50$, and otherwise retains the original boundary of 1.96. For $z\ge1.50$, the adapted conditional rejection probability is

a(z)=1-\Phi\!\left(\frac{1.80-z/\sqrt{2}}{1/\sqrt{2}}\right).

Worked Example: A Rule That Fails the Conditional Audit

Evaluate the proposed boundary of 1.80 at the interim value $z=1.50$, which lies in the rule’s adaptation region. The original conditional error at this value is approximately 0.1017. Under the new boundary:

a(1.50) =1-\Phi\!\left(\frac{1.80-\sqrt{0.5}(1.50)}{\sqrt{0.5}}\right) =1-\Phi(1.0456) \approx 0.1479.

Since $0.1479>0.1017$, the adapted rule uses more conditional rejection probability than the original test allows at this interim value. The pointwise criterion fails. This is not merely a harmless discrepancy at an isolated value: the probabilities vary continuously with $z$, so the inequality also fails in an interval around 1.50, which has positive probability under the continuous null model. The simple conditional-error argument therefore cannot certify Type I error preservation for this rule.

Assumptions, Diagnostics, and Scope

A reliable audit starts by writing down four items before doing arithmetic: the null hypothesis being protected; the conditioning information available when the change is made; the original test’s conditional rejection probability; and the adapted procedure’s conditional rejection probability. State the data model and the distribution of future data conditional on the interim information. In the Gaussian illustration, independence of increments is what lets the future increment retain its standard normal distribution after conditioning on $Z_1$.

For a rule with multiple branches, calculate or bound the adapted conditional probability in every branch and compare it with the original conditional error throughout that branch. Useful diagnostics include checking boundary values, checking whether each probability changes in the expected direction as interim evidence changes, and verifying calculations with unrounded intermediate quantities. A comparison at a few selected interim values can reveal a problem, but it does not establish a pointwise inequality over a continuous range.

A common mistake is to compare only the overall average rejection probabilities, or to argue that a change is safe because it is more stringent for most interim outcomes. The correction is to condition on the full information driving the change and verify $a(\mathcal{F})\le e_0(\mathcal{F})$ almost surely. Positive and negative differences at different interim values can cancel in an average, even though the proposed rule exceeds the original conditional error in some region.

For a composite null hypothesis, the audit must protect every parameter value in the null, not just one convenient value. The simple-null calculation above is exact for the specified model. In a composite-null problem, the conditional probabilities may depend on nuisance parameters. A sufficient route is to establish the conditional inequality for every null parameter value (with the probability-one qualification under each corresponding null distribution), and then use the original test’s Type I error bound for each value. If a common conditional-error bound is used instead, its relationship to the original conditional rejection probability must be shown explicitly; substituting a point estimate of a nuisance parameter does not by itself establish uniform control.

This audit is a comparison with a specified original test under stated assumptions. If the conditional distribution used in the calculation is wrong, the calculated error bound may not apply. Likewise, passing this audit does not excuse an incomplete description of which data determine the adaptation, nor does it establish control for hypotheses or rejection claims not included in the null being audited. Its strength is precise: when the conditional probabilities are correctly specified and the pointwise inequality holds, the expectation argument shows why the adaptation does not increase rejection probability under that null.

Key takeaway: Define the interim information and null first. Then compare the adapted conditional rejection probability with the original conditional error for every relevant interim outcome. A pointwise conditional bound, together with a valid original test, gives a direct Type I error preservation argument.

Check Your Understanding

Answer each question, then compare with the brief solutions.

In the Gaussian example, what is the conditioning information at the interim look, and which quantity remains random after conditioning?
Why does $a(\mathcal{F})\le e_0(\mathcal{F})$ imply that the adapted test’s unconditional rejection probability is no greater than the original test’s?
At $Z_1=1.50$, the original conditional error is approximately 0.1017. A proposed boundary of 1.80 gives an adapted conditional rejection probability of approximately 0.1479. Does the rule pass the pointwise audit at this value?
Why is checking the inequality at only a few interim values insufficient for a rule defined over a continuous range?
What additional issue arises when the null hypothesis is composite?

Solutions. (1) The conditioning information is $\mathcal{F}=\sigma(Z_1)$, or the observed interim statistic $Z_1=z$; the future increment $Z_2$ remains random. (2) Taking expectations of the pointwise inequality gives $E_{H_0}\{a(\mathcal{F})\}\le E_{H_0}\{e_0(\mathcal{F})\}$, which equals the comparison of the two unconditional rejection probabilities by conditional expectation. (3) No: $0.1479>0.1017$, so the adapted probability exceeds the original conditional error. (4) The inequality may fail between the checked values; a complete audit needs a proof or valid bound over the entire relevant region, apart from null sets. (5) Conditional rejection probabilities can depend on nuisance parameters, so the argument must protect every parameter value in the null or establish an appropriate common bound.