Simulating Type I Error Across Adaptation Rules

Why Simulate Type I Error After Re-Estimation?

When a trial changes its stage-two sample size in response to interim data, it is useful to check the whole decision procedure under the null hypothesis. A simulation can expose errors in the adaptation rule, the stage-two test, or the final rejection calculation. It also clarifies an important distinction: a procedure can have the intended overall Type I error even though its rejection probability conditional on a particular interim result is not equal to that overall rate.

The goal here is not to find a new critical value by trial and error. Instead, specify a method whose null behavior is justified, simulate that method as it will actually be used, and estimate its probability of rejecting $H_0$ when $H_0$ is true. The introductory material on combination tests and conditional error provides the relevant test constructions; the focus here is designing and interpreting the validation simulation.

Null Model and Procedures to Compare

Use a simple two-stage normal model so that the simulation target is explicit. Suppose observations have known standard deviation $\sigma$, and test $H_0:\mu=0$ against $H_a:\mu>0$ at one-sided $\alpha=0.025$. Stage 1 has $n_1=50$ observations, and the planned stage 2 also has 50. Under $H_0$, with independent observations across stages, the standardized stage statistics

Z_1=\frac{\sqrt{n_1}\bar{X}_1}{\sigma}, \qquad Z_2=\frac{\sqrt{n_2}\bar{X}_2}{\sigma}

are independent standard normal variables, including when $n_2$ is selected using stage-one information. This statement requires the stage-two observations to be new and independent, the variance to be known, and the stage-two statistic to be standardized using the selected $n_2$. Thus, conditional on the interim data and the selected sample size, $Z_2\sim N(0,1)$ under the null.

The fixed-size benchmark uses $n_2=50$ and rejects when the planned-design statistic exceeds the one-sided standard normal critical value $c=1.959964$. For adaptive procedures, consider two illustrative rules:

Rule A: set $n_2=100$ if $Z_1<0.5$; otherwise retain $n_2=50$.
Rule B: set $n_2=150$ if $Z_1<1$; otherwise retain $n_2=50$.

These rules are deliberately based on interim data. In a real design, the thresholds, caps, test, and adaptation algorithm would need to be specified before use. The rules here let us compare methods under both a modest and a larger possible increase.

Definition: For $M$ simulated trials generated under $H_0$, the empirical Type I error estimate is the proportion that reject $H_0$: $$ \widehat{\alpha}_{\mathrm{sim}}=\frac{1}{M}\sum_{m=1}^{M} I_m, $$ where $I_m=1$ if simulated trial $m$ rejects and $I_m=0$ otherwise. It estimates the rejection probability of the specified procedure under the simulated null model.

What Each Procedure Does

For the fixed-size benchmark, the stage sizes do not change. Its statistic is

Z_{\mathrm{fixed}}=\frac{Z_1+Z_2}{\sqrt{2}}, \qquad \text{reject }H_0\text{ if } Z_{\mathrm{fixed}}>1.959964.

The inverse-normal procedure uses the planned equal-stage weights, held fixed regardless of the interim-based sample-size choice:

Z_{\mathrm{IN}}=\frac{\Phi^{-1}(1-p_1)+\Phi^{-1}(1-p_2)}{\sqrt{2}} =\frac{Z_1+Z_2}{\sqrt{2}}, \qquad \text{reject if }Z_{\mathrm{IN}}>1.959964.

Here $p_1=1-\Phi(Z_1)$ and $p_2=1-\Phi(Z_2)$ are one-sided stagewise p-values. The weights are not changed after seeing $Z_1$. Under the stated model, each stagewise p-value is null-uniform and stage 2 remains independent of stage 1, even though its sample size is selected using stage 1.

For the conditional-error procedure, preserve the conditional rejection probability of the original fixed-size test. Given $Z_1=z_1$, the original test rejects if $Z_2>\sqrt{2}c-z_1$. After re-estimation, use that same stage-two boundary with the valid standardized stage-two statistic. The boundary therefore depends on the interim result, not on a new, unadjusted fixed-design critical value.

Simulation Plan and Justification

A simulation replicate follows the design in its actual order. First generate stage 1 under the null. Next apply the sample-size rule to the resulting $Z_1$. Then generate stage 2 at the selected sample size and calculate each procedure’s rejection indicator. Repeat this process $M$ times and divide each procedure’s rejection count by $M$.

Generate the interim result.
Draw $Z_1\sim N(0,1)$, as implied by the null model and the stage-one assumptions.

Apply the adaptation rule.
Use the simulated $Z_1$ to select $n_2$. Do not select the sample size using information that would not be available at the interim analysis.

Generate stage 2 and analyze.
Generate independent null stage-two data at the selected $n_2$, calculate its standardized statistic and p-value, and apply each procedure’s specified rejection rule.

Record and summarize.
Store a separate zero-or-one rejection indicator for each procedure, then calculate its rejection proportion across all replicates.

There is a short justification for why the inverse-normal procedure has the target overall error in this setting. Conditional on any stage-one result and the resulting $n_2$, the stage-two statistic is standard normal. Because that conditional distribution does not depend on the stage-one result, $Z_2$ is independent of $Z_1$ under the null. Consequently, $(Z_1+Z_2)/\sqrt{2}\sim N(0,1)$, so its upper-tail probability beyond $1.959964$ is $0.025$. This conclusion depends on the fixed weights and the stated independence and calibration assumptions; it is not a general guarantee for weights chosen after inspecting the interim data.

The conditional-error procedure has a different justification. Define the original conditional error at $z_1$ as

\gamma(z_1)=P(Z_2>\sqrt{2}c-z_1\mid Z_1=z_1) =1-\Phi(\sqrt{2}c-z_1).

The adaptive procedure uses a valid stage-two test with conditional rejection probability $\gamma(z_1)$. Averaging over the null distribution of $Z_1$ gives

P(\text{reject }H_0) =E\{\gamma(Z_1)\} =P(Z_{\mathrm{fixed}}>c) =\alpha.

This argument explains why a conditional rejection probability can be above or below $0.025$ for a particular interim result while the overall Type I error remains $0.025$.

Worked Example: Applying an Interim Rule to One Simulated Trial

Suppose a hypothetical null replicate produces $Z_1=0.8$ and $Z_2=2.0$. Under Rule A, $0.8$ is not less than $0.5$, so $n_2=50$. Under Rule B, $0.8<1$, so $n_2=150$. The stage-two statistic is standardized using whichever sample size was selected.

For the inverse-normal test, the observed statistic is

Z_{\mathrm{IN}}=\frac{0.8+2.0}{\sqrt{2}} =\frac{2.8}{1.4142} =1.980.

Since $1.980>1.959964$, this replicate rejects under the inverse-normal procedure for either rule. The decision is the same for both rules because the prespecified inverse-normal weights are fixed and the standardized stage-two result is the same; the sample-size rule affects the distribution of raw stage-two data, not the rejection formula for this replicate.

The fixed-size benchmark, by contrast, would require a trial with its planned second-stage sample size of 50. It is a reference design, not a separate adaptive decision for the $n_2=150$ branch.

Worked Example: Calculating a Conditional-Error Boundary

For a hypothetical interim value $Z_1=0.8$, the original fixed-size test’s stage-two boundary is

\sqrt{2}(1.959964)-0.8 =2.7718-0.8 =1.9718.

Thus the conditional-error procedure rejects this replicate if its valid stage-two statistic exceeds $1.9718$. For the $Z_2=2.0$ in the previous example, it rejects because $2.0>1.9718$. Under the null, its conditional rejection probability at this interim value is $1-\Phi(1.9718)$, approximately $0.0243$. That conditional probability is close to, but need not equal, the nominal overall rate.

For another hypothetical interim value, $Z_1=1.2$, the boundary is $2.7718-1.2=1.5718$, and the conditional rejection probability is approximately $1-\Phi(1.5718)=0.0580$. The larger conditional probability is not itself evidence of Type I error inflation: the interim result $1.2$ is only one possible value, and the overall error averages the conditional probabilities over all null interim results.

Worked Example: Turning Simulated Decisions Into an Error Estimate

Suppose a hypothetical simulation runs $M=100{,}000$ null replicates under Rule A. The inverse-normal procedure rejects in 2,508 replicates. Its empirical Type I error is

\widehat{\alpha}_{\mathrm{sim}} =\frac{2{,}508}{100{,}000} =0.02508.

The estimate is $2.508\%$, compared with the nominal $2.5\%$. This single simulation result is compatible with the target, but it does not prove exact control. A different random run would generally yield a slightly different rejection proportion. The immediate validation question is whether the simulation faithfully implemented the null model, adaptation rule, stage-two test, and rejection criterion—not whether one run landed exactly on $0.025000$.

Comparing Procedures and Reading Results

The following hypothetical output illustrates how results might be organized. The counts are illustrative, not empirical claims. The fixed-size benchmark has no sample-size adaptation; the other rows apply each adaptation rule to the adaptive procedure.

Procedure and design	Rejections out of 100,000	Empirical Type I error
Fixed-size benchmark, no adaptation	2,490	0.02490
Inverse-normal, Rule A	2,508	0.02508
Conditional error, Rule A	2,486	0.02486
Inverse-normal, Rule B	2,519	0.02519
Conditional error, Rule B	2,497	0.02497

In this simple model, with equal planned information in each stage, fixed inverse-normal weights, and a conditional-error boundary inherited from the same fixed-size test, the inverse-normal and conditional-error rejection conditions coincide. Their simulated rates should therefore be similar. That agreement is a useful implementation check, not a general claim that all inverse-normal and conditional-error constructions are interchangeable.

The methods differ in how they express the final test. The inverse-normal method combines stagewise evidence using prespecified weights. The conditional-error method states how much conditional null rejection probability remains after the interim result and preserves that allowance. The fixed-size procedure is the nonadaptive reference. A simulation should represent each method as specified; silently changing weights or substituting an unadjusted pooled statistic changes the procedure being evaluated.

Diagnostics, Pitfalls, and Limitations

Check the simulation at several levels before interpreting its overall rejection rate. Under the null, the simulated $Z_1$ values should have mean near zero and variance near one. For each adaptation branch, verify that the selected $n_2$ matches the rule’s threshold. Within those branches, check that standardized stage-two statistics behave as $N(0,1)$ under the assumed model. Also check that the one-sided p-values use the correct tail and that the rejection comparison uses the intended critical value.

Wrong reasoning: “Every interim subgroup must reject at rate $0.025$, or the method is invalid.” Correction: Overall Type I error is averaged over the null distribution of interim results. A conditional-error method deliberately allows the conditional rejection probability to vary with $Z_1$.
Wrong reasoning: “If $n_2$ changes, the stage-two statistic is still valid without any recalibration.” Correction: Standardize using the selected sample size and verify the stage-two null distribution conditional on the interim data. The conclusion here relies on known variance and independent new observations.
Wrong reasoning: “A simulated estimate slightly above $0.025$ proves inflation.” Correction: A rejection proportion varies from run to run. Check the implementation and interpret the estimate as a simulation result, not as an exact probability.

This validation is only as trustworthy as its data-generating model. Known variance, independent stages, a continuous normal test statistic, and a one-sided alternative make the example especially transparent. With estimated nuisance parameters, discrete outcomes, dependence, missingness, or model misspecification, stagewise p-values may not have the conditional null distribution assumed here. The simulation must then reflect the actual design and analysis, including those complications. A reassuring result under a simplified model does not establish control under a different one.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Why can $n_2$ depend on $Z_1$ while the stage-two statistic remains standard normal under the null in the stated model?
For $Z_1=0.8$, calculate the conditional-error boundary using $c=1.959964$.
If a simulation has 1,250 rejections among 50,000 null replicates, what is its empirical Type I error?
Does a valid conditional-error procedure require the conditional rejection probability to equal $0.025$ at every interim result? Explain.
Name two assumptions in the simulation model that would need to be revisited for an analysis with estimated variance or discrete outcomes.

Solutions. (1) Stage 2 consists of new independent observations, and its statistic is standardized using the selected $n_2$; conditional on stage 1 and $n_2$, it is $N(0,1)$. (2) $\sqrt{2}(1.959964)-0.8=2.7718-0.8=1.9718$. (3) $1{,}250/50{,}000=0.025$, or $2.5\%$. (4) No. The conditional probability may vary with the interim result; the null average of those probabilities is the overall Type I error. (5) For example, the known-variance normal calibration and continuous uniform stagewise p-values may no longer hold, so the null distribution and stage-two conditional calibration must be specified appropriately.