Why Type I Error Comes Before Power
A design can appear to have greater power simply because its test rejects too often when there is no treatment effect. For that reason, a power comparison is informative only after checking how often each procedure declares a difference under the null. This is a fairness check on the statistical comparison: it asks whether the procedures are being judged at comparable false-positive rates, not whether the treatment assignments are fair to individual participants.
The nominal significance level \(\alpha\) is a target chosen for a test, often \(0.05\). The actual Type I error rate depends on the full combination of design, data-generating process, test statistic, and rejection rule. In particular, response-adaptive randomization makes later assignment probabilities depend on earlier outcomes. A test that would have its intended error rate under fixed allocation need not retain it automatically when used with adaptive data.
For a treatment comparison, \(H_0\) commonly states that the treatments have the same outcome distribution, or, for a binary endpoint, the same response probability. Under a sharp no-treatment-effect null, each participant’s outcome would be the same whichever treatment they received. A Type I error occurs if the analysis rejects that null. The relevant probability is across repetitions of the entire trial, including its random assignments and outcomes; it is not the fraction of participants assigned to either arm.
What a Null Audit Measures
A null audit repeats the planned trial many times under a specified no-effect model and applies the proposed analysis to every simulated trial. If \(R\) of \(M\) trials reject at nominal level \(\alpha\), the empirical false-positive rate is \(\hat r=R/M\). This estimates the actual rejection probability for that null model and design. It is a diagnostic for the particular model examined, not a universal guarantee across all possible outcomes or assumptions.
A response-adaptive procedure must be simulated as it would actually be run: initialize the trial, update the allocation rule using the available outcomes, assign each next participant according to the resulting probability, and continue to the planned endpoint. Simulating only final arm sizes, without reproducing how those sizes arose, can miss the dependence between observed outcomes and later assignments.
The key comparison is between the empirical rejection rates under the same null model. A rate above \(\alpha\) indicates that the test rejects too often for that setting; a rate below \(\alpha\) indicates conservatism. Neither conclusion should be based on tiny discrepancies without considering Monte Carlo uncertainty. Increasing \(M\) reduces that uncertainty, but it cannot repair a misspecified simulation model.
Worked Example: A Nominal Test in One Fixed-Allocation Trial
Suppose a hypothetical fixed-randomization trial assigns 200 participants to each of two treatments. The binary outcome is success, with 54 successes in arm A and 60 in arm B. For illustration, use a two-sided pooled two-proportion \(z\)-test of \(H_0:p_A=p_B\) at nominal \(\alpha=0.05\). With independent participants and fixed allocation, the pooled standard error under \(H_0\) is appropriate for this test.
The estimated difference in success proportions, taking A minus B, is \(54/200-60/200=0.270-0.300=-0.030\). Under the null, the pooled success proportion is \((54+60)/(200+200)=114/400=0.285\). Thus,
The test statistic is \(z=-0.030/0.04514\approx-0.6646\), giving a two-sided normal-approximation \(p\)-value of about \(0.506\). This trial does not reject \(H_0\) at \(0.05\). That result does not estimate the test’s Type I error rate: one trial supplies one rejection decision, while Type I error is a probability across repeated trials under the null.
A Two-Arm Adaptive-Design Audit
Consider the following fully hypothetical simulation setup. Each trial enrolls 400 participants, with a binary outcome generated independently as Bernoulli with success probability \(0.30\) for every participant, regardless of treatment. Thus the two treatments have no effect under this model. In the fixed design, assignment is equal throughout. In the adaptive design, the first 40 participants are assigned equally; thereafter, the probability of assigning the next participant to A is \(0.5+(\hat p_A-\hat p_B)\), limited to the range from \(0.20\) to \(0.80\), where the proportions use outcomes observed so far. The remaining probability is assigned to B. Both designs are analyzed using the same planned two-sided pooled two-proportion \(z\)-test at nominal \(0.05\).
Because later assignments in the adaptive design respond to earlier outcomes, it is important not to assume in advance that the usual test has its nominal size there. Instead, reproduce the assignment rules and analysis in each simulated trial. The following made-up results illustrate how to report such an audit; they are not claims about a real study or a universal property of adaptive randomization.
Worked Example: Estimating the False-Positive Rates
Suppose 20,000 independent null trials are simulated for each design. The fixed design rejects in 1,012 trials; the adaptive design rejects in 1,230. Each rejection is a false positive because the simulated success probability is \(0.30\) in both arms.
For fixed randomization, the estimated rate is \(1{,}012/20{,}000=0.0506\). Its Monte Carlo standard error is \(\sqrt{0.0506(1-0.0506)/20{,}000}\approx0.001550\). For the adaptive design, the estimated rate is \(1{,}230/20{,}000=0.06150\), with Monte Carlo standard error \(\sqrt{0.06150(1-0.06150)/20{,}000}\approx0.001699\).
The adaptive estimate exceeds the fixed estimate by \(0.06150-0.05060=0.01090\), or 1.09 percentage points. Since the two sets of simulated trials are independent, the Monte Carlo standard error of this difference is \(\sqrt{0.001550^2+0.001699^2}\approx0.002300\). The estimated difference is about \(0.01090/0.002300=4.74\) Monte Carlo standard errors from zero. In this hypothetical audit, the gap is larger than ordinary simulation noise would readily explain. The adaptive test’s estimated false-positive rate is also above the nominal 5% target.
These estimates describe the specified null model and simulation procedure. They do not prove that every adaptive design inflates Type I error; another allocation rule, endpoint, test, or sample size could behave differently. They do show why comparing power without checking the null rejection rate can be misleading.
Why Unequal Error Rates Distort a Power Comparison
Power is the probability of rejecting \(H_0\) when a specified alternative is true. A rejection rule with a more liberal threshold tends to reject more often both under the null and under alternatives. Therefore, if an adaptive procedure has higher power but also a higher Type I error rate than a fixed design, that power difference may reflect the more permissive rejection behavior rather than a genuine advantage in detecting treatment effects at a common error rate.
A fair comparison first checks whether each procedure controls Type I error at the chosen level for relevant null scenarios. If one does not, possible responses include using an analysis designed for the adaptive assignment mechanism or recalibrating a rejection threshold using null simulations, then validating the resulting procedure independently. Comparisons should report both the false-positive rates and power; a correction that changes a rejection rule can change power as well.
A simple justification for this order follows from the definitions. Let \(D\) denote a design and \(C_D\) its rejection rule. Under the null, its size is \(P_0(C_D\text{ rejects})\); under a specified alternative, its power is \(P_1(C_D\text{ rejects})\). If \(C_D\) is made more liberal, its rejection region expands, so the probability of rejection can increase under both \(P_0\) and \(P_1\). Thus \(P_1(C_D\text{ rejects})\) alone does not separate a design’s ability to detect effects from its willingness to reject. Checking \(P_0(C_D\text{ rejects})\) is necessary to interpret a power contrast.
Worked Example: Calibrating and Validating a Cutoff
Suppose a null calibration simulation for the adaptive design suggests using \(|Z|>2.17\), rather than the usual large-sample two-sided cutoff of about \(1.96\), for the same pooled test statistic \(Z\). The threshold is selected using calibration trials; it must then be checked using separate null validation trials to avoid judging the cutoff on the same simulation noise that selected it.
In 10,000 independent validation trials under the same specified no-effect model, suppose 492 trials exceed 2.17 in absolute value. The validated empirical rate is \(492/10{,}000=0.04920\). Its Monte Carlo standard error is \(\sqrt{0.04920(1-0.04920)/10{,}000}\approx0.002163\). This provides evidence that the recalibrated rule is near the 5% target for this model, subject to simulation uncertainty. It does not establish control under other null distributions, allocation rules, or sample sizes.
The numerical cutoff is illustrative, not a recommended universal value. A valid procedure might instead use a test that incorporates the known adaptive assignment probabilities, or a randomization-based method appropriate to the null being tested. The method must match the design and the scientific null.
Alternatives, Diagnostics, and Limitations
The main approaches answer related but distinct questions. An ordinary model-based test uses a reference distribution justified by its assumptions; null simulation checks its actual behavior under a specified design and model; and a randomization-based test uses the assignment mechanism to construct a null reference distribution when its conditions are met. Simulation is an audit tool, not automatically a substitute for a valid inferential procedure.
| Approach | What it uses | Important caution |
|---|---|---|
| Ordinary model-based test | A test statistic and an assumed reference distribution, such as a normal approximation | Adaptive assignment may affect the distribution used by the test; nominal \(\alpha\) alone does not verify size. |
| Null simulation audit | Repeated trials generated under a specified null and the exact proposed design and analysis | Conclusions are limited to the simulated assumptions and are subject to Monte Carlo error. |
| Randomization-based inference | The assignment probabilities and a null reference distribution generated by the randomization mechanism | The null and recalculation of adaptive assignments must be specified correctly; this is not automatically a test of every weaker null. |
Useful diagnostics include checking that simulated assignment probabilities obey their bounds, final sample sizes follow the intended rule, outcomes are generated independently of assignment under the no-effect model, and every replicate is analyzed with the planned stopping and testing procedure. Examine more than one plausible null outcome rate when appropriate, especially for binary outcomes, because a single null scenario may not reveal sensitivity to the baseline response probability. Report the number of replicates, rejection count, rate, and Monte Carlo standard error.
A common mistake is to say, “The test uses \(\alpha=0.05\), so its Type I error is 5% and its power can be compared directly.” The correction is that \(\alpha=0.05\) is the nominal target; actual size depends on whether the test’s assumptions hold under the design. Audit rejection under no effect first. Another mistake is to interpret a slightly different simulated rate as definitive evidence of a design problem without considering Monte Carlo error. The rate is an estimate, and its precision depends on the number of simulated trials.
No finite collection of simulations establishes error control for every possible data-generating process. Results depend on the outcome model, allocation algorithm, sample size, analysis, and stopping rule used in the audit. A careful report therefore states these choices and limits the conclusion accordingly. The purpose of the check is to make a power comparison interpretable, not to declare one randomization design universally superior.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- In 5,000 null simulations, a test rejects 275 times. What is the empirical Type I error rate?
- Why does one trial with a large \(p\)-value not demonstrate that a test controls Type I error?
- A design has estimated null rejection rate \(0.0615\) at nominal \(\alpha=0.05\). What does the difference say, and what does it not establish?
- Why can higher power be misleading if the corresponding Type I error rate is also higher?
Solutions. (1) The empirical rate is \(275/5{,}000=0.0550\), or 5.50%. (2) Type I error is a repeated-trial probability under \(H_0\); one trial records only whether that particular run rejected. (3) The estimated rate is 1.15 percentage points above the nominal target for the audited model. Whether this is more than simulation noise requires the Monte Carlo standard error, and the result does not establish behavior for other models or designs. (4) A more liberal rule may reject more often under both null and alternative, so the extra power may be partly or wholly attributable to excess false positives rather than better detection at a common error rate.