Why Calibration Diagnostics Need More Than One Rejection Rate
A simulated null rejection rate is a useful check on an adaptive test, but a single pooled number can conceal how the test behaves. Rejections may be concentrated among particular interim-statistic values, sample-size choices, nuisance-parameter settings, or adaptation paths. Conversely, a conditional rate that differs from the nominal level is not automatically a defect: adaptive procedures can have different conditional rejection probabilities in different parts of the interim-data space while retaining the intended overall Type I error rate.
The diagnostic goal is therefore twofold: estimate the marginal rejection probability under the null, and examine how that probability is distributed across prespecified conditions. A valid simulation must reproduce the null model, the interim analysis, the adaptation rule, and the final rejection rule. It must also represent the nuisance parameters and other design features relevant to the claim being checked.
For a composite null hypothesis, a design may need to control Type I error for every null configuration, not just one convenient setting. A corresponding calibration target is \(P_{\theta}(R=1)\leq\alpha\) for all relevant \(\theta\) under \(H_0\). A grid of nuisance values can reveal sensitivity, but checking finitely many points does not by itself prove control over a continuous parameter space.
What Stratified Rates Can—and Cannot—Show
Suppose the diagnostic groups \(B=1,\ldots,J\) form a partition: every simulated trial belongs to exactly one group. The law of total probability gives
Thus, the overall rate is a weighted average of the conditional rates. In general, the conditional rates are not each required to equal \(\alpha\). Their appropriate benchmarks may differ across groups, particularly when the design has a valid but nonconstant conditional rejection probability. The relevant diagnostic question is whether the observed conditional behavior agrees with the procedure’s conditional behavior under the null, and whether the weighted marginal rate meets the overall target.
For example, a very high interim statistic may leave little conditional error available for the second stage under a particular testing rule, while a low interim statistic may leave more. The resulting conditional rejection rates can differ even when their weighted average is calibrated. Demanding a rate of exactly \(\alpha\) in every interim range would confuse unconditional Type I error control with conditional error control.
Monte Carlo Uncertainty Is Part of the Diagnosis
At a fixed null configuration, simulate \(M\) independent trials using the complete adaptive algorithm. If \(K\) trials reject, the estimated rejection rate is \(\hat r=K/M\). Under independent, identically generated replicates, \(K\) has a binomial distribution with parameters \(M\) and \(r=P_{\theta}(R=1)\). The estimated Monte Carlo standard error is
This standard error describes simulation uncertainty, not uncertainty about whether the scientific null hypothesis is true. It is useful to compare \(\hat r\) with \(\alpha\), report a Monte Carlo interval, and check whether a discrepancy is large relative to simulation noise. When rejection counts are small, a normal approximation can be unreliable; use a binomial interval suited to sparse counts. If many diagnostic cells are inspected, apparent outliers are expected by chance, so interpret them as a pattern and consider simultaneous uncertainty rather than treating every cell as an independent confirmatory test.
Worked Example: Pooled Null Rejection Rate
Suppose a hypothetical validation run uses \(M=100{,}000\) independent trials at a specified null configuration. The test rejects in \(K=2{,}600\) trials, with nominal level \(\alpha=0.025\). The estimated rate is
The estimated Monte Carlo standard error is
Using this estimated-rate standard error, the excess over the nominal level is \(0.0010/0.0005032\approx 1.99\) standard errors. A useful alternative diagnostic uses the standard error under the nominal benchmark:
The observed rate is about 2.025 nominal-benchmark standard errors above \(0.025\). A rough 95% interval using the estimated-rate standard error is \(0.0260\pm1.96(0.0005032)\), or approximately \((0.02501,0.02699)\). This is a signal to investigate, not a standalone proof of a defect: the interval accounts for Monte Carlo error under the specified simulation, but it cannot detect a misspecified null generator or an incorrectly implemented adaptation rule.
Interim-Statistic Ranges: Compare Like With Like
A useful stratification divides the interim statistic into ranges chosen before examining the simulation results. The ranges should be broad enough to provide adequate numbers of trials, and they should cover the relevant interim-statistic distribution. For each range, record the number of trials, the number of rejections, the estimated conditional rate, and its uncertainty. If the test’s conditional rejection probabilities can be calculated or independently simulated under the null, compare each observed rate with that conditional benchmark rather than with \(\alpha\) automatically.
The following hypothetical example illustrates a calibrated pattern with deliberately varying conditional rates. Assume that an independent null calculation for this particular procedure gives the listed conditional benchmarks. The rates are examples, not universal targets for interim-statistic bins.
Worked Example: Rejection Rates Across Interim Ranges
In \(60{,}000\) hypothetical null simulations, trials are divided into four prespecified ranges of interim statistic \(Z_1\). The observed counts and independently obtained conditional benchmarks are:
| Interim-statistic range | Trials | Rejections | Observed rate | Conditional benchmark |
|---|---|---|---|---|
| \(Z_1<-1\) | 8,000 | 80 | 0.0100 | 0.010 |
| \(-1\leq Z_1<0\) | 22,000 | 440 | 0.0200 | 0.020 |
| \(0\leq Z_1<1\) | 22,000 | 660 | 0.0300 | 0.030 |
| \(Z_1\geq1\) | 8,000 | 320 | 0.0400 | 0.040 |
The pooled rejection rate is
The conditional rates are not all \(0.025\), but they agree with this example’s conditional benchmarks. The weighted benchmark is also \(0.025\):
For scale, the estimated standard error in the first range is \(\sqrt{0.010(0.990)/8{,}000}\approx0.00111\); in the last range it is \(\sqrt{0.040(0.960)/8{,}000}\approx0.00219\). These are much larger than the pooled standard error because each range contains fewer trials. The table supports checking whether the observed pattern follows the conditional benchmark; it would be incorrect to declare the last range miscalibrated simply because its rate exceeds the nominal level.
Sample-Size Choices, Nuisance Values, and Adaptation Paths
Stratification should also reflect how the procedure adapts. Useful summaries include the selected stage-two sample size or information level, the nuisance-parameter value used to generate data, and the realized adaptation path. A path may be defined by the sequence of prespecified decisions the algorithm makes—for example, which sample-size category is selected after the interim analysis. Record both the marginal rejection rate for each null configuration and the conditional rates in adequately populated path or sample-size groups.
A path-specific rate is not automatically expected to equal \(\alpha\), for the same reason that an interim-range rate is not. Its interpretation depends on the procedure’s conditional behavior. However, a sharp and unexplained change across paths, or an overall excess at a nuisance value, can expose an implementation error, a misspecified model, or sensitivity that the pooled result hides. A path can also be rare: its conditional estimate may be highly imprecise even when the total simulation count is large.
Worked Example: A Nuisance-Parameter and Sample-Size Diagnostic
Consider a hypothetical adaptive design with three stage-two sample-size categories: low, medium, and high. At each of two null nuisance settings, \(20{,}000\) independent trials are simulated, with \(5{,}000\), \(10{,}000\), and \(5{,}000\) trials, respectively, in those categories. The results are:
| Null nuisance setting | Stage-two category | Trials | Rejections | Conditional rate |
|---|---|---|---|---|
| \(\theta=1\) | Low | 5,000 | 75 | 0.015 |
| \(\theta=1\) | Medium | 10,000 | 250 | 0.025 |
| \(\theta=1\) | High | 5,000 | 175 | 0.035 |
| \(\theta=1.5\) | Low | 5,000 | 100 | 0.020 |
| \(\theta=1.5\) | Medium | 10,000 | 300 | 0.030 |
| \(\theta=1.5\) | High | 5,000 | 200 | 0.040 |
At \(\theta=1\), the marginal rate is \((75+250+175)/20{,}000=500/20{,}000=0.0250\). At \(\theta=1.5\), it is \((100+300+200)/20{,}000=600/20{,}000=0.0300\). The latter is \(0.0050\) above the nominal level. Its estimated standard error is
This is a substantial diagnostic signal relative to Monte Carlo noise at that setting. The conditional rates also rise across the sample-size categories at each nuisance value, but those differences alone do not establish miscalibration: they need to be compared with the procedure’s expected conditional behavior. The marginal excess at \(\theta=1.5\), by contrast, warrants investigation if that value is within the null parameter space. One should check the data generator, nuisance estimation, adaptation rule, final test calculation, and any unmodeled features that could differ across settings. Averaging the two nuisance-specific rates to \(0.0275\) would obscure the worse result and would not establish control at either setting.
A Practical Diagnostic Sequence
A disciplined review makes it easier to locate a problem without mistaking random fluctuation for a failure. The simulation should use independent replicates, and all grouping rules should be defined before inspecting rejection results where practicable.
State the null configuration, nuisance values, nominal \(\alpha\), and the complete adaptive design being simulated. Confirm that every replicate follows the intended data-generating model and stopping or sample-size rules.
Report \(K\), \(M\), \(\hat r=K/M\), and Monte Carlo uncertainty for each relevant null configuration. Do not pool distinct nuisance settings before checking them separately.
Tabulate interim-statistic ranges, selected sample sizes, and adaptation paths. Include cell counts and conditional uncertainty; avoid strong conclusions from sparse cells.
For the overall rate, compare with the marginal target \(\alpha\). For a conditional rate, use the procedure’s conditional benchmark when available; do not substitute \(\alpha\) merely for convenience.
Reproduce suspicious cells, check the adaptation and rejection code, and examine whether discrepancies track a particular interim range, sample-size choice, nuisance value, or path.
Alternatives, Limitations, and Common Mistakes
| Approach | What it can tell you | Main limitation |
|---|---|---|
| Pooled null rejection rate | Whether the simulated marginal rate is near the nominal target at a specified configuration | Can conceal localized problems or nuisance sensitivity |
| Stratified rates | Where rejections occur across interim ranges, sample sizes, or paths | Conditional rates need not equal \(\alpha\); small cells are noisy |
| Analytic calibration argument | Can establish control under stated mathematical assumptions | Does not detect coding errors in a separate implementation |
Simulation is a diagnostic, not a general proof of Type I error control. It is conditional on the accuracy of the simulated null model and the implementation. A finite grid cannot guarantee behavior between nuisance values, and a finite number of replicates cannot rule out a small excess. Where available, a mathematical calibration argument and carefully designed simulation checks answer complementary questions.
- Mistake: require every interim bin to have rejection rate \(\alpha\). This imposes a conditional requirement that is not implied by marginal Type I error control. Compare each bin with the appropriate conditional benchmark.
- Mistake: trust a pooled rate because it is near \(\alpha\). Different strata can offset one another. Inspect relevant nuisance settings and adaptation paths as well as the pooled result.
- Mistake: treat a small difference as meaningful without its Monte Carlo error. Report counts and uncertainty, and avoid interpreting isolated fluctuations across many cells as confirmed defects.
- Mistake: interpret a simulation result as proof. A simulated rate assesses the code and model that were actually used; it does not establish that either matches the intended design.
Check Your Understanding
Answer each question, then check the short solutions below.
- In \(40{,}000\) null simulations, a procedure rejects \(1{,}040\) times. Find the estimated rate and its approximate excess, in standard errors, above \(\alpha=0.025\), using the nominal-benchmark standard error.
- Two interim groups have probabilities \(0.40\) and \(0.60\), with conditional rejection probabilities \(0.010\) and \(0.035\). What is the overall rejection probability? Does either group need to have conditional rate \(0.025\)?
- Why is it not enough to report one pooled rate across two nuisance-parameter values if one value is inside the null parameter space and may be more problematic?
- A high stage-two sample-size path has a conditional rejection rate above \(\alpha\). What additional information is needed before calling that path miscalibrated?
Solutions. (1) The estimated rate is \(1{,}040/40{,}000=0.0260\). The nominal standard error is \(\sqrt{0.025(0.975)/40{,}000}\approx0.0007806\); the excess is \(0.0010/0.0007806\approx1.281\) standard errors. (2) The overall rate is \(0.40(0.010)+0.60(0.035)=0.004+0.021=0.025\). Neither conditional rate must equal \(0.025\); their weighted average is the marginal rate. (3) A pooled value can hide an excess at one configuration, and control over a composite null concerns the relevant nuisance values individually or through a justified worst-case assessment. (4) Check the path’s sample size, number of trials, Monte Carlo uncertainty, and the procedure’s conditional benchmark. A rate above \(\alpha\) alone is not evidence of failure because conditional rates need not equal the nominal marginal level.