Diagnose Type I Error And Coverage Failures

Why Calibration Needs Its Own Check

A simulation can produce a plausible-looking estimate of a treatment effect while the associated test or interval procedure is poorly calibrated. Calibration asks whether a method’s inferential statements behave as intended across repeated trials. In particular, under a scenario where the null hypothesis is true, does the test reject at approximately its stated significance level? When the target parameter is known in the simulation, do confidence intervals contain it at approximately their stated confidence level?

These are properties of the entire simulated procedure: the data-generating mechanism, adaptive allocation, stopping and analysis rules, and the test or interval construction. A discrepancy from the target can reflect a genuine problem, but simulated rates themselves fluctuate because only a finite number of replicates are run. The diagnostic task is to assess both the size of the discrepancy and the uncertainty in the estimated rate.

Define the Rates Before Calculating Them

Suppose a simulation uses \(B\) independent replicates under a prespecified scenario. In each replicate, the adaptive trial is generated and analyzed according to the same specified rules. For a test with significance level \(\alpha\), let \(R_b=1\) if replicate \(b\) rejects \(H_0\), and \(R_b=0\) otherwise. In this diagnostic, the scenario must satisfy \(H_0\); otherwise, its rejection rate is not a Type I error rate.

For a confidence interval procedure with nominal confidence level \(1-\alpha\), let \(C_b=1\) if the interval from replicate \(b\) contains the true target parameter \(\theta\), and \(C_b=0\) otherwise. The target \(\theta\) must be known from the data-generating scenario, and the interval must be evaluated for that same estimand.

These definitions treat each complete simulation replicate as one Bernoulli outcome: rejection or not, coverage or not. If replicates are generated independently under an unchanged scenario, then \(R\) has a binomial distribution with size \(B\) and probability \(q_R\), and \(C\) has a binomial distribution with size \(B\) and probability \(q_C\). This is the basis for quantifying Monte Carlo uncertainty in the rates.

Monte Carlo Uncertainty Around a Nominal Rate

For a binomial count \(K\) from \(B\) independent replicates, the estimated rate is \(\hat q=K/B\). Since the variance of a Bernoulli outcome with probability \(q\) is \(q(1-q)\), the variance of the average of \(B\) independent outcomes is \(q(1-q)/B\). Thus the standard error of \(\hat q\), if \(q\) were known, is \(\sqrt{q(1-q)/B}\). For a calibration check, the nominal target can be substituted to assess how much Monte Carlo variation would be expected if the method were calibrated.

A useful standardized discrepancy is the difference between the estimated and nominal rate divided by this nominal standard error. It is a diagnostic, not a verdict: a large value warrants investigation, while a modest value is compatible with ordinary Monte Carlo fluctuation. A confidence interval for the unknown rate provides a complementary view. For example, a Wilson interval or an exact binomial interval is generally preferable to a simple Wald interval, especially when the rate is near zero or one or the number of replicates is small.

A rough 95% Monte Carlo range around a nominal rate is \(q_0\pm1.96\sqrt{q_0(1-q_0)/B}\), when the normal approximation is reasonable. It describes the variability of the simulated rate under the assumption that the procedure truly has rate \(q_0\). It is not a confidence interval for the treatment effect, nor does it account for uncertainty about whether the simulated model represents the intended trial.

Checking Confidence-Interval Coverage

Coverage is assessed similarly, but the indicator is whether the interval contains the known \(\theta\), not whether an interval includes a hypothesized null value in some informal sense. The nominal target is \(1-\alpha\). A rate below that target indicates undercoverage in the scenario: intervals contain \(\theta\) too infrequently in the repeated simulation. A rate above it indicates overcoverage, which may reflect conservative intervals or other features of the procedure. Neither direction alone identifies the cause.

Separate Random Fluctuation From a Persistent Pattern

A single estimated rate is rarely enough to diagnose a failure. Consider the scale of Monte Carlo uncertainty, then examine whether departures recur across independent runs or relevant null scenarios. If the simulation budget is increased while the scenario and method remain fixed, Monte Carlo standard errors decrease at approximately the rate \(1/\sqrt{B}\). A discrepancy that remains similar in size as \(B\) grows becomes harder to attribute to random simulation noise alone.

For a composite null hypothesis, checking one null parameter value is not a guarantee of Type I error control over all null values. Likewise, coverage may vary with the true effect, event rate, allocation behavior, or other features of the data-generating mechanism. Report the scenario-specific rates and their Monte Carlo uncertainty rather than pooling scenarios with different true rates into one number that can conceal a failure.

The following sequence makes the diagnostic interpretable:

For example, if a nominal 5% test rejects in 27 of 500 null replicates, its estimated rate is \(27/500=0.0540\). The nominal standard error is \(\sqrt{0.05(0.95)/500}\approx0.00975\), so the 0.0040 excess is only about 0.41 nominal standard errors. That result is compatible with Monte Carlo fluctuation; it does not establish exact calibration, but it supplies little evidence of a discrepancy at this simulation size.

Failure Modes and Appropriate Comparisons

When a rate looks unusual, first verify that the diagnostic itself is correctly constructed. Common failure modes include:

• Wrong scenario. A rejection rate is called a Type I error rate even though the simulated parameter values do not satisfy \(H_0\). Recheck the data-generating values and the null statement.
• Wrong target. Coverage is evaluated against a parameter different from the interval’s estimand, such as comparing an interval for an average effect with a subgroup effect.
• Wrong denominator. Failed or incomplete runs are silently omitted, or different numbers of replicates are used without being reported. Define in advance how such runs are handled and report the number analyzed.
• Wrong indicator. A rejection rule, tail convention, or interval endpoint check is coded incorrectly. Confirm that rejection and inclusion follow the prespecified analysis exactly.
• Dependent replicates treated as independent. Reusing data or random outcomes across nominally separate replicates can invalidate the simple binomial Monte Carlo standard error. The independence assumption must match the simulation design.
• Overreading a point estimate. A rate that differs from nominal is not automatically evidence of systematic miscalibration; its Monte Carlo uncertainty matters.

A point estimate alone is the simplest comparison, but it can make small random departures appear important. A Monte Carlo interval shows plausible uncertainty in the rate, while a standardized discrepancy summarizes distance from nominal in standard-error units. These approaches answer closely related diagnostic questions; neither explains why a departure occurred. A binomial interval is useful when normal approximations are questionable, especially with few replicates or rates near the boundaries.

Also distinguish a calibration diagnostic from a formal claim about all operating conditions. A test may appear calibrated at one null configuration and fail at another. A confidence interval may have reasonable coverage for one parameter value and not another. Repeating the check over prespecified scenarios gives a more informative picture, but each scenario’s result still has Monte Carlo uncertainty. The primary purpose here is to detect and investigate calibration concerns, not to summarize other performance goals such as power or estimation accuracy.

Why the Distinction Matters

A simulation study can be internally consistent and still give misleading reassurance if it reports only an empirical rate without uncertainty. Conversely, treating every small departure from nominal as a failure can lead to unnecessary conclusions driven by finite simulation size. Reporting the target, replicate count, estimated rate, and Monte Carlo uncertainty makes clear what the simulation supports.