Evaluating Estimator Bias and Interval Coverage

Why Evaluate Estimators Across Simulated Trials?

A trial analysis produces an estimate and often a confidence interval, but one simulated trial cannot show how the analysis performs over repeated trials. In clinical trial simulation, each replicate represents a possible trial under a specified scenario. Comparing its estimate with the scenario’s known true value makes it possible to evaluate systematic error, overall estimation error, and whether the reported intervals include that truth at the intended rate.

These performance summaries answer different questions. An estimator can have little average bias but substantial variability; a method can have modest mean squared error while its intervals cover too infrequently; and an empirical coverage rate can differ from its nominal target simply because the number of replicates is limited. The summaries therefore need to be computed and interpreted together.

For a given scenario, let \(\theta\) be the fixed true value of the target parameter and let \(\hat{\theta}_r\) be its estimate in replicate \(r\), for \(r=1,\ldots,R\). The scenario must specify the target clearly, including its scale and units. For example, \(\theta\) might be a treatment difference of 2.0 points on a clinical outcome scale. The repeated-trial performance summaries describe the distribution of estimates and intervals under that scenario; they do not describe uncertainty about the fixed value \(\theta\).

Estimating Bias and Mean Squared Error

In a simulation with \(R\) completed replicates, the empirical mean estimate is \(\bar{\hat{\theta}}=R^{-1}\sum_{r=1}^{R}\hat{\theta}_r\). The estimated bias is \(\bar{\hat{\theta}}-\theta\). A positive value means that estimates average above the truth; a negative value means that they average below it. Report the sign, magnitude, and units, rather than only saying that an estimator is “biased.”

The empirical MSE is the average squared error from the truth:

It is in squared units. Its square root, the root mean squared error (RMSE), is in the same units as \(\theta\) and can be easier to interpret as a typical combined estimation error. Bias and MSE should not be conflated: bias describes the average signed error, whereas MSE penalizes errors in either direction and incorporates both systematic displacement and variability.

To see the decomposition, write \(\hat{\theta}-\theta=(\hat{\theta}-E(\hat{\theta}))+ (E(\hat{\theta})-\theta)\). Squaring and taking expectations gives a variance term, a squared bias term, and a cross-product. The expected cross-product is zero because \(E[\hat{\theta}-E(\hat{\theta})]=0\):

For simulated replicates, the empirical MSE also equals the replicate variance calculated with divisor \(R\), plus the squared empirical bias. If instead the usual sample variance \(s_{\hat{\theta}}^2\) with divisor \(R-1\) is used, then the empirical MSE is \(\frac{R-1}{R}s_{\hat{\theta}}^2+(\bar{\hat{\theta}}-\theta)^2\). The small divisor distinction matters when reproducing exact calculations.

Empirical Coverage and Its Monte Carlo Uncertainty

For a nominal 95% confidence-interval procedure, the target coverage is 0.950 under conditions where the procedure is calibrated for that scenario. For each replicate, record whether the interval contains the scenario’s true value. If \(I_r\) is 1 when \(L_r\leq\theta\leq U_r\) and 0 otherwise, empirical coverage is \(\hat{c}=R^{-1}\sum I_r\). Its interpretation is the proportion of simulated intervals that included the truth, not the probability that the fixed parameter lies in any particular interval.

When replicates are independently generated under the same scenario, the coverage indicators are Bernoulli observations with probability \(c\). The Monte Carlo standard error (MCSE) of \(\hat{c}\) is estimated by \(\sqrt{\hat{c}(1-\hat{c})/R}\). It quantifies simulation noise in the estimated coverage rate; it is not the standard error of an individual trial’s treatment estimate.

Comparing Performance and Diagnosing Discrepancies

A single summary rarely identifies why a procedure performs as it does. Compare bias, variability, MSE, coverage, interval width, and analysis completion rates across methods and scenarios. Use the same truth, replicate set where feasible, and clearly stated denominator for each comparison. For intervals, also inspect how often estimates or interval endpoints are extreme, whether intervals are unusually wide or narrow, and whether failures are concentrated in particular types of replicate.

Nominal coverage is a repeated-sampling target for a specified interval procedure and scenario. It is not guaranteed simply because an interval is labelled “95%.” Finite-sample approximations, the data-generating conditions, and the interval construction can all affect actual coverage. Nor does a difference between empirical and nominal coverage automatically establish a systematic defect: the number of replicates controls the precision of the empirical rate.

Analysis failures require particular care. If an estimator or interval is unavailable for some simulated trials, calculating coverage only among successful trials estimates coverage conditional on successful analysis. That may be useful, but it is not automatically the performance of the full analysis strategy. Report the number and proportion of failures, explain the denominator, and assess whether failure depends on trial characteristics. A policy that defines how failed analyses are handled should be evaluated according to that policy; silently dropping failures can make performance look better or worse.

Useful checks include verifying that the same parameter scale and true value are used in estimates, intervals, and calculations; confirming that every replicate belongs to the intended scenario; and comparing direct summaries against replicate-level calculations. Coverage close to nominal with a wide MCSE is inconclusive, not evidence of precise calibration. Conversely, a large apparent gap relative to MCSE deserves investigation, but the gap alone does not identify its cause. Examine interval construction, finite-sample behavior, implementation, and the handling of failed fits before drawing conclusions.

Bias, MSE, and coverage complement rather than replace other operating characteristics. Type I error concerns rejection under a null hypothesis; interval coverage concerns inclusion of a parameter value by an interval procedure. Their calibration questions are related but not interchangeable. Likewise, minimizing MSE does not ensure nominal coverage: a narrow procedure may estimate efficiently on average and still produce intervals that are too short.

Common Pitfalls and Practical Interpretation

• Reporting bias without units or direction. State, for example, “the mean estimate was 0.120 points above the true difference,” rather than simply “bias was 0.120.”
• Confusing bias with MSE. Positive and negative errors can cancel in the mean while still producing large squared errors. Inspect variability and MSE as well as signed bias.
• Calling Monte Carlo uncertainty trial-level uncertainty. The MCSE describes the precision of a simulation summary across replicates, not the uncertainty attached to one participant or one trial estimate.
• Ignoring the denominator. If some analyses fail, state whether coverage is among completed intervals or uses an all-replicate policy, and report failures separately.
• Treating nominal coverage as a guarantee. The nominal label is a target under specified conditions. Empirical performance must be evaluated in the scenario of interest.
• Overinterpreting a small difference. Compare the gap with its Monte Carlo uncertainty and consider whether the replicate count is adequate before attributing the discrepancy to the analysis procedure.