Estimator Bias and Interval Coverage in Simulated Trials

Why Evaluate Estimation Across Simulated Trials?

A simulated adaptive trial produces more than a design decision. It can also produce an estimate of a treatment effect, response probability, or other parameter, together with a confidence interval. A procedure that often selects an appropriate action may still estimate the target poorly, and an interval that looks precise in a typical replicate may fail to include the truth too often. Evaluating estimation performance therefore requires checking the estimates and intervals against the truth used to generate each scenario.

The key unit of evaluation is the replicate: one complete simulated trial run under a specified scenario and protocol. The scenario supplies a known parameter value, while the algorithm and estimator are applied as they would be in an actual trial. Repeating this process reveals how the estimation procedure behaves across possible trial outcomes, including the variability caused by adaptive decisions.

Definition: For a fixed scenario, let \(\theta\) be the parameter the estimator targets, and let \(\hat{\theta}_r\) be its estimate from simulated replicate \(r\), for \(r=1,\ldots,R\). The estimator's bias is \(E(\hat{\theta})-\theta\); its mean squared error is \(E[(\hat{\theta}-\theta)^2]\). For a confidence interval \([L_r,U_r]\), coverage in replicate \(r\) is the indicator \(I(L_r\leq\theta\leq U_r)\). In simulation, expectations and coverage probabilities are approximated by averages across replicates generated under that same scenario.

Bias and Mean Squared Error

Bias describes systematic displacement from the target: positive bias means estimates tend to exceed \(\theta\), and negative bias means they tend to fall below it. The empirical bias across \(R\) replicates is

\widehat{\operatorname{Bias}}=\frac{1}{R}\sum_{r=1}^{R}\hat{\theta}_r-\theta.

The empirical mean squared error (MSE) measures squared estimation error, combining systematic displacement and replicate-to-replicate variability:

\widehat{\operatorname{MSE}}=\frac{1}{R}\sum_{r=1}^{R}(\hat{\theta}_r-\theta)^2.

MSE is in squared units of \(\theta\). Its square root, the root mean squared error (RMSE), returns to the original units and is often easier to interpret. Bias alone does not tell the whole story: estimates can be centered correctly yet highly variable, or consistently close to one another but displaced from the truth.

Why MSE Includes Both Bias and Variability

The decomposition of MSE explains why both components matter. Write \(\hat{\theta}-\theta=(\hat{\theta}-E(\hat{\theta}))+(E(\hat{\theta})-\theta)\), and square both sides. Taking expectations, the cross-product has expectation zero because \(E[\hat{\theta}-E(\hat{\theta})]=0\). Thus:

\begin{aligned} E[(\hat{\theta}-\theta)^2] &=E[(\hat{\theta}-E(\hat{\theta}))^2] +(E(\hat{\theta})-\theta)^2\\ &\quad+2(E(\hat{\theta})-\theta)E[\hat{\theta}-E(\hat{\theta})]\\ &=\operatorname{Var}(\hat{\theta})+\operatorname{Bias}(\hat{\theta})^2. \end{aligned}

This identity holds when the estimator has a finite second moment. It shows why a small bias does not guarantee small MSE: variance may still be large. Conversely, a modest bias can sometimes accompany low MSE if variability is reduced sufficiently. The empirical MSE and empirical variance-plus-squared-bias need not match exactly in a finite simulation, because the empirical variance convention and the use of finite averages affect the calculation.

Worked Example: Bias and MSE for a Continuous Treatment Effect

Suppose a hypothetical scenario has a true treatment effect of \(\theta=10\) days, where positive values represent days of improvement. Five simulated trial replicates produce estimates of 8, 11, 9, 12, and 10 days. Treating these as the complete set of replicates for this calculation, find empirical bias and MSE.

The mean estimate is \((8+11+9+12+10)/5=50/5=10\) days. Therefore:

\widehat{\operatorname{Bias}}=10-10=0\text{ days}.

The replicate errors are \(-2,1,-1,2,0\) days. Squaring and averaging gives:

\widehat{\operatorname{MSE}} =\frac{(-2)^2+1^2+(-1)^2+2^2+0^2}{5} =\frac{10}{5}=2\text{ days}^2.

The empirical RMSE is \(\sqrt{2}=1.414\) days. The estimates have no average displacement in these five replicates, but they are not identical to the truth; the MSE captures that replicate-level error. Zero empirical bias here does not establish that the procedure has exactly zero bias in the simulation distribution.

Worked Example: Error for a Scenario-Specific Response Probability

In a second hypothetical scenario, the true response probability is \(p=0.30\). Four replicate estimates are \(\hat{p}=0.25, 0.35, 0.30,\) and \(0.40\). Calculate empirical bias, MSE, and RMSE.

The mean estimate is \((0.25+0.35+0.30+0.40)/4=1.30/4=0.325\), so empirical bias is \(0.325-0.30=0.025\), or 2.5 percentage points. The errors are \(-0.05,0.05,0,0.10\), giving:

\widehat{\operatorname{MSE}} =\frac{(-0.05)^2+(0.05)^2+0^2+(0.10)^2}{4} =\frac{0.015}{4}=0.00375.

The RMSE is \(\sqrt{0.00375}=0.06124\), or about 6.12 percentage points. The parameter and estimates are probabilities, so their differences are probability-scale differences; reporting them in percentage points can make the magnitude clearer. This performance statement applies to the stated scenario, not automatically to other true response probabilities.

Confidence-Interval Coverage

A confidence interval procedure is intended to include the target parameter at a stated long-run rate under its assumptions. To evaluate a nominal 95% interval in simulation, check in each replicate whether the interval contains the scenario-specific truth. Equality at either endpoint counts as inclusion. The empirical coverage is the fraction of intervals that include \(\theta\):

\widehat{C}=\frac{1}{R}\sum_{r=1}^{R}I(L_r\leq\theta\leq U_r).

For example, a coverage estimate of 0.90 means that 90% of the simulated intervals included the fixed scenario truth in the evaluated replicates. It does not mean there is a 90% probability that the fixed parameter lies in any particular observed interval. In practice, simulation averages are subject to Monte Carlo variation, so a small difference from the nominal level is not, by itself, proof of a defect.

Worked Example: Counting Covered Intervals

A hypothetical scenario has true value \(\theta=20\). Six simulated replicates yield intervals \([17,23]\), \([19,24]\), \([15,19]\), \([20,25]\), \([16,20]\), and \([18,22]\). Find empirical coverage and compare it with a nominal 95% target.

The first, second, fourth, fifth, and sixth intervals contain 20. The third interval ends at 19, so it does not contain the truth. Thus five of six intervals cover:

\widehat{C}=\frac{5}{6}=0.8333.

The observed coverage is 83.3%, below the nominal 95% target. With only six replicates, this is a highly imprecise performance summary; it signals what happened in this small set, not a precise estimate of long-run coverage. The boundary values 20 in the fourth and fifth intervals count as covered because the inclusion rule is \(L_r\leq\theta\leq U_r\).

Assumptions, Diagnostics, and Comparisons

These summaries are meaningful only when the target and scenario are defined carefully. The true value \(\theta\) must be the parameter that the reported estimator and interval are intended to address. Each replicate must be generated under the scenario being summarized, and the same estimation and interval-construction rules must be applied consistently. If the target changes by scenario, calculate performance against that scenario's own truth; pooling estimates from different truths can create apparent bias or conceal it.

Adaptive trials can have replicates in which an estimate is unavailable—for example, because an outcome is not observed or a fitting procedure fails. Do not silently drop such replicates and describe the resulting average as unconditional performance. Report the frequency of unavailable estimates separately, and state whether bias, MSE, or coverage is calculated among all replicates under a defined failure rule or only among replicates with an available result. A complete-case summary is conditional on availability and can be misleading if availability is related to trial outcomes or estimate values.

Useful diagnostics include the distribution of replicate estimates relative to \(\theta\), not just their average; bias and MSE across distinct scenarios; coverage alongside interval widths; and the proportion of failed or unavailable results. Strong skew, boundary pile-up, or a few extreme estimates can make a mean-based summary unrepresentative. Low coverage can arise from bias, excessive variability, or intervals that are too narrow; high coverage can come with very wide intervals. Coverage and precision therefore answer different questions.

The main summaries are complementary. Bias isolates average displacement, while MSE penalizes both displacement and variability. RMSE expresses overall error in the parameter's units. Coverage evaluates an interval procedure, while average interval width describes its typical scale but not whether it succeeds in including the truth. Alternative summaries, such as median error or absolute error, can be informative when errors are skewed or dominated by outliers, but they do not replace the requested bias, MSE, and coverage measures.

These assessments are limited to the scenarios and trial algorithms simulated. Good results under a specified set of data-generating assumptions do not establish good performance if those assumptions are wrong. Performance should be examined across scientifically relevant scenarios, and conclusions should distinguish the simulation's finite-replicate results from exact properties of the procedure.

Common pitfall: “The mean estimate equals the truth, so the estimator is accurate and its intervals are reliable.” This reasoning confuses zero average displacement with low estimation error and adequate interval coverage. Check MSE to include variability, and calculate coverage directly from the replicate intervals; neither follows from bias alone.

Check Your Understanding

Answer each question using the definitions above. Short solutions follow the questions.

A scenario has \(\theta=5\), with replicate estimates 4, 5, and 8. Calculate empirical bias and MSE.
Why must estimates from scenarios with different true values generally be assessed separately?
For \(\theta=12\), intervals are \([10,12]\), \([12.1,14]\), and \([11,13]\). What is empirical coverage?
What does zero bias establish, and what does it not establish about MSE?
If some simulated trials produce no estimate, why might reporting bias only among successful fits be misleading?

Solutions. 1. The mean is \((4+5+8)/3=17/3=5.667\), so bias is \(5.667-5=0.667\). The MSE is \([(-1)^2+0^2+3^2]/3=10/3=3.333\). 2. Bias and error must be measured relative to the relevant scenario truth; pooling different truths can manufacture or hide displacement. 3. The first and third intervals include 12, while the second does not, so coverage is \(2/3=0.6667\). 4. Zero bias establishes that the estimator's expectation equals the target, if the statement holds for the underlying distribution; it does not rule out high variance, and therefore does not ensure low MSE. 5. If fit availability depends on trial outcomes or estimate values, the successful subset can differ systematically from all simulated trials. The conditional summary should be labelled, with failure frequency reported separately.