Why Simulate an Adaptive Estimator?
In an adaptive trial, the final estimate can depend on an interim result that also determines which participants are enrolled, which treatment is selected, or whether the trial continues. The estimator’s sampling distribution can therefore differ from the one expected under a fixed design. A simulation can expose that difference by repeating the entire trial process under specified conditions—not by simulating only the final analysis as if the design had been fixed.
The object of study must be defined before the simulation begins. For example, “the estimated treatment effect” is not enough: one must state the treatment-effect parameter, the adaptation rule, the estimator used on each path, and whether performance is evaluated across all trials or only among trials meeting a selection condition. The earlier material on conditional versus unconditional interval guarantees distinguishes these target populations; simulation must preserve the distinction.
Build the Simulation Around the Design
A defensible simulation is a controlled experiment on a statistical procedure. Prespecify a set of true effect sizes, including values near important decision boundaries, and apply the complete trial algorithm at every value. The data-generating assumptions should describe the endpoint distribution, treatment allocation, dependence between observations, missingness if relevant, and any nuisance parameters. They should also state which parts of the design are fixed and which are randomized.
The adaptation rule must be programmed as it would operate in the trial. If an interim statistic determines whether enrollment continues, calculate that statistic from interim data and apply the stated threshold. If allocation probabilities change, use those probabilities to generate later allocations. Then calculate the estimator and interval using exactly the prespecified analysis. Omitting the decision step or replacing it with a fixed sample size estimates a different procedure.
Choose the effect sizes, nuisance-parameter values, adaptation rules, estimators, and intervals to evaluate. State whether each target is unconditional or conditional on a particular path or selection event.
Generate outcomes and allocations under the stated model, apply every interim decision, and record the resulting path, estimate, interval, and any selection indicator.
Repeat the complete trial process \(R\) times at each prespecified scenario, using independent random draws for the repetitions.
Calculate performance measures from replicate-level results, quantify their Monte Carlo error, and check that the simulated paths and analysis match the design specification.
For replicate \(r\), let \(\hat{\theta}_r\) be the estimate and \(C_r\) the interval. The usual Monte Carlo summaries are
where \(\bar{\hat{\theta}}=R^{-1}\sum_r\hat{\theta}_r\), and \(I(\cdot)\) is 1 when its statement is true and 0 otherwise. The variance uses denominator \(R-1\), while the direct Monte Carlo estimate of MSE uses \(R\). Equivalently, MSE is the variance around the replicate mean plus squared bias; with the finite simulation summaries, that identity uses variance with denominator \(R\).
Bias and Mean Squared Error Under Selection
Selection can create bias even when an estimator is unbiased before selection. Consider a hypothetical trial with a single estimate \(X\), where \(X\sim N(\theta,1)\). The trial reports the estimate only if \(X>0\), and otherwise stops without selecting the treatment for further study. This is a deliberately simple model: it isolates the effect of a data-dependent selection rule. It assumes the normal model and unit standard deviation are correct, and that the threshold and reporting rule are applied exactly.
Worked Example: Estimating Bias Among Selected Trials
Set \(\theta=0.5\). In a hypothetical simulation with \(R=10{,}000\) trials, suppose 6,915 trials have \(X>0\), and the sum of their reported estimates is 6,978.4. The selected-trial mean is
The estimated bias among selected trials is therefore \(1.009-0.5=0.509\) units of the treatment-effect scale. This is a conditional quantity: it describes the estimator’s average error given \(X>0\), not its average error over all trial repetitions. In the full, unselected model, \(E(X)=\theta\), so the unconditional bias is zero. The discrepancy is caused by conditioning on a favorable noisy result.
This simulation result is plausible under the model. For a unit-variance normal variable, \(E(X\mid X>0)=\theta+\phi(-\theta)/[1-\Phi(-\theta)]\), where \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. At \(\theta=0.5\), this is approximately \(1.009\), a conditional bias of approximately \(0.509\). The analytic value is a useful benchmark for checking the simulation code.
At each effect size, estimate bias and MSE separately. A small bias does not imply good overall estimation: an estimator may have high variance. Conversely, an estimator with some bias can have lower MSE if its variance is sufficiently reduced. MSE summarizes squared error in the parameter’s units squared, while bias and standard deviation are in the parameter’s units.
Worked Estimation of Bias, Variance, and MSE
A second hypothetical design shows why the adaptation rule itself belongs inside the simulation. Let \(X_1\) and \(X_2\) be independent interim and additional estimates, each distributed as \(N(\theta,1)\). If \(X_1\leq0\), stop and use \(X_1\). If \(X_1>0\), continue and use \((X_1+X_2)/2\). The model assumes independent, equally precise stage estimates and a known unit variance; the rule is deterministic given \(X_1\).
A short derivation supplies a benchmark for bias. With \(A=\{X_1>0\}\), the adaptive estimator can be written \(\hat{\theta}=X_1-\tfrac12X_1I(A)+\tfrac12X_2I(A)\). Since \(X_2\) is independent of \(A\), \(E[X_2I(A)]=\theta P(A)\). Also, writing \(X_1=\theta+Z\), where \(Z\sim N(0,1)\), gives \(E[X_1I(A)]=\theta P(A)+\phi(\theta)\). Therefore
At \(\theta=0.5\), \(\phi(0.5)\approx0.3521\), so the theoretical bias is approximately \(-0.1760\). The negative sign arises because a high interim estimate triggers averaging with a second estimate, while a low interim estimate is retained without averaging.
Worked Example: Calculating Simulation Summaries From Replicates
Suppose a small illustrative run at \(\theta=0.5\) produces five adaptive estimates: \(-0.3,\ 0.1,\ 0.2,\ 0.4,\ 1.2\). These five values are hypothetical replicate results; their purpose is to show the calculations, not to provide a precise performance evaluation.
The errors relative to \(\theta\) are \(-0.8,\ -0.4,\ -0.3,\ -0.1,\ 0.7\). Their sum is \(-0.9\), so the estimated bias is \(-0.9/5=-0.180\) units. The average estimate is \(0.320\), also giving \(0.320-0.500=-0.180\).
The estimated MSE is the mean squared error:
The deviations of the errors from their mean \(-0.18\) are \(-0.62,-0.22,-0.12,0.08,0.88\). Their squared sum is \(1.228\), giving variance around the replicate mean with denominator 5 of \(1.228/5=0.2456\). Thus \(0.2456+(-0.180)^2=0.278\), agreeing with the directly calculated MSE. The conventional sample variance with denominator \(5-1\) is \(1.228/4=0.307\); it is not the variance term in that finite-run MSE identity. The estimated bias of \(-0.180\) is close to the theoretical \(-0.1760\), but five replicates are far too few to establish that the simulation is accurate.
Coverage Must Match the Conditioning Target
For each replicate, record whether the interval contains the true parameter. The proportion of indicators equal to 1 estimates coverage for the population of repetitions represented in its denominator. If intervals are assessed only among selected trials, divide the number of selected trials whose intervals cover by the number of selected trials—not by all simulated trials. If selection itself is part of the event of interest, report the joint probability as well and label it clearly.
Worked Example: Checking Coverage After a Selection Threshold
Use the first model, \(X\sim N(\theta,1)\), and suppose the reported interval is \(X\pm1.96\). At \(\theta=0.5\), let \(Z=X-0.5\), so \(Z\sim N(0,1)\). Selection requires \(X>0\), or \(Z>-0.5\). Interval coverage requires \(|X-0.5|\leq1.96\), equivalently \(-1.96\leq Z\leq1.96\). Among selected trials, the coverage event is their intersection, \(-0.5<Z\leq1.96\).
Consequently, conditional coverage among selected trials is
The unconditional probability that a trial is both selected and covered is the numerator, \(0.9750-0.3085=0.6665\). This joint probability is not the conditional coverage rate. The distinction matters because only about \(0.6915\) of all trials are selected in this model.
For a simulation check, suppose the 10,000 repetitions yield 6,915 selected trials and 6,664 selected trials whose intervals cover. Then the conditional coverage estimate is \(6{,}664/6{,}915\approx0.9637\); the joint selection-and-coverage estimate is \(6{,}664/10{,}000=0.6664\). Both are consistent with the model’s probabilities after rounding. They answer different questions, and neither should be labelled simply “coverage” without specifying the denominator.
Monte Carlo Error, Diagnostics, and Alternatives
Simulation summaries are themselves estimates. If replicate estimates are independent and have finite variance, the Monte Carlo standard error of the estimated mean is approximately \(s/\sqrt{R}\), where \(s\) is the replicate standard deviation. The estimated bias has the same Monte Carlo standard error because \(\theta\) is fixed. For an estimated coverage proportion \(\hat{q}\), its approximate Monte Carlo standard error is \(\sqrt{\hat{q}(1-\hat{q})/R}\). Conditional coverage uses the number of selected trials, \(R_{\mathrm{sel}}\), in place of \(R\); if few trials are selected, its Monte Carlo error can be large even when \(R\) is substantial.
Increase \(R\) until the Monte Carlo uncertainty is small enough for the decision being made. More repetitions cannot repair an incorrect data-generating model, an incorrectly coded adaptation rule, or a mismatch between the simulated estimator and the intended analysis. Useful diagnostics include checking selection frequencies against known or independently calculated probabilities, verifying that estimates and intervals are recorded on the correct paths, and confirming that summaries use the intended denominator. Inspect the distribution of estimates as well as its mean: a mean can conceal skewness, heavy tails, or rare extreme estimates that drive MSE.
A simulation can compare an adaptive estimator with a fixed-design estimator, a bias-corrected estimator, or another prespecified method. Apply each method to the same simulated trial where that comparison is meaningful, and compare bias, variance, MSE, and coverage at the same \(\theta\) and under the same data-generating conditions. A closed-form derivation or exact calculation, when available, is a valuable check; it is not a replacement for evaluating complicated decision rules. The main limitation is scope: results apply to the assumptions and parameter scenarios simulated. Prespecifying plausible sensitivity scenarios is more informative than presenting one favorable model as universally representative.
Check Your Understanding
Answer each question, then compare with the concise solutions below.
- At each fixed \(\theta\), what must be repeated in a simulation of an adaptive trial?
- In 8,000 repetitions, 2,000 trials meet a selection criterion and 1,700 selected-trial intervals cover. What are the conditional coverage estimate and the joint selection-and-coverage estimate?
- An estimator has estimated bias \(0.2\) units and variance \(1.5\) units squared. What is its MSE, using the population identity?
- Why can a simulation with few selected trials give imprecise conditional coverage even if the total number of repetitions is large?
Solutions. (1) Generate a complete trial, including allocation, interim decisions, continuation or stopping, final estimator, and interval, under the specified model. (2) Conditional coverage is \(1{,}700/2{,}000=0.850\); the joint probability estimate is \(1{,}700/8{,}000=0.2125\). The different denominators answer different questions. (3) \(\operatorname{MSE}=\operatorname{Var}(\hat{\theta})+\operatorname{Bias}(\hat{\theta})^2=1.5+0.2^2=1.54\) units squared. (4) Its Monte Carlo precision depends on the selected denominator, not simply the total \(R\); with few selected trials, the conditional proportion has greater sampling variability.