Simulation Bias In Treatment Effect Estimates

What Simulation Bias Measures

A treatment-effect estimator can vary from trial to trial even when every trial follows the same design and data-generating assumptions. Some estimates will be above the generating treatment effect and others below it. Simulation bias summarizes the average direction and size of this discrepancy across repeated trials. It is a property of an estimator under a specified generating scenario, not a description of how far one particular trial estimate is from the truth.

For a given scenario, let \(\tau\) be the treatment effect used to generate trial data. In replicate \(r\), apply the planned trial analysis and record its treatment-effect estimate \(\hat{\tau}_r\), for \(r=1,\ldots,R\). The estimator may, for example, be the posterior mean under a specified Bayesian model. The definition does not require a particular estimator, but the analysis procedure must be held fixed across replicates.

The generating scenario includes the design and data-generating values that determine the distribution of trial results. Consequently, bias is conditional on those choices: it can differ across sample sizes, outcome scenarios, or other features of the design. If several scenarios are combined into a single average, the weighting scheme must be specified; that average is not the conditional bias for any one scenario.

Estimating Bias and Its Monte Carlo Uncertainty

The average of the simulated estimates approximates their expected value, and subtracting \(\tau\) compares that average with the known generating value. When replicate estimates are independent and generated under the same scenario, the Monte Carlo standard error (MCSE) of the estimated bias is

Here \(s\) measures the spread of treatment-effect estimates across simulated trials, while \(s/\sqrt{R}\) measures the Monte Carlo uncertainty in their average. Increasing the number of replicates reduces this uncertainty, but does not reduce the estimator’s underlying sampling variability in actual trials. A small MCSE makes the estimated bias more precise; it does not make the estimator less biased.

This MCSE formula assumes independent replicate estimates. If simulation methods deliberately make replicates dependent, such as through shared random numbers, the mean’s Monte Carlo uncertainty must account for that dependence rather than applying the independent-replicate formula automatically. The replication and random-number design should be recorded so the uncertainty calculation matches the simulation.

Why Bias Can Depend on Sample Size

An estimator’s behavior can change with sample size even when the generating effect and analysis model remain fixed. One reason in a Bayesian analysis is shrinkage: a posterior estimate can be pulled toward the prior center, with the strength of that pull depending on the information in the data. The following simple normal model makes this dependence explicit. It is a derivation for a particular estimator, not a general formula for all Bayesian treatment-effect estimates.

Assume an observed estimate \(Y\) has sampling distribution \(Y\mid\tau\sim N(\tau,v)\), where \(v\) is known, and the analysis uses prior \(\tau\sim N(0,s_0^2)\). The posterior mean is $$ E(\tau\mid Y)=\lambda Y, \qquad \lambda=\frac{s_0^2}{s_0^2+v}. $$ Because repeated data generation has \(E(Y\mid\tau)=\tau\), the repeated-sampling expectation of this posterior-mean estimator is \(\lambda\tau\). Its bias conditional on the fixed generating value is therefore $$ E(\lambda Y\mid\tau)-\tau =\lambda\tau-\tau =-(1-\lambda)\tau. $$

When \(\tau>0\), this bias is negative: the posterior mean averages below the generating effect. Since \(v\) often decreases as sample size increases, \(\lambda\) moves closer to one and this particular shrinkage bias tends to become smaller. The direction and size of bias are not universal; they depend on the estimator, prior, likelihood, scenario, and generating value.

Comparing Bias Across Scenarios

A useful calibration report gives a separate estimated bias for each design or generating scenario of interest. For each one, record the generating effect, number of participants, analysis estimator, number of simulation replicates, average estimate, estimated bias, and MCSE. Keeping these quantities visible helps distinguish a real pattern in the estimated biases from noise caused by a finite simulation.

For example, consider two hypothetical scenarios with the same generating effect, \(\tau=1.50\) units, but different sample sizes. Suppose 1,000 independent replicates give:

The MCSEs are \(0.80/\sqrt{1000}=0.0253\) and \(0.55/\sqrt{1000}=0.0174\) units. Both estimated biases are negative, and the larger-sample estimate is closer to zero. The difference between the estimated biases is \(0.06\) units. If the two simulation runs are independent, its MCSE is approximately $$ \sqrt{0.0253^2+0.0174^2}=0.0307\text{ units}. $$ The observed difference is about \(0.06/0.0307=1.95\) MCSEs. This suggests a possible sample-size pattern, but the estimate is not infinitely precise; more replicates or additional checks may be warranted before drawing a firm conclusion.

The same comparison across outcome scenarios can reveal that an estimator behaves differently under different generating conditions. The scenarios must be defined in advance and interpreted on the same estimand and scale. A change in the estimate’s reported scale, target population, or treatment-effect definition can create a numerical difference that is not a meaningful change in bias.

Diagnostics, Alternatives, and Limitations

Before interpreting a bias estimate, check that the simulated data were generated at the stated treatment effect and that the analysis returns an estimate of that same effect. Verify treatment coding, units, the direction of subtraction, and the analysis settings across replicates. Compare a small number of individual simulated trials with expected behavior, and, where available, use analytic benchmarks or unit tests to check simulation components. These checks help identify an implementation problem that could otherwise look like estimator bias.

A useful diagnostic is to inspect estimated bias across a planned grid of sample sizes and generating scenarios. A changing pattern may be scientifically important, but abrupt reversals or unexpectedly large shifts should prompt checks of the generating values, analysis convergence, and estimator definition. If some replicates fail to produce an estimate, report how failures are handled: an average over only successful fits estimates a different quantity if failure is related to the data or estimate.

Bias is not the only way to compare estimators. The median simulated estimate describes a different center and is not the mean-bias definition given here. The variability \(s\) describes spread, not average displacement from \(\tau\). A decision-rule operating characteristic, such as the probability of meeting a posterior-probability threshold, answers a decision-performance question rather than an estimation-bias question. Each measure should be selected for the question being asked rather than substituted for another.

• Few replicates: an apparent bias can be Monte Carlo noise. Report the MCSE and increase \(R\) if the uncertainty is too large for the comparison.
• Different estimands or scales: subtracting quantities that do not represent the same treatment effect does not yield interpretable bias. Align the target and units first.
• Confusing spread with bias: a large across-trial standard deviation does not show that the estimator is biased. Compare the average estimate with the generating value.
• Overgeneralizing a scenario: bias under one design does not automatically describe other sample sizes or outcome-generating conditions. Report the scenario to which each result applies.

A finite simulation estimates bias with uncertainty, and the result is conditional on the simulation assumptions and analysis procedure. It cannot by itself show that the real trial’s data-generating process matches the simulated one. Bias assessment is therefore most informative when the scenarios are scientifically plausible, the estimand is clearly stated, and the simulation has been checked for coding and implementation errors.