Interpreting Bias and Root Mean Squared Error

Why Summarize Parameter Estimates Across Replicates?

A simulation study can produce hundreds or thousands of fitted parameter estimates for each candidate design. Looking at one fitted value at a time does not answer the design question: how close are the estimates to the parameter value that generated the data, and how much do they vary from one simulated study to another? Three summaries provide complementary answers: bias, empirical standard deviation, and root mean squared error (RMSE).

These are summaries of an estimator's behavior across simulated replicates under a specified scenario. They are not summaries of variability among individuals in a single simulated trial. For example, the empirical standard deviation of estimated clearance across replicates describes how much the estimated typical clearance varies between simulated studies; it is not the standard deviation of individual clearances.

Definitions and Interpretation

Suppose a fixed simulation scenario has a known generating value \(\theta_0\) for a parameter \(\theta\). Let \(\hat{\theta}_r\) be the estimate from replicate \(r\), for \(r=1,\ldots,R\), and let \(\bar{\hat{\theta}}=R^{-1}\sum_{r=1}^{R}\hat{\theta}_r\). Each replicate must use the same target value and analysis definition for these summaries to have a clear interpretation.

Bias has the same units as the parameter. A positive estimated bias means that the replicate estimates average above \(\theta_0\); a negative bias means they average below it. Empirical standard deviation and RMSE are nonnegative and also have the parameter's units. A small bias alone does not imply accurate individual estimates: positive and negative errors can cancel in the average.

The empirical standard deviation uses \(R-1\) in its denominator, as a sample standard deviation. RMSE instead averages squared errors over the \(R\) observed replicates, using denominator \(R\). RMSE is therefore not generally equal to the square root of estimated bias squared plus empirical standard deviation squared when those finite-\(R\) definitions are used. The exact relationship is derived below.

Why RMSE Combines Bias and Variability

RMSE is useful because it penalizes both systematic displacement from the target and random spread around the average estimate. The finite-replicate relationship follows by writing each error as the sum of its deviation from the sample mean and the estimated bias:

Squaring and summing gives a cross-product term. That term is zero because deviations from the sample mean sum to zero:

Taking square roots yields the exact sample identity:

For the underlying estimator across repeated simulated data sets, the corresponding population identity is mean squared error \(=\operatorname{Var}(\hat{\theta})+\operatorname{Bias}(\hat{\theta})^2\). The sample identity above reflects the use of \(R-1\) for the empirical standard deviation. As \(R\) grows, the factor \((R-1)/R\) approaches 1.

Scale, Units, and Comparing Parameters

Absolute bias, empirical standard deviation, and RMSE are most directly interpretable when designs estimate the same parameter on the same scale. When parameters have substantially different magnitudes or units, absolute errors cannot be compared as though they represented the same relative performance. A relative summary can help, provided its denominator is meaningful and nonzero.

Assumptions, Diagnostics, and Comparisons

These summaries are meaningful only when the simulation output matches the intended performance question. Replicates should represent repeated data sets generated under the same defined scenario, with the same target value and analysis procedure. If the scenario changes between replicates, a single pooled bias or RMSE may mix different targets and sources of variation. Summarize such scenarios separately unless the performance estimand explicitly defines a weighted combination.

Check the estimates and errors, not just the final summaries. A histogram or dot plot across replicates can reveal skewness, outliers, multiple modes, or a few extreme errors that strongly influence RMSE. Compare the mean estimate with \(\theta_0\), and check that the reported parameter units and transformations are consistent. RMSE is sensitive to large errors because it squares them; a small number of extreme estimates can dominate it.

Also report how many replicates contributed. If some fits fail or return no usable estimate, calculating summaries only among successful fits describes performance conditional on obtaining an estimate. It does not describe the overall behavior of the analysis pipeline unless that conditional target is what the study intends. Record the number excluded and the rule used; do not silently treat failed fits as if they never occurred. The detailed diagnosis of estimation failures is a separate evaluation task.

Bias, empirical standard deviation, and RMSE answer different questions, while alternatives such as median error or mean absolute error emphasize different features. The signed bias can hide offsetting positive and negative errors; the median error is less affected by extreme values but does not measure average squared error. Mean absolute error gives errors a linear rather than squared penalty. No one summary replaces inspection of the distribution when unusual behavior matters.

The estimated summaries also have Monte Carlo error because they are calculated from a finite number \(R\) of simulated replicates. A reported bias close to zero is not proof of exactly zero underlying bias, especially when \(R\) is small. The replicate count and Monte Carlo precision should therefore accompany interpretation, following the principles of the earlier material on Monte Carlo error. These summaries do not assess confidence interval coverage; that is a distinct performance measure.

Common Mistakes

• Calling empirical standard deviation “bias.” Spread around the mean estimate is not displacement from the generating value. Calculate bias as \(\bar{\hat{\theta}}-\theta_0\).
• Choosing a design by standard deviation alone. A tightly clustered set of estimates can be tightly clustered around the wrong value. Examine bias and RMSE as well.
• Using the wrong denominator without explanation. The empirical standard deviation uses \(R-1\); the defined RMSE averages squared errors over \(R\).
• Comparing raw errors across unlike scales. State units and consider a clearly defined relative measure when the parameter values are nonzero and scientifically appropriate.
• Dropping failed fits without disclosure. This changes the target to performance among successful fits and can make a design appear better than its full pipeline performance.