Measuring Bias, Variance, And MSE In Simulation Studies

Introduction

A simulation study can produce hundreds or thousands of estimates of the same actuarial quantity. A list of results is informative only after it is summarized in a way that answers the relevant question: are the estimates centered on the target, how much do they vary, and how far are they from the target overall? Bias, variance, and mean squared error (MSE) describe these distinct aspects of estimator performance.

The calculations depend on the target being fixed and clearly defined, and on repeated estimates being generated under the same data-generating conditions. In a simulation study, the true target is usually known because it is specified by the simulation model. That known value makes it possible to assess the errors of the estimator directly.

Definition: Let \(\theta\) be the fixed target and let \(T\) be an estimator, regarded as a random variable across repeated data sets generated under the same conditions. The bias of \(T\) is \(E[T]-\theta\); its variance is \(E[(T-E[T])^2]\); and its mean squared error is \(E[(T-\theta)^2]\). Bias measures systematic displacement from the target, variance measures spread around the estimator’s own mean, and MSE measures average squared error from the target.

Calculating the Empirical Summaries

Suppose a simulation produces \(R\) estimates \(T_1,T_2,\ldots,T_R\) of a known target \(\theta\). Their average is \(\bar T\). The empirical bias is the average signed error, the sample variance describes spread around \(\bar T\), and the empirical MSE averages squared errors from \(\theta\).

\bar T=\frac{1}{R}\sum_{r=1}^{R}T_r,\qquad \widehat{\operatorname{Bias}}=\bar T-\theta

s_T^2=\frac{1}{R-1}\sum_{r=1}^{R}(T_r-\bar T)^2,\qquad \widehat{\operatorname{MSE}}=\frac{1}{R}\sum_{r=1}^{R}(T_r-\theta)^2

The different denominators are deliberate. The sample variance \(s_T^2\), with denominator \(R-1\), is the usual unbiased sample estimate of the estimator’s variance. The empirical MSE is the average of the \(R\) squared errors from the target, so its denominator is \(R\). Do not substitute deviations from \(\bar T\) for errors from \(\theta\) when calculating MSE.

These formulas summarize the simulation results; they do not change the estimator itself. For example, a positive empirical bias means the average simulated estimate is above the target, while a negative one means it is below. A small bias alone does not imply that individual estimates are close to the target: substantial variability may remain.

Why MSE Combines Bias and Variance

The population relationship between MSE, variance, and bias follows by separating an estimator’s deviation from the target into two parts: its deviation from its own expectation, and the displacement of that expectation from the target. Write \(b=E[T]-\theta\). Then \(T-\theta=(T-E[T])+b\). Squaring and taking expectations gives the decomposition below.

\begin{aligned} E[(T-\theta)^2] &=E\left[\left((T-E[T])+b\right)^2\right]\\ &=E[(T-E[T])^2]+2bE[T-E[T]]+b^2\\ &=\operatorname{Var}(T)+b^2, \end{aligned}

The middle term is zero because \(E[T-E[T]]=0\). Thus, MSE is the sum of variance and squared bias. Squaring matters: bias may be positive or negative, but its contribution to MSE cannot be negative.

There is a corresponding exact identity for the finite set of simulation results. If empirical variance is calculated with denominator \(R\), then the average squared error equals that variance plus the squared empirical bias. Since \(s_T^2\) uses denominator \(R-1\), the equivalent identity in terms of the usual sample variance is:

\widehat{\operatorname{MSE}} =\frac{R-1}{R}s_T^2+(\bar T-\theta)^2

This identity is also a useful arithmetic check. It explains why \(\widehat{\operatorname{MSE}}\) will not generally equal \(s_T^2+\widehat{\operatorname{Bias}}^2\): the sample variance requires the factor \((R-1)/R\) to match the empirical average squared error.

Worked Example: An Unbiased Severity-Cost Estimator

A hypothetical insurer studies an estimator of mean claim severity \(\theta\). Claims are undiscounted, with no mortality component. Under the specified data-generating model, individual severities are independent and identically distributed as Gamma with shape 100 and scale 1 dollar, so \(E[X]=100\) dollars and \(\operatorname{Var}(X)=100\) dollars squared. The estimator is the sample mean from five claims in each simulated data set. Five simulated estimates, in dollars, are \(92,96,100,104,108\). Calculate empirical bias, sample variance, and MSE. Method: calculate deviations from the target for bias and MSE, and deviations from the simulation average for sample variance. Common mistake avoided: using \(R-1\) rather than \(R\) in the empirical MSE.

Step 1: Find the average estimate. Here \(R=5\), so

\bar T=\frac{92+96+100+104+108}{5}=\frac{500}{5}=100.

Step 2: Find empirical bias. The target is \(\theta=100\), so \(\widehat{\operatorname{Bias}}=100-100=0\) dollars.

Step 3: Find sample variance. The deviations from \(\bar T=100\) are \(-8,-4,0,4,8\). Their squared values sum to \(64+16+0+16+64=160\). Therefore,

s_T^2=\frac{160}{5-1}=40\text{ dollars}^2.

Step 4: Find empirical MSE. Because the average estimate equals the target, the errors from the target are also \(-8,-4,0,4,8\). Thus,

\widehat{\operatorname{MSE}}=\frac{160}{5}=32\text{ dollars}^2.

Check. The finite-simulation identity gives \(\frac{4}{5}(40)+0^2=32\) dollars squared, matching the direct calculation. The empirical bias is zero, but individual simulation estimates still vary.

Worked Example: Lower Variance Does Not Guarantee Lower MSE

A hypothetical study targets a mean claim cost of \(\theta=50\) dollars. Undiscounted claim costs have a Gamma distribution with shape 25 and scale 2 dollars, giving mean 50 dollars and variance 100 dollars squared; there is no mortality component. In separate repeated simulations, estimator A (the sample mean of ten claims) has estimates \(48,50,52,50\), while estimator B is that sample mean plus a fixed 2-dollar adjustment and has estimates \(51,53,51,53\). Calculate each estimator’s empirical bias, sample variance, and MSE. Method: apply the same target-based error calculations to each set. Common mistake avoided: choosing an estimator based only on its variance.

Estimator A. Its average is \((48+50+52+50)/4=200/4=50\), so its empirical bias is \(50-50=0\) dollars. Deviations from its average are \(-2,0,2,0\); their squares sum to \(4+0+4+0=8\). Hence \(s_A^2=8/3=2.6667\) dollars squared. The squared errors from the target also sum to 8, so \(\widehat{\operatorname{MSE}}_A=8/4=2\) dollars squared.

Estimator B. Its average is \((51+53+51+53)/4=208/4=52\), so its empirical bias is \(52-50=2\) dollars. Deviations from its average are \(-1,1,-1,1\); their squares sum to 4. Thus \(s_B^2=4/3=1.3333\) dollars squared. Its errors from the target are \(1,3,1,3\), whose squared values sum to \(1+9+1+9=20\). Therefore, \(\widehat{\operatorname{MSE}}_B=20/4=5\) dollars squared.

Check and interpretation. For A, \(\frac{3}{4}(2.6667)+0^2=2\), and for B, \(\frac{3}{4}(1.3333)+2^2=5\), with rounding in the displayed intermediate values. B has the smaller sample variance, but its positive bias makes its MSE larger. In this simulation, A has the lower average squared error from the target.

Worked Example: Summarizing Error in a Small-Claim Portfolio

A hypothetical portfolio study estimates the mean of an undiscounted claim severity distribution. The data-generating model is exponential with mean and scale \(\theta=1{,}200\) dollars; equivalently, it is Gamma with shape 1 and scale 1,200 dollars. There is no mortality component. In each simulated data set, the estimator is the average severity from four claims. Five simulated estimates, in dollars, are \(1{,}100,1{,}150,1{,}200,1{,}250,1{,}300\). Calculate and interpret the three empirical summaries. Method: measure centering against \(1{,}200\) and spread around the observed simulation average. Common mistake avoided: treating variance around the average as if it were MSE from the target.

Step 1: Average the estimates.

\bar T=\frac{1{,}100+1{,}150+1{,}200+1{,}250+1{,}300}{5} =\frac{6{,}000}{5}=1{,}200\text{ dollars}.

Step 2: Calculate empirical bias. The average equals the target, so \(\widehat{\operatorname{Bias}}=1{,}200-1{,}200=0\) dollars.

Step 3: Calculate sample variance. Deviations from \(1{,}200\) are \(-100,-50,0,50,100\). Their squared values sum to \(10{,}000+2{,}500+0+2{,}500+10{,}000=25{,}000\). Therefore, \(s_T^2=25{,}000/4=6{,}250\) dollars squared.

Step 4: Calculate MSE. These same values are also the errors from the target in this case. The average squared error is \(25{,}000/5=5{,}000\) dollars squared.

Check and interpretation. The identity gives \(\frac{4}{5}(6{,}250)+0^2=5{,}000\) dollars squared. The simulation shows no empirical signed bias, but it does show variation across repeated samples. If the average had not equaled the target, the deviations used for sample variance and MSE would differ.

Interpreting the Results Carefully

Bias, variance, and MSE answer different questions, so they should be reported and interpreted separately rather than treated as interchangeable labels for “accuracy.”

Bias concerns average direction. A positive empirical bias means the average estimate exceeds the target; a negative empirical bias means it falls below. Positive and negative errors can cancel in the average.
Variance concerns repeatability around the estimator’s mean. A low variance means the estimates are concentrated around their own average, not necessarily around \(\theta\).
MSE concerns squared distance from the target. It combines variance with squared bias and is expressed in squared units of the target.
The target is essential. Empirical bias and MSE cannot be calculated from the estimates alone unless the target is also known or a stated reference value is used.

A simulation study should use a consistent target and data-generating scenario for all repetitions being summarized. If the simulation conditions change between repetitions, the estimates need not represent repeated performance under one fixed experiment, and a single bias or variance summary may be misleading. Likewise, if the reference value is only an approximation to the true target, the reported empirical bias and MSE are relative to that approximation.

Common Exam Traps

Using the wrong center. Sample variance uses \(T_r-\bar T\); MSE uses \(T_r-\theta\). These are not generally the same deviations.
Using the wrong denominator. The conventional sample variance divides by \(R-1\), whereas empirical MSE divides by \(R\).
Dropping the bias term. Low variance does not ensure low MSE when an estimator is systematically displaced from the target.
Forgetting units. Bias has the same units as the estimator; variance and MSE have squared units.
Confusing a simulation summary with a known population quantity. The empirical bias and MSE are summaries from a finite number of runs; they need not equal the estimator’s exact bias and MSE.

Key takeaway: For repeated estimates of a known target, calculate empirical bias from the difference between the simulation average and the target, sample variance from deviations around the simulation average, and MSE from squared errors against the target. Verify the results using \(\widehat{\operatorname{MSE}}=\frac{R-1}{R}s_T^2+(\bar T-\theta)^2\).

Check Your Understanding

For each problem, treat the listed estimates as repeated simulation results under one fixed hypothetical scenario. Use sample variance with denominator \(R-1\) and empirical MSE with denominator \(R\).

A hypothetical estimator targets an undiscounted mean severity of 40 dollars. The claim-severity model is Gamma with shape 10 and scale 4 dollars, so its mean is 40 dollars; there is no mortality component. Its three simulated estimates are 38, 40, and 42 dollars. Find empirical bias, sample variance, and MSE. (Answer: bias \(0\) dollars; sample variance \(4\) dollars squared; MSE \(2.6667\) dollars squared.)
A hypothetical estimator targets a frequency parameter \(\theta=25\). Counts follow a Poisson distribution with mean 25; interest and mortality are not applicable. Four simulated estimates are 27, 27, 27, and 27. Find the three summaries. (Answer: bias \(2\); sample variance \(0\); MSE \(4\).)
A hypothetical severity study targets \(\theta=500\) dollars. Undiscounted severities follow an exponential distribution with mean 500 dollars; there is no mortality component. Four simulated estimates are 480, 500, 520, and 500 dollars. Calculate empirical bias, sample variance, and MSE. (Answer: bias \(0\) dollars; sample variance \(266.6667\) dollars squared; MSE \(200\) dollars squared.)
A hypothetical estimator targets a mean annual claim cost of 80 dollars. Annual costs follow a Gamma distribution with shape 16 and scale 5 dollars, so the target is 80 dollars; amounts are undiscounted and mortality is not modeled. Three estimates are 78, 82, and 84 dollars. Calculate empirical bias, sample variance, and MSE, then check the finite-simulation identity. (Answer: bias \(1.3333\) dollars; sample variance \(9.3333\) dollars squared; MSE \(8\) dollars squared; \(\frac{2}{3}(9.3333)+(1.3333)^2\approx8\).)
A hypothetical estimator targets \(\theta=10\) under a Poisson model with mean 10; interest and mortality are not applicable. Four simulated estimates are 8, 10, 12, and 14. Calculate bias, sample variance, and MSE, and explain why MSE is not the same as sample variance. (Answer: bias \(1\); sample variance \(6.6667\); MSE \(6\); they differ because sample variance is centered on \(\bar T=11\), while MSE is centered on \(\theta=10\), and the denominators also differ.)

What You'll Learn