Estimating Output Variance and Its Uncertainty

Why Quantify Uncertainty in an Estimated Variance?

A simulation may be used to estimate an expected loss, a reserve distribution, or another quantity of interest. Its output variance describes how dispersed individual simulated outcomes are. But when the variance itself is estimated from a finite number of runs, that estimate is also uncertain. The uncertainty matters when the simulation is being used to compare variability across scenarios or to report how precisely variability has been measured.

The variance of the outputs and the standard error of the estimated variance are different quantities. If simulated outputs are measured in dollars, the output variance is measured in dollars squared, and the standard error of that variance estimate is also measured in dollars squared. Keeping these units straight is a useful check on calculations.

Definition: Let $X_1,\ldots,X_n$ be independent, identically distributed simulation outputs with mean $\mu$, variance $\sigma^2$, and finite fourth central moment $\mu_4=E[(X-\mu)^4]$. The sample variance is $$ S^2=\frac{1}{n-1}\sum_{j=1}^n(X_j-\bar X)^2,\qquad \bar X=\frac{1}{n}\sum_{j=1}^nX_j. $$ Its standard error is $\sqrt{\operatorname{Var}(S^2)}$, the standard deviation of the sample variance across repeated samples of $n$ simulation outputs.

The divisor $n-1$ makes $S^2$ an unbiased estimator of $\sigma^2$ under the stated independent, identically distributed model. The formula for its uncertainty depends on $\mu_4$, not only on $\sigma^2$. Consequently, knowing the output variance alone is generally insufficient to determine the standard error of the estimated variance.

The General Formula and Why the Fourth Moment Appears

For independent, identically distributed outputs with finite fourth central moment, the variance of the sample variance is

\operatorname{Var}(S^2)=\frac{1}{n}\left(\mu_4-\frac{n-3}{n-1}\sigma^4\right).

Here $\sigma^4$ means $(\sigma^2)^2$. The fourth central moment measures how much probability lies far from the mean, so distributions with heavier tails can produce more variable estimates of variance, even when their output variances are the same.

Derivation: Write $Z_j=X_j-\mu$, $A=\sum_{j=1}^n Z_j^2$, and $B=\sum_{j=1}^n Z_j$. Then $(n-1)S^2=A-B^2/n$. Independence and $E[Z_j]=0$ give $$ E[A^2]=E[AB^2]=n\mu_4+n(n-1)\sigma^4,\qquad E[B^4]=n\mu_4+3n(n-1)\sigma^4. $$ Expanding the square, $$ E[((n-1)S^2)^2]=E[A^2]-\frac{2}{n}E[AB^2]+\frac{1}{n^2}E[B^4]. $$ Substituting the three expectations and subtracting $\{E[(n-1)S^2]\}^2=(n-1)^2\sigma^4$ yields $$ \operatorname{Var}((n-1)S^2)=\frac{(n-1)^2}{n}\mu_4-\frac{(n-1)(n-3)}{n}\sigma^4. $$ Dividing by $(n-1)^2$ gives the stated formula. The $\mu_4$ term enters because squaring the sum of squared deviations produces fourth powers of deviations.

When the output distribution is not fully specified, estimate the fourth central moment with the sample fourth central moment $m_4=\frac{1}{n}\sum_{j=1}^n(X_j-\bar X)^4$, and estimate $\sigma^2$ with $S^2$. A common plug-in estimate of the variance of $S^2$, and its corresponding standard error estimate, are

\widehat{\operatorname{Var}}(S^2)=\frac{1}{n}\left(m_4-\frac{n-3}{n-1}(S^2)^2\right), \qquad \widehat{\operatorname{SE}}(S^2)=\sqrt{\widehat{\operatorname{Var}}(S^2)}.

This plug-in estimate is an approximation because the unknown moments have been replaced by sample quantities. It is not a guarantee of high precision for a small sample or a strongly skewed, heavy-tailed output distribution. If the estimated quantity under the square root is negative, do not report its square root as a real standard error; treat that result as a warning that the plug-in estimate is unstable for the available sample.

Worked Examples

Worked Example: Uncertainty for a Two-Point Output Distribution

A hypothetical simulation produces independent outputs $X$, with $P(X=0)=0.5$ and $P(X=2)=0.5$. For $n=10$ runs, find the standard error of the sample variance using the known output distribution. No interest rate or mortality assumption applies to this simulation-output model.

Method and moments. The output mean is $\mu=(0+2)/2=1$. The variance is $\sigma^2=0.5(0-1)^2+0.5(2-1)^2=1$, so $\sigma^4=1$. The fourth central moment is $\mu_4=0.5(0-1)^4+0.5(2-1)^4=1$.

Apply the general formula. With $n=10$, the coefficient $(n-3)/(n-1)$ is $7/9$. Therefore $$ \operatorname{Var}(S^2)=\frac{1}{10}\left(1-\frac{7}{9}\cdot1\right) =\frac{1}{10}\cdot\frac{2}{9}=\frac{1}{45}. $$

The standard error is $\sqrt{1/45}=0.1491$, to four decimal places. As a check, the variance estimate is in squared output units, and its standard error has the same units. The method uses the fourth moment of the stated two-point distribution; substituting the normal-output formula would be inappropriate.

Common mistake avoided: Using only $\sigma^2=1$ does not determine the general standard error. Here it is the additional fact $\mu_4=1$ that completes the calculation.

Worked Example: Estimating Uncertainty From Three Simulated Outputs

A hypothetical simulation of an unknown output distribution gives the independent outputs $1,3,5$, in output units. Estimate the standard error of the sample variance using the sample fourth central moment. No interest rate or mortality assumption is involved; no parametric distribution is assumed.

Find the sample mean and variance. The mean is $\bar X=(1+3+5)/3=3$. The deviations are $-2,0,2$, so the sum of squared deviations is $4+0+4=8$. Thus $$ S^2=\frac{8}{3-1}=4. $$

Find the sample fourth central moment. The sum of fourth powers of the deviations is $16+0+16=32$. Dividing by $n=3$ gives $m_4=32/3$. The fourth-power sum is $32$; it is not the sample variance, which is $4$.

Estimate the variance and standard error. For $n=3$, $(n-3)/(n-1)=0$, so $$ \widehat{\operatorname{Var}}(S^2)=\frac{1}{3}\left(\frac{32}{3}-0\cdot 4^2\right)=\frac{32}{9}. $$ Consequently, $$ \widehat{\operatorname{SE}}(S^2)=\sqrt{\frac{32}{9}}=\frac{\sqrt{32}}{3}=1.8856. $$

This is the plug-in estimate to four decimal places. The calculation is internally consistent: the estimated variance of $S^2$ is $32/9$, whose square root is $1.8856$. With only three observations, the estimate can be quite unstable.

Common mistake avoided: Do not confuse the sum of fourth powers, $32$, the sample fourth central moment, $32/3$, and the sample variance, $4$. They are different quantities with different roles in the formula.

The Normal-Output Special Case

If the simulation outputs are independent and normally distributed, $X_j\sim N(\mu,\sigma^2)$, the sample variance has a particularly simple exact result:

\frac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}, \qquad \operatorname{Var}(S^2)=\frac{2\sigma^4}{n-1}, \qquad \operatorname{SE}(S^2)=\sigma^2\sqrt{\frac{2}{n-1}}.

The second argument in $N(\mu,\sigma^2)$ is the variance. For a normal output, $\mu_4=3\sigma^4$; substituting this into the general formula gives $\operatorname{Var}(S^2)=\frac{1}{n}(3\sigma^4-\frac{n-3}{n-1}\sigma^4)=2\sigma^4/(n-1)$. The chi-square result provides the exact distributional justification under normality.

In practice, $\sigma^2$ is usually unknown. Replacing it with $S^2$ gives the usual sample-based estimated standard error:

\widehat{\operatorname{SE}}(S^2)=S^2\sqrt{\frac{2}{n-1}}.

This formula estimates uncertainty in the estimated variance, not uncertainty in the sample mean. It is exact as a standard-error formula when the true $\sigma^2$ is used and the outputs are normal; replacing $\sigma^2$ with $S^2$ makes it an estimate.

Worked Example: Normal Outputs and a Sample-Based Standard Error

A hypothetical model produces independent, approximately normal simulation outputs. In $n=25$ runs, the sample variance is $S^2=36$ in thousands of dollars squared. Estimate the standard error of $S^2$ using the normal-output formula. No interest rate or mortality assumption is part of the output model.

Method. Use the sample-based normal formula, with $n-1=24$: $$ \widehat{\operatorname{SE}}(S^2)=36\sqrt{\frac{2}{24}} =36\sqrt{\frac{1}{12}}=\frac{36}{\sqrt{12}}=10.3923. $$

The estimated standard error is $10.39$ thousand dollars squared, to two decimal places. Its square, $108$, is the plug-in estimate of the variance of $S^2$, in units of thousands of dollars to the fourth power. This unit check confirms that the standard error itself is in thousands of dollars squared.

Common mistake avoided: The factor $2/(n-1)$ belongs under the square root, and the multiplier is $S^2=36$, not the output standard deviation $6$. Using $6$ would produce an answer in the wrong scale.

Choosing and Reporting an Uncertainty Estimate

Use the general fourth-moment formula when the output distribution is not assumed normal and its fourth central moment is known or can be estimated. Use the simpler normal formula only when normality is a reasonable model for the outputs. A large sample can make a normal approximation for $S^2$ useful under suitable finite-moment conditions, but it does not make heavy tails or dependence disappear.

Both formulas require independent, identically distributed simulation outputs. If runs are dependent, for example because outputs come from a correlated simulation process, these formulas do not generally apply as written. Dependence changes the sampling variability and requires a method that accounts for the output structure.

When reporting results, identify the sample size, the estimated variance, the method used for its standard error, and any important distributional assumption. Do not describe the standard error of $S^2$ as the spread of individual outputs: that spread is $\sigma^2$, while the standard error describes run-to-run fluctuation in the estimate of that spread.

Key takeaway: For independent, identically distributed outputs with a finite fourth central moment, uncertainty in the sample variance depends on both $\sigma^2$ and $\mu_4$. Under normality, the standard-error estimate simplifies to $S^2\sqrt{2/(n-1)}$. State the assumptions and keep the variance estimate distinct from its standard error.

Check Your Understanding

Try these original problems before checking the answers.

A sample of simulation outputs is $1,3,5$. Find the sample mean, sample variance, and sum of fourth powers of deviations from the sample mean. (Answer: $\bar X=3$, $S^2=4$, and the fourth-power sum is $32$.)
Independent outputs have $P(X=0)=P(X=2)=0.5$, with $n=5$. Find $\sigma^2$, $\mu_4$, and $\operatorname{Var}(S^2)$. (Answer: $\sigma^2=1$, $\mu_4=1$, and $\operatorname{Var}(S^2)=1/10$.)
A sample of $n=9$ independent normal outputs has $S^2=16$. Estimate the standard error of $S^2$. (Answer: $16\sqrt{2/8}=8.0000$.)
For $n=6$, suppose the output moments are $\sigma^2=4$ and $\mu_4=48$. Calculate the standard error of $S^2$ using the general formula. (Answer: $\sqrt{\frac{1}{6}(48-\frac{3}{5}\cdot16)}=\sqrt{6.4}=2.5298$.)
A colleague applies $S^2\sqrt{2/(n-1)}$ to strongly right-skewed, independent outputs without assessing normality. State the assumption behind this formula and the appropriate general alternative. (Answer: The simple formula assumes normal outputs; use the fourth-moment formula $\operatorname{Var}(S^2)=\frac{1}{n}(\mu_4-\frac{n-3}{n-1}\sigma^4)$, or its sample plug-in estimate, when the fourth central moment is available.)

What You'll Learn