Tutorials › Statistical Decision Theory › Normal Variance Estimation Under Squared Error

Home  /  Tutorials  /  Quadratic-Risk Estimation Problems

Quadratic-Risk Estimation Problems · Tutorial 407 of 982

Normal Variance Estimation Under Squared Error

Use the chi-square distribution of the centered sum of squares to calculate and compare the bias, variance, and squared-error risk of two standard normal variance estimators.

Intermediate 8 min read

What You'll Learn

  • Define the usual unbiased and maximum-likelihood estimators of variance when the normal mean is unknown
  • Derive the first two moments of the relevant chi-square statistic
  • Calculate each estimator’s bias, variance, and squared-error risk
  • Compare their risks for every sample size of at least two
  • Distinguish risk for estimating variance from realized error in a particular sample

Why Compare Variance Estimators Under Squared Error?

In a normal sample, two familiar estimators of the population variance use the same centered sum of squares but divide it by different quantities. The usual unbiased estimator divides by \(n-1\); the maximum-likelihood estimator divides by \(n\). Unbiasedness distinguishes them in one way, but does not by itself determine which has smaller expected squared error.

Here the target parameter is the variance \(\sigma^2\), and the loss for reporting \(d\) is \((d-\sigma^2)^2\). We will calculate the bias, variance, and risk of each estimator under the model, then compare them. Throughout, \(X_1,\ldots,X_n\) are independent observations from \(N(\mu,\sigma^2)\), with both \(\mu\) and \(\sigma^2>0\) unknown and \(n\geq 2\).

Definition: The usual unbiased sample variance and the normal maximum-likelihood estimator of variance are, respectively, $$ S^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)^2, \qquad \hat{\sigma}_{\mathrm{MLE}}^2=\frac{1}{n}\sum_{i=1}^n(X_i-\bar X)^2 =\frac{n-1}{n}S^2, $$ where \(\bar X=n^{-1}\sum_{i=1}^nX_i\). Under squared-error loss for estimating \(\sigma^2\), the risk of an estimator \(\delta\) is \(R(\mu,\sigma^2;\delta)=E_{\mu,\sigma^2}[(\delta-\sigma^2)^2]\), provided this expectation is finite.

The terminology “unbiased” means \(E_{\mu,\sigma^2}[\delta]=\sigma^2\) for every allowed \((\mu,\sigma^2)\). It does not mean that an estimate from one particular sample must be close to \(\sigma^2\). Risk averages the squared error over repeated samples from the specified model.

The Centered Sum of Squares and Its Moments

The distributional fact that makes the calculations direct is $$ Q=\frac{1}{\sigma^2}\sum_{i=1}^n(X_i-\bar X)^2 =\frac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}. $$ The centering by \(\bar X\) costs one degree of freedom. For the risk calculations, we need the first two moments of this chi-square variable.

Result: If \(Q\sim\chi^2_\nu\), where \(\nu\) is a positive integer, then \(E[Q]=\nu\), \(\operatorname{Var}(Q)=2\nu\), and \(E[Q^2]=\nu(\nu+2)\).

To verify the moments, write \(Q=\sum_{j=1}^{\nu}Z_j^2\), where the \(Z_j\) are independent \(N(0,1)\) variables. The standard normal moments are \(E[Z_j^2]=1\) and \(E[Z_j^4]=3\). Thus

$$ E[Q]=\sum_{j=1}^{\nu}E[Z_j^2]=\nu. $$

Independence gives \(\operatorname{Var}(Z_j^2)=E[Z_j^4]-(E[Z_j^2])^2=3-1=2\), and the variance of a sum of independent variables is the sum of their variances. Therefore

$$ \operatorname{Var}(Q)=\sum_{j=1}^{\nu}\operatorname{Var}(Z_j^2)=2\nu, \qquad E[Q^2]=\operatorname{Var}(Q)+(E[Q])^2=2\nu+\nu^2=\nu(\nu+2). $$

Taking \(\nu=n-1\) yields \(E[Q]=n-1\) and \(E[Q^2]=(n-1)(n+1)\). Since \(S^2=\sigma^2 Q/(n-1)\), these moments give

$$ E[S^2]=\sigma^2, \qquad \operatorname{Var}(S^2) =\frac{\sigma^4}{(n-1)^2}\operatorname{Var}(Q) =\frac{2\sigma^4}{n-1}. $$

Worked Example: The Unbiased Estimator at \(n=4\)

Suppose \(n=4\) and the observations follow \(N(\mu,9)\), with \(\mu\) unknown. Then \(Q=3S^2/9\sim\chi^2_3\), so \(E[Q]=3\) and \(\operatorname{Var}(Q)=6\).

The unbiased estimator is \(S^2=3Q\). Its expectation is \(E[S^2]=3E[Q]=9\), so its bias is \(9-9=0\). Its variance, and therefore its squared-error risk, is \(\operatorname{Var}(S^2)=3^2\operatorname{Var}(Q)=9(6)=54\).

The units of this risk are squared variance units: the estimation error is in variance units, and the loss squares that error. The calculation is a repeated-sampling risk calculation, not the squared error of any particular observed estimate.

Bias, Variance, and Risk

For any estimator \(\delta\) with finite second moment, its squared-error risk decomposes as

$$ E[(\delta-\sigma^2)^2] =\operatorname{Var}(\delta)+\big(E[\delta]-\sigma^2\big)^2. $$

This is the scalar bias-variance identity applied with target \(\sigma^2\). For \(S^2\), the chi-square moments already show that the bias is zero and the risk is \(2\sigma^4/(n-1)\). The maximum-likelihood estimator is a fixed multiple of \(S^2\), so its bias and variance follow from the same calculations.

Theorem: For an iid \(N(\mu,\sigma^2)\) sample with unknown \(\mu\), \(\sigma^2>0\), and \(n\geq2\), the usual unbiased estimator and the maximum-likelihood estimator of variance have biases and squared-error risks $$ \begin{aligned} \operatorname{Bias}(S^2)&=0, & R(\mu,\sigma^2;S^2)&=\frac{2\sigma^4}{n-1},\\ \operatorname{Bias}(\hat{\sigma}_{\mathrm{MLE}}^2)&=-\frac{\sigma^2}{n}, & R(\mu,\sigma^2;\hat{\sigma}_{\mathrm{MLE}}^2)&=\frac{2n-1}{n^2}\sigma^4. \end{aligned} $$ For every \(n\geq2\), the maximum-likelihood estimator has strictly smaller squared-error risk than \(S^2\).

Proof. The likelihood, as a function of \(\mu\) and \(\sigma^2\), is proportional to \((\sigma^2)^{-n/2}\exp\{-\sum_i(X_i-\mu)^2/(2\sigma^2)\}\). For fixed \(\sigma^2\), maximizing it in \(\mu\) is equivalent to minimizing \(\sum_i(X_i-\mu)^2\), whose minimizer is \(\bar X\). Substituting \(\bar X\), the log-likelihood as a function of \(\sigma^2\), up to a constant, is \(-\frac n2\log(\sigma^2)-\frac{\sum_i(X_i-\bar X)^2}{2\sigma^2}\). When the centered sum of squares is positive, differentiating with respect to \(\sigma^2\) gives the maximizer \(\sum_i(X_i-\bar X)^2/n\), the stated maximum-likelihood estimator. Under the continuous normal model, the centered sum is positive with probability one for \(n\geq2\).

Because \(\hat{\sigma}_{\mathrm{MLE}}^2=((n-1)/n)S^2\) and \(E[S^2]=\sigma^2\),

$$ E[\hat{\sigma}_{\mathrm{MLE}}^2]=\frac{n-1}{n}\sigma^2, \qquad \operatorname{Bias}(\hat{\sigma}_{\mathrm{MLE}}^2) =\left(\frac{n-1}{n}-1\right)\sigma^2 =-\frac{\sigma^2}{n}. $$

Its variance is the squared scaling factor times the variance of \(S^2\):

$$ \operatorname{Var}(\hat{\sigma}_{\mathrm{MLE}}^2) =\left(\frac{n-1}{n}\right)^2\frac{2\sigma^4}{n-1} =\frac{2(n-1)}{n^2}\sigma^4. $$

The bias-variance identity now gives

$$ R(\mu,\sigma^2;\hat{\sigma}_{\mathrm{MLE}}^2) =\frac{2(n-1)}{n^2}\sigma^4+\frac{\sigma^4}{n^2} =\frac{2n-1}{n^2}\sigma^4. $$

For \(S^2\), zero bias makes its risk equal to its variance, \(2\sigma^4/(n-1)\). To compare the two positive coefficients, use

$$ \frac{2}{n-1}-\frac{2n-1}{n^2} =\frac{2n^2-(2n-1)(n-1)}{n^2(n-1)} =\frac{3n-1}{n^2(n-1)}>0 \qquad(n\geq2). $$

Thus the maximum-likelihood estimator has strictly smaller squared-error risk for every allowed sample size. \(\square\)

Worked Example: Comparing Both Risks at \(n=10\)

Let the model be \(N(\mu,4)\), with unknown \(\mu\), and take \(n=10\). The unbiased estimator has bias \(0\) and risk \[ R(S^2)=\frac{2(4^2)}{9}=\frac{32}{9}\approx3.556. \]

The maximum-likelihood estimator has bias \(-4/10=-0.4\). Its variance and risk are \[ \operatorname{Var}(\hat{\sigma}_{\mathrm{MLE}}^2) =\frac{2(9)}{100}(16)=2.88, \qquad R(\hat{\sigma}_{\mathrm{MLE}}^2)=2.88+(-0.4)^2=3.04. \]

The risk reduction is \(32/9-3.04\approx0.516\). It arises despite the maximum-likelihood estimator’s downward bias: its smaller variance more than compensates for the squared bias in this comparison.

Reading the Comparison Correctly

The risk formulas depend on \(\sigma^2\) through \(\sigma^4\), but not on \(\mu\). This follows from the centered sum of squares: translating every observation by the same amount leaves it unchanged. For a fixed sample size, the risk comparison is therefore the same for every positive variance and every mean in the model.

The result does not say that the maximum-likelihood estimate is closer to the true variance in every sample. It says that its expected squared error, averaged over samples from the model, is smaller. Nor does it establish that it is preferable under every loss function or every decision criterion; the comparison proved here is specifically for squared error in estimating \(\sigma^2\).

Worked Example: A Realized Estimate Is Not Its Risk

Consider the observed sample \(1,3,4,4\), from a model \(N(3,4)\). The sample mean is \(\bar X=3\), and the centered sum of squares is \[ (1-3)^2+(3-3)^2+(4-3)^2+(4-3)^2=4+0+1+1=6. \]

Thus \(S^2=6/3=2\), while \(\hat{\sigma}_{\mathrm{MLE}}^2=6/4=1.5\). With true variance \(4\), the realized squared errors are \((2-4)^2=4\) and \((1.5-4)^2=6.25\). In this particular sample, the unbiased estimate has the smaller squared error.

That observation does not contradict the risk comparison. For \(n=4\) and \(\sigma^2=4\), the risks over repeated samples are \(2(16)/3=32/3\) for \(S^2\) and \((7/16)(16)=7\) for the maximum-likelihood estimator. The risk ordering concerns averages over all possible samples, not the ordering in one realized sample.

Common Mistakes

  • Calling the maximum-likelihood estimator unbiased. Its expectation is \((n-1)\sigma^2/n\), so its bias is \(-\sigma^2/n\), not zero.
  • Comparing only variances. The maximum-likelihood estimator has lower variance, but its risk must include the squared bias. Here the full calculation still gives it lower risk.
  • Using \(n\) degrees of freedom for the centered sum. Estimating the mean and centering at \(\bar X\) gives \(n-1\) degrees of freedom, which is why the unbiased denominator is \(n-1\).
  • Confusing variance estimation with standard-deviation estimation. The estimators and risks above target \(\sigma^2\) under loss \((d-\sigma^2)^2\); they do not calculate risk for estimating \(\sigma\).
  • Treating risk as the error from one data set. Risk is an expectation under repeated sampling; a single sample can favor either estimator.
Key takeaway: In a normal sample with unknown mean, the chi-square moments give \(R(S^2)=2\sigma^4/(n-1)\) and \(R(\hat{\sigma}_{\mathrm{MLE}}^2)=(2n-1)\sigma^4/n^2\). The maximum-likelihood estimator is biased downward but has strictly smaller squared-error risk for every \(n\geq2\).

Check Your Understanding

Answer each question using the model and squared-error target specified above.

  1. Why does the centered sum of squares have \(n-1\), rather than \(n\), chi-square degrees of freedom?
  2. For \(n=5\), write the maximum-likelihood estimator as a multiple of \(S^2\), and give its bias as a function of \(\sigma^2\).
  3. For \(n=3\) and \(\sigma^2=2\), calculate the risk of \(S^2\).
  4. For \(n=3\) and \(\sigma^2=2\), calculate the risk of the maximum-likelihood estimator by adding its variance and squared bias.
  5. Explain why observing a sample where \(S^2\) has smaller squared error does not refute the general risk comparison.

Previous

Next

Back to Tutorials
Clinical Biostats