Worked Risk Comparison In A Gaussian Mean Problem

Risk Curves in a Gaussian Mean Problem

An estimator is not judged by its value at one observed data set alone. In decision theory, we compare procedures by their expected loss at each possible parameter value. A risk curve makes that comparison visible: it shows how the expected loss changes as the unknown mean changes. Here we calculate and compare three procedures in a scalar Gaussian model: the sample mean, a fixed estimate, and a fixed-factor shrinkage estimate.

Suppose $X_1,\ldots,X_n$ are independent and identically distributed as $N(\theta,\sigma^2)$, with known $\sigma^2$, and let $\bar X=n^{-1}\sum_{i=1}^n X_i$. Then $\bar X\sim N(\theta,v)$, where $v=\sigma^2/n$. We use squared-error loss: estimating $\theta$ by $d$ incurs loss $(d-\theta)^2$.

Definition: The pointwise risk of an estimator $\delta$ at parameter value $\theta$, under squared-error loss, is $R(\theta,\delta)=E_\theta[(\delta-\theta)^2]$. The expectation is taken over the sampling distribution when the true mean is $\theta$. A risk curve plots this expected loss against $\theta$; it is not the loss for one observed sample.

Three Estimators and Their Risks

The sample mean uses $\delta_{\mathrm{mean}}(\bar X)=\bar X$. A fixed estimate uses $\delta_a(\bar X)=a$, where $a$ is chosen in advance and does not depend on the data. A fixed-factor shrinkage estimate toward zero uses $\delta_\lambda(\bar X)=\lambda\bar X$, with a fixed factor $\lambda$. In the usual shrinkage comparison $0<\lambda<1$: the estimate is pulled toward zero, but the factor is not selected using the unknown $\theta$.

The formulas below follow from the mean and variance of $\bar X$. For shrinkage, write $\lambda\bar X-\theta=\lambda(\bar X-\theta)-(1-\lambda)\theta$. Since $E_\theta[\bar X-\theta]=0$, the cross-product has expectation zero. The remaining variance and squared-bias terms give the risk.

Theorem: In this model, for every real $\theta$, fixed $a$, and fixed $\lambda$, $$ R(\theta,\delta_{\mathrm{mean}})=v,\qquad R(\theta,\delta_a)=(a-\theta)^2,\qquad R(\theta,\delta_\lambda)=\lambda^2v+(1-\lambda)^2\theta^2. $$

Proof. For the sample mean, $E_\theta[(\bar X-\theta)^2]=\operatorname{Var}_\theta(\bar X)=v$. For the fixed estimate, the error is the nonrandom value $a-\theta$, so its squared error and risk are both $(a-\theta)^2$. For shrinkage, expand the squared error:

\begin{aligned} R(\theta,\delta_\lambda) &=E_\theta\!\left[\{\lambda(\bar X-\theta)-(1-\lambda)\theta\}^2\right]\\ &=\lambda^2E_\theta[(\bar X-\theta)^2] -2\lambda(1-\lambda)\theta E_\theta[\bar X-\theta] +(1-\lambda)^2\theta^2\\ &=\lambda^2v+(1-\lambda)^2\theta^2. \end{aligned}

The middle term vanishes because the centered sample mean has expectation zero. This proves all three formulas. $\square$

The sample mean has constant risk. The fixed estimate has a parabola with its minimum, zero, at $\theta=a$. Shrinkage has a parabola centered at zero, with risk $\lambda^2v$ at $\theta=0$; for $0<\lambda<1$, its risk increases with $\theta^2$. These differences in shape determine where the curves cross.

Worked Example: Plotting the Three Risk Curves

Take $v=1$, use the fixed estimate $a=0$, and shrink by $\lambda=1/2$. The three risks are $1$, $\theta^2$, and $\theta^2/4+1/4$. The table gives exact values at equally spaced parameter values.

$\theta$	Sample mean	Fixed estimate $0$	Shrinkage $\bar X/2$
$-3$	1	9	2.50
$-2$	1	4	1.25
$-1$	1	1	0.50
0	1	0	0.25
1	1	1	0.50
2	1	4	1.25
3	1	9	2.50

A compact plot of these values is shown below. Each column corresponds, in order, to $\theta=-3,-2,-1,0,1,2,3$. The symbols give approximate vertical levels on a common risk scale from 0 to 9; consult the table for exact heights.

Risk curve	Approximate plotted levels from left to right
Fixed estimate $0$	█▄▂▁▂▄█
Shrinkage $\bar X/2$	▃▂▁▁▁▂▃
Sample mean	▂▂▂▂▂▂▂

The curves are symmetric about zero because these choices give risks that depend only on $\theta^2$. At $\theta=0$, the fixed estimate is best; at $\theta=1$, shrinkage is best; and at $\theta=3$, the sample mean is best. The crossings that explain these changes can be calculated exactly.

Finding and Interpreting Crossings

For this example, the fixed estimate and shrinkage risks agree when $\theta^2=\theta^2/4+1/4$. Thus $3\theta^2/4=1/4$, or $\theta^2=1/3$, giving crossings at $\theta=\pm1/\sqrt{3}$. Below this absolute value the fixed estimate has lower risk; beyond it shrinkage has lower risk than the fixed estimate.

Shrinkage and the sample mean agree when $\theta^2/4+1/4=1$, which gives $\theta^2=3$ and $\theta=\pm\sqrt{3}$. Shrinkage has lower risk between those crossings, while the sample mean has lower risk outside them. Consequently, the lowest-risk procedure among these three is the fixed estimate for $|\theta|<1/\sqrt{3}$, shrinkage for $1/\sqrt{3}<|\theta|<\sqrt{3}$, and the sample mean for $|\theta|>\sqrt{3}$. At the boundaries, the corresponding risks tie.

The fixed estimate and sample mean also cross at $\theta^2=1$. That crossing does not change which of the three has the smallest risk: at $\theta=\pm1$, shrinkage has risk $1/2$, less than the risk $1$ shared by the other two. Pairwise crossings therefore need to be interpreted against all the curves, not treated automatically as changes in the best procedure.

Worked Example: A Fixed Estimate Away From Zero

Keep $v=1$ and $\lambda=1/2$, but now use the fixed estimate $a=1$. Its risk is $(1-\theta)^2$, while shrinkage has risk $\theta^2/4+1/4$. To find their crossings, set these expressions equal and expand:

(1-\theta)^2=\frac{\theta^2}{4}+\frac14 \quad\Longleftrightarrow\quad 3\theta^2-8\theta+3=0.

The quadratic formula gives $\theta=(8\pm\sqrt{64-36})/6=(4\pm\sqrt7)/3$, approximately $0.451$ and $2.215$. Because the difference between the fixed-estimate risk and shrinkage risk is an upward-opening quadratic, it is positive outside these roots and negative between them. Thus shrinkage has lower risk than the fixed estimate outside the interval between the crossings; the fixed estimate has lower risk inside it.

For comparison with the sample mean, the fixed estimate crosses its constant risk of 1 when $(1-\theta)^2=1$, giving $\theta=0$ or $\theta=2$. Shrinkage crosses the sample-mean risk at $\theta=\pm\sqrt3$, as before. At $\theta=1$, the three risks are $1$ for the sample mean, $0$ for the fixed estimate, and $1/2$ for shrinkage. This check confirms that the fixed estimate is best there, even though its risk is not generally smallest.

Worked Example: Changing the Variance and Shrinkage Factor

Suppose $v=1/4$, take the fixed estimate $a=0$, and set $\lambda=3/5$. The risks are $R_{\mathrm{mean}}=1/4$, $R_0(\theta)=\theta^2$, and $R_{3/5}(\theta)= (9/25)(1/4)+(4/25)\theta^2=9/100+4\theta^2/25$. The shrinkage and sample-mean risks cross where $9/100+4\theta^2/25=1/4$. Hence $4\theta^2/25=16/100$, so $\theta^2=1$. Shrinkage is lower for $|\theta|<1$, and the sample mean is lower for $|\theta|>1$.

For the fixed estimate and shrinkage, solve $\theta^2=9/100+4\theta^2/25$. This gives $21\theta^2/25=9/100$, or $\theta^2=3/28$, so the crossings are $\theta=\pm\sqrt{3/28}$, approximately $\pm0.327$. The fixed estimate is lower between these crossings, while shrinkage is lower outside them. At $\theta=0$, their risks are $0$ and $0.09$, respectively; at $\theta=0.5$, the risks are $0.25$, $0.25$, and $0.13$ for the sample mean, fixed estimate, and shrinkage, respectively. At $\theta=1$, shrinkage and the sample mean both have risk $0.25$, while the fixed estimate has risk $1$. The risk ordering therefore changes with the mean, even after the variance and shrinkage factor have been specified.

What the Comparison Does—and Does Not—Show

A lower risk at some values of $\theta$ is a pointwise advantage, not a guarantee that an estimator is better for every possible mean. For example, in the first plot the fixed estimate of zero has zero risk at $\theta=0$, but its risk grows without bound as $|\theta|$ grows. The sample mean has constant risk, so it eventually has lower risk than either fixed zero or a shrinkage rule with $0<\lambda<1$. Shrinkage can improve on the sample mean near zero, but its risk also grows with $\theta^2$.

This comparison concerns frequentist risk at each fixed parameter value. It does not average risk over a distribution for $\theta$, and the table does not reveal which parameter value is true. In particular, selecting whichever curve is lowest at the unknown true $\theta$ is not itself an implementable decision rule. A risk plot describes the performance of already specified rules; it does not provide the unknown mean needed to choose among them.

Common pitfall: Seeing that shrinkage has smaller risk than the sample mean near zero and concluding that it is uniformly better. The calculation shows the opposite outside the crossing: for $0<\lambda<1$, the shrinkage risk increases quadratically with $|\theta|$, while the sample-mean risk stays at $v$. A local improvement is not uniform dominance.

Check Your Understanding

Use the risk formulas and compare values at the same parameter value.

If $v=2$, what is the risk of the sample mean for any value of $\theta$? Why does it not change with $\theta$?
For $v=1$, fixed estimate $a=0$, and $\lambda=1/2$, solve for the values of $\theta$ where the fixed-estimate risk equals the sample-mean risk.
In the same setting, at what values of $\theta$ do shrinkage and the sample mean have equal risk? Which has lower risk between the crossings?
For $v=1$ and fixed estimate $a=1$, calculate the fixed-estimate risk at $\theta=1$. Compare it with the risks of the sample mean and $\lambda=1/2$ shrinkage.
Explain why a risk curve with the smallest value near zero does not, by itself, establish that an estimator is uniformly better.

\(\theta\)	Sample mean	Fixed estimate \(0\)	Shrinkage \(\bar X/2\)
\(-3\)	1	9	2.50
\(-2\)	1	4	1.25
\(-1\)	1	1	0.50
0	1	0	0.25
1	1	1	0.50
2	1	4	1.25
3	1	9	2.50

Risk curve	Approximate plotted levels from left to right
Fixed estimate \(0\)	█▄▂▁▂▄█
Shrinkage \(\bar X/2\)	▃▂▁▁▁▂▃
Sample mean	▂▂▂▂▂▂▂

Worked Risk Comparison In A Gaussian Mean Problem

What You'll Learn