Constant-Factor Shrinkage Of A Gaussian Mean

Why Compare a Fixed Multiple with the Observation?

When an observation is noisy, multiplying it by a constant between zero and one pulls its values toward zero. This can reduce variability, but it also introduces bias when the true mean is not zero. The relevant question is not whether the estimator is biased, or whether its variance is smaller, but how those effects combine in its risk at each possible value of the unknown mean.

We study that tradeoff for one Gaussian observation and a factor chosen in advance. The comparison is pointwise in the unknown mean: a factor may lower risk near zero while raising it farther away. No averaging over possible parameter values is involved.

Model and loss: Let $X\sim N(\theta,\sigma^2)$, where $\theta\in\mathbb R$ is unknown and $\sigma^2>0$ is known. Under squared-error loss, the loss for estimating $\theta$ by a value $d$ is $L(\theta,d)=(d-\theta)^2$. The risk of an estimator $\delta(X)$ is $R(\theta,\delta)=E_\theta[(\delta(X)-\theta)^2]$.

The observation itself is the estimator $\delta_1(X)=X$. Its risk is $R(\theta,\delta_1)=\sigma^2$ for every $\theta$: $X$ is unbiased for $\theta$ and has variance $\sigma^2$. We compare it with $\delta_a(X)=aX$, where $a$ is a fixed real number. The usual shrinkage case has $0\leq a<1$, but allowing any real $a$ makes the comparison complete.

Deriving the Risk of a Fixed-Factor Rule

Write the observation as $X=\theta+\varepsilon$, where $E_\theta[\varepsilon]=0$ and $E_\theta[\varepsilon^2]=\sigma^2$. Then the estimation error for $\delta_a$ is $aX-\theta=(a-1)\theta+a\varepsilon$. Squaring and taking expectations gives the risk.

Theorem: For every fixed $a\in\mathbb R$ and every $\theta\in\mathbb R$, $$ R(\theta,\delta_a)=(a-1)^2\theta^2+a^2\sigma^2. $$ The first term is squared bias and the second is variance.

Proof. Expanding the squared error and using the zero mean and variance of $\varepsilon$,

\begin{aligned} R(\theta,\delta_a) &=E_\theta\left[\big((a-1)\theta+a\varepsilon\big)^2\right]\\ &=(a-1)^2\theta^2+2a(a-1)\theta E_\theta[\varepsilon] +a^2E_\theta[\varepsilon^2]\\ &=(a-1)^2\theta^2+a^2\sigma^2. \end{aligned}

The cross term is zero because $E_\theta[\varepsilon]=0$. The squared-bias term is $\{E_\theta[aX]-\theta\}^2=(a-1)^2\theta^2$, and the variance is $\operatorname{Var}_\theta(aX)=a^2\sigma^2$. Thus the formula also agrees with the bias-variance identity. $\square$

For $a=1$, the squared-bias term vanishes and the risk is $\sigma^2$. For $a\neq1$, the factor generally creates bias unless $\theta=0$. At the same time, if $|a|<1$, the variance term $a^2\sigma^2$ is less than $\sigma^2$. Whether the total risk falls depends on the size of $\theta$.

Worked Example: Halving an Observation with Variance Four

Suppose $X\sim N(\theta,4)$ and use $\delta_{1/2}(X)=X/2$. The risk formula gives

R(\theta,\delta_{1/2}) =\left(\frac12-1\right)^2\theta^2 +\left(\frac12\right)^2(4) =\frac{\theta^2}{4}+1.

The observation itself has risk $4$. At $\theta=2$, the shrinkage risk is $2^2/4+1=2$, which is below $4$. At $\theta=4$, it is $4^2/4+1=5$, which is above $4$. The same factor helps at the first parameter value and hurts at the second.

Exactly When Is the Risk Lower?

Subtract the risk of the observation from the risk of the fixed-factor rule. Factoring the result makes the direction of the comparison transparent.

\begin{aligned} R(\theta,\delta_a)-R(\theta,\delta_1) &=(a-1)^2\theta^2+(a^2-1)\sigma^2\\ &=(a-1)\left\{(a-1)\theta^2+(a+1)\sigma^2\right\}. \end{aligned}

Theorem: The strict risk-improvement condition is $$ R(\theta,\delta_a)<R(\theta,\delta_1) \quad\Longleftrightarrow\quad -1<a<1 \ \text{and}\ \theta^2<\frac{1+a}{1-a}\sigma^2. $$ For $-1<a<1$, the risks are equal at the boundary $\theta^2=\frac{1+a}{1-a}\sigma^2$, and the fixed-factor rule has greater risk beyond that boundary.

Proof. If $a=1$, the estimators are identical, so their risks are equal. If $a>1$, then $a-1>0$, while both terms inside the braces are positive or nonnegative and the second is strictly positive. The risk difference is therefore positive. If $a<-1$, then $a-1<0$, and the expression in braces is strictly negative, so the risk difference is again positive. At $a=-1$, the difference is $4\theta^2$, which is never negative.

It remains to consider $-1<a<1$. Here $a-1<0$, so the risk difference is negative exactly when the expression in braces is positive. Rearranging that condition,

(a-1)\theta^2+(a+1)\sigma^2>0 \quad\Longleftrightarrow\quad (1-a)\theta^2<(1+a)\sigma^2 \quad\Longleftrightarrow\quad \theta^2<\frac{1+a}{1-a}\sigma^2.

At equality the risk difference is zero; beyond the boundary it is positive. This proves the stated comparison. $\square$

In the ordinary shrinkage range $0\leq a<1$, the cutoff is positive and can be expressed as $\lvert\theta\rvert<\sigma\sqrt{(1+a)/(1-a)}$. The risk improvement is therefore largest in relevance near $\theta=0$, where the bias is small and the variance reduction matters. As $\lvert\theta\rvert$ grows, the squared-bias term grows quadratically, eventually outweighing that reduction.

Worked Example: A Factor of 0.8

Let $X\sim N(\theta,1)$ and take $a=0.8$. The risk is

R(\theta,\delta_{0.8})=(0.8-1)^2\theta^2+0.8^2 =0.04\theta^2+0.64.

The observation has risk $1$. The improvement condition is

\theta^2<\frac{1+0.8}{1-0.8}(1)=9, \qquad\text{or equivalently}\qquad |\theta|<3.

At $\theta=1$, the risk is $0.04+0.64=0.68<1$. At $\theta=4$, it is $0.64+0.64=1.28>1$. At $\theta=3$, it is $0.04(9)+0.64=1$, exactly matching the observation’s risk.

Worked Example: A Negative Factor

Take $X\sim N(\theta,9)$ and $a=-0.5$. This rule reverses the sign of the observation as well as reducing its magnitude. Its risk is

R(\theta,\delta_{-0.5}) =(-1.5)^2\theta^2+(-0.5)^2(9) =2.25\theta^2+2.25.

The observation has risk $9$. The comparison theorem gives

\theta^2<\frac{1-0.5}{1+0.5}(9)=3, \qquad\text{or}\qquad |\theta|<\sqrt{3}.

For example, at $\theta=1$ the risk is $2.25+2.25=4.5<9$. At $\theta=2$, it is $2.25(4)+2.25=11.25>9$. A negative factor can reduce risk near zero, but its squared-bias cost grows quickly as the true mean moves away from zero.

What the Pointwise Comparison Does—and Does Not—Say

For every $-1<a<1$, the risk difference is negative for some parameter values near zero and positive for sufficiently large $\lvert\theta\rvert$. Consequently, no such fixed factor has lower risk than $X$ at every real value of $\theta$. To verify the second claim directly, the coefficient of $\theta^2$ in the risk difference is $(a-1)^2>0$, so the difference tends to positive infinity as $\theta^2$ increases. A factor outside this interval does not give a strict improvement at any parameter value, as the theorem’s case analysis shows.

This is a pointwise risk calculation for a specified estimator, not a claim that one factor is best overall. The threshold depends on both the chosen factor and the known variance. Changing either changes the range of means for which the comparison favors shrinkage. The calculation also does not assign probabilities to different values of $\theta$; it evaluates risk separately at each fixed value.

Common Mistake: Comparing Variances Alone

A tempting but incorrect argument is: “If $0<a<1$, then $\operatorname{Var}(aX)=a^2\sigma^2<\sigma^2$, so $aX$ is better than $X$.” This omits the bias. In fact, $E_\theta[aX]=a\theta$, so the bias is $(a-1)\theta$, and its square contributes $(a-1)^2\theta^2$ to risk. For large $\lvert\theta\rvert$, that contribution can more than offset the variance reduction.

The correction is to compare the full risks, not just their variance components. For example, with $a=0.8$ and $\sigma^2=1$, the variance falls from $1$ to $0.64$, but at $\theta=4$ the total risk is $1.28$, which exceeds the observation’s risk of $1$. Lower variance alone does not imply lower squared-error risk.

Key takeaway: The fixed-factor estimator has risk $(a-1)^2\theta^2+a^2\sigma^2$. It strictly improves on the observation only when $-1<a<1$ and $\theta^2<\frac{1+a}{1-a}\sigma^2$. The comparison balances variance reduction against squared bias and depends on the unknown mean.

Check Your Understanding

Use the risk formula and the pointwise comparison to answer the following.

For $X\sim N(\theta,5)$ and $\delta(X)=0.6X$, derive the risk as a function of $\theta$.
For $a=0.6$ and $\sigma^2=5$, find the exact condition on $\theta^2$ for strict improvement over $X$.
At what value or values of $\theta$ do $\delta_{1/2}$ and $X$ have equal risk when $\sigma^2=4$?
Why does a variance reduction for $0<a<1$ not guarantee lower risk for every $\theta$?
Can any fixed factor $a$ strictly improve on $X$ at every real value of $\theta$? Explain using the risk difference.