Why Deliberate Bias Can Improve an Estimate
An estimator is a random variable: its value changes from sample to sample, even when the parameter being estimated stays fixed. In a finite sample, an estimator that is unbiased can still vary substantially. An alternative estimator may introduce a systematic shift, called bias, while reducing that variation. Whether this tradeoff is worthwhile depends on the resulting mean squared error, not on bias or variance alone.
This comparison is useful when an estimate is based on limited data and there is a defensible target toward which to pull it. For example, a hypothetical analyst might moderate a noisy estimate of a small portfolio's average claim size toward a reference amount. The moderation is not automatically beneficial: the reference may be far from the true mean, and the estimator's bias may outweigh its variance reduction.
An estimator is unbiased at \(\theta\) if \(E_\theta[T]=\theta\). It is biased if this equality fails. These are statements about repeated samples under a specified model, not descriptions of whether one observed estimate happens to be above or below the parameter.
The Bias-Variance Decomposition
The key comparison uses squared-error loss: an estimate \(T\) incurs loss \((T-\theta)^2\). The mean squared error is the expected loss over repeated samples. It separates into a variance term and a squared-bias term.
The decomposition applies to any estimator with a finite second moment. It does not say that bias is always undesirable, or that smaller variance alone guarantees better performance. It says that, under squared-error loss, the two contributions must be added, with bias squared.
Write \(T-\theta=(T-E_\theta[T])+(E_\theta[T]-\theta)\).
The squared expression has two squared terms and a cross-product.
Since \(E_\theta[T-E_\theta[T]]=0\), the expected cross-product is zero. The first expected square is the variance; the second is the squared bias.
In equations, the justification is
The cross-product vanishes because \(E_\theta[T-E_\theta[T]]=0\). Thus, an unbiased estimator has MSE equal to its variance, while a biased estimator has MSE equal to its variance plus a nonnegative penalty for bias.
Scaling an Unbiased Estimator
Suppose \(T\) is unbiased for \(\theta\), with variance \(V\), and consider the scaled estimator \(cT\), where \(c\) is fixed. Its expectation is \(c\theta\), so its bias is \((c-1)\theta\); its variance is \(c^2V\). Therefore,
Scaling by a factor between zero and one pulls the estimator toward zero. This reduces variance, since variance is multiplied by \(c^2\), but generally creates bias. A smaller variance is not enough to establish that the scaled estimator is better; its MSE must be compared with \(V\), the MSE of the original unbiased estimator.
Worked Example: Scaling a Sample Mean
Suppose \(X_1,\ldots,X_{16}\) are independent observations from \(N(\theta,16)\), where the second parameter is the variance. Compare the sample mean \(\bar X\) with \(T=0.8\bar X\) when the true value is \(\theta=10\). Find the bias, variance, and MSE of each estimator, and state which has the smaller MSE.
Method. Use the sampling distribution of the mean and the bias-variance decomposition. Since the observations are independent with variance 16, \(\operatorname{Var}(\bar X)=16/16=1\), and \(E[\bar X]=10\).
For the unbiased estimator \(\bar X\), the bias is \(10-10=0\), and its variance is 1. Therefore,
For \(T=0.8\bar X\), its expectation is \(0.8(10)=8\), so its bias is \(8-10=-2\). Its variance is
Consequently,
The scaled estimator has lower variance but larger MSE: \(4.64>1\). Check: its expected squared error must include both the variance and the squared bias; the squared bias of 4 is the dominant contribution. The common mistake is to conclude that the scaled estimator is better solely because its variance, 0.64, is below 1.
Shrinking Toward a Fixed Target
Scaling toward zero is one form of shrinkage. More generally, let \(T\) be unbiased for \(\theta\), with variance \(V\), and let \(m\) be a fixed target. For a fixed shrinkage factor \(a\), define
When \(0\leq a\leq1\), this estimator lies between the target \(m\) and the original estimate \(T\). If \(a=1\), there is no shrinkage; if \(a=0\), the estimate is always \(m\). Since \(E_\theta[T]=\theta\), the bias and variance are
The corresponding MSE is
The first term falls as \(a\) is reduced from 1, while the second term reflects the cost of pulling away from the true value. The formula also gives a theoretical factor minimizing MSE when \(V\), \(m\), and the true \(\theta\) are specified.
To verify the result, write \(D=(\theta-m)^2\), which is nonnegative. The MSE as a function of \(a\) is \(a^2V+(1-a)^2D\). Differentiating gives \(2aV-2(1-a)D\). Setting this derivative to zero yields \(aV=(1-a)D\), hence \(a^*=D/(V+D)\) when \(V+D>0\). The second derivative is \(2(V+D)>0\), so this value minimizes the MSE. Substitution gives \(V D/(V+D)\). If \(D=0\), the target already equals \(\theta\), and the minimum occurs at \(a=0\), with MSE zero. The minimum is no larger than the original MSE \(V\); when \(D>0\), it is strictly smaller.
Worked Example: Shrinking a Noisy Mean Toward a Reference
In a hypothetical model, \(X_1,\ldots,X_4\) are independent observations from \(N(\theta,16)\). The sample mean estimates \(\theta\), and an analyst considers shrinking it toward the fixed reference \(m=10\). For this risk comparison, take the true value to be \(\theta=12\). Find the oracle shrinkage factor, then compare the two estimators' bias, variance, and MSE.
Method. The sample mean is unbiased, and its variance is \(V=16/4=4\). The squared distance between the true parameter and the target is \((12-10)^2=4\). Thus,
The resulting estimator is \(T_{0.5}=10+0.5(\bar X-10)\). Its expectation is \(10+0.5(12-10)=11\), so its bias is \(11-12=-1\). Its variance is
Its MSE is therefore \(1+(-1)^2=2\). By comparison, \(\bar X\) has bias 0, variance 4, and MSE 4. The shrunken estimator has the smaller MSE, \(2<4\). Check: the optimal factor is between zero and one, and the minimum MSE formula gives \(4(4)/(4+4)=2\), agreeing with the direct calculation. This factor is optimal for the stated true value; since \(\theta\) is usually unknown, one cannot simply assume that this oracle factor is available in practice.
Shrinkage for a Bernoulli Proportion
The same reasoning applies to estimators other than sample means. Suppose \(Z_1,\ldots,Z_n\) are independent Bernoulli observations with success probability \(p\), and let \(K=\sum_{j=1}^n Z_j\). The sample proportion \(K/n\) is unbiased for \(p\), with variance \(p(1-p)/n\). Shrinking it toward a fixed reference proportion \(m\) gives \(m+a(K/n-m)\). Its bias is \((1-a)(m-p)\), and its variance is \(a^2p(1-p)/n\). These quantities depend on the true \(p\), even though the estimator need not use \(p\).
Worked Example: Shrinking a Small-Sample Proportion
A hypothetical study records 10 independent Bernoulli outcomes with success probability \(p\). Suppose 3 successes are observed. Consider the estimator \(T=0.8(K/10)+0.2(0.5)\), which shrinks the observed proportion toward one-half. Evaluate its observed estimate and, at the specified true value \(p=0.2\), its bias, variance, and MSE. Compare its MSE with that of the sample proportion.
Observed estimate. Here \(K/10=3/10=0.3\), so \(T=0.8(0.3)+0.2(0.5)=0.24+0.10=0.34\).
Repeated-sampling performance at \(p=0.2\). The expected sample proportion is \(0.2\), so
The bias is \(0.26-0.2=0.06\). The sample proportion has variance \(0.2(0.8)/10=0.016\), so the variance of \(T\) is \(0.8^2(0.016)=0.64(0.016)=0.01024\). Therefore,
The sample proportion has zero bias, so its MSE is its variance, 0.016. The shrunken estimator has smaller MSE at this specified \(p\), because \(0.01384<0.016\). Check: the observed estimate is 0.34, while the bias and MSE describe performance over repeated samples, not error in this one observed result. The true value \(p=0.2\) is used to evaluate risk; it is not needed to calculate the observed estimate.
How to Judge the Tradeoff
A bias-variance comparison is always tied to a model, parameter value, and loss criterion. The calculations above use squared-error loss and compare MSE at specified parameter values. An estimator can have lower MSE at some values of the parameter and higher MSE at others. Consequently, a claim that an estimator is “better” should specify the criterion and the conditions of the comparison.
| Estimator property | What it tells you | What it does not establish by itself |
|---|---|---|
| Bias | The difference between the estimator's expectation and the parameter | Whether typical squared error is small |
| Variance | How much the estimator varies around its own expectation | Whether its expectation is close to the parameter |
| MSE | Expected squared distance from the parameter | Performance under a loss function other than squared error |
When evaluating a proposed shrinkage rule, calculate its expectation and variance under the model, form its bias, and then add squared bias to variance. If the shrinkage factor is chosen using a theoretical formula, check whether that formula involves an unknown parameter. A factor derived from the true parameter describes an oracle benchmark, not necessarily a procedure that can be implemented from the data.
Common Exam Traps
- Adding bias rather than squared bias. The decomposition is \(\operatorname{MSE}=\operatorname{Var}+\operatorname{Bias}^2\), not variance plus bias. A negative bias still contributes a nonnegative amount.
- Scaling variance by \(c\) instead of \(c^2\). For a constant \(c\), \(\operatorname{Var}(cT)=c^2\operatorname{Var}(T)\).
- Confusing one estimate with estimator performance. Bias and variance are expectations over repeated samples. They cannot be read off from a single estimate alone.
- Assuming shrinkage must help. It reduces variance in common fixed-factor settings, but may create enough bias to increase MSE.
- Treating an oracle choice as available. An optimal factor that contains the unknown parameter may be useful for theoretical comparison, but it is not automatically a usable data-based rule.
Check Your Understanding
For each problem, calculate the requested quantities under the stated model. Answers are given for checking.
- Let \(T\) be unbiased for \(\theta=8\), with variance 1. For \(U=0.7T\), find the bias, variance, and MSE. (Answer: bias \(-2.4\), variance \(0.49\), MSE \(6.25\).)
- An unbiased estimator has variance 4. It is shrunk toward \(m=5\) when the true parameter is \(\theta=9\). Find the MSE-minimizing oracle factor and the resulting bias, variance, and MSE. (Answer: \(a^*=0.8\), bias \(-0.8\), variance \(2.56\), MSE \(3.20\).)
- For \(n=20\) Bernoulli observations with \(p=0.4\), let \(T=0.75(K/20)+0.25(0.5)\). Find its bias, variance, and MSE. (Answer: bias \(0.025\), variance \(0.00675\), MSE \(0.007375\).)
- An unbiased estimator has variance 2. A fixed shrinkage rule toward \(m=3\) uses \(a=0.6\), and the true parameter is \(\theta=5\). Find the shrunken estimator's bias, variance, and MSE, and compare its MSE with the original estimator's. (Answer: bias \(-0.8\), variance \(0.72\), MSE \(1.36\), lower than the original MSE of \(2\).)
- For an unbiased estimator with variance \(V=3\), the squared distance from a fixed target to the true parameter is \(D=6\). Find the oracle factor and minimum MSE. (Answer: \(a^*=2/3\), minimum MSE \(2\).)