Bias and Unbiasedness

Why Bias Is a Long-Run Property

An estimator is a rule for using observed data to estimate an unknown parameter. Before the data are observed, the estimator is a random variable: different possible samples can produce different estimates. Bias describes the estimator’s average error over its sampling distribution when the parameter is held fixed. It does not tell us whether one estimate from one finite sample will be close to the target.

This distinction is useful in evaluating statistical procedures. An estimate can happen to be close to the parameter even when the procedure is biased; conversely, an unbiased procedure can produce an estimate far from the parameter in a particular sample. The calculation must therefore begin with the estimator’s expected value under the stated model, not with a single observed estimate.

Definition: Let $T$ be an estimator of a parameter $\theta$, and suppose $E_\theta[|T|]$ is finite. The bias of $T$ at $\theta$ is $$ \operatorname{Bias}_\theta(T)=E_\theta[T]-\theta. $$ The estimator is unbiased for $\theta$ if $E_\theta[T]=\theta$ for every $\theta$ in the parameter space. For a target $g(\theta)$, replace $\theta$ by $g(\theta)$: the bias is $E_\theta[T]-g(\theta)$, and unbiasedness requires $E_\theta[T]=g(\theta)$ throughout the parameter space.

The subscript on $E_\theta$ emphasizes that expectation is taken under the sampling distribution corresponding to a fixed value of the parameter. In frequentist estimation, $\theta$ is not treated as a random variable in this calculation. The estimator is random because the data are random.

Bias has the same units as the target. A positive bias means that the estimator’s expected value is above the target; a negative bias means it is below. A zero bias means the positive and negative errors balance in expectation. It does not mean that each possible error is zero, or that the estimate is likely to be close in a particular sample.

A Short Justification: Expected Error and Bias

The bias formula is the expected estimation error written with its sign. If the error is defined as estimator minus target, then, because $\theta$ is fixed under $E_\theta$, linearity of expectation gives:

E_\theta[T-\theta] =E_\theta[T]-E_\theta[\theta] =E_\theta[T]-\theta =\operatorname{Bias}_\theta(T).

Thus an estimator is unbiased exactly when its signed error has expectation zero. This does not require the error to be zero for every sample. For example, errors of different signs can cancel in the average.

Checking Unbiasedness Across the Parameter Space

A calculation at one parameter value is not enough to establish that an estimator is unbiased. The condition is an identity in $\theta$ that must hold throughout the model’s parameter space. An estimator may have zero bias at one special value and nonzero bias elsewhere; it is then unbiased at that value only, not an unbiased estimator for the full model.

The following example uses a discrete sampling model. Its probabilities are specified as functions of the parameter, so the expectation can be calculated directly.

Worked Example: Finding an Unbiased Estimator in a Discrete Model

A hypothetical instrument produces an observation $X$ in $\{0,1,2\}$. The model has parameter space $0\leq\theta\leq2$, with $P_\theta(X=0)=(2-\theta)/4$, $P_\theta(X=1)=1/2$, and $P_\theta(X=2)=\theta/4$. Find constants $a$ and $b$ such that $T=aX+b$ is unbiased for $\theta$. Then verify the result at $\theta=1.2$.

Method. First check that the probabilities form a valid distribution:

\frac{2-\theta}{4}+\frac12+\frac{\theta}{4} =\frac{2-\theta+2+\theta}{4}=1.

They are nonnegative for $0\leq\theta\leq2$. Next calculate the expected value of $X$:

E_\theta[X] =0\left(\frac{2-\theta}{4}\right) +1\left(\frac12\right) +2\left(\frac{\theta}{4}\right) =\frac12+\frac{\theta}{2}.

Therefore $E_\theta[aX+b]=a(1/2+\theta/2)+b$. For this to equal $\theta$ for every $\theta$, the coefficient of $\theta$ must be 1 and the constant term must be 0:

\frac{a}{2}=1 \quad\text{and}\quad \frac{a}{2}+b=0.

The first equation gives $a=2$; substituting it into the second gives $b=-1$. Hence $T=2X-1$, and its expectation is $2(1/2+\theta/2)-1=\theta$ throughout the parameter space.

At $\theta=1.2$, the probabilities of $X=0,1,2$ are $0.2,0.5,0.3$, respectively. The estimator $T=2X-1$ takes values $-1,1,3$, so

E_{1.2}[T]=(-1)(0.2)+(1)(0.5)+(3)(0.3) =-0.2+0.5+0.9=1.2.

The check agrees with the general result. The common mistake to avoid is checking only the value at $\theta=1.2$: unbiasedness here follows from the identity for every $\theta\in[0,2]$, not from this one numerical verification.

Normalization and the Unbiasedness Condition Are Different

When an estimator’s sampling distribution is given by possible values and probabilities, two separate requirements matter. The probabilities must first add to 1. Then, using those valid probabilities, the expected estimator must equal its target for unbiasedness. Satisfying one requirement does not guarantee the other.

Worked Example: A Valid Distribution That Cannot Be Made Unbiased

In a hypothetical location model, one observation $X$ equals $\theta-2$ with probability $0.4$, $\theta+4$ with probability $0.2$, and $\theta+1$ with probability $r$. The estimator is $T=X$, and the target is $\theta$. Determine whether any value of $r$ makes $T$ unbiased while retaining the two stated probabilities.

Step 1: Apply probability normalization. The three probabilities must sum to 1:

0.4+0.2+r=1, \qquad r=0.4.

Step 2: Calculate the expectation using the valid probability.

\begin{aligned} E_\theta[T] &=0.4(\theta-2)+0.2(\theta+4)+0.4(\theta+1)\\ &=(0.4+0.2+0.4)\theta-0.8+0.8+0.4\\ &=\theta+0.4. \end{aligned}

The bias is $E_\theta[T]-\theta=0.4$. Thus the normalized distribution gives a positively biased estimator.

Check whether another $r$ could work. The expected displacement from $\theta$, before applying normalization, is $0.4(-2)+0.2(4)+r(1)=r$. Unbiasedness would require $r=0$, but normalization requires $r=0.4$. No value of $r$ satisfies both conditions.

The common mistake is to find $r=0.4$ from normalization and then call the estimator unbiased. That value makes the probabilities valid, but the resulting bias is $0.4$, not zero. Probability normalization and unbiasedness are distinct checks.

Unbiasedness Does Not Guarantee Accuracy in a Finite Sample

An expectation summarizes the long-run average of estimates over repeated samples generated under the same model and parameter. A single sample need not produce an estimate near that average. To assess finite-sample accuracy, one would need information about the estimator’s possible values and how probable they are, not merely its expectation. The next example makes the distinction explicit without confusing a realized estimate with its long-run mean.

Worked Example: An Unbiased Estimate Can Miss the Target

A single observation $X$ has a Bernoulli distribution with parameter $0\leq\theta\leq1$: $P_\theta(X=1)=\theta$ and $P_\theta(X=0)=1-\theta$. Consider $T_U=X$ as an estimator of $\theta$. At $\theta=0.6$, calculate its bias and the probability that the realized estimate differs from $\theta$ by at least $0.4$.

Step 1: Find the sampling expectation.

E_\theta[T_U]=E_\theta[X] =0(1-\theta)+1(\theta)=\theta.

So $T_U$ is unbiased for every $\theta\in[0,1]$, and its bias at $\theta=0.6$ is $0.6-0.6=0$.

Step 2: Examine the finite-sample outcomes. At $\theta=0.6$, $T_U=0$ with probability $0.4$ and $T_U=1$ with probability $0.6$. If $T_U=0$, the absolute error is $|0-0.6|=0.6$. If $T_U=1$, it is $|1-0.6|=0.4$. Both outcomes have error at least $0.4$, so

P_{0.6}\bigl(|T_U-\theta|\geq0.4\bigr) =0.4+0.6=1.

The expectation is exactly the target, yet every possible estimate in this one-observation problem differs from the target by at least $0.4$. This does not contradict unbiasedness: unbiasedness concerns signed errors averaged over repetitions, not closeness in each repetition.

Bias Can Be Parameter-Specific

The definition also clarifies why an estimator can be centered correctly at one parameter value without being unbiased for the model. Consider the constant estimator $C=0.75$ in the Bernoulli model above. It is a valid rule, even though it ignores the data. Its bias as an estimator of $\theta$ is

\operatorname{Bias}_\theta(C)=E_\theta[C]-\theta=0.75-\theta.

It has zero bias when $\theta=0.75$, but is not unbiased across $0\leq\theta\leq1$. The same caution applies whenever an expectation calculation happens to match the target at one parameter value: report whether the result holds at that value or throughout the parameter space.

Worked Example: A Biased Rule Can Be Closer in a Particular Sample

Use the one-observation Bernoulli model with $\theta=0.5$. Compare the unbiased estimator $T_U=X$ with the constant estimator $C=0.75$. Calculate each estimator’s bias and its absolute error for every possible sample, then compare their closeness to the target.

Unbiased estimator. Since $E_{0.5}[X]=0.5$, $T_U$ has bias zero. Its two possible estimates are 0 and 1, each with probability $0.5$. Their absolute errors are $|0-0.5|=0.5$ and $|1-0.5|=0.5$. Thus its absolute error is $0.5$ for either sample.

Constant estimator. The estimator $C=0.75$ has expectation $0.75$, so its bias at $\theta=0.5$ is $0.75-0.5=0.25$. It always estimates $0.75$, giving absolute error $|0.75-0.5|=0.25$.

At this parameter value, the biased estimator is closer in every possible sample: its absolute error is $0.25$, compared with $0.5$ for the unbiased estimator. This comparison is specific to $\theta=0.5$; the constant rule’s bias is $0.75-\theta$ at other parameter values. It shows why zero bias alone is not a guarantee of finite-sample accuracy.

Common Exam Traps

Checking only one parameter value. To establish unbiasedness, show $E_\theta[T]=\theta$ for all permitted $\theta$, unless the question explicitly asks about bias at a particular value.
Using an invalid probability distribution. Verify that probabilities are nonnegative and sum to 1 before calculating an expectation.
Confusing expectation with a realized estimate. $E_\theta[T]=\theta$ does not say that the observed $T$ equals $\theta$, or even that it is close.
Dropping the sign of bias. Bias is estimator expectation minus target. Reversing the subtraction reverses the sign.
Calling a pointwise match unbiasedness for the whole model. Zero bias at one value is not the same as zero bias throughout the parameter space.

Key takeaway: Bias is the expected signed error under the sampling distribution at a fixed parameter value. An estimator is unbiased only when that expected error is zero throughout the parameter space. Unbiasedness is a statement about long-run centering, not a promise of accuracy for a finite sample.

Check Your Understanding

For each problem, state the relevant expectation and distinguish a model-wide conclusion from a result at one parameter value.

Let $X\sim\operatorname{Bernoulli}(\theta)$, $0\leq\theta\leq1$, and let $T=2X-1$ estimate $2\theta-1$. Find the bias as an estimator of this target. (Answer: $E_\theta[T]=2\theta-1$, so the bias is 0 for every $\theta$.)
In the same Bernoulli model, let $U=X+0.2$ estimate $\theta$. Find its bias and determine whether it is unbiased. (Answer: $\operatorname{Bias}_\theta(U)=0.2$; it is not unbiased for any $\theta$ in the model.)
Suppose $X\in\{0,1,2\}$ has probabilities $(2-\theta)/4, 1/2,\theta/4$, respectively, for $0\leq\theta\leq2$. If $W=X$ estimates $\theta$, find its bias and identify any parameter value where the bias is zero. (Answer: $E_\theta[W]=(1+\theta)/2$, so the bias is $(1-\theta)/2$; it is zero at $\theta=1$ only, and $W$ is not unbiased over the full parameter space.)
Let $X\sim\operatorname{Bernoulli}(0.4)$. Compare $T=X$ and the constant estimator $C=0.7$ as estimates of $0.4$. For each rule, give its bias and the probability that its absolute error is less than $0.35$. (Answer: $T$ has bias 0 and probability 0; $C$ has bias $0.3$ and probability 1.)

What You'll Learn