Tutorials › VEE Mathematical Statistics › Calculating Bias in Standard Estimators

Home  /  Tutorials  /  VEE Mathematical Statistics / Bias, Variance, and Efficiency

VEE Mathematical Statistics · Bias, Variance, and Efficiency · Tutorial 484 of 880

Calculating Bias in Standard Estimators

Calculate bias by finding an estimator’s expected value under the stated model, then subtracting the parameter it is intended to estimate.

Intermediate 10 min read

What You'll Learn

  • Calculate the bias of an estimator of a Bernoulli probability, including a pseudocount estimator.
  • Find bias by comparing the expectation of a proposed population-mean estimator with the population mean.
  • Derive the expected value and bias of the variance estimator that divides by the sample size.
  • Distinguish an estimator’s bias from the error in its observed value.
  • Identify how a denominator mistake changes an estimator’s expected value.

Bias Is a Property of an Estimator

An estimator is a statistic used to estimate an unknown population parameter. Because its value depends on a random sample, it is itself a random variable before the sample is observed. Its bias measures the difference between its expected value over repeated samples and the parameter it is intended to estimate. The calculation is therefore about the estimator’s sampling behavior, not just the result from one particular data set.

Definition: Let \(T\) be an estimator of a parameter \(\theta\), under a model indexed by \(\theta\). If \(E_\theta[T]\) exists, the bias of \(T\) at \(\theta\) is $$ \operatorname{Bias}_\theta(T)=E_\theta[T]-\theta. $$ The estimator is unbiased for \(\theta\) if \(E_\theta[T]=\theta\) for every allowed value of \(\theta\). A positive bias means the expectation is above the target; a negative bias means it is below the target.

The subscript on \(E_\theta\) emphasizes that the expectation is taken under the model with parameter value \(\theta\). Thus, bias may depend on the unknown parameter. To find it, state the estimator, calculate its expectation under the assumed model, and subtract the target parameter. Do not substitute the observed data for the expectation.

Bias for a Bernoulli Probability Estimator

Suppose \(X_1,\ldots,X_n\) are independent and identically distributed Bernoulli random variables with success probability \(p\). Each \(X_i\) equals 1 with probability \(p\) and 0 with probability \(1-p\), so \(E[X_i]=p\). The usual sample proportion is

$$ \hat p=\frac{1}{n}\sum_{i=1}^{n}X_i. $$

By linearity of expectation, \(E[\hat p]=p\), so this estimator is unbiased. A different estimator can have a different expectation even when it is calculated from the same Bernoulli observations. In particular, adding a fixed pseudocount and changing the denominator can pull the estimate toward a fixed value.

Worked Example: A Pseudocount Estimator of a Bernoulli Probability

Let \(X_1,\ldots,X_4\) be independent Bernoulli observations with success probability \(p=0.30\). A proposed estimator is \(T=(X_1+X_2+X_3+X_4+1)/6\). Find its bias at \(p=0.30\). If the observed outcomes are \(1,0,1,0\), also calculate the estimator’s observed value and its error. Method: take the expectation under the Bernoulli model before evaluating bias. Trap avoided: treating one sample’s error as the estimator’s bias.

First, use \(E[X_i]=p\) for each of the four observations and linearity of expectation:

$$ E_p[T] =\frac{E_p[X_1]+E_p[X_2]+E_p[X_3]+E_p[X_4]+1}{6} =\frac{4p+1}{6}. $$

Subtract the target \(p\) to obtain the bias as a function of \(p\):

$$ \operatorname{Bias}_p(T) =\frac{4p+1}{6}-p =\frac{1-2p}{6}. $$

At \(p=0.30\), this is \((1-0.60)/6=0.40/6=1/15\), or approximately \(0.0667\) to four decimal places. The estimator has positive bias at this parameter value.

For the observed outcomes, the number of successes is \(1+0+1+0=2\). Thus \(T=(2+1)/6=0.50\). The observed estimation error is \(T-p=0.50-0.30=0.20\). This is not the bias: the bias is the repeated-sampling expectation minus \(p\), which is \(1/15\). As a check, the usual sample proportion for these outcomes is \(2/4=0.50\), and its expectation is \(p\), even though its observed error here is \(0.20\).

The example also shows why a single parameter value is not enough to establish unbiasedness. The pseudocount estimator has bias \((1-2p)/6\), which is zero at \(p=0.50\), but is not zero for all \(p\) in the Bernoulli model.

Bias for an Estimator of a Population Mean

Let \(X_1,\ldots,X_n\) be independent observations from the same population with finite mean \(\mu\). Then \(E[X_i]=\mu\), and the sample mean \(\bar X=(1/n)\sum_{i=1}^{n}X_i\) has expectation \(\mu\). A denominator error can change that expectation. More generally, if a proposed estimator is a weighted sum \(T=\sum_{i=1}^{n}a_iX_i\), then

$$ E[T]=\sum_{i=1}^{n}a_iE[X_i] =\mu\sum_{i=1}^{n}a_i, \qquad \operatorname{Bias}_\mu(T) =\mu\left(\sum_{i=1}^{n}a_i-1\right). $$

The weights must sum to 1 for this weighted estimator to be unbiased for every possible \(\mu\). In the common denominator-error case \(T=(\sum X_i)/(n-1)\), the weights sum to \(n/(n-1)\), not 1.

Worked Example: A Denominator Error in a Mean Estimator

Suppose \(X_1,\ldots,X_4\) are independent hypothetical claim amounts from an exponential distribution with mean \(\mu=\$500\) and rate \(\lambda=1/500\) per dollar. A candidate uses \(T=(X_1+X_2+X_3+X_4)/3\) to estimate the population mean. For observed amounts \(\$300,\$400,\$500,\$800\), calculate \(T\), its bias, and its observed error. Method: use linearity of expectation under the specified exponential model. Trap avoided: mistaking an observed result for the estimator’s expected value.

The model gives \(E[X_i]=\mu=\$500\). Therefore,

$$ E[T] =\frac{E[X_1]+E[X_2]+E[X_3]+E[X_4]}{3} =\frac{4(\$500)}{3} =\$666.666\ldots. $$

The bias is the expected value minus the target mean:

$$ \operatorname{Bias}_\mu(T) =\frac{4\mu}{3}-\mu =\frac{\mu}{3} =\frac{\$500}{3} =\$166.67. $$

For the observed data, the sum is \(\$300+\$400+\$500+\$800=\$2{,}000\), so \(T=\$2{,}000/3=\$666.67\), rounded to cents. Its observed error is \(\$666.67-\$500=\$166.67\), rounded to cents. The numerical equality of this sample error and the rounded bias is coincidental; the former uses one sample, while the latter comes from the model’s expectation. As a check, the correctly scaled sample mean is \(\$2{,}000/4=\$500\), and its expected value is also \(\$500\).

Bias for an Estimator of a Population Variance

Suppose \(X_1,\ldots,X_n\) are independent and identically distributed with finite mean \(\mu\) and finite variance \(\sigma^2\). The estimator that divides the sum of squared deviations from the sample mean by \(n\) is

$$ T_n=\frac{1}{n}\sum_{i=1}^{n}(X_i-\bar X)^2. $$

This estimator is generally biased for \(\sigma^2\). The expectation can be found by relating deviations from \(\bar X\) to deviations from the population mean \(\mu\). The identity below is the key step; it holds for every set of observations.

Derivation: Write \(X_i-\bar X=(X_i-\mu)-(\bar X-\mu)\), expand the squares, and use \(\sum_{i=1}^{n}(X_i-\mu)=n(\bar X-\mu)\). The cross terms combine with the final squared term to give $$ \sum_{i=1}^{n}(X_i-\bar X)^2 =\sum_{i=1}^{n}(X_i-\mu)^2-n(\bar X-\mu)^2. $$ Taking expectations, \(E[(X_i-\mu)^2]=\sigma^2\) and \(E[(\bar X-\mu)^2]=\operatorname{Var}(\bar X)=\sigma^2/n\), where the last equality uses independence and a common variance. Hence $$ E\left[\sum_{i=1}^{n}(X_i-\bar X)^2\right] =n\sigma^2-n\left(\frac{\sigma^2}{n}\right) =(n-1)\sigma^2. $$

Dividing by \(n\) gives \(E[T_n]=((n-1)/n)\sigma^2\), so its bias is \(-\sigma^2/n\). Dividing the same sum of squared deviations by \(n-1\) instead gives the usual sample variance \(S^2\), whose expectation is \(\sigma^2\) under these assumptions. This result requires independent observations with a common finite variance; it does not automatically apply to dependent observations or to data with different variances.

Worked Example: Bias of the Variance Estimator with Denominator \(n\)

Let \(X_1,\ldots,X_5\) be independent observations from a normal population \(N(10,25)\), where the second parameter is the variance. For the observed values \(5,8,10,12,15\), calculate \(T_5\), its theoretical bias, and the corrected sample variance. Method: use the expectation derived from the iid model, then calculate the observed sums of squares. Trap avoided: confusing the realized value of \(T_5\) with its bias.

The model parameters are \(\mu=10\) and \(\sigma^2=25\). For \(n=5\), the expectation and bias of the denominator-\(n\) estimator are

$$ E[T_5]=\frac{5-1}{5}(25)=20, \qquad \operatorname{Bias}(T_5)=20-25=-5. $$

The observed sample mean is \(\bar x=(5+8+10+12+15)/5=50/5=10\). The squared deviations from this mean are \(25,4,0,4,25\), with sum \(25+4+0+4+25=58\). Thus the observed denominator-\(n\) estimator is \(T_5=58/5=11.6\).

The corrected sample variance for these same observations is \(S^2=58/(5-1)=58/4=14.5\). The theoretical bias of \(T_5\) is \(-5\), not \(11.6-25=-13.4\); that last quantity is the observed error for this sample. As a check on the correction, the general expectation formula gives \(E[S^2]=((n-1)\sigma^2)/(n-1)=\sigma^2=25\).

A Reliable Calculation and Common Traps

For each bias problem, identify the target parameter and the exact random statistic before calculating. Then use the stated sampling model to find the expectation. Linearity of expectation applies to sums whether or not the summands are independent; independence is needed for some results about variances, including \(\operatorname{Var}(\bar X)=\sigma^2/n\) in the variance-estimator derivation.

1
State the model and target.
For example, specify iid Bernoulli observations with probability \(p\), or iid observations with mean \(\mu\) and variance \(\sigma^2\). Identify whether the estimator targets \(p\), \(\mu\), or \(\sigma^2\).
2
Write the estimator exactly.
Check its coefficients and denominator. A sum divided by \(n-1\) is not the same statistic as a sum divided by \(n\).
3
Find its expectation under the model.
Use the expectation of each observation, or an appropriate identity for the statistic. Keep the parameter symbolic until the bias expression is obtained.
4
Subtract the target.
Calculate \(E_\theta[T]-\theta\), not \(\theta-E_\theta[T]\). Substitute a given parameter value only after obtaining the general expression when that is useful.

A frequent wrong calculation is to call \(T_{\text{observed}}-\theta\) the bias. That is the error in one realized sample. Bias is \(E_\theta[T]-\theta\), which averages the estimator across all samples possible under the model. Another common trap is to call an estimator unbiased because its bias happens to be zero at one parameter value. Unbiasedness requires zero bias throughout the parameter space.

Key takeaway: Bias is signed and model-based: calculate the estimator’s expectation first, then subtract the parameter it targets. For the standard iid sample mean, this gives zero bias; for the sum of squared deviations divided by \(n\), it gives \(-\sigma^2/n\). These calculations are aligned with the SOA VEE Mathematical Statistics syllabus.

Check Your Understanding

For each problem, calculate the expectation first and then subtract the target parameter. Answers are given for checking.

  1. Let \(X_1,\ldots,X_6\) be independent Bernoulli observations with success probability \(p\). For \(T=(\sum_{i=1}^{6}X_i+2)/10\), find the bias as a function of \(p\), then evaluate it at \(p=0.40\). (Answer: \(\operatorname{Bias}_p(T)=0.2-0.4p\); at \(p=0.40\), the bias is \(0.04\).)
  2. Let \(X_1,\ldots,X_5\) be iid observations from a population with mean \(\mu=\$80\). A proposed mean estimator is \(T=(\sum_{i=1}^{5}X_i)/4\). Find its expectation and bias. (Answer: \(E[T]=\$100\); bias \(=\$20\).)
  3. For an iid sample of size 8 from a population with variance \(\sigma^2=36\), let \(T=(1/8)\sum_{i=1}^{8}(X_i-\bar X)^2\). Find its expectation and bias for \(\sigma^2\). (Answer: \(E[T]=31.5\); bias \(=-4.5\).)
  4. Let \(X_1,X_2,X_3\) be independent Bernoulli observations with success probability \(p=0.70\). A proposed estimator is \(T=(2\bar X+1)/3\). Find its expectation and bias at this \(p\). (Answer: \(E[T]=(2p+1)/3=0.80\); bias \(=0.10\).)

Previous

Next

Back to Tutorials
Clinical Biostats