Why Bias and Variance Can Prove Consistency
An estimator can be biased for every finite sample size and still become reliable as the sample grows. The key question is whether its systematic error and its random fluctuation both shrink. This tutorial gives a general proof: if the bias tends to zero and the variance tends to zero, then the estimator converges in probability to the parameter.
The argument is useful because it turns two quantities that describe an estimator’s performance into a direct statement about its probability of being far from the target. It also provides a finite-sample upper bound, although that bound can be conservative.
Consistency is a statement about the probability of error, not a requirement that \(T_n\) equal \(\theta\) for any particular sample. The parameter is treated as fixed. Expectations, variances, and probabilities are calculated under the assumed model at that fixed parameter value.
The Bias-and-Variance Consistency Result
The theorem does not require \(T_n\) to be unbiased at every sample size. It requires the bias to vanish in the limit. The variance condition controls random fluctuation around the estimator’s own mean; the bias condition ensures that this mean approaches the target.
Write \(T_n-\theta=(T_n-E[T_n])+(E[T_n]-\theta)\). The first term is centered random fluctuation, and the second is the bias.
If \(|T_n-\theta|>\varepsilon\) and \(|E[T_n]-\theta|\leq\varepsilon/2\), then \(|T_n-E[T_n]|>\varepsilon/2\).
For a finite-variance random variable \(W\), \(P(|W-E[W]|>c)\leq\operatorname{Var}(W)/c^2\) for \(c>0\). Apply it to \(T_n\) with \(c=\varepsilon/2\).
The bias condition eventually makes the split valid, and the resulting upper bound tends to zero when the variance tends to zero.
Derivation of the Probability Bound
Fix any \(\varepsilon>0\). Since \(|b_n(\theta)|\to 0\), there is an index \(N\) such that \(|b_n(\theta)|\leq\varepsilon/2\) whenever \(n\geq N\). For these \(n\), the triangle inequality gives \(|T_n-\theta|\leq |T_n-E[T_n]|+|b_n(\theta)|\). Therefore, if \(|T_n-\theta|>\varepsilon\), then \(|T_n-E[T_n]|>\varepsilon/2\). In event notation, \(\{|T_n-\theta|>\varepsilon\}\subseteq\{|T_n-E[T_n]|>\varepsilon/2\}\).
Chebyshev’s inequality now yields the bound
For each fixed \(\varepsilon>0\), the right-hand side tends to zero if \(\operatorname{Var}(T_n)\to0\). Thus \(P(|T_n-\theta|>\varepsilon)\to0\), which is exactly consistency. This proves the theorem.
The same reasoning also gives a useful bound before taking a limit: whenever \(|b_n(\theta)|<\varepsilon\),
Indeed, an error larger than \(\varepsilon\) requires centered fluctuation larger than \(\varepsilon-|b_n(\theta)|\), and Chebyshev’s inequality applies with that threshold. The split-at-\(\varepsilon/2\) bound is often simpler for proving a limit; the sharper threshold can be more informative for a particular \(n\).
Worked Example: A Noisy Calibration Reading
A hypothetical calibration procedure produces an estimator \(T_n\) of a fixed instrument level \(\theta\). Under the model, \(T_n\sim N(\theta+1/n,4/n)\), where the second argument is the variance. Determine whether \(T_n\) is consistent, and find a sufficient sample size for the Chebyshev bound to be at most \(0.05\) when \(\varepsilon=0.5\).
Method: compute bias and variance, then apply the split-at-half-threshold bound. The model gives \(E[T_n]=\theta+1/n\), so \(b_n(\theta)=E[T_n]-\theta=1/n\). Its variance is \(4/n\). Thus the bias tends to zero and the variance tends to zero, which proves consistency.
For the requested finite-sample bound, first ensure the bias is at most \(\varepsilon/2=0.25\): \(1/n\leq0.25\), or \(n\geq4\). For such \(n\),
To make this upper bound at most \(0.05\), solve \(64/n\leq0.05\), giving \(n\geq1280\). This also satisfies the bias condition. Therefore \(n=1280\) is a sufficient sample size; the bound at that value is \(64/1280=0.05\). The actual probability may be smaller, because Chebyshev’s inequality need not be tight.
Common mistake avoided: using the centered variance as if it were the mean squared error. Here the estimator is biased by \(1/n\); the proof accounts for that bias before applying Chebyshev’s inequality.
Worked Example: A Shrinkage Estimator for a Bernoulli Probability
Suppose \(X_1,\ldots,X_n\) are independent Bernoulli random variables with common success probability \(p\), where \(0\leq p\leq1\). Consider the estimator \(T_n=(\sum_{i=1}^{n}X_i+1)/(n+2)\) for \(p\). Find its bias and variance, establish consistency for every \(p\), and calculate a Chebyshev upper bound at \(p=0.4\), \(n=1000\), and \(\varepsilon=0.1\).
Method: use independence to calculate the mean and variance of the sum, then apply the theorem. Each \(X_i\) has mean \(p\) and variance \(p(1-p)\). Therefore, \(E[\sum X_i]=np\), and independence gives \(\operatorname{Var}(\sum X_i)=np(1-p)\). It follows that
Since \(p\) is fixed, \(|1-2p|\leq1\), so \(|b_n(p)|\leq1/(n+2)\to0\). Also \(0\leq p(1-p)\leq1/4\), which implies \(0\leq\operatorname{Var}(T_n)\leq n/[4(n+2)^2]\to0\). Both conditions hold for every \(p\in[0,1]\), so the estimator is consistent, including at the boundary values \(p=0\) and \(p=1\).
At \(p=0.4\) and \(n=1000\), the bias is \((1-0.8)/1002=0.2/1002\approx0.0001996\), which is less than \(\varepsilon/2=0.05\). The variance is
The split-at-half-threshold bound is \(4(0.0002390)/(0.1)^2\approx0.0956\). Thus the probability of an absolute error greater than \(0.1\) is at most approximately \(0.0957\) (to four decimal places). This is an upper bound, not a calculation of the exact error probability.
Common mistake avoided: treating a finite-sample bias as evidence that an estimator cannot be consistent. The bias here is generally nonzero, but its absolute value tends to zero.
Worked Example: A Sample-Size Guarantee From Given Performance Formulas
A hypothetical estimator \(T_n\) for a parameter \(\theta\) has bias \(b_n(\theta)=3/(n+3)\) and variance \(\operatorname{Var}(T_n)=1/(n+1)\) for positive integer \(n\). Find a sufficient sample size to ensure \(P(|T_n-\theta|>0.2)\leq0.01\) using the split-at-half-threshold argument.
Method: satisfy the bias threshold and the Chebyshev probability bound separately. The half-threshold is \(0.2/2=0.1\). The bias condition is \(3/(n+3)\leq0.1\). Multiplying by the positive quantity \(n+3\) gives \(3\leq0.1(n+3)\), so \(n+3\geq30\), or \(n\geq27\).
For \(n\geq27\), the probability bound is
To make this at most \(0.01\), require \(100/(n+1)\leq0.01\), so \(n+1\geq10000\), or \(n\geq9999\). This is stronger than the bias requirement. At \(n=9999\), the bias is \(3/10002\approx0.0002999\), and the bound is \(100/10000=0.01\). Thus \(n=9999\) is the smallest positive integer meeting both conditions for this particular sufficient-bound method.
Common mistake avoided: checking only that the variance bound is small. The triangle-inequality argument also needs the bias to be smaller than the error threshold; both conditions have been checked here.
What the Result Does and Does Not Say
The theorem gives a sufficient route to consistency. It does not say that vanishing bias is necessary: an estimator might be consistent without satisfying the particular moment conditions used in this proof. Nor does it say that either condition alone is enough. The argument controls the centered fluctuation using variance and controls the location of the estimator’s mean using bias.
A related quantity is mean squared error. For an estimator with finite second moment,
To see this, write \(T_n-\theta=(T_n-E[T_n])+b_n(\theta)\), square, and take expectations. The cross term is zero because \(E[T_n-E[T_n]]=0\). Consequently, if the bias and variance both tend to zero, the mean squared error tends to zero as well. Markov’s inequality applied to \((T_n-\theta)^2\) then gives \(P(|T_n-\theta|>\varepsilon)\leq E[(T_n-\theta)^2]/\varepsilon^2\), another proof of consistency. The triangle-and-Chebyshev proof makes explicit how bias and centered variability contribute separately.
For exam calculations, keep the target parameter fixed and make the limit statements explicit. A formula such as \(\operatorname{Var}(T_n)=c/n\) tends to zero only when \(c\) is finite and does not grow with \(n\). If a variance is infinite, Chebyshev’s inequality cannot be applied in this form. Similarly, a bias formula must be compared with the target \(\theta\), not merely with zero unless the estimator is intended to estimate zero.
Common Exam Traps
- Bias alone is not enough. An estimator can have zero bias at every \(n\) and still retain substantial random variation. The consistency theorem needs variance to tend to zero as well.
- Variance alone is not enough. A highly concentrated estimator can concentrate around the wrong value if its bias does not vanish.
- Do not omit the bias threshold. The split bound \(4\operatorname{Var}(T_n)/\varepsilon^2\) applies after \(|b_n(\theta)|\leq\varepsilon/2\) has been established.
- Do not confuse an upper bound with an exact probability. Chebyshev’s inequality may produce a loose bound, and a bound exceeding one gives no useful improvement over the trivial probability bound of one.
Check Your Understanding
For each problem, identify the relevant bias and variance conditions before using the consistency bound.
- An estimator has \(E[T_n]=\theta+2/n\) and \(\operatorname{Var}(T_n)=9/n\). Is it consistent for \(\theta\)? (Answer: Yes; the bias is \(2/n\to0\) and the variance is \(9/n\to0\).)
- An estimator has bias \(1/(n+1)\) and variance \(3/(n+1)\). State whether the bias-and-variance result proves consistency. (Answer: Yes; both quantities tend to zero.)
- An estimator has \(E[T_n]=\theta\) for every \(n\), but \(\operatorname{Var}(T_n)=2\) for every \(n\). Does the bias-and-variance result establish consistency? (Answer: No; the variance does not tend to zero, so this result cannot establish consistency.)
- An estimator has bias \(2/(n+2)\) and variance \(5/n\). Using the split-at-half-threshold bound, find a sufficient integer \(n\) for \(P(|T_n-\theta|>0.5)\leq0.02\). (Answer: \(n\geq4000\); the bias condition requires \(n\geq6\), and the probability bound is \(80/n\leq0.02\).)
- At a particular \(n\), an estimator has bias \(0.03\) and variance \(0.0004\). Find the split-at-half-threshold bound for an error threshold of \(0.1\). (Answer: The bias is at most \(0.05\), and the bound is \(4(0.0004)/(0.1)^2=0.16\).)