Why Conditional Variance Helps Compare Estimators
Two estimators can target the same parameter and both be unbiased, yet have very different sampling variability. A useful comparison starts with an unbiased estimator \(U\) and a statistic \(T\) that contains information about the parameter. Replacing \(U\) by its conditional expectation given \(T\) averages out the part of \(U\) that varies after \(T\) is known. The result remains unbiased and cannot have greater variance, provided the estimators have finite second moments.
This comparison is especially useful when a candidate estimator depends on the full sample, but a sufficient statistic captures the sample information about the parameter. The conditional expectation can then produce an estimator based only on that sufficient statistic. The key question is not just whether the variance falls, but whether it falls strictly.
The Variance Decomposition and Strict Improvement
For any random variable \(U\) with finite variance and any statistic \(T\), the law of total variance states that the variance of \(U\) is the variance of its conditional mean plus its expected conditional variance. Applying this identity to an estimator shows exactly how much variability is removed by conditioning.
The identity follows by writing \(U-E_\theta[U]\) as the sum of two terms:
When the square is expanded and expectations are taken, the cross-product has expectation zero. In fact, conditional on \(T\), the second term is fixed and \(E_\theta[U-E_\theta[U\mid T]\mid T]=0\). The expected square of the first term is \(E_\theta[\operatorname{Var}_\theta(U\mid T)]\); the expected square of the second is \(\operatorname{Var}_\theta(E_\theta[U\mid T])\). This gives the displayed identity.
Conditional expectation preserves the mean, since \(E_\theta[E_\theta(U\mid T)]=E_\theta[U]\). Thus \(U^*\) is unbiased whenever \(U\) is unbiased. The decomposition then gives
The reduction is strict at a parameter value \(\theta\) precisely when \(E_\theta[\operatorname{Var}_\theta(U\mid T)]>0\). Since conditional variances are nonnegative, this is equivalent to having positive conditional variance with positive probability. If \(U\) is already determined by \(T\) with probability one, its conditional variance is zero and conditioning does not change it. When \(T\) is sufficient, this result is the variance comparison in the Rao–Blackwell theorem.
Worked Comparisons
Worked Example: A Single Bernoulli Observation Versus the Sample Proportion
Let \(X_1,\ldots,X_5\) be independent Bernoulli random variables with success probability \(p=0.4\), so \(P(X_i=1)=0.4\) and \(P(X_i=0)=0.6\). The target is \(p\). Compare the unbiased estimator \(U=X_1\) with its conditional estimator given \(T=\sum_{i=1}^5X_i\), and calculate the variance reduction.
Method. Use exchangeability of the Bernoulli observations to calculate \(E[X_1\mid T]\). Given \(T=t\), exactly \(t\) of the five observations equal 1. Each observation has the same conditional probability of being one, so \(P(X_1=1\mid T=t)=t/5\). Therefore,
The original estimator is unbiased because \(E[X_1]=p=0.4\). Its variance is
The conditional estimator is also unbiased: \(E[T/5]=5p/5=0.4\). Since \(T\) is binomial with parameters 5 and 0.4, its variance is \(5(0.4)(0.6)=1.2\), giving
The variance falls by \(0.24-0.048=0.192\), so the reduction is strict. As a check on the conditional calculation, if the observed total is \(T=3\), then \(E[X_1\mid T=3]=3/5=0.6\). Common mistake avoided: using \(T\) itself as the estimator of \(p\). Its expectation is \(5p\), not \(p\); the scale factor \(1/5\) is required.
Worked Example: A Poisson Indicator and a Nonlinear Conditional Estimator
Let \(X_1,X_2,X_3\) be independent Poisson random variables with common parameter \(\lambda=\ln(3/2)\). Thus \(P(X_i=k)=e^{-\lambda}\lambda^k/k!\) for nonnegative integers \(k\). The target is \(e^{-\lambda}\). Compare \(U=I(X_1=0)\) with \(E[U\mid T]\), where \(T=X_1+X_2+X_3\), and calculate both variances.
The original estimator is unbiased because \(E[U]=P(X_1=0)=e^{-\lambda}\). Conditional on \(T=t\), the three counts have a multinomial allocation with total \(t\) and equal cell probabilities \(1/3\). In particular, \(X_1\mid T=t\) has a binomial distribution with parameters \(t\) and \(1/3\). Hence
Because \(e^{-\lambda}=2/3\), the variance of \(U\), a zero-one indicator, is
The total \(T\) is Poisson with parameter \(3\lambda\). For a Poisson random variable with parameter \(a\), its probability-generating function gives \(E[s^T]=\exp(a(s-1))\). Apply it first with \(s=2/3\):
Now apply the same formula to the second moment, using \((U^*)^2=(4/9)^T\):
Thus the conditional estimator is unbiased and has smaller variance: approximately \(0.06432\), compared with \(0.22222\) for \(U\). Common mistake avoided: the second moment is \(E[(4/9)^T]\), not \(E[(2/3)^T]\). The latter is the mean, and subtracting its square from the correct second moment is essential for the variance.
Worked Example: Estimating the Square of a Normal Mean
Suppose \(X_1,X_2\) are independent and each has distribution \(N(\mu,1)\), where the variance is known to be 1 and \(\mu\) is unknown. The target is \(\mu^2\). Consider \(U=X_1^2-1\), which is unbiased because \(E[X_1^2]=\mu^2+1\). Find its conditional estimator given \(T=X_1+X_2\), then compare variances at \(\mu=0\).
For two independent normal observations with equal variance, conditional on their sum \(T=t\), \(X_1\) is normal with mean \(t/2\) and variance \(1/2\). For a normal variable, its second moment is its variance plus the square of its mean. Therefore,
This estimator is unbiased: \(T\) has distribution \(N(2\mu,2)\), so \(E[T^2]=2+(2\mu)^2\). It follows that \(E[U^*]=(2+4\mu^2)/4-1/2=\mu^2\).
At \(\mu=0\), \(X_1\sim N(0,1)\), and \(U=X_1^2-1\) has variance \(\operatorname{Var}(X_1^2)=2\). Also, at \(\mu=0\), \(T/\sqrt{2}\) has distribution \(N(0,1)\). Write \(Z=T/\sqrt{2}\); then \(U^*=(Z^2-1)/2\). Since a standard normal variable satisfies \(\operatorname{Var}(Z^2)=2\),
The variance is reduced from 2 to 0.5, strictly, at \(\mu=0\). Common mistake avoided: conditioning \(X_1^2\) requires adding the conditional variance \(1/2\) to the squared conditional mean \(t^2/4\). Using only \(t^2/4\) would omit part of the conditional second moment and would not yield the correct unbiased estimator.
How to Read a Variance Comparison
Unbiasedness alone does not tell you which of two estimators is preferable. A variance comparison must be made under the stated model, and the variances can depend on the unknown parameter. The Rao–Blackwell comparison is stronger than an informal claim that averaging is helpful: for every parameter value for which the required moments exist, it gives a nonnegative variance difference with an exact expression.
When comparing two arbitrary unbiased estimators, do not assume one is better merely because it is a conditional expectation. The guarantee applies to the estimator \(U\) and its conditional version \(E[U\mid T]\), under the stated conditions. For a different competitor, calculate or otherwise justify the variance comparison. If conditioning leaves no randomness in \(U\), there is no reduction; if conditional variability remains on a set of positive probability, that variability is removed.
A Common Variance-Decomposition Trap
A frequent error is to label the expected conditional variance as the variance of the Rao–Blackwellized estimator. The decomposition shows the opposite role: it is the part subtracted from the original variance. For the Bernoulli example, the conditional variance of \(X_1\) given \(T=t\) is \((t/5)(1-t/5)\). Its expectation is \(0.192\), since the original variance is \(0.24\) and the conditional estimator variance is \(0.048\).
The incorrect calculation would be \(\operatorname{Var}(E[X_1\mid T])=E[\operatorname{Var}(X_1\mid T)]=0.192\). The correction is \(\operatorname{Var}(E[X_1\mid T])=\operatorname{Var}(X_1)-E[\operatorname{Var}(X_1\mid T)]=0.24-0.192=0.048\). Keeping the two terms of the decomposition distinct prevents this error.
Check Your Understanding
For each problem, identify the conditional estimator and use the model assumptions to compare its variance with the starting estimator.
- Let \(X_1,\ldots,X_4\) be independent Bernoulli variables with \(p=0.3\). For estimating \(p\), start with \(U=X_1\) and condition on \(T=\sum X_i\). Find the conditional estimator and both variances. (Answer: \(U^*=T/4\); \(\operatorname{Var}(U)=0.21\), \(\operatorname{Var}(U^*)=0.0525\), a strict reduction.)
- Let \(X_1,X_2\) be independent Poisson variables with parameter \(\lambda>0\). For estimating \(e^{-\lambda}\), take \(U=I(X_1=0)\) and \(T=X_1+X_2\). Find \(U^*\) and its variance. (Answer: \(U^*=(1/2)^T\); \(\operatorname{Var}(U^*)=e^{-3\lambda/2}-e^{-2\lambda}\), compared with \(\operatorname{Var}(U)=e^{-\lambda}(1-e^{-\lambda})\).)
- Let \(X_1,X_2,X_3\) be independent \(N(\mu,4)\) variables, with known variance 4 and unknown mean \(\mu\). To estimate \(\mu\), compare \(U=X_1\) with its conditional estimator given \(T=X_1+X_2+X_3\). (Answer: \(U^*=T/3=\bar X\); the variances are \(4\) and \(4/3\), respectively, so the reduction is strict.)
- Suppose \(U\) has finite variance and is already a function of \(T\) with probability one. State the conditional variance and the effect of replacing \(U\) by \(E[U\mid T]\). (Answer: \(\operatorname{Var}(U\mid T)=0\) with probability one; \(E[U\mid T]=U\) with probability one, so there is no variance reduction.)
- For an unbiased estimator \(U\) with finite variance, suppose \(E[\operatorname{Var}(U\mid T)]=0.07\) and \(\operatorname{Var}(U)=0.31\). Find the variance of the conditional estimator and determine whether the reduction is strict. (Answer: \(\operatorname{Var}(E[U\mid T])=0.31-0.07=0.24\); the reduction is strict.)