The Rao–Blackwell Theorem and Its Variance Proof

Why Conditioning Can Improve an Estimator

An estimator may use all the observed data even when some variation in those data carries no information about the parameter of interest. If a statistic $T$ is sufficient, conditioning an unbiased estimator on $T$ averages out variation among data sets that have the same value of $T$. The result keeps the same expectation but cannot have greater variance, provided the assumptions of the theorem hold.

This is the Rao–Blackwell theorem. Its conclusion is useful only when the conditional expectation can be written as a statistic that does not depend on the unknown parameter. Sufficiency supplies that guarantee; conditioning alone does not.

Definition: Let $X$ denote the observed data, with distribution indexed by a parameter $\theta$, and let $T=T(X)$ be a statistic. The statistic $T$ is sufficient for $\theta$ if the conditional distribution of $X$ given $T$ does not depend on $\theta$. An estimator $U=U(X)$ is unbiased for $g(\theta)$ if $E_\theta[U]=g(\theta)$ for every parameter value in the model.

The conditional distribution definition applies to discrete and continuous models. In a discrete model, it says that the conditional probabilities of the possible data sets, given $T=t$, are parameter-free whenever $P_\theta(T=t)>0$. In a continuous model, the corresponding conditional distribution must likewise be parameter-free. The factorization theorem, studied in the earlier material on sufficiency, is another way to establish sufficiency in common models.

Theorem: Suppose $T$ is sufficient for $\theta$, $U$ is an unbiased estimator of $g(\theta)$, and $U$ has finite variance under every parameter value under consideration. Then $U^*=E_\theta[U\mid T]$ has a version that is a function of $T$ alone, independent of $\theta$. It is unbiased for $g(\theta)$, and $$ \operatorname{Var}_\theta(U) = \operatorname{Var}_\theta(U^*) + E_\theta\!\left[\operatorname{Var}_\theta(U\mid T)\right]. $$ Consequently, $\operatorname{Var}_\theta(U^*)\leq \operatorname{Var}_\theta(U)$.

Why the Theorem Holds

There are two distinct claims to justify. First, the conditional expectation must define an estimator that can be computed without knowing $\theta$. Second, it must preserve the expectation and reduce variance. Sufficiency addresses the first claim; conditional expectation and variance identities establish the second.

Use sufficiency.
Because the conditional distribution of $X$ given $T$ is parameter-free, averaging $U(X)$ over that conditional distribution gives a function $h(T)$ that does not involve the unknown parameter. Set $U^*=h(T)=E_\theta[U\mid T]$.

Preserve the expectation.
The law of iterated expectation gives $E_\theta[U^*]=E_\theta[E_\theta(U\mid T)]=E_\theta[U]=g(\theta)$. Thus $U^*$ is unbiased.

Decompose the variance.
Write $U-E_\theta[U]=(U-U^*)+(U^*-E_\theta[U])$. The two terms have zero covariance, so the variance of their sum is the sum of their variances.

For completeness, the zero-covariance statement follows by conditioning on $T$. Since $U^*$ is a function of $T$, $$ E_\theta\!\left[(U-U^*)(U^*-E_\theta[U])\right] = E_\theta\!\left[(U^*-E_\theta[U])E_\theta(U-U^*\mid T)\right] =0. $$ The last equality holds because $E_\theta(U-U^*\mid T)=0$. Squaring the decomposition and taking expectations therefore gives the variance identity:

\begin{aligned} \operatorname{Var}_\theta(U) &=\operatorname{Var}_\theta(U^*)+ E_\theta\!\left[(U-U^*)^2\right]\\ &=\operatorname{Var}_\theta(U^*)+ E_\theta\!\left[\operatorname{Var}_\theta(U\mid T)\right]. \end{aligned}

The second equality uses $E_\theta[(U-U^*)^2\mid T]=\operatorname{Var}_\theta(U\mid T)$. Conditional variances are nonnegative, so the difference between the original variance and the conditioned variance cannot be negative. The variance proof is also a special case of the general variance decomposition: total variance is the variance of the conditional mean plus the expected conditional variance.

Worked Examples

Worked Example: A Finite Model With a Measurable Variance Reduction

Consider a finite model with parameter $\theta\in(1/3,5/9)$. The statistic $T$ takes values $1,2,3,4,5$, with probabilities $$ \begin{aligned} P_\theta(T=1)&=5/9-\theta,& P_\theta(T=2)&=2/9,& P_\theta(T=3)&=3/9,\\ P_\theta(T=4)&=2/9,& P_\theta(T=5)&=\theta-1/3. \end{aligned} $$ Given $T=j$, the data include an auxiliary label chosen uniformly from $n_j$ possibilities, where $(n_1,n_2,n_3,n_4,n_5)=(1,2,3,2,1)$. Define $U$ by assigning the following values to those labels: for $T=1$, the value is $0$; for $T=2$, the values are $0,1$; for $T=3$, the values are $0,0,1$; for $T=4$, the values are $0,1$; and for $T=5$, the value is $1$. Find the Rao–Blackwell estimator and compare the variances at $\theta=4/9$.

Method and sufficiency. The conditional distribution of the auxiliary label given $T$ is uniform and does not involve $\theta$. Therefore $T$ is sufficient, and the conditional expectation of $U$ given $T$ can be found by averaging within each group.

Step 1: Compute the conditional means. For $T=1$, the mean is $0$. For $T=2$, it is $(0+1)/2=1/2$. For $T=3$, it is $(0+0+1)/3=1/3$. For $T=4$, it is $(0+1)/2=1/2$. For $T=5$, it is $1$. Thus $U^*=E_\theta[U\mid T]$ takes values $0,1/2,1/3,1/2,1$, respectively.

Step 2: Check unbiasedness in the model. The expectation of $U$ is the sum of the probability of each group times its conditional mean: $$ E_\theta[U] =(2/9)(1/2)+(3/9)(1/3)+(2/9)(1/2)+(\theta-1/3)(1) =1/9+1/9+1/9+\theta-1/3 =\theta. $$ Thus $U$ is unbiased for $\theta$, and the theorem implies that $U^*$ is also unbiased.

Step 3: Evaluate the estimator at $\theta=4/9$. The probabilities of $T=1,2,3,4,5$ are $1/9,2/9,3/9,2/9,1/9$. Therefore $$ E_{4/9}[U^*] =(1/9)(0)+(2/9)(1/2)+(3/9)(1/3)+(2/9)(1/2)+(1/9)(1) =0+1/9+1/9+1/9+1/9 =4/9. $$ To four decimal places, this is $0.4444$, which agrees with the target $4/9$.

Step 4: Compare variances. At this parameter value, the second moment of $U$ is $$ E_{4/9}[U^2] =(2/9)(1/2)+(3/9)(1/3)+(2/9)(1/2)+(1/9)(1) =4/9. $$ Hence $\operatorname{Var}_{4/9}(U)=4/9-(4/9)^2=20/81$. For the conditioned estimator, $$ E_{4/9}[(U^*)^2] =(2/9)(1/4)+(3/9)(1/9)+(2/9)(1/4)+(1/9)(1) =1/18+1/27+1/18+1/9 =7/27, $$ so $\operatorname{Var}_{4/9}(U^*)=7/27-16/81=5/81$. The reduction is $20/81-5/81=15/81=5/27$.

Check and trap avoided. The within-group variances are $0,1/4,2/9,1/4,0$, so their probability-weighted average is $(2/9)(1/4)+(3/9)(2/9)+(2/9)(1/4)=5/27$, matching the variance reduction. A common arithmetic error is to report $1/3$ for the weighted mean in Step 3; the five terms sum to $4/9$, not $1/3$.

Worked Example: When Conditioning Does Not Improve Variance

Suppose one observation $T$ takes values $a,b,c$, with probabilities $\theta/2,1/2,(1-\theta)/2$, respectively, where $0<\theta<1$. Let $U=2$ when $T=a$, and $U=0$ otherwise. Verify unbiasedness and compare $U$ with its Rao–Blackwell version at $\theta=2/5$.

Method. Since $T$ is the entire observed data, its conditional distribution given itself is degenerate and parameter-free. Thus $T$ is sufficient. Also $U$ is already a function of $T$, so conditioning cannot change it.

Step 1: Verify the target. The probability that $T=a$ is $\theta/2$, hence $E_\theta[U]=2(\theta/2)=\theta$. The estimator is unbiased for $\theta$.

Step 2: Condition on $T$. For each possible value of $T$, $U$ is known exactly. Therefore $E_\theta[U\mid T]=U$.

Step 3: Calculate the variance at $\theta=2/5$. Here $P(T=a)=1/5$. Thus $E[U^2]=4(1/5)=4/5$, and $$ \operatorname{Var}(U)=4/5-(2/5)^2=20/25-4/25=16/25. $$ The conditioned estimator is identical, so its variance is also $16/25$.

Check and trap avoided. Equality is correct because $\operatorname{Var}(U\mid T)=0$: knowing $T$ already determines $U$. Rao–Blackwellization guarantees no increase, not a strict decrease in every problem.

Worked Example: Why Sufficiency Is Essential

Let $X$ take values $0$ and $1$, with $P_\theta(X=1)=\theta/3$, where $\theta$ is either $1$ or $2$. The target is $\theta$, and $U=3X$. Consider conditioning on the constant statistic $T=0$. Explain why this does not give a valid Rao–Blackwell improvement.

Method. First verify the starting estimator, then check the sufficiency condition before interpreting its conditional expectation as a new estimator.

Step 1: Verify unbiasedness. Since $E_\theta[X]=\theta/3$, $E_\theta[U]=3(\theta/3)=\theta$.

Step 2: Check whether $T$ is sufficient. Because $T$ is constant, conditioning on it leaves the distribution of $X$ unchanged. In particular, $P_\theta(X=1\mid T=0)=\theta/3$, which is $1/3$ at $\theta=1$ and $2/3$ at $\theta=2$. The conditional distribution depends on $\theta$, so $T$ is not sufficient.

Step 3: Identify the failure. For each fixed parameter value, $E_\theta[U\mid T]=\theta$. But these values do not define one statistic that can be calculated from the observation without knowing $\theta$. Calling this parameter-dependent quantity a new estimator would use the unknown target itself.

Check and trap avoided. Conditioning on an arbitrary statistic does not automatically produce a usable estimator. The crucial condition is that the conditional distribution given the statistic be parameter-free.

When Is the Variance Reduction Strict?

The variance identity gives an exact criterion. For a fixed parameter value, the variances are equal if and only if $E_\theta[\operatorname{Var}_\theta(U\mid T)]=0$. Because conditional variance is nonnegative, this occurs precisely when $U$ is determined by $T$ with probability one under that parameter value. If $U$ still varies among possible data sets with the same value of $T$ on a set of positive probability, then the expected conditional variance is positive and the variance reduction is strict.

The theorem does not say that every sufficient statistic yields an improvement for every estimator. If the original estimator is already a function of $T$, its conditional expectation is itself. Nor does the theorem say that the new estimator is automatically the best possible estimator in every class. It establishes a specific comparison: among these two unbiased estimators, the conditioned one has no greater variance.

Common Mistakes and Exam Traps

Forgetting to check sufficiency. The expression $E_\theta[U\mid T]$ can depend on $\theta$ when $T$ is insufficient. Such an expression is not necessarily an implementable estimator.
Claiming a strict reduction without checking equality. The reduction is $E_\theta[\operatorname{Var}_\theta(U\mid T)]$, which can be zero.
Mixing up the conditional mean and the variance. The improved estimator is $E_\theta[U\mid T]$; the amount removed from the variance is the expected conditional variance.
Leaving out the moment condition. The variance comparison requires finite variances. Unbiasedness alone does not guarantee that the variance identity is a finite numerical comparison.
Using the wrong weights. To compute an expectation of a function of $T$, weight each conditional value by $P_\theta(T=t)$. The conditional mean itself is not generally the unconditional mean.

Key takeaway: If $T$ is sufficient and $U$ is unbiased with finite variance, then $E_\theta[U\mid T]$ is a parameter-free function of $T$, remains unbiased, and has variance smaller by exactly $E_\theta[\operatorname{Var}_\theta(U\mid T)]$.

Check Your Understanding

For each problem, use the theorem’s assumptions carefully. Answers are provided for checking.

Suppose $T$ is sufficient, $U$ is unbiased for $g(\theta)$, and $E_\theta[\operatorname{Var}_\theta(U\mid T)]=0.12$. If $\operatorname{Var}_\theta(U^*)=0.31$, find $\operatorname{Var}_\theta(U)$. (Answer: $0.43$)
In a finite sufficient-statistic model, $P(T=t_1)=1/4$, $P(T=t_2)=3/4$, and $E[U\mid T=t_1]=2$, $E[U\mid T=t_2]=0$. Find $E[U^*]$. (Answer: $1/2$)
An unbiased estimator $U$ has conditional variance $0$ for every possible value of a sufficient statistic $T$. What relation holds between $U$ and $U^*=E[U\mid T]$ almost surely, and what does that imply about their variances? (Answer: $U=U^*$ almost surely, so their variances are equal.)
A statistic $T$ is constant, and the conditional distribution of the data given $T$ changes with $\theta$. Can one invoke the Rao–Blackwell theorem to claim that $E_\theta[U\mid T]$ is a parameter-free improved estimator? (Answer: No; $T$ is not sufficient, and the conditional expectation may depend on the unknown parameter.)
For a sufficient statistic, $\operatorname{Var}(U)=0.40$ and $\operatorname{Var}(E[U\mid T])=0.25$. Find the expected conditional variance. If it is positive, what does that say about the improvement? (Answer: $0.15$; the variance reduction is strict.)

What You'll Learn