Rao–Blackwellization of Bernoulli Estimators

Why Condition a Single Bernoulli Observation?

Suppose the quantity of interest is a success probability $p$, and a sample contains several independent observations of the same success-or-failure outcome. The first observation alone gives an unbiased estimate of $p$, but it uses only one data point. Conditioning that observation on the total number of successes uses the sample’s information more effectively: the resulting estimate is the sample proportion.

This is a concrete application of the Rao–Blackwell theorem. The key work here is finding the conditional expectation correctly and checking what it does to the estimator’s variance. The setup and notation below make explicit which quantity is conditioned on; conditioning on the wrong information can remove the improvement.

Model and definition: Let $X_1,\ldots,X_n$ be independent, identically distributed Bernoulli random variables with $P(X_i=1)=p$ and $P(X_i=0)=1-p$, where $0\leq p\leq 1$ and $n\geq 1$. A success is coded as 1 and a failure as 0. The target is $p$, and the starting estimator is $T=X_1$. It is unbiased because $E[X_1]=p$. Define the total number of successes by $S=\sum_{i=1}^{n}X_i$.

The total $S$ has a binomial distribution with parameters $n$ and $p$. In this model, the distribution of the observations given $S$ does not depend on $p$: once the total is fixed at $s$, all arrangements with $s$ successes are equally likely. This makes $S$ a sufficient statistic for $p$, and it provides the appropriate statistic on which to condition $T$.

Deriving the Conditioned Estimator

Fix a possible total $s$, where $s$ is an integer from 0 through $n$. There are $\binom{n}{s}$ possible arrangements of $n$ observations with exactly $s$ successes. Among them, the arrangements with $X_1=1$ must place the other $s-1$ successes among the remaining $n-1$ observations, giving $\binom{n-1}{s-1}$ arrangements. Thus, for $s\geq 1$,

P(X_1=1\mid S=s) =\frac{\binom{n-1}{s-1}}{\binom{n}{s}} =\frac{s}{n}.

If $s=0$, then $X_1=0$ with certainty, so the same result, $P(X_1=1\mid S=0)=0=s/n$, holds. Since $X_1$ takes only the values 0 and 1, its conditional expectation is its conditional probability of being 1. Therefore:

E[X_1\mid S=s]=\frac{s}{n}, \qquad E[X_1\mid S]=\frac{S}{n}.

The conditioned estimator is consequently the sample proportion of successes. The calculation depends on the exchangeability of the observations: given their total, each observation has the same conditional chance of being one of the $s$ successes.

Result: Under the independent, identically distributed Bernoulli model, Rao–Blackwellizing $T=X_1$ with respect to $S$ gives $$ T^*=E[T\mid S]=\frac{S}{n}=\overline{X}. $$ The estimator $T^*$ is unbiased for $p$, and its variance is no greater than the variance of $T$.

Why the Variance Cannot Increase

The variance comparison follows from the law of total variance. For any estimator $T$ with finite variance and any statistic $S$,

\operatorname{Var}(T) = E[\operatorname{Var}(T\mid S)] + \operatorname{Var}(E[T\mid S]).

Both terms on the right are nonnegative. The second is the variance of the conditioned estimator, so $\operatorname{Var}(E[T\mid S])\leq\operatorname{Var}(T)$. In this Bernoulli example, the comparison can also be calculated directly. Since $\operatorname{Var}(X_i)=p(1-p)$, independence gives $\operatorname{Var}(S)=np(1-p)$. Hence:

\operatorname{Var}(T)=p(1-p), \qquad \operatorname{Var}(T^*)= \operatorname{Var}\left(\frac{S}{n}\right) =\frac{np(1-p)}{n^2} =\frac{p(1-p)}{n}.

For $n>1$ and $0<p<1$, the variance falls by the factor $1/n$, a strict reduction. The variance difference is $p(1-p)(1-1/n)$. If $n=1$, conditioning makes no change. At $p=0$ or $p=1$, both estimators have variance zero, so there is no strict improvement. Unbiasedness is preserved because iterated expectation gives $E[T^*]=E[E[T\mid S]]=E[T]=p$.

Specify the starting estimator.
Use $T=X_1$, which is unbiased for $p$ under the stated Bernoulli model.

Choose the statistic to condition on.
Use the total $S=\sum_{i=1}^{n}X_i$, not the entire ordered sample.

Find the conditional expectation.
Given $S=s$, the chance that the first observation is one is $s/n$, so $E[X_1\mid S]=S/n$.

Check unbiasedness and variance.
Iterated expectation gives unbiasedness; the binomial variance gives $\operatorname{Var}(S/n)=p(1-p)/n$.

Worked Applications

Worked Example: Estimating a Hypothetical Defect Probability

A hypothetical inspection process records whether each tested component has a specified defect. Assume $n=8$ tested components have independent, identically distributed defect indicators $X_i\sim\operatorname{Bernoulli}(p)$, where $p$ is the unknown defect probability. The observed outcomes contain three defects, and the first component is not defective. Find the original estimate and its Rao–Blackwellized version. Then compare their variances if $p=0.20$.

Method. Start with $T=X_1$. The recorded first outcome is zero, so the original estimate is $T=0$. The total is $S=3$, and conditioning on that total gives:

T^*=E[X_1\mid S=3]=\frac{3}{8}=0.375.

For the variance comparison, $p(1-p)=0.20(0.80)=0.16$. Therefore:

\operatorname{Var}(T)=0.16, \qquad \operatorname{Var}(T^*)=\frac{0.16}{8}=0.02.

The reduction is $0.16-0.02=0.14$, or $0.14/0.16=0.875$, which is 87.5%. The conditioned estimate is $0.375$, consistent with the observed fraction $3/8$; the variance calculation also agrees with the factor $1/n$. Common mistake avoided: the first outcome being zero does not mean the conditioned estimate is zero. The estimator conditions on the total, which also reflects the other seven observations.

Worked Example: Counting Arrangements Given the Total

In a hypothetical screening study, $n=5$ independent participants each have a positive or negative result, with positive-result indicators $X_i\sim\operatorname{Bernoulli}(p)$. Given that exactly two results are positive, calculate the conditional probability that the first result is positive. Then give the Rao–Blackwellized estimate of $p$ for this observed total and compare the two estimators’ variances when $p=0.35$.

Method. With two positives among five participants, there are $\binom{5}{2}=10$ possible arrangements. If the first result is positive, one more positive must be placed among the other four participants, giving $\binom{4}{1}=4$ arrangements. Thus:

P(X_1=1\mid S=2)=\frac{\binom{4}{1}}{\binom{5}{2}} =\frac{4}{10}=0.4.

Because $X_1$ is an indicator, this probability is $E[X_1\mid S=2]$. The conditioned estimate is $0.4=2/5$. For $p=0.35$, $p(1-p)=0.35(0.65)=0.2275$, so:

\operatorname{Var}(T)=0.2275, \qquad \operatorname{Var}(T^*)=\frac{0.2275}{5}=0.0455.

The variance ratio is $0.0455/0.2275=0.20=1/5$, as required. Common mistake avoided: using four as the denominator counts only arrangements with a positive first result. The conditional probability’s denominator must count all ten arrangements consistent with $S=2$.

Worked Example: Checking a Claimed Variance Reduction

A hypothetical reliability audit uses $n=12$ independent component-failure indicators $X_i\sim\operatorname{Bernoulli}(p)$. The observed sample contains five failures. For a model value $p=0.25$, calculate the Rao–Blackwellized estimate, the variances of $X_1$ and the conditioned estimator, and the percentage reduction in variance.

Method. The conditioned estimate uses the total count, regardless of whether the first observed component failed:

T^*=\frac{S}{n}=\frac{5}{12}=0.416666\ldots.

For the variance calculation, $p(1-p)=0.25(0.75)=0.1875$. Thus:

\operatorname{Var}(X_1)=0.1875, \qquad \operatorname{Var}(T^*)=\frac{0.1875}{12}=0.015625.

The absolute variance reduction is $0.1875-0.015625=0.171875$. The percentage reduction is:

\frac{0.171875}{0.1875}\times100\% =91.6666\ldots\%.

Rounded to one decimal place, the reduction is 91.7%. As a check, the variance ratio is $0.015625/0.1875=1/12$, so the reduction is $1-1/12=11/12$, the same percentage. Common mistake avoided: the observed estimate $5/12$ and the model-based variance use different inputs. The estimate uses the observed total; the variance uses the specified model value $p=0.25$.

Exam Traps and Interpretation

A frequent error is to condition on all the observations rather than on the total. Given the full ordered sample $(X_1,\ldots,X_n)$, the value of $X_1$ is already known. Consequently:

E[X_1\mid X_1,\ldots,X_n]=X_1,

which simply returns the original estimator and gives no variance reduction. The useful conditioning is on $S$, which retains the number of successes but not their positions. Given $S=s$, the locations remain uncertain, and the conditional chance that any particular observation is a success is $s/n$.

Also distinguish the conditional estimate from the unknown parameter. Once data are observed, $S/n$ is a numerical estimate; before sampling, it is a random variable with expectation $p$. The variance formula $p(1-p)/n$ describes repeated samples under the assumed model, not the variability of a fixed observed value.

Key takeaway: For independent, identically distributed Bernoulli observations, conditioning the unbiased single-observation estimator $X_1$ on the total number of successes produces $S/n$. This estimator remains unbiased and, for $n>1$ and $0<p<1$, has strictly smaller variance by the factor $1/n$.

Check Your Understanding

For each problem, use the stated independent, identically distributed Bernoulli model and condition on the total $S$.

A sample of $n=4$ binary outcomes contains three successes. What is $E[X_1\mid S=3]$? (Answer: $3/4=0.75$)
For $n=9$ and $p=0.20$, find the variance of $X_1$ and the variance of its Rao–Blackwellized estimator. (Answer: $0.16$ and $0.16/9=0.0178$, respectively, to four decimals)
For $n=6$, exactly two outcomes are successes. Using arrangements, what is $P(X_1=1\mid S=2)$? (Answer: $\binom{5}{1}/\binom{6}{2}=5/15=1/3$)
For $n=10$ and $p=0.40$, calculate the absolute and percentage variance reductions from $X_1$ to $S/10$. (Answer: $0.216$ and $90\%$)
A sample of $n=8$ outcomes has $S=0$. Give the conditioned estimate and explain the conditional value of $X_1$. (Answer: estimate $0$; given $S=0$, $X_1=0$ with probability 1)

What You'll Learn