Rao–Blackwellization in a Geometric Model

Why Condition on the Sample Total?

Suppose repeated independent trials each produce a waiting time until success. A single waiting time is an unbiased estimate of the mean waiting time, but it can vary substantially from one observation to another. If several waiting times are observed, their total contains information about their common mean. Conditioning one observation on that total averages away some of its randomness while preserving its expectation.

This is a direct application of Rao–Blackwellization. In the geometric model, the conditional expectation has a particularly simple form: it is the sample mean. The key step is to identify the conditional distribution correctly; it is not the original geometric distribution, because the observed total is being held fixed.

Model and notation: Let $X_1,\ldots,X_n$ be independent and identically distributed geometric random variables with success probability $p$, where $0<p<1$. Here $X_i$ counts the number of trials up to and including the first success, so $X_i\in\{1,2,3,\ldots\}$ and $$ P(X_i=x)=p(1-p)^{x-1},\qquad x=1,2,3,\ldots. $$ The parameter of interest is the reciprocal success probability, $1/p$, which is the mean number of trials to the first success. Define the sample total $T=\sum_{i=1}^n X_i$ and the sample mean $\bar X=T/n$.

The assumptions matter: the observations must be independent and have the same success probability $p$. The support convention matters, too. This tutorial counts the successful trial, so the smallest possible observation is 1. A different convention that counts failures before success starts at 0 and changes the corresponding conditional formulas.

Conditioning One Observation on the Total

Start with $X_1$, a simple unbiased estimator of $1/p$, because $E[X_1]=1/p$. We want to calculate $E[X_1\mid T]$. For a fixed total $T=s$, the possible sample outcomes are positive-integer vectors $(x_1,\ldots,x_n)$ whose coordinates sum to $s$.

For every such vector, independence gives the same joint probability: $$ P(X_1=x_1,\ldots,X_n=x_n) =p^n(1-p)^{(x_1-1)+\cdots+(x_n-1)} =p^n(1-p)^{s-n}. $$ Thus, conditional on $T=s$, all positive-integer vectors summing to $s$ are equally likely. This cancellation is a consequence of the common success probability and the fixed total; it would not hold in this form if the observations had different success probabilities.

Conditional distribution: For $n\geq 2$, $T=s$ is possible only for $s\geq n$. For $k=1,\ldots,s-n+1$, $$ P(X_1=k\mid T=s) =\frac{\binom{s-k-1}{n-2}}{\binom{s-1}{n-1}}. $$ The denominator counts the positive-integer vectors of length $n$ summing to $s$; the numerator counts the ways to choose the remaining $n-1$ positive observations to sum to $s-k$.

The binomial coefficients count compositions into positive integers. In particular, the number of ways to write $s$ as a sum of $n$ positive integers is $\binom{s-1}{n-1}$: place $n-1$ separators in the $s-1$ gaps between $s$ objects. Fixing $X_1=k$ leaves $s-k$ to be written as a sum of $n-1$ positive integers, giving $\binom{s-k-1}{n-2}$ possibilities.

Why the Conditional Expectation Is the Sample Mean

There is a short symmetry argument that avoids summing the conditional probabilities term by term. Given $T=s$, the coordinates $X_1,\ldots,X_n$ are interchangeable: every positive-integer vector with total $s$ has the same conditional probability. Consequently, their conditional expectations are equal. Their sum, however, is fixed at $s$.

\sum_{i=1}^n E[X_i\mid T=s] =E\left[\sum_{i=1}^n X_i\mid T=s\right] =E[T\mid T=s] =s.

Since the $n$ conditional expectations in the sum are equal, each must be $s/n$. This gives the conditional expectation as a function of the observed total:

Rao–Blackwellized estimator: For $n\geq 1$, $$ E[X_1\mid T]=\frac{T}{n}=\bar X. $$ For $n=1$, this identity holds directly because $T=X_1$; the composition-count formula is needed only when $n\geq 2$.

Worked Example: Finding the Conditional Distribution at a Fixed Total

Suppose three independent geometric observations, each counting trials through the first success, have an observed total of 7. Find the conditional distribution of $X_1$ given $T=7$, and use it to calculate $E[X_1\mid T=7]$. The method is to count the positive-integer compositions; a common mistake is to assign the five possible values of $X_1$ equal probabilities.

Step 1: List the possible values. The other two observations must each be at least 1, so $X_1$ can be $1,2,3,4,$ or $5$.

Step 2: Count the completions for each value. If $X_1=k$, the other two observations must sum to $7-k$. The number of ordered positive pairs with that sum is $7-k-1$. The counts for $k=1,2,3,4,5$ are therefore $5,4,3,2,1$, respectively. The total number of possible triples is $5+4+3+2+1=15$.

Step 3: Form the conditional probabilities. The conditional probabilities are $5/15,4/15,3/15,2/15,1/15$, in order. They are not equal: a small value of $X_1$ leaves more ways to allocate the remaining total to $X_2$ and $X_3$.

Step 4: Calculate the conditional mean. $$ E[X_1\mid T=7] =1\left(\frac{5}{15}\right)+2\left(\frac{4}{15}\right) +3\left(\frac{3}{15}\right)+4\left(\frac{2}{15}\right) +5\left(\frac{1}{15}\right) =\frac{5+8+9+8+5}{15} =\frac{35}{15} =\frac{7}{3}. $$ This checks against the symmetry result $T/n=7/3$. The mistake of treating the five possible values as equally likely would give 3, not the correct conditional expectation.

Unbiasedness and Variance Improvement

The original estimator $X_1$ is unbiased for $1/p$. The conditional expectation remains unbiased by the law of total expectation:

E[\bar X] =E\big[E[X_1\mid T]\big] =E[X_1] =\frac{1}{p}.

To quantify the improvement, under the stated geometric model $E[X_i]=1/p$ and $\operatorname{Var}(X_i)=(1-p)/p^2$. Independence then gives the variance of the sample mean:

\operatorname{Var}(X_1)=\frac{1-p}{p^2}, \qquad \operatorname{Var}(\bar X) =\operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^n X_i\right) =\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(X_i) =\frac{1-p}{np^2}.

The variance is reduced by a factor of $n$ relative to using $X_1$ alone. The general reason conditioning cannot increase variance is the conditional variance identity:

\operatorname{Var}(X_1) =E[\operatorname{Var}(X_1\mid T)] +\operatorname{Var}(E[X_1\mid T]).

Since $E[X_1\mid T]=\bar X$, the second term is $\operatorname{Var}(\bar X)$, and $E[\operatorname{Var}(X_1\mid T)]\geq 0$. Hence $\operatorname{Var}(\bar X)\leq\operatorname{Var}(X_1)$. For $n>1$ and $0<p<1$, the inequality is strict: for example, totals greater than $n$ can be split among the observations in more than one way, so $X_1$ has positive conditional variance for totals that occur with positive probability.

Worked Example: Checking a Conditional Probability and Mean

Four independent geometric observations share an unknown success probability, and their observed total is 12. Find $P(X_1=3\mid T=12)$ and $E[X_1\mid T=12]$. Use composition counts rather than the marginal geometric probabilities; the common factor involving $p$ cancels after conditioning.

Step 1: Count all compatible vectors. The number of ways to write 12 as four positive integers is $$ \binom{12-1}{4-1}=\binom{11}{3}=165. $$

Step 2: Count the vectors with $X_1=3$. The other three observations must be positive integers summing to $12-3=9$. There are $$ \binom{9-1}{3-1}=\binom{8}{2}=28 $$ such triples. Therefore, $$ P(X_1=3\mid T=12)=\frac{28}{165}. $$

Step 3: Find the conditional expectation. Given the total, the four coordinates are exchangeable and sum to 12. Thus, $$ E[X_1\mid T=12]=\frac{12}{4}=3. $$ This is the conditional estimate of $1/p$ for the observed total. The probability $28/165$ and the conditional mean 3 answer different questions: the first is the chance that $X_1$ equals 3, while the second averages all its possible values.

Worked Example: Comparing the Single-Observation and Improved Estimators

A hypothetical process is modeled by five independent geometric waiting times with success probability $p=0.2$, counting the successful trial. The observed waiting times are $2,1,5,3,4$. Compare the single-observation estimator $X_1$ with its Rao–Blackwellized version, and calculate both variances. The method is to condition on the total for the realized estimates and use the geometric variance formulas for the repeated-sampling comparison.

Step 1: Compute the total and the two estimates. The total is $$ T=2+1+5+3+4=15. $$ The original estimate is $X_1=2$. Its conditional expectation given the total is the sample mean: $$ E[X_1\mid T=15]=\bar X=\frac{15}{5}=3. $$ Thus, for these data, conditioning changes the estimate from 2 to 3.

Step 2: Find the target mean. Under $p=0.2$, the reciprocal success probability is $$ \frac{1}{p}=\frac{1}{0.2}=5. $$ Both estimators are unbiased for this value over repeated samples; a particular estimate need not equal 5.

Step 3: Compare variances. Since $p=0.2$, $p^2=0.04$ and $1-p=0.8$. Therefore, $$ \operatorname{Var}(X_1)=\frac{0.8}{0.04}=20, \qquad \operatorname{Var}(\bar X)=\frac{0.8}{5(0.04)}=\frac{0.8}{0.2}=4. $$ The sample mean has one fifth the variance of the single observation, a reduction of $(20-4)/20=0.8$, or 80%. The common mistake is to interpret the realized value 3 as biased because it differs from 5; unbiasedness describes the average over repeated samples, not each observed estimate.

Common Traps and Scope of the Result

The central calculation is compact, but several details can change the answer or invalidate the argument. Check the support, the conditioning event, and the model assumptions before applying the result.

Using equal probabilities for the possible values of $X_1$. Given $T=s$, the possible values of $X_1$ are generally not equally likely. Their probabilities depend on the number of ways the remaining observations can make up $s-X_1$, as in the composition count.
Using the marginal geometric probability after fixing the total. The conditional distribution is obtained by counting compatible vectors. The joint probability of every vector with total $s$ has the same factor $p^n(1-p)^{s-n}$, which cancels in the conditional probabilities.
Changing the geometric support without adjusting the formula. Here observations count trials through success and start at 1. If a variable instead counts failures before success, add 1 to each observation to use this model; in particular, the reciprocal-probability target is then one more than the mean number of failures.
Applying the symmetry argument to nonidentical observations. If observations have different success probabilities, the joint probabilities of vectors with the same total need not be equal. The conditional expectation need not be $T/n$.
Confusing conditional expectation with a conditional probability. $E[X_1\mid T=s]=s/n$ is an average of possible $X_1$ values, not the probability that $X_1=s/n$.

The derivation illustrates Rao–Blackwellization in a discrete model: start with an unbiased estimator, condition on the sample total, and obtain the sample mean. This result is aligned with the SOA VEE Mathematical Statistics syllabus. The argument establishes an unbiased estimator with no larger variance; it does not, by itself, require a claim that the estimator is the unique best unbiased estimator.

Check Your Understanding

For each problem, use the convention that a geometric observation counts trials through the first success. Assume independent observations with a common success probability unless the problem says otherwise.

Two geometric observations have total $T=7$. Find $E[X_1\mid T=7]$ and state the resulting Rao–Blackwellized estimate of $1/p$. (Answer: $7/2=3.5$)
Three observations have total $T=8$. Find $P(X_1=2\mid T=8)$ by counting compositions. (Answer: $5/21$)
Four observed waiting times are $1,3,2,6$. Find the original estimate $X_1$, the conditional estimate $E[X_1\mid T]$, and name the method producing the latter. (Answer: Original estimate $1$; conditional estimate $12/4=3$; Rao–Blackwellization.)
For $n=5$ and known $p=0.4$, calculate the variance of $X_1$ and the variance of the sample-mean estimator. (Answer: $\operatorname{Var}(X_1)=3.75$; $\operatorname{Var}(\bar X)=0.75$)
Three observations have total $T=7$. Find both $P(X_1=2\mid T=7)$ and $E[X_1\mid T=7]$, and explain in one phrase why they are different quantities. (Answer: $P(X_1=2\mid T=7)=4/15$; $E[X_1\mid T=7]=7/3$; one is a probability and the other is a conditional mean.)

What You'll Learn