Selecting Unbiased Starting Estimators

Why the Starting Estimator Matters

A UMVU estimator is not usually guessed in one step. A practical route is to first find any estimator whose expectation equals the target, then condition it on a suitable sufficient statistic. The first choice is a starting estimator: it need not be efficient or even symmetric in the observations. Its main job is to be unbiased and simple enough to condition.

Two useful starting points are event indicators and functions of individual sample observations. An indicator records whether an event occurred, so its expectation is the probability of that event. A sample observation can estimate a target whenever its expected value equals that target. After checking unbiasedness, the key decision is which sufficient statistic to condition on. For a UMVU conclusion, completeness matters as well as sufficiency.

Definition: An estimator \(U\) of a parameter function \(\tau(\theta)\) is unbiased if \(E_\theta[U]=\tau(\theta)\) for every allowed parameter value \(\theta\). A starting estimator is any unbiased estimator chosen as the input to a later conditioning step. If \(T\) is a statistic, the conditioned estimator is \(E_\theta[U\mid T]\).

The conditioning step is useful only after the starting estimator has been checked. In particular, a statistic that looks like a natural estimate may have bias, and conditioning does not in general remove that bias. For a sufficient statistic, the conditional distribution of the sample given the statistic does not depend on the unknown parameter. This lets us compute the conditional expectation from the observed value of the statistic without inserting an unknown parameter into the estimator.

Constructing an Unbiased Starting Estimator

Suppose the target is a probability, such as the probability that a specified pattern occurs in several observations. If \(A\) is that event, then \(\mathbf{1}\{A\}\) equals 1 when \(A\) occurs and 0 otherwise, so

E_\theta[\mathbf{1}\{A\}]=P_\theta(A).

Thus an event indicator is unbiased for the event probability. More generally, if \(g(X_1)\) has expectation \(\tau(\theta)\), then \(g(X_1)\) itself is an unbiased starting estimator. These constructions are often deliberately based on only part of the sample; the conditioning step can then use information from the whole sample.

A useful check is to calculate \(E_\theta[U]\) from the model, not from one observed data set. Unbiasedness is a property across repeated samples at each parameter value. The numerical value of \(U\) for a particular sample is a separate calculation.

Worked Example: Estimating a Bernoulli Pattern Probability

Let \(X_1,\ldots,X_4\) be independent Bernoulli random variables with success probability \(\theta\), where \(0<\theta<1\). Estimate \(\theta^2(1-\theta)\). Start with the indicator \(U=\mathbf{1}\{X_1=1,X_2=1,X_3=0\}\), and condition on \(T=\sum_{i=1}^4 X_i\).

Method and unbiasedness check. By independence, the specified pattern has probability \(\theta\cdot\theta\cdot(1-\theta)=\theta^2(1-\theta)\). Therefore \(E_\theta[U]=\theta^2(1-\theta)\), so \(U\) is an unbiased starting estimator. The total \(T\) is complete and sufficient for this Bernoulli model.

Condition on the total. Given \(T=t\), the \(t\) successes occupy a uniformly selected set of \(t\) of the four positions. The event in \(U\) requires successes in positions 1 and 2 and a failure in position 3; position 4 is unrestricted. If \(t=2\), only the success set \(\{1,2\}\) works, among \(\binom{4}{2}=6\) possible sets. If \(t=3\), the success set must be \(\{1,2,4\}\), among \(\binom{4}{3}=4\) possible sets. For \(t=0,1,4\), the event is impossible. Hence

E[U\mid T=t]= \begin{cases} 1/6, & t=2,\\ 1/4, & t=3,\\ 0, & t=0,1,4. \end{cases}

Check the conditioned estimator’s expectation. Since \(T\sim\operatorname{Binomial}(4,\theta)\),

E[E(U\mid T)] =\frac{1}{6}\binom{4}{2}\theta^2(1-\theta)^2 +\frac{1}{4}\binom{4}{3}\theta^3(1-\theta) =\theta^2(1-\theta)^2+\theta^3(1-\theta) =\theta^2(1-\theta).

The result is unbiased for the target, as required. For an observed sample \((1,0,1,1)\), \(T=3\), so the conditioned estimate is \(1/4\). The raw indicator is 0 for this sample; the conditional estimate uses the information in the total rather than treating this one arrangement as decisive. Common mistake avoided: given \(T=3\), there are four possible locations for the single failure, not six; the six-way count applies to \(T=2\).

Choosing the Statistic to Condition On

Sufficiency and completeness answer different questions. A sufficient statistic retains all the sample information about the parameter in the sense that the conditional distribution of the sample given the statistic is parameter-free. A complete statistic has the additional property that no nonzero function of it has expectation zero for every parameter value.

Definition: A statistic \(T\) is complete for a model if, for every function \(h\) for which the expectations exist, \(E_\theta[h(T)]=0\) for all allowed \(\theta\) implies \(P_\theta(h(T)=0)=1\) for all allowed \(\theta\). A complete sufficient statistic is both complete and sufficient.

When a complete sufficient statistic is available, conditioning an unbiased estimator on it gives an unbiased function of that statistic, and the Lehmann–Scheffé theorem identifies that estimator as the unique UMVU estimator (up to almost-sure equality). If several sufficient statistics are in view, it is therefore important to identify whether the chosen one is complete. Sufficiency alone does not provide the same uniqueness guarantee.

Here is the basic expectation argument. For any integrable starting estimator \(U\), the law of iterated expectation gives

E_\theta[E_\theta(U\mid T)]=E_\theta[U].

Thus conditioning preserves the target expectation. If \(T\) is sufficient, the conditional expectation can be expressed without the unknown parameter. If \(T\) is also complete, the unbiased function of \(T\) is unique: the difference of any two unbiased functions of \(T\) has expectation zero for all parameter values, so completeness forces that difference to be zero almost surely. The Lehmann–Scheffé theorem supplies the UMVU conclusion. These statements assume the relevant expectations exist and that the statistic is complete and sufficient for the stated model.

Match a simple statistic to the target.
For a probability target, try an indicator of an event with that probability. For a moment target, try a function of one observation whose expectation is the target.

Verify unbiasedness across the model.
Compute \(E_\theta[U]\) and show it equals \(\tau(\theta)\) for every allowed parameter value.

Identify a sufficient statistic and check completeness.
Use the model’s sufficient statistic; do not assume that a minimal or convenient sufficient statistic is automatically complete.

Condition and interpret.
Calculate \(E(U\mid T)\) from the conditional sample distribution. If \(T\) is complete sufficient, the resulting unbiased estimator is UMVU.

Poisson Indicators and Conditional Allocation

For independent Poisson observations with a common mean, conditioning on their sum gives a multinomial allocation of that fixed total among the observations. This fact is especially useful when the starting estimator is an indicator about one observation. The allocation probabilities are determined by the sample sizes; the unknown Poisson mean disappears after conditioning.

Worked Example: Estimating a Zero-Count Probability

Let \(X_1,X_2,X_3\) be independent Poisson random variables, each with mean \(\lambda>0\). Estimate \(P_\lambda(X_1=0)=e^{-\lambda}\). Use \(U=\mathbf{1}\{X_1=0\}\) and condition on \(T=X_1+X_2+X_3\).

Method and unbiasedness check. Since \(X_1\sim\operatorname{Poisson}(\lambda)\), \(E_\lambda[U]=P_\lambda(X_1=0)=e^{-\lambda}\). Thus the indicator is unbiased. The total \(T\) is complete sufficient for \(\lambda\) in this independent, common-mean Poisson model.

Conditional calculation. Given \(T=t\), the \(t\) events are allocated among three equally likely observations. Thus \(X_1\mid T=t\sim\operatorname{Binomial}(t,1/3)\), including the degenerate case \(t=0\). Therefore

E[U\mid T=t]=P(X_1=0\mid T=t)=\left(\frac{2}{3}\right)^t.

Verify the expectation. The probability generating function of a Poisson random variable gives \(E[s^T]=\exp(\lambda(s-1))\). Taking \(s=2/3\),

E\left[\left(\frac{2}{3}\right)^T\right] =\exp\left(\lambda\left(\frac{2}{3}-1\right)\right) =e^{-\lambda}.

For observed counts \((0,2,1)\), \(T=3\), and the conditioned estimate is \((2/3)^3=8/27\), or approximately \(0.2963\) to four decimals. The raw indicator equals 1 for these data. Both values are sample estimates; the expectation calculation, not this particular sample, verifies unbiasedness. Common mistake avoided: conditional on the total, the probability that \(X_1=0\) is \((2/3)^t\), not \(e^{-\lambda}\); the latter is the unconditional probability and still contains the unknown parameter.

Using a Sample Observation for a Moment Target

A starting estimator need not be an indicator. For example, if the target is \(E_\theta[g(X_1)]\), then \(g(X_1)\) is unbiased by definition. In a symmetric sample, conditioning on a statistic built from symmetric summaries can sometimes be computed by symmetry: if the sum of several conditional expectations is already fixed, and they are equal, each is the fixed sum divided by the sample size.

Worked Example: Estimating a Normal Second Moment

Let \(X_1,X_2,X_3\) be independent observations from \(N(\mu,\sigma^2)\), with \(\mu\in\mathbb{R}\) and \(\sigma^2>0\) both unknown. Estimate \(\mu^2+\sigma^2\), using \(U=X_1^2\), and condition on \(T=(T_1,T_2)=(\sum_{i=1}^3X_i,\sum_{i=1}^3X_i^2)\).

Method and unbiasedness check. For \(X_1\sim N(\mu,\sigma^2)\), its second moment is \(E[X_1^2]=\operatorname{Var}(X_1)+(E[X_1])^2=\sigma^2+\mu^2\). Hence \(U\) is unbiased for the target. In this normal model with both parameters unknown, \(T\) is complete sufficient.

Condition using symmetry. Given \(T\), the conditional distribution treats the three observation labels symmetrically. Also, \(\sum_{i=1}^3 X_i^2=T_2\) is fixed given \(T\). The three conditional expectations of \(X_i^2\) are therefore equal, and their sum is \(T_2\). It follows that

E[X_1^2\mid T]=\frac{T_2}{3}.

This conditioned estimator is unbiased because

E\left[\frac{T_2}{3}\right] =\frac{1}{3}\sum_{i=1}^3E[X_i^2] =\frac{1}{3}\cdot 3(\mu^2+\sigma^2) =\mu^2+\sigma^2.

For observed values \((1,3,5)\), \(T_2=1^2+3^2+5^2=1+9+25=35\), so the conditioned estimate is \(35/3=11.6667\) to four decimals. The raw estimate is \(X_1^2=1\). The calculation is consistent with the defining conditional identity: the conditioned value averages the squared observations given the symmetric summaries. Common mistake avoided: the target is the raw second moment \(\mu^2+\sigma^2\), not the variance \(\sigma^2\) alone.

Common Selection Errors

Starting with an unverified statistic. A plausible-looking function of the data is not automatically unbiased. Calculate its expectation under the stated model first.
Confusing a realized value with an expectation. An indicator may be 0 or 1 in the observed sample while still having the required target as its repeated-sampling expectation.
Conditioning on a statistic without checking its role. The conditional calculation may be manageable for a sufficient statistic, but completeness is what supports the standard UMVU uniqueness conclusion.
Assuming minimal sufficiency implies completeness. These are distinct properties. Verify completeness using the model’s established results rather than inferring it from minimality.
Leaving an unknown parameter in the final estimator. A conditional expectation given a sufficient statistic should be written as a function of the data and known model structure, not as a formula requiring the unknown parameter.

Key takeaway: Construct an unbiased starting estimator by matching an indicator or sample-observation function to the target’s expectation. Then condition on a sufficient statistic; when that statistic is complete as well, the Lehmann–Scheffé theorem gives the UMVU conclusion. This problem-solving sequence is aligned with the SOA VEE Mathematical Statistics syllabus.

Check Your Understanding

For each problem, identify a starting estimator and find the requested conditional estimator. Answers are provided for checking.

Let \(X_1,\ldots,X_5\) be independent Bernoulli\((\theta)\) observations. Use the indicator that \(X_1=X_2=X_3=1\) to estimate \(\theta^3\). Given \(T=\sum_{i=1}^5X_i=k\), what is its conditional expectation? (Answer: \(\frac{k(k-1)(k-2)}{5\cdot4\cdot3}\) for \(k\geq3\), and \(0\) for \(k=0,1,2\).)
For four independent Bernoulli\((\theta)\) observations, use \(U=\mathbf{1}\{X_1=1,X_2=0\}\) to estimate \(\theta(1-\theta)\). Find \(E[U\mid T=k]\), where \(T\) is the total number of successes. (Answer: \(\frac{k(4-k)}{12}\), for \(k=0,1,2,3,4\).)
Let \(X_1,\ldots,X_4\) be independent Poisson\((\lambda)\) observations. Use \(\mathbf{1}\{X_1=2\}\) to estimate \(e^{-\lambda}\lambda^2/2\). Given \(T=\sum X_i=t\), find the conditional expectation. (Answer: \(\binom{t}{2}(1/4)^2(3/4)^{t-2}\) for \(t\geq2\), and \(0\) for \(t=0,1\).)
Let \(X_1,X_2,X_3\) be independent Uniform\((0,\theta)\) observations, with \(\theta>0\). Since \(E[2X_1]=\theta\), use \(U=2X_1\) to estimate \(\theta\). Given the complete sufficient statistic \(M=\max(X_1,X_2,X_3)=m\), find \(E[U\mid M=m]\). (Answer: \(4m/3\).)
Let \(X_1,\ldots,X_n\) be independent observations with \(E_\theta[X_i^2]=\tau(\theta)\), and suppose \(T_2=\sum_{i=1}^nX_i^2\) is complete sufficient. Use \(U=X_1^2\). What is the conditioned estimator, and why is it UMVU? (Answer: \(T_2/n\); it is unbiased and is the conditional expectation given a complete sufficient statistic, so the Lehmann–Scheffé theorem gives the UMVU conclusion.)

What You'll Learn