Binomial Estimation with a Least-Favorable Prior

Why Calculate a Prior’s Weights?

In a binomial estimation problem, the observed count contains the information about the unknown success probability $\theta$. A prior distribution turns that count into a posterior distribution, and under squared-error loss the posterior mean is the Bayes estimator. But a Bayes estimator is not automatically minimax: its risk may be small on average under the prior and much larger at a parameter value the prior scarcely represents.

A useful certificate arises when a prior’s Bayes rule has risk no greater than a common value $r$ everywhere in the parameter set, while its Bayes risk is exactly $r$. The maximum risk and Bayes risk then agree, proving that the rule is minimax and that the prior is least favorable. This tutorial develops that check through explicit finite-sample binomial calculations.

Binomial Model and Posterior-Mean Rule

Definition: Let $X\sim\operatorname{Bin}(n,\theta)$, where $n$ is fixed and $\theta$ belongs to a finite set $\Theta=\{\theta_1,\ldots,\theta_k\}\subseteq[0,1]$. An estimator $\delta$ maps the observed count $X$ to a real estimate of $\theta$. Its squared-error risk is $$ R(\theta,\delta)=E_\theta[(\delta(X)-\theta)^2] =\sum_{x=0}^{n}{n\choose x}\theta^x(1-\theta)^{n-x}(\delta(x)-\theta)^2. $$ A prior assigns weights $\pi_j\geq 0$, with $\sum_j\pi_j=1$, to the parameter values. Its Bayes risk is the prior-weighted risk of its Bayes rule.

For an observed count $x$ with positive prior predictive probability, the posterior weight on $\theta_j$ is proportional to its prior weight times its binomial likelihood:

P(\theta=\theta_j\mid X=x) =\frac{\pi_j {n\choose x}\theta_j^x(1-\theta_j)^{n-x}} {\sum_{\ell=1}^{k}\pi_\ell {n\choose x}\theta_\ell^x(1-\theta_\ell)^{n-x}}.

When the denominator is positive, the posterior-mean rule is therefore

\delta_\pi(x)=E[\theta\mid X=x] =\sum_{j=1}^{k}\theta_jP(\theta=\theta_j\mid X=x).

If an outcome has zero prior predictive probability, the posterior is undefined there; the value assigned to the rule at that outcome does not affect its Bayes risk. To establish a minimax risk bound over all $\theta\in\Theta$, however, the rule’s risk must still be evaluated under every parameter value.

Result: Under squared-error loss, the posterior mean minimizes posterior expected loss. For any posterior distribution with finite second moment and any action $a$, $$ E[(a-\theta)^2\mid X=x] =\operatorname{Var}(\theta\mid X=x)+(a-E[\theta\mid X=x])^2. $$

To verify the identity, write $(a-\theta)=(a-m)+(m-\theta)$, where $m=E[\theta\mid X=x]$. On squaring and taking the conditional expectation, the cross term vanishes because $E[m-\theta\mid X=x]=0$. The remaining terms are $(a-m)^2$ and $E[(\theta-m)^2\mid X=x]$. The latter is the posterior variance, so the expression is minimized at $a=m$. This proves the posterior-mean rule without requiring a special form for the prior. $\square$

Worked Example: Posterior Means from a Two-Point Prior

Let $n=2$, let $\Theta=\{1/5,4/5\}$, and assign prior probability $1/2$ to each value. Find the posterior-mean rule for every possible count.

For $X=0$, the likelihoods are $P(X=0\mid\theta=1/5)=16/25$ and $P(X=0\mid\theta=4/5)=1/25$. The posterior probabilities are therefore $16/17$ and $1/17$, giving $$ \delta_\pi(0)=\frac{16}{17}\frac15+\frac{1}{17}\frac45=\frac{4}{17}. $$

For $X=1$, both likelihoods equal $8/25$, so the posterior probabilities remain $1/2,1/2$ and $\delta_\pi(1)=1/2$. By symmetry, for $X=2$ the posterior probabilities are $1/17,16/17$, giving $\delta_\pi(2)=13/17$.

The risks are equal at the two parameter values. At $\theta=1/5$, the three count probabilities are $16/25,8/25,1/25$, so $$ R(1/5,\delta_\pi) =\frac{16}{25}\left(\frac{3}{85}\right)^2 +\frac{8}{25}\left(\frac{3}{10}\right)^2 +\frac{1}{25}\left(\frac{48}{85}\right)^2 =\frac{306}{7225}. $$

Replacing each count $x$ by $2-x$ and each parameter $\theta$ by $1-\theta$ leaves squared error unchanged, so $R(4/5,\delta_\pi)=306/7225$ as well. Thus the Bayes risk, the average of the two risks under the equal-weight prior, is also $306/7225$; the maximum risk equals the Bayes risk.

Equalizing Risks to Choose Prior Weights

The next calculation shows how the weights themselves can be selected to make a candidate prior useful as a minimax certificate. Take one Bernoulli observation, so $n=1$, and let $\Theta=\{1/4,3/4\}$. Give $\theta=1/4$ prior probability $w$ and $\theta=3/4$ probability $1-w$. At $X=0$, the likelihoods are $3/4$ and $1/4$; at $X=1$, they are $1/4$ and $3/4$. Thus the posterior means are

\delta_\pi(0)=\frac{3w+3(1-w)}{3w+(1-w)}\frac14 \ \text{with the numerator understood as the weighted parameter sum,}

More explicitly, at $X=0$ the posterior mean is

\delta_\pi(0) =\frac{w(3/4)(1/4)+(1-w)(1/4)(3/4)} {w(3/4)+(1-w)(1/4)}.

For equal prior weights $w=1/2$, this simplifies to $3/8$; similarly, $\delta_\pi(1)=5/8$. The risk at $\theta=1/4$ is

R(1/4,\delta_\pi) =\frac34\left(\frac38-\frac14\right)^2 +\frac14\left(\frac58-\frac14\right)^2 =\frac{3}{256}+\frac{9}{256} =\frac{3}{64}.

The other risk is equal by symmetry. Consequently the equal-weight prior’s Bayes risk and the rule’s maximum risk both equal $3/64$. This is an equal-risk certificate, not merely an observation that the model and prior look symmetric.

Worked Example: Solving for a Least-Favorable Prior Weight

Keep $n=1$ and $\Theta=\{0,1/2\}$. Let the prior probability of $\theta=0$ be $w$, and write $m=\delta_\pi(0)$. At $X=0$, the likelihoods under $0$ and $1/2$ are $1$ and $1/2$, respectively; hence $$ m=\frac{(1-w) (1/2)(1/2)}{w+(1-w)/2} =\frac{1-w}{1+w}. $$ At $X=1$, the likelihood under $\theta=0$ is zero, so $\delta_\pi(1)=1/2$.

Under $\theta=0$, $X=0$ with probability one, and therefore $R(0,\delta_\pi)=m^2$. Under $\theta=1/2$, $X=0$ and $X=1$ each have probability $1/2$. The estimate at $X=1$ is exact, giving $$ R(1/2,\delta_\pi)=\frac12\left(\frac12-m\right)^2. $$

Choose the weight to equalize these risks. Since $m$ must be nonnegative, the equation $m^2=\frac12(1/2-m)^2$ gives $$ m=\frac{1/2-m}{\sqrt{2}}, \qquad m=\frac{1}{2(\sqrt{2}+1)}=\frac{\sqrt{2}-1}{2}. $$

The corresponding prior weight is $$ w=\frac{1-m}{1+m}=4\sqrt{2}-5\approx 0.65685. $$ For this prior, both risks equal $$ m^2=\left(\frac{\sqrt{2}-1}{2}\right)^2 =\frac{3-2\sqrt{2}}{4} \approx 0.04289. $$

The Bayes risk is the prior-weighted average of these two equal risks, so it is also $0.04289$. Since the parameter set consists only of these two values, the maximum risk is $0.04289$ as well. The square is essential: $m\approx0.20711$ is the estimation error scale in this calculation, whereas the squared-error risk is $m^2\approx0.04289$.

Why Matching the Two Risks Proves Minimaxity

Theorem: Suppose a prior $\pi$ on a finite parameter set has Bayes rule $\delta_\pi$ and Bayes risk $r_\pi$. If $R(\theta,\delta_\pi)\leq r_\pi$ for every $\theta\in\Theta$, then $R(\theta,\delta_\pi)=r_\pi$ at every parameter value with positive prior weight, and $\delta_\pi$ is minimax with minimax risk $r_\pi$. The prior $\pi$ is least favorable.

Here the Bayes risk is $r_\pi=\sum_{\theta\in\Theta}\pi(\theta)R(\theta,\delta_\pi)$. Since this is a weighted average of risks all bounded above by $r_\pi$, every risk receiving positive weight must equal $r_\pi$; otherwise the average would be strictly smaller. For any estimator $\delta$, its maximum risk is at least its prior-weighted risk, and the latter is at least the minimum prior-weighted risk $r_\pi$. Thus

\sup_{\theta\in\Theta}R(\theta,\delta)\geq \sum_{\theta\in\Theta}\pi(\theta)R(\theta,\delta)\geq r_\pi.

Taking the infimum over $\delta$ shows that the minimax risk is at least $r_\pi$. For $\delta_\pi$, the assumed bound gives $\sup_\theta R(\theta,\delta_\pi)\leq r_\pi$, so the minimax risk is at most $r_\pi$. The two bounds prove equality and establish that $\delta_\pi$ is minimax. Finally, the Bayes risk of any prior is at most the minimax risk, because its minimum average risk cannot exceed the maximum risk of a minimax rule. Since $\pi$’s Bayes risk equals the minimax risk, $\pi$ is least favorable. $\square$

In the two-point example with parameter values $0$ and $1/2$, both parameter values have positive prior weight and both risks equal $r_\pi$, so the theorem applies directly. In larger finite sets, equalizing only the risks on the prior’s support is not enough: risks at all unsupported parameter values must also be checked against $r_\pi$.

Common Pitfall and Practical Check

A common mistake is to calculate a Bayes risk and call it the minimax risk without checking the rule’s risks parameter by parameter. The Bayes risk is an average under one prior; a large risk at a low-weight or zero-weight state can make the maximum risk larger than that average. The correction is to calculate $R(\theta,\delta_\pi)$ for every $\theta\in\Theta$, then compare the maximum with the Bayes risk.

A second error is to confuse an error magnitude with squared-error risk. In the equal-risk example, $m=(\sqrt{2}-1)/2$ is approximately $0.20711$, but the risk is its square, approximately $0.04289$, not half that value. Squaring the exact expression gives $(3-2\sqrt{2})/4$.

Compute the posterior rule.
Multiply each prior weight by its binomial likelihood and normalize to obtain the posterior mean at every possible count.

Calculate each state’s risk.
Use the binomial probabilities under that parameter value and the squared error of the rule at each count.

Compare average and maximum.
Verify that the prior-weighted average is the Bayes risk and that no parameter risk exceeds it. Only then does the least-favorable-prior certificate apply.

These calculations make prior weights more than a device for averaging risk. A well-chosen prior can expose the parameter values that determine the worst-case performance and provide a concise lower bound that meets the Bayes rule’s upper bound. The finite sample size matters throughout: the posterior weights and risks use the exact binomial probabilities, not a large-sample approximation.

Check Your Understanding

Use the posterior-mean and risk calculations developed above.

For $n=1$, $\Theta=\{1/4,3/4\}$, and equal prior weights, what are the posterior means after $X=0$ and $X=1$?
Why does equality of the risks on the prior’s support not by itself prove minimaxity when $\Theta$ has additional values?
In the example with $\Theta=\{0,1/2\}$, explain why the posterior estimate after $X=1$ is $1/2$ for every prior weight $0<w<1$.
State the two risk comparisons needed to conclude that a prior’s Bayes rule is minimax with risk equal to its Bayes risk.
If an equalized error magnitude is $m=(\sqrt{2}-1)/2$, what is the corresponding squared-error risk, exactly and approximately?

Binomial Estimation with a Least-Favorable Prior

What You'll Learn