Least-Favorable Priors in Two-Value Gaussian Scale Models

Why the Prior Weight Matters

When a Gaussian scale parameter can take only two values, a prior assigns probability to each possibility. The posterior mean under squared-error loss depends on that weight, so changing the prior changes the estimator and its risk at both parameter values. The least-favorable weight is the one for which neither state has a higher risk than the other: the two risks are equal.

This is a useful finite-parameter setting in which the Bayes calculation and the minimax calculation meet. We derive the estimator, express the equal-risk condition as an integral equation, and prove that its solution gives a minimax rule. In general, the integral equation does not simplify to an elementary formula for the prior weight; it can nevertheless specify that weight exactly and be solved by numerical integration and a one-dimensional root search.

Model, Prior, and Bayes Estimator

Use $\theta$ for the standard deviation, so the Gaussian scale model is $X\sim N(0,\theta^2)$. Suppose the parameter space is $\{a,b\}$, where $0<a<b$, and the observation is one draw. Squared-error loss for estimating $\theta$ is $L(\theta,\delta)=(\delta-\theta)^2$. Write $f_a$ and $f_b$ for the densities of $X$ at the two parameter values.

Definition: Put prior probability $p$ on $b$ and $1-p$ on $a$, where $0\leq p\leq1$. The posterior probability of $b$, given $X=x$, is $$ q_p(x)=\frac{p f_b(x)}{(1-p)f_a(x)+p f_b(x)}. $$ For $0<p<1$, the posterior mean estimator under squared-error loss is $$ \delta_p(x)=a\{1-q_p(x)\}+bq_p(x)=a+(b-a)q_p(x). $$

The posterior mean minimizes posterior expected squared error: for any posterior distribution with mean $m$, $E[(\theta-d)^2\mid X=x]=\operatorname{Var}(\theta\mid X=x)+(d-m)^2$, which is minimized at $d=m$. Here the posterior has mass only at $a$ and $b$, giving the displayed formula. At $p=0$ or $p=1$, the Bayes estimator is the corresponding constant $a$ or $b$.

Let $R_a(p)$ and $R_b(p)$ denote the frequentist risks of $\delta_p$ when the true scale is $a$ and $b$, respectively. With $d=b-a$, the estimation errors are $d q_p(X)$ at $a$, and $-d\{1-q_p(X)\}$ at $b$. Therefore

R_a(p)=d^2\int_{\mathbb R} f_a(x)q_p(x)^2\,dx,\qquad R_b(p)=d^2\int_{\mathbb R} f_b(x)\{1-q_p(x)\}^2\,dx.

These formulas are the key calculation: the equal-risk weight is a solution of $R_a(p)=R_b(p)$. The densities are strictly positive, and $q_p(x)$ is bounded between zero and one, so both risks are finite.

Worked Example: The Posterior Estimate at an Observation of Zero

Take $a=1$, $b=2$, and assign equal prior probabilities, $p=1/2$. For a single observation, the Gaussian densities at zero are $f_1(0)=1/\sqrt{2\pi}$ and $f_2(0)=1/(2\sqrt{2\pi})$. Thus

q_{1/2}(0) =\frac{(1/2)f_2(0)}{(1/2)f_1(0)+(1/2)f_2(0)} =\frac{1}{3}.

The posterior mean estimate is consequently $$ \delta_{1/2}(0)=1+(2-1)\frac13=\frac43. $$ This is between the two possible scales, as a posterior mean must be. It is not yet established that equal prior weights equalize the frequentist risks; that requires the risk calculation below.

The Equal-Risk Prior

For a Gaussian scale, the density at scale $\theta$ is $$ f_\theta(x)=\frac{1}{\theta\sqrt{2\pi}}\exp\!\left(-\frac{x^2}{2\theta^2}\right). $$ Consequently the likelihood ratio, and hence the posterior probability, can be written explicitly:

\frac{f_b(x)}{f_a(x)} =\frac{a}{b}\exp\!\left\{\frac{x^2}{2}\left(\frac{1}{a^2}-\frac{1}{b^2}\right)\right\}, \qquad q_p(x)=\frac{p\,f_b(x)/f_a(x)}{1-p+p\,f_b(x)/f_a(x)}.

Substituting this $q_p$ into the two risk integrals gives a one-dimensional equation in $p$. The following result establishes that there is exactly one interior solution, and explains why it is least favorable.

Theorem: Suppose $0<a<b$, the two sampling densities $f_a,f_b$ are positive everywhere, and both integrate to one. Under squared-error loss, the Bayes risk for the prior probability $p$ on $b$ is strictly concave on $[0,1]$. It has a unique maximizer $p^*\in(0,1)$, characterized by $R_a(p^*)=R_b(p^*)$. The corresponding Bayes estimator has equal risks at $a$ and $b$, and is minimax over $\{a,b\}$.

Proof. For a fixed observation $x$, the prior-weighted expected loss is minimized by the posterior mean. The minimum value is the posterior variance multiplied by the prior predictive density. For a distribution on $a,b$, that value is

h_p(x)=d^2\frac{p(1-p)f_a(x)f_b(x)} {(1-p)f_a(x)+p f_b(x)}.

Thus the Bayes risk is $B(p)=\int_{\mathbb R}h_p(x)\,dx$. To establish strict concavity, set $A=f_a(x)>0$, $C=f_b(x)>0$, and $D=A+p(C-A)>0$. Direct differentiation gives

\frac{d^2}{dp^2}\left\{\frac{p(1-p)}{A+p(C-A)}\right\} =-\frac{2AC}{\{A+p(C-A)\}^3}<0.

It follows that $h_p(x)$ is strictly concave in $p$ for every $x$, and its integral is strictly concave as well. The integral is finite: $h_p(x)\leq d^2 f_a(x)$ for all $p$, since its denominator is at least $p f_b(x)$ when $p>0$; the endpoint $p=0$ gives zero. The same conclusion follows by continuity at the endpoints.

At $p=0$, the rule is the constant $a$, so its risk under $a$ is zero and its risk under $b$ is $d^2$. At $p=1$, the rule is the constant $b$, with risks $d^2$ under $a$ and zero under $b$. In particular, the Bayes risk is zero at both endpoints and positive in the interior: for $0<p<1$, the integrand $h_p(x)$ is positive everywhere. Continuity and strict concavity therefore give a unique interior maximizer.

It remains to identify its condition. For an interior $p$, differentiating the minimized prior-weighted loss with respect to $p$ gives

$$ B'(p)=R_b(p)-R_a(p). $$

To see why the derivative has this form, before minimization the prior-weighted loss at $x$ is $(1-p)f_a(x)(\delta-a)^2+p f_b(x)(\delta-b)^2$. Its derivative with respect to $p$, holding $\delta$ fixed at the minimizing posterior mean, is $f_b(x)(\delta-b)^2-f_a(x)(\delta-a)^2$. The derivative arising from the change in the minimizing $\delta$ is zero, because the expression is minimized there. Integrating gives the displayed identity. Differentiation under the integral is valid here: $\delta_p\in[a,b]$, and on any closed subinterval of $0<p<1$, the derivative is bounded in absolute value by a constant times $f_a+f_b$, an integrable function. At the interior maximizer $B'(p^*)=0$, so the two risks are equal.

Finally, let their common value be $r$. The estimator’s maximum risk over the two states is $r$. For any other rule, its maximum risk is at least its risk averaged under the prior $p^*$, and that prior-averaged risk is at least the Bayes risk $B(p^*)=r$. Thus no rule has maximum risk smaller than $r$, proving minimaxity and least favorability. $\square$

Worked Example: Specify the Equalizing Weight for Scales One and Two

Let $X\sim N(0,\theta^2)$ with $\theta\in\{1,2\}$, and write $p=P(\theta=2)$. The likelihood ratio is $$ \frac{f_2(x)}{f_1(x)}=\frac12\exp(3x^2/8), $$ so the posterior probability of scale 2 is $$ q_p(x)=\frac{p\exp(3x^2/8)}{2(1-p)+p\exp(3x^2/8)}. $$ The estimator is $1+q_p(x)$. Since $d=1$, the equal-risk weight is the unique $p^*\in(0,1)$ satisfying $$ \int_{\mathbb R}\frac{e^{-x^2/2}}{\sqrt{2\pi}}\, \left\{\frac{p^*e^{3x^2/8}}{2(1-p^*)+p^*e^{3x^2/8}}\right\}^{\!2}dx = \int_{\mathbb R}\frac{e^{-x^2/8}}{2\sqrt{2\pi}}\, \left\{\frac{2(1-p^*)}{2(1-p^*)+p^*e^{3x^2/8}}\right\}^{\!2}dx. $$ The left side is $R_1(p^*)$; the right side is $R_2(p^*)$. This equation, together with the theorem’s uniqueness result, determines the prior weight exactly. For numerical work, evaluate both integrals for a trial $p$, take their difference, and find its zero on $(0,1)$. The equal-prior calculation at $x=0$ does not replace this integration: risks average squared errors over all possible observations, not just the observation zero.

More Than One Observation

With $n$ independent observations $X_1,\ldots,X_n\sim N(0,\theta^2)$, the same posterior-mean and equal-risk arguments apply. The likelihood depends on the observations through $T=\sum_{i=1}^n X_i^2$. For $a=1,b=2$, the likelihood ratio is

\frac{f_2(x_1,\ldots,x_n)}{f_1(x_1,\ldots,x_n)} =2^{-n}\exp(3T/8).

This changes the posterior probability but not the form of the Bayes estimator: $\delta_p=1+q_p(T)$, where $q_p(T)=p e^{3T/8}/\{2^n(1-p)+p e^{3T/8}\}$. For two observations, $T$ has density $\frac12e^{-t/2}$ at scale 1 and $\frac18e^{-t/8}$ at scale 2, for $t>0$. The equal-risk equation is therefore explicitly

\int_0^\infty \frac12e^{-t/2} \left\{\frac{p e^{3t/8}}{4(1-p)+p e^{3t/8}}\right\}^{\!2}dt = \int_0^\infty \frac18e^{-t/8} \left\{\frac{4(1-p)}{4(1-p)+p e^{3t/8}}\right\}^{\!2}dt.

The factor $1/4$ in the likelihood ratio comes from $2^{-n}$ with $n=2$; the second density is the $T/4$ rescaling of a chi-squared random variable with two degrees of freedom. The unique root in $(0,1)$ is the equal-risk prior weight for this two-observation problem. It need not equal the one-observation weight: additional data change the likelihood ratio distribution and hence both risk integrals.

Why Equal Risks Identify the Least-Favorable Prior

The prior-weighted risk is an average of the two state-specific risks. No estimator can have maximum risk below its average risk under any prior. Choosing a prior that maximizes the Bayes risk is therefore a way to obtain a lower bound on the minimax risk. In this two-state problem, strict concavity makes that maximizing prior unique, and the derivative condition says its two active risks are equal. The Bayes estimator then attains the lower bound because its maximum risk is exactly its Bayes risk.

The equal-risk equation is generally an integral equation, not an algebraic equation in $p$. A reliable numerical procedure is to evaluate $R_a(p)-R_b(p)$ by quadrature and use a bracketing root method on $[0,1]$. The endpoint risk differences have opposite signs, and the theorem guarantees a unique interior zero. This provides both a computational check and a precise interpretation of the numerical answer.

Common Mistakes

Using the prior weight itself as the posterior weight. The posterior probability depends on the observed likelihood. Use $p f_b(x)/\{(1-p)f_a(x)+p f_b(x)\}$, not simply $p$.
Confusing scale and variance. Here $\theta$ is the standard deviation, so the model is $N(0,\theta^2)$, and the loss is squared error in $\theta$. Estimating the variance would be a different decision problem.
Equalizing posterior losses at one observation. Frequentist risk is an expectation under each true parameter value. The equation must integrate over the full sampling distribution at $a$ and at $b$.
Assuming equal prior probabilities equalize risks. The weight $p=1/2$ is only a candidate. It is least favorable when it solves the risk equation, not merely because it is symmetric as a prior.
Claiming minimaxity from equal risks alone. The argument also needs the rule to be Bayes for the equalizing prior. Then its common risk equals its Bayes risk, while every rule’s maximum risk is at least its risk averaged under that prior.

Key takeaway: For two Gaussian scales and squared-error loss, the Bayes rule is the posterior mean $a+(b-a)q_p(X)$. The unique interior prior weight solving $R_a(p)=R_b(p)$ maximizes Bayes risk, and the resulting equal-risk Bayes rule is minimax.

Check Your Understanding

Work through these questions to check the estimator, risk equation, and minimax argument.

For scales $a=2$ and $b=5$, write the posterior mean estimator using the posterior probability of $b$.
Why is $R_a(p)$ proportional to an expectation of $q_p(X)^2$, while $R_b(p)$ is proportional to an expectation of $\{1-q_p(X)\}^2$?
In the one-observation model with scales 1 and 2, derive the likelihood ratio $f_2(x)/f_1(x)$.
What endpoint risk differences ensure that the equal-risk equation has an interior solution?
Explain why a Bayes estimator whose risks are equal at both parameter values is minimax in this two-state problem.