Least-Favorable Priors in Finite Exponential Families

Why Likelihood Ratios Simplify the Calculation

When the parameter can take only finitely many values, a prior is a vector of weights and the posterior is a reweighting of that vector by the likelihood. In a one-parameter exponential family, the likelihood ratios often depend on the data through a single statistic. This makes it possible to move systematically from a prior to posterior probabilities, from those probabilities to a Bayes rule, and finally from the rule to an optimization over prior weights.

The goal here is to carry out those calculations explicitly. The key is to keep two operations distinct: the posterior calculation uses the sampling model and the prior, while the Bayes action minimizes posterior expected loss. The resulting Bayes risk is then a function of the prior weights.

Finite Subsets of an Exponential Family

Definition: A one-parameter exponential family has densities or mass functions, with respect to a common dominating measure, of the form $$ f_\theta(x)=h(x)\exp\{\eta(\theta)T(x)-A(\theta)\}. $$ Here $T(x)$ is a statistic, $\eta(\theta)$ is the natural parameter, and $A(\theta)$ is the log-partition function. A finite parameter subset is $\{\theta_1,\ldots,\theta_k\}$. A prior on this subset assigns weights $\pi_j\geq 0$, with $\sum_{j=1}^k\pi_j=1$.

For independent observations, the joint density has the same form, with the statistic equal to the sum of the individual statistics and the log-partition term multiplied by the sample size. More generally, the calculations below apply directly whenever the joint model can be written in exponential-family form. Write $f_j(x)=f_{\theta_j}(x)$. At an observation $x$ for which the marginal density is positive, Bayes' rule gives

P(\theta=\theta_j\mid X=x) =\frac{\pi_j f_j(x)}{\sum_{r=1}^k \pi_r f_r(x)} =\frac{\pi_j\exp\{\eta(\theta_j)T(x)-A(\theta_j)\}} {\sum_{r=1}^k\pi_r\exp\{\eta(\theta_r)T(x)-A(\theta_r)\}}.

The common factor $h(x)$ cancels. In particular, for two parameter values with positive likelihoods, posterior odds equal prior odds times a likelihood ratio:

\frac{P(\theta=\theta_2\mid X=x)} {P(\theta=\theta_1\mid X=x)} = \frac{\pi_2}{\pi_1} \exp\!\left\{[\eta(\theta_2)-\eta(\theta_1)]T(x) -[A(\theta_2)-A(\theta_1)]\right\}.

If a likelihood is zero, the weight assigned to that parameter becomes zero in the posterior whenever the marginal density is positive; the normalized-weight formula still applies. Posterior odds are especially useful for deciding when a Bayes rule switches between actions.

Posterior Expected Loss Determines the Bayes Rule

Let $a$ denote an action and $L(\theta_j,a)$ the loss when the true parameter is $\theta_j$. For observed data $x$, the posterior expected loss of action $a$ is

\rho(a\mid x)=\sum_{j=1}^k P(\theta=\theta_j\mid X=x)L(\theta_j,a).

Theorem: Suppose the action space is finite, the losses are finite, and the posterior probabilities are measurable functions of the data. A Bayes rule chooses, at each $x$, an action minimizing $\rho(a\mid x)$. Ties may be resolved by any fixed ordering of the actions.

Proof. For any measurable rule $\delta$, its prior-averaged risk is the expectation, under the prior predictive distribution of $X$, of $\rho(\delta(X)\mid X)$. At every data value, a posterior-minimizing action has posterior expected loss no greater than that of $\delta(X)$. Integrating this pointwise inequality shows that the minimizing rule has prior-averaged risk no greater than any other rule. Because the action space is finite, one can select the first minimizing action in a fixed ordering; the comparisons between finitely many measurable posterior losses ensure this selection is measurable. $\square$

Under squared-error loss $L(\theta,a)=(\theta-a)^2$, the minimizing action is the posterior mean. To see this, let $m(x)=E[\theta\mid X=x]$. Expanding the square gives

E[(\theta-a)^2\mid X=x] =E[(\theta-m(x))^2\mid X=x]+(a-m(x))^2.

The cross term vanishes because $E[\theta-m(x)\mid X=x]=0$, so the expression is minimized at $a=m(x)$. Under zero-one loss for a two-state classification problem, the Bayes rule instead chooses the state with the larger posterior probability.

Worked Example: A Poisson Estimate from Posterior Odds

Let $X\sim\operatorname{Poisson}(\lambda)$, with $\lambda\in\{1,3\}$. Assign prior probability $0.4$ to $\lambda=1$ and $0.6$ to $\lambda=3$. Under squared-error loss for estimating $\lambda$, calculate the Bayes estimate when $x=0$ and when $x=2$.

The Poisson mass is $f_\lambda(x)=e^{-\lambda}\lambda^x/x!$. Thus the likelihood ratio of $\lambda=3$ against $\lambda=1$ is

\frac{f_3(x)}{f_1(x)} =e^{-2}3^x.

Let $q(x)=P(\lambda=3\mid X=x)$. Posterior odds are prior odds $0.6/0.4$ multiplied by this likelihood ratio, so

q(x)=\frac{0.6e^{-2}3^x}{0.4+0.6e^{-2}3^x}, \qquad \hat{\lambda}_{B}(x)=1\{1-q(x)\}+3q(x)=1+2q(x).

At $x=0$, $e^{-2}\approx0.1353353$, so $q(0)=0.6e^{-2}/(0.4+0.6e^{-2})\approx0.1687468$. Therefore $\hat{\lambda}_{B}(0)=1+2q(0)\approx1.3374937$, or approximately 1.3375.

At $x=2$, $e^{-2}3^2\approx1.218017$, so $q(2)=0.6e^{-2}9/(0.4+0.6e^{-2}9)\approx0.6462714$. Hence $\hat{\lambda}_{B}(2)=1+2q(2)\approx2.2925427$, or approximately 2.2925. The posterior estimate moves toward $3$ as the observation makes the likelihood ratio in favor of $3$ larger.

Worked Example: Optimizing Prior Weights for Bernoulli Estimation

Observe one Bernoulli random variable $Y$, with success probability $\theta\in\{1/4,3/4\}$. Let $p=P(\theta=3/4)$, and estimate $\theta$ under squared-error loss. Find the Bayes rule and the value of $p$ that maximizes the Bayes risk.

For $Y=1$, the probability of success is $(1+2p)/4$, and the posterior probability of $\theta=3/4$ is $3p/(1+2p)$. For $Y=0$, the probability of failure is $(3-2p)/4$, and the corresponding posterior probability is $p/(3-2p)$. The posterior-mean Bayes rule is therefore

\delta_p(1)=\frac14+\frac12\frac{3p}{1+2p} =\frac{1+8p}{4(1+2p)},\qquad \delta_p(0)=\frac14+\frac12\frac{p}{3-2p} =\frac{3}{4(3-2p)}.

For a two-point posterior with values separated by $1/2$, its variance is $(1/2)^2q(1-q)=q(1-q)/4$. The minimum posterior squared-error loss is this variance. Weighting by the probabilities of the two observations gives the Bayes risk

\begin{aligned} r(p) &=\frac{1+2p}{4}\cdot\frac14 \frac{3p}{1+2p}\frac{1-p}{1+2p} +\frac{3-2p}{4}\cdot\frac14 \frac{p}{3-2p}\frac{3(1-p)}{3-2p}\\ &=\frac{3p(1-p)}{16} \left(\frac{1}{1+2p}+\frac{1}{3-2p}\right) =\frac{3p(1-p)}{4\{3+4p(1-p)\}}. \end{aligned}

Set $t=p(1-p)$. Then $0\leq t\leq1/4$, with the maximum $t=1/4$ attained at $p=1/2$. The risk is $3t/[4(3+4t)]$, which is strictly increasing in $t$, since its derivative is $9/[4(3+4t)^2]>0$. Thus the maximizing prior weight is $p=1/2$, and the maximum Bayes risk is $3/64$. This calculation optimizes the prior rather than assuming that equal weights must be optimal.

Worked Example: A Binomial Classification Rule and Its Thresholds

Suppose $X\sim\operatorname{Binomial}(2,\theta)$, where $\theta\in\{1/3,2/3\}$. The action is to classify the parameter as low ($1/3$) or high ($2/3$), with loss one for an incorrect classification and zero otherwise. Let $p=P(\theta=2/3)$. Find the rule for $p=0.4$ and calculate its Bayes risk.

For an observed count $x$, the likelihood ratio for high against low is $\{(2/3)/(1/3)\}^x\{(1/3)/(2/3)\}^{2-x}=2^{2x-2}$. Posterior odds of high are $p/(1-p)$ times this ratio. With $p=0.4$, the prior odds are $2/3$. For $x=0,1,2$, the likelihood ratios are respectively $1/4,1,4$, so the posterior odds are $1/6,2/3,8/3$. The Bayes rule chooses high only at $x=2$, because only there do the posterior odds exceed one.

Under $\theta=1/3$, the probability of $X=2$ is $1/9$, which is the classification error probability for that state. Under $\theta=2/3$, the probability of $X\ne2$ is $1-4/9=5/9$. Averaging these errors using the prior weights gives

r(0.4)=\frac35\cdot\frac19+\frac25\cdot\frac59 =\frac{3+10}{45}=\frac{13}{45}\approx0.2889.

The thresholds also follow directly from posterior odds. At $x=0$, choose high when $p/(1-p)\cdot1/4>1$, or $p>4/5$. At $x=1$, the threshold is $p>1/2$; at $x=2$, it is $p>1/5$. Equality gives a posterior tie, where either action is Bayes.

Why the Prior Optimization Is Concave

For a rule $\delta$, write $R_j(\delta)=E_{\theta_j}[L(\theta_j,\delta(X))]$. Its risk averaged under prior weights $\pi=(\pi_1,\ldots,\pi_k)$ is $\sum_j\pi_jR_j(\delta)$. The Bayes risk is the smallest such average over all admissible rules.

Theorem: If the risks $R_j(\delta)$ are finite for every parameter value and admissible rule, then the Bayes risk $$ r(\pi)=\inf_\delta\sum_{j=1}^k\pi_jR_j(\delta) $$ is concave in the prior vector $\pi$.

Proof. Take two prior vectors $\pi,\tilde\pi$ and $0\leq s\leq1$. For any rule $\delta$, its average risk at $s\pi+(1-s)\tilde\pi$ is $s\sum_j\pi_jR_j(\delta)+(1-s)\sum_j\tilde\pi_jR_j(\delta)$. Each of the two sums is at least its infimum over rules. Therefore, for every $\delta$, this risk is at least $s r(\pi)+(1-s)r(\tilde\pi)$. Taking the infimum over $\delta$ preserves the inequality, giving $r(s\pi+(1-s)\tilde\pi)\geq s r(\pi)+(1-s)r(\tilde\pi)$. This is concavity. $\square$

Concavity does not by itself identify the maximizing prior, but it provides useful structure for optimization over the simplex, including its boundary. In a symmetric problem, it can identify a maximizer immediately: if exchanging two states leaves the Bayes risk unchanged, concavity implies that averaging a prior with its exchanged version cannot reduce the risk. The Bernoulli example above gives a direct formula and verifies the maximizing weight explicitly.

Common Pitfall and Practical Use

A frequent mistake is to choose the most likely parameter value and call that the Bayes estimate regardless of the loss. The most likely value is the Bayes action for zero-one classification between parameter states, but not generally for estimation under squared-error loss. In the Poisson example, the posterior probability of $\lambda=1$ at $x=0$ exceeds that of $\lambda=3$; the squared-error Bayes estimate is nevertheless the posterior mean, approximately $1.3375$, not $1$. The action follows from the loss function, not from the posterior probabilities alone.

For a finite subset of an exponential family, a practical calculation proceeds by writing the posterior weights, minimizing posterior expected loss, and then evaluating the resulting Bayes risk as a function of the prior. Likelihood ratios often reduce the posterior step to a simple function of $T(x)$; the loss determines the action step; and concavity helps organize the prior optimization. Keeping these steps separate prevents a common source of errors and makes the final prior-weight calculation checkable.

Check Your Understanding

Work through these questions using the likelihood, posterior, and loss as separate ingredients.

In a finite exponential family, why does the common factor $h(x)$ disappear from posterior probabilities?
For two parameter values, write posterior odds as prior odds multiplied by a likelihood ratio. What part of the exponential-family likelihood determines its data dependence?
Under squared-error loss, why is the posterior mean the Bayes action, while under zero-one classification loss the action is a posterior mode?
In the Bernoulli estimation example, which feature of the risk formula establishes that $p=1/2$ maximizes it?
State the concavity inequality for Bayes risk at a mixture of two prior vectors, and explain why taking an infimum preserves the required inequality.