Discrete Parameter Estimation with Absolute-Error Loss

Why Medians Matter Under Absolute-Error Loss

When an estimate is penalized by its absolute distance from the true parameter, the posterior mean is not generally the Bayes action. The relevant summary is a posterior median. In finite problems, the posterior can put exactly half its mass on either side of an interval, so there may be more than one Bayes action. That flexibility can matter: among posterior medians, an action can sometimes be selected to balance the estimator’s risks across parameter states.

The aim here is to connect that calculation to minimax certification. We will first identify posterior medians, then calculate statewise risks for explicit models, and finally use prior-weighted risk to verify a minimax rule.

Definition: Let \(\Theta\) be a finite subset of the real line, let \(Y\) have a finite sample space, and let the action \(a\) be a real-valued estimate of \(\theta\in\Theta\). Under absolute-error loss, \(L(\theta,a)=|a-\theta|\). A rule \(\delta\) maps each observation \(y\) to an action \(\delta(y)\), and its risk at state \(\theta\) is \(R(\theta,\delta)=E_\theta[|\delta(Y)-\theta|]\). Given a prior and an observation with positive prior-predictive probability, a posterior median is any action minimizing the posterior expected absolute loss.

Posterior Medians: Definition and Characterization

For a fixed observation, write the posterior probability at state \(\theta_j\) as \(w_j=P(\theta=\theta_j\mid Y=y)\), where the \(w_j\) are nonnegative and sum to one. The posterior expected loss of action \(a\) is

m(a)=\sum_j w_j|a-\theta_j|.

A posterior median is an action with at least half the posterior mass at or below it and at least half at or above it. In terms of a posterior distribution function \(F(\,\cdot\mid y)\), this means \(F(a\mid y)\geq 1/2\) and \(F(a-\mid y)\leq 1/2\). The set of medians can be an interval: if the cumulative posterior mass reaches exactly one half between two consecutive parameter values, every action in that gap is a median.

Result: For a finite posterior distribution on the real line, an action \(a\) minimizes \(m(a)\) if and only if \(F(a-\mid y)\leq 1/2\leq F(a\mid y)\). Thus posterior medians, not necessarily posterior means, are the Bayes actions under absolute-error loss.

To justify the characterization, \(m\) is a convex, piecewise-linear function. At an action \(a\) that is not one of the posterior support points, its slope is the posterior mass below \(a\) minus the posterior mass above \(a\), or \(2F(a\mid y)-1\). At a support point, the left and right slopes are \(2F(a-\mid y)-1\) and \(2F(a\mid y)-1\). A convex piecewise-linear function is minimized exactly where its left slope is nonpositive and its right slope is nonnegative. These two inequalities are precisely \(F(a-\mid y)\leq 1/2\leq F(a\mid y)\). This also explains why a whole interval may minimize the posterior loss when the cumulative mass is exactly one half.

Worked Example: A Posterior Median Need Not Be the Mean

Suppose there is no informative observation, and the prior assigns probabilities \(0.2, 0.5, 0.3\) to parameter values \(0,1,4\), respectively. The posterior is therefore the same as this prior. Find its posterior median and calculate the risks of using that fixed estimate at each state.

The cumulative probabilities at \(0,1,4\) are \(0.2,0.7,1\). The first value at which the cumulative probability is at least \(1/2\) is \(1\), so the unique posterior median is \(1\). Its posterior expected absolute loss is

$$ 0.2|1-0|+0.5|1-1|+0.3|1-4| =0.2+0+0.9=1.1. $$

The posterior mean, by contrast, is \(0.2(0)+0.5(1)+0.3(4)=1.7\); it is not a posterior median. The fixed rule \(\delta=1\) has statewise risks \(R(0,\delta)=|1-0|=1\), \(R(1,\delta)=0\), and \(R(4,\delta)=3\). Their prior-weighted average is \(0.2(1)+0.5(0)+0.3(3)=1.1\), matching the posterior expected loss.

From Bayes Actions to a Minimax Certificate

A posterior median minimizes conditional expected loss, but that fact alone does not say that its risks are balanced across parameter values. To make the minimax connection, calculate the statewise risks \(R(\theta,\delta)\) for a candidate rule, not just its average under a prior. The following finite-problem criterion makes the required comparison precise.

Theorem: Let \(\pi\) be a prior on a finite parameter set, and let \(\delta^*\) be a Bayes rule for \(\pi\), with Bayes risk \(r_\pi\). If \(R(\theta,\delta^*)\leq r_\pi\) for every state \(\theta\), then \(\delta^*\) is minimax and its maximum risk is \(r_\pi\). In particular, it is sufficient that every state with positive prior probability have risk \(r_\pi\), and every state outside the prior’s support have risk at most \(r_\pi\).

For any rule \(\delta\), its maximum risk is at least its prior-weighted risk:

\max_{\theta\in\Theta} R(\theta,\delta) \ \geq\ \sum_{\theta\in\Theta}\pi(\theta)R(\theta,\delta) \ \geq\ r_\pi.

The second inequality holds because \(r_\pi\) is the smallest prior-weighted risk among all rules. Thus no rule can have maximum risk below \(r_\pi\). For \(\delta^*\), the theorem’s risk bound gives \(\max_\theta R(\theta,\delta^*)\leq r_\pi\), while its prior-weighted risk is \(r_\pi\), so its maximum risk must equal \(r_\pi\). It attains the lower bound and is minimax. \(\square\)

Worked Example: A Symmetric Channel Gives Equal Risks

Let \(\Theta=\{0,2\}\), and let the observation be \(H\) or \(T\), with \(P(H\mid\theta=0)=0.2\) and \(P(H\mid\theta=2)=0.8\). Use the prior \(\pi(0)=\pi(2)=1/2\). Find a posterior-median rule and check whether it is minimax.

After observing \(H\), the two unnormalized posterior probabilities are \(0.5(0.2)=0.1\) for \(\theta=0\) and \(0.5(0.8)=0.4\) for \(\theta=2\). The posterior probability of \(\theta=2\) is \(0.4/(0.1+0.4)=0.8\), so the unique median is \(2\). After \(T\), the corresponding probabilities are \(0.4\) and \(0.1\), giving posterior probability \(0.2\) on \(\theta=2\); the unique median is \(0\). Take \(\delta(H)=2\) and \(\delta(T)=0\).

At \(\theta=0\), only \(H\) causes loss, which is \(2\), so \(R(0,\delta)=0.2(2)=0.4\). At \(\theta=2\), only \(T\) causes loss, also \(2\), so \(R(2,\delta)=0.2(2)=0.4\). The rule is Bayes under the stated prior and both statewise risks equal its Bayes risk, \(0.4\). The theorem gives a lower bound of \(0.4\) on the maximum risk of every rule, while this rule attains maximum risk \(0.4\). It is therefore minimax.

Using a Posterior-Median Tie to Balance Risks

A tie in the posterior-median condition can be useful for risk balancing. The posterior expected loss is then flat across an interval of actions, so choosing a particular point in that interval does not change the Bayes risk. It can, however, change the statewise risks. The next example selects a prior weight that creates such a tie, then uses a median from the tied interval to obtain equal risks.

Worked Example: A Nonuniform Prior and an Equal-Risk Median

Let \(\Theta=\{0,2\}\), with observations \(A\) and \(B\). Suppose \(P(A\mid0)=0.8\) and \(P(A\mid2)=0.3\), so \(P(B\mid0)=0.2\) and \(P(B\mid2)=0.7\). Choose a prior that makes the posterior at \(A\) a tie, and find a Bayes rule with equal statewise risks.

Write \(\pi(2)=p\), so \(\pi(0)=1-p\). A posterior tie at \(A\) requires equal unnormalized probabilities:

0.8(1-p)=0.3p, \qquad 0.8=1.1p, \qquad p=\frac{8}{11}.

Thus \(\pi(2)=8/11\) and \(\pi(0)=3/11\). At \(A\), the two unnormalized posterior probabilities are both \(0.8(3/11)=0.3(8/11)=2.4/11\). Every action in \([0,2]\) is a posterior median. At \(B\), the unnormalized probabilities are \(0.2(3/11)=0.6/11\) for state \(0\) and \(0.7(8/11)=5.6/11\) for state \(2\). The posterior probability on state \(2\) is \(5.6/(5.6+0.6)=28/31>1/2\), so the posterior median is \(2\).

Let the action after \(A\) be \(c\in[0,2]\), and set \(\delta(B)=2\). The risk at state \(0\) is \(0.8c+0.2(2)=0.8c+0.4\). The risk at state \(2\) is \(0.3(2-c)+0.7(0)=0.6-0.3c\). Equalizing gives

0.8c+0.4=0.6-0.3c, \qquad 1.1c=0.2, \qquad c=\frac{2}{11}.

This action is in the tied median interval, as required. Substituting it into both risks gives

R(0,\delta)=0.8\left(\frac{2}{11}\right)+0.4 =\frac{16}{110}+\frac{44}{110} =\frac{6}{11}, \qquad R(2,\delta)=0.3\left(2-\frac{2}{11}\right) =0.3\left(\frac{20}{11}\right) =\frac{6}{11}.

The rule is Bayes under the prior: it uses a posterior median at each observation. Its Bayes risk is the prior-weighted risk, \((3/11)(6/11)+(8/11)(6/11)=6/11\). Both states have risk \(6/11\), so the certificate theorem proves that the rule is minimax, with maximum risk \(6/11\).

What the Risk Calculations Establish

The examples separate three calculations that are easy to confuse. First, posterior probabilities determine which actions are Bayes: under absolute-error loss, these are posterior medians. Second, the sampling distributions determine the statewise risks of the resulting rule. Third, a prior-weighted risk supplies a lower bound on the maximum risk of every rule. A minimax certificate is obtained when a Bayes rule’s statewise risks do not exceed that lower bound.

The prior in the last example was not chosen by averaging the two parameter values. Its weight was selected to make the posterior at one observation exactly balanced. The resulting interval of posterior medians allowed a further choice of action that equalized the risks. The prior-weighted lower bound and the statewise risk calculation then matched. This is why posterior-median ties can be more than a technical detail in finite minimax problems.

Common Pitfall

Wrong reasoning: “The posterior median minimizes posterior expected loss, so the rule must be minimax.” Being Bayes under one prior does not by itself establish minimaxity; its maximum statewise risk might exceed the prior Bayes risk.
Correction: Compute \(R(\theta,\delta)\) at every parameter value. Compare the maximum to the Bayes risk under the candidate prior. The certificate applies when the rule’s risks are all at most that Bayes risk; equal risks on the prior support provide the common case used here.
Another error: Treating a posterior tie as if it forced an arbitrary endpoint action. Every action in the median interval is Bayes, but different choices can have different statewise risks. Calculate those risks before selecting among the tied actions.

Key takeaway: Under absolute-error loss, posterior medians minimize posterior expected loss. To certify minimaxity, pair a Bayes rule with a prior-weighted lower bound and verify that its statewise risks are no larger than that bound.

Check Your Understanding

Use the posterior-median condition and the statewise-risk criterion to answer the following.

A posterior puts probability \(0.4\) at \(0\) and \(0.6\) at \(3\). What is its posterior median?
A posterior puts probability \(1/2\) at each of \(1\) and \(5\). Describe its full set of posterior medians.
For a rule Bayes under a prior, why is its Bayes risk a lower bound on the maximum risk of every rule?
In the nonuniform-prior example, why does choosing \(2/11\) after \(A\) preserve Bayes optimality?
What statewise risk condition completes the minimax certificate when the prior has positive probability at every parameter value?