Why the Loss Function Can Change the Conclusion
An estimator is judged by its risk: its expected loss at each possible parameter value. Changing the loss changes what counts as a costly error. Absolute-error loss charges in proportion to the size of an error; squared-error loss gives relatively greater weight to large errors. Consequently, the same estimator can have different risk comparisons, and even different admissibility status, under the two losses.
We will use a small discrete model so that every risk can be calculated directly. The example also makes an important distinction precise: squaring a loss is an increasing transformation of each individual loss, but it does not generally preserve comparisons between expected losses.
Model, Risk, and Admissibility
Let the parameter space and action space both be \(\{0,1,2\}\). We observe \(X\in\{a,b\}\), with distribution depending on \(\theta\) as follows.
| \(\theta\) | \(P_\theta(X=a)\) | \(P_\theta(X=b)\) |
|---|---|---|
| 0 | 0.6 | 0.4 |
| 1 | 0.5 | 0.5 |
| 2 | 0.4 | 0.6 |
A deterministic estimator is a function \(\delta:\{a,b\}\to\{0,1,2\}\). A randomized procedure may instead choose a random action from \(\{0,1,2\}\) after observing \(X\). We consider both kinds when checking admissibility. The two losses are \(L_1(\theta,d)=|d-\theta|\) and \(L_2(\theta,d)=(d-\theta)^2\).
In this finite model, risk is a finite sum. For a deterministic estimator, write \(\delta(a)=u\) and \(\delta(b)=v\). Then
This follows by partitioning the expectation according to the two possible observations: the loss is \(L(\theta,u)\) on \(X=a\), and \(L(\theta,v)\) on \(X=b\). For a randomized procedure, the same calculation applies with an additional expectation over its action randomization conditional on each observation.
Worked Risk Comparisons
Worked Example: A Squared-Error Dominance
Consider \(\delta(a)=0,\ \delta(b)=2\), which estimates by an endpoint, and \(\gamma(a)=\gamma(b)=1\), which always estimates the midpoint. Calculate their risks under both losses.
Under absolute-error loss, at \(\theta=0\), \(\delta\) has risk \(0.6(0)+0.4(2)=0.8\). At \(\theta=1\), either action of \(\delta\) has error 1, so its risk is 1. At \(\theta=2\), its risk is \(0.4(2)+0.6(0)=0.8\). Thus its absolute-error risk vector, in parameter order \(0,1,2\), is \((0.8,1,0.8)\).
Under squared-error loss, the endpoint errors of size 2 contribute \(2^2=4\). The risks are \(0.6(0)+0.4(4)=1.6\) at \(\theta=0\), \(1\) at \(\theta=1\), and \(0.4(4)+0.6(0)=1.6\) at \(\theta=2\). The squared-error risk vector is \((1.6,1,1.6)\).
The constant estimator \(\gamma\) has absolute-error risk \((1,0,1)\) and squared-error risk \((1,0,1)\): its errors have size 1 at either endpoint and size 0 at the midpoint. It does not dominate \(\delta\) under absolute loss, because its endpoint risks, 1, exceed 0.8. It does dominate \(\delta\) under squared loss: \(1<1.6\) at both endpoints and \(0<1\) at the midpoint.
Worked Example: The Same Error Distribution, Two Different Comparisons
Focus on \(\theta=2\) in the preceding example. The estimator \(\delta\) outputs 0 with probability 0.4 and 2 with probability 0.6. Its error magnitude is therefore 2 with probability 0.4 and 0 with probability 0.6.
Its expected absolute error is \(0.4(2)+0.6(0)=0.8\), while its expected squared error is \(0.4(2^2)+0.6(0^2)=1.6\). The constant estimator \(\gamma=1\) has error magnitude 1 with certainty, so both its expected absolute error and expected squared error are 1.
Thus \(\delta\) has lower risk than \(\gamma\) under absolute loss at this parameter, but higher risk under squared loss. The reversal comes from the relatively large errors that occur with \(\delta\): squaring gives those errors more influence.
Worked Example: A Third Rule and Its Risk Vector
Let \(\eta(a)=0,\ \eta(b)=1\). Under absolute loss, at \(\theta=0\) its risk is \(0.6(0)+0.4(1)=0.4\). At \(\theta=1\), the errors are 1 on \(a\) and 0 on \(b\), giving risk \(0.5(1)+0.5(0)=0.5\). At \(\theta=2\), the errors are 2 on \(a\) and 1 on \(b\), giving risk \(0.4(2)+0.6(1)=1.4\). Its absolute-error risk vector is \((0.4,0.5,1.4)\).
For squared loss, the corresponding risks are \(0.6(0)+0.4(1)=0.4\) at \(\theta=0\), \(0.5(1)+0.5(0)=0.5\) at \(\theta=1\), and \(0.4(4)+0.6(1)=2.2\) at \(\theta=2\). Its squared-error risk vector is \((0.4,0.5,2.2)\).
This rule improves on \(\delta\) at \(\theta=0\) under either loss, but has greater risk at \(\theta=2\) under either loss. It therefore does not dominate \(\delta\) under either criterion. Comparing just one parameter value, or just one estimator’s risk at a time, cannot establish dominance.
Admissibility Can Change with the Loss
The calculations show that \(\delta\) is dominated under squared-error loss. To conclude that its admissibility actually changes, we must also rule out every possible procedure dominating it under absolute-error loss. The next result does this, including for randomized procedures.
Proof. Inadmissibility under squared loss follows from the risk vectors already calculated: \(\gamma\)'s risk is strictly smaller than \(\delta\)'s at all three parameter values.
For absolute loss, suppose a randomized procedure weakly improves on \(\delta\) at every parameter. Let \(A\) denote its random action conditional on \(X=a\), and \(B\) its random action conditional on \(X=b\). Both take values in \(\{0,1,2\}\). At \(\theta=0\), the loss is the action itself, so
At \(\theta=2\), the loss is 2 minus the action, giving
For \(\delta\), each of these endpoint risks is 0.8. A weak improvement at both endpoints must therefore satisfy \(R_1(0)+R_1(2)\leq 1.6\). Adding the displayed expressions gives
Because \(0\leq E[A]\leq2\) and \(0\leq E[B]\leq2\), we also have \(E[A]-E[B]\geq-2\). Hence equality must hold: \(E[A]=0\) and \(E[B]=2\). Since \(A\) and \(B\) take values between 0 and 2, these expectations force \(A=0\) with probability 1 and \(B=2\) with probability 1. The proposed procedure is therefore exactly \(\delta\), and cannot have strictly smaller risk at any parameter. No randomized procedure dominates \(\delta\) under absolute loss. This proves its admissibility under that loss. \(\square\)
Why Increasing the Loss Does Not Preserve Risk Order
For any fixed error magnitude \(e\geq0\), \(e^2\) increases as \(e\) increases. But a risk averages losses over the sampling distribution, and in general \(E[e^2]\) is not determined by \(E[e]\). The first worked example makes the distinction visible: an estimator can have a smaller average absolute error while having a larger average squared error at the same parameter.
The common mistake is to argue that squared loss is just an increasing transformation of absolute loss, and therefore must preserve every risk comparison. That reasoning compares losses at a single error size, not expectations over potentially different error distributions. Here, \(\delta\)'s endpoint error is sometimes 2 and sometimes 0, while \(\gamma\)'s is always 1. The rare larger error matters more after squaring.
Admissibility is consequently always relative to the loss and the class of procedures being considered. In this model, \(\delta\) is admissible under absolute loss but inadmissible under squared loss. Neither calculation says that one loss is universally preferable; each expresses a different penalty for estimation errors.
Check Your Understanding
Use the model and definitions above to answer the following questions.
- For \(\delta(a)=0,\delta(b)=2\), calculate its absolute-error risk at each parameter value directly from the sampling probabilities.
- Why does the constant rule \(\gamma=1\) dominate \(\delta\) under squared loss but not under absolute loss?
- In the admissibility proof, why do endpoint risk inequalities alone force \(E[A]=0\) and \(E[B]=2\)?
- Does a lower risk for one rule at one parameter suffice to establish that it dominates another rule? Explain.
- In what sense is admissibility relative to a loss function and a procedure class?