Squared, Absolute, And Huber Estimation Losses

One Distribution, Several Sensible Targets

An estimate is not judged in isolation: it is judged by the loss assigned to its error. Squared loss penalizes large errors especially strongly, absolute loss grows at a constant rate with error size, and Huber loss combines a quadratic penalty near zero with a linear penalty in the tails. Consequently, minimizing expected loss need not select the same action under each choice.

We first study a general location problem. Let \(Y\) be a random variable representing the quantity to be summarized, and let \(a\in\mathbb{R}\) be an action. The expected loss of \(a\) is \(E[\rho(Y-a)]\), when this expectation is finite. In an estimation problem, the analogous risk at parameter value \(\theta\) is \(R(\theta,\delta)=E_\theta[\rho(\delta(X)-\theta)]\). The first problem selects a representative location of a distribution; the second evaluates an estimator’s sampling error. The same loss functions apply, but the random quantities and the decision problem must not be confused.

The Three Loss Functions

For an error \(u\), squared loss is \(\rho_2(u)=u^2\), and absolute loss is \(\rho_1(u)=|u|\). We use the following convention for Huber loss with tuning constant \(c>0\):

Huber loss is quadratic for errors no larger than \(c\) in magnitude and linear for larger errors. Its derivative, called the Huber score here, is the clipped function

The score remains bounded, so a very large residual contributes no more than \(c\) to the derivative of the objective. This limits the influence of extreme observations on the minimizing action.

Squared Loss Selects the Mean

Suppose \(E[Y^2]<\infty\), and write \(\mu=E[Y]\). Expanding the square gives an exact decomposition for every action:

The cross-product has expectation zero because \(E[Y-\mu]=0\). The variance term does not depend on \(a\), and the remaining term is minimized uniquely at \(a=\mu\).

The result is not just a convenient formula: it explains why using squared error makes the mean the target. If extreme values make large contributions to squared error, they can pull that target substantially.

Absolute Loss Selects a Median

A median of \(Y\) is any value \(m\) satisfying \(P(Y\leq m)\geq 1/2\) and \(P(Y\geq m)\geq 1/2\). There may be more than one median, especially for a discrete distribution. The complete condition for minimizing expected absolute loss is:

To verify the condition, let \(G(a)=E[|Y-a|]\), a finite convex function. Its left and right derivatives are

These expressions follow by taking the left or right derivative of \(|y-a|\) for each fixed \(y\); the difference quotients are bounded in absolute value by 1, so dominated convergence justifies taking expectations. A point minimizes a convex function exactly when its left derivative is at most zero and its right derivative is at least zero. Applying that criterion gives \(P(Y<a)\leq1/2\leq P(Y\leq a)\), which is precisely the median condition. This also explains why the minimizer may be an interval rather than a single point.

Huber Loss Selects a Clipped-Score Location

Assume \(E[|Y|]<\infty\), and define \(H(a)=E[\rho_c(Y-a)]\). This expectation is finite: Huber loss is bounded above by a constant plus \(c|Y-a|\). Its derivative can be taken inside the expectation because \(\rho_c\) is differentiable and its derivative is bounded in magnitude by \(c\). The chain rule therefore gives

Since Huber loss is convex, any solution of this score equation minimizes the expected loss. A solution exists: \(H\) is continuous and tends to infinity as \(|a|\) tends to infinity, since \(\rho_c(u)\geq c|u|-c^2/2\). Hence it attains a minimum, and differentiability makes the derivative zero at a minimizer. The minimizer need not always be unique. A sufficient condition for uniqueness is that \(Y\) have a density positive everywhere: for any \(a_1<a_2\), there is positive probability that \(Y\) lies in an interval where \(\psi_c(a_2-Y)>\psi_c(a_1-Y)\); thus \(H'\) is strictly increasing.

What Changes in Risk Assessment?

In estimation, a rule \(\delta\) has risk \(R(\theta,\delta)=E_\theta[\rho(\delta(X)-\theta)]\). Changing \(\rho\) changes the penalty assigned to each possible estimation error, and hence changes the risk function used to assess the rule. It may also change which rule is optimal for a decision problem. A rule optimized for squared-error risk is not thereby optimal under absolute or Huber risk.

Keep two questions separate: which action minimizes expected loss? and how much expected loss does a specified estimator incur? The first identifies a target or an optimal action for a stated distribution. The second evaluates an estimator under a sampling model and a stated parameter. The mean, median, and Huber location answer the first question for their respective losses; they do not, by themselves, calculate the sampling risk of an estimator.

Common Mistakes

• Assuming every loss targets the mean. The mean minimizes squared loss, but absolute loss targets medians, and Huber loss uses a clipped-score equation.
• Assuming the median is always unique. Discrete distributions can have an interval of medians, as the four-value example shows.
• Calling Huber loss “the median loss.” Huber loss has bounded score in the tails, but its minimizer generally differs from the median.
• Comparing risk numbers across losses without qualification. Risk is an expectation of the chosen loss; changing the loss changes the quantity being averaged.
• Confusing a distributional location with an estimator’s sampling risk. A location minimizes expected loss over \(Y\); an estimator’s risk averages loss over the sampling distribution at a fixed parameter.