Squared, Absolute, and Huber Losses Compared

Three Ways to Penalize a Residual

A loss function translates a prediction error into a numerical penalty. Here the comparison is about the shape of the loss as a function of the residual, not about which prediction minimizes an expected loss. Write \(e=y-a\), where \(y\) is the observed response and \(a\) is a proposed prediction. Positive and negative residuals have opposite signs, but the three losses below penalize their magnitudes symmetrically.

Definition: The squared loss is \(\ell_{\mathrm{sq}}(e)=e^2\), and the absolute loss is \(\ell_{\mathrm{abs}}(e)=|e|\). For a threshold \(\delta>0\), the Huber loss is \(\ell_{\mathrm{H},\delta}(e)=\frac12 e^2\) when \(|e|\leq\delta\), and \(\ell_{\mathrm{H},\delta}(e)=\delta(|e|-\frac12\delta)\) when \(|e|>\delta\). As a loss in the prediction \(a\), each is evaluated at \(e=y-a\).

The factor \(\frac12\) in the Huber definition is conventional: it makes the derivative in the central region equal to \(e\). The threshold \(\delta\) sets the boundary between its quadratic central region and its linear tails. It must be positive and expressed in the same units as the residual.

Worked Penalties at Specific Residuals

Worked Example: Comparing Penalties at Four Residuals

Take \(\delta=1\) and residuals \(e=0.5,1,2,-4\). For the squared and absolute losses, the calculations are direct: at \(e=0.5\), the penalties are \(0.5^2=0.25\) and \(|0.5|=0.5\); at \(e=1\), they are \(1\) and \(1\); at \(e=2\), they are \(4\) and \(2\); and at \(e=-4\), they are \(16\) and \(4\).

For Huber loss, use the quadratic formula when \(|e|\leq1\) and the linear formula otherwise. The penalties are \(\frac12(0.5)^2=0.125\), \(\frac12(1)^2=0.5\), \(1(2-\frac12)=1.5\), and \(1(4-\frac12)=3.5\). The negative residual gives the same penalty as its positive counterpart of equal magnitude.

For the large residuals, the squared loss grows much faster: at magnitude \(4\), it is \(16\), compared with \(4\) for absolute loss and \(3.5\) for Huber loss. This does not make Huber loss bounded; it still increases without limit as the residual magnitude increases.

Shape, Derivatives, and Curvature

All three losses are nonnegative and symmetric about \(e=0\). Their main differences concern how their slopes change. For squared loss, \(\ell'_{\mathrm{sq}}(e)=2e\) and \(\ell''_{\mathrm{sq}}(e)=2\) everywhere. Its slope grows without bound in magnitude as \(|e|\) grows.

Absolute loss has derivative \(1\) for \(e>0\) and \(-1\) for \(e<0\), but no ordinary derivative at \(e=0\). Its graph has a corner at zero and is linear on either side, so its curvature is zero wherever the second derivative exists.

Huber loss is differentiable everywhere. Its derivative, including the two threshold points, is

\ell'_{\mathrm{H},\delta}(e)= \begin{cases} -\delta, & e<-\delta,\\ e, & -\delta\leq e\leq\delta,\\ \delta, & e>\delta. \end{cases}

The formulas agree at \(e=-\delta\) and \(e=\delta\). The second derivative is \(1\) for \(|e|<\delta\), and \(0\) for \(|e|>\delta\); it is not defined at the thresholds because the slope changes there. Thus Huber loss is continuously differentiable but not twice differentiable everywhere. Absolute loss is not differentiable at zero, while squared loss is twice differentiable everywhere.

The following result verifies the convexity comparison and the differentiability claims for Huber loss directly.

Theorem: For every \(\delta>0\), squared loss, absolute loss, and Huber loss are convex functions of \(e\). Huber loss is differentiable everywhere, with the derivative displayed above. It is strictly convex on \([-\delta,\delta]\), but not strictly convex on the whole real line.

Proof. Squared loss has second derivative \(2>0\), so it is convex. Absolute loss is the maximum of the two affine functions \(e\) and \(-e\); the pointwise maximum of affine functions is convex.

For Huber loss, the two pieces have the same value at each join: at \(e=\delta\), both give \(\frac12\delta^2\), and at \(e=-\delta\), both give \(\frac12\delta^2\). Their derivatives also agree at the joins: the derivative is \(-\delta\) at \(-\delta\) and \(\delta\) at \(\delta\). This proves differentiability and the stated derivative formula.

That derivative is nondecreasing: it is constant at \(-\delta\) to the left of \(-\delta\), increases as \(e\) from \(-\delta\) to \(\delta\), and is constant at \(\delta\) to the right. A differentiable function on an interval with nondecreasing derivative is convex, so Huber loss is convex. Its derivative increases strictly within \([-\delta,\delta]\), which gives strict convexity there. But for \(e>\delta\), the loss is affine with slope \(\delta\), so it is not strictly convex on the whole real line. \(\square\)

Convexity also holds as a function of the prediction \(a\) for fixed \(y\). Indeed, substituting \(e=y-a\) composes each convex loss with an affine function of \(a\), which preserves convexity. The derivative with respect to \(a\) is the negative of the derivative with respect to \(e\); the second-derivative values, where they exist, are unchanged because the affine substitution has slope \(-1\).

Bounded Values and Changing Slopes

None of these losses is bounded above on the real line. Squared loss tends to infinity quadratically; absolute loss tends to infinity linearly; and, once \(|e|>\delta\), Huber loss also tends to infinity linearly. They are all bounded below by zero, with their minimum value attained at \(e=0\). Bounded below is not the same as bounded above.

Their slopes tell a different story. The squared-loss derivative is unbounded as the residual grows. The absolute-loss slope has magnitude \(1\) wherever it is defined, and its subgradients at zero lie between \(-1\) and \(1\). Huber’s derivative is bounded in magnitude by \(\delta\), because it equals the residual in the central region and is capped at \(\pm\delta\) in the tails. Therefore, “Huber loss is less sensitive to very large residuals than squared loss” describes its slope or rate of increase in the tails; it does not mean the Huber loss values are bounded.

Key distinction: Boundedness asks whether the loss values have a finite upper bound. Curvature describes how the slope changes. A loss can have unbounded values and bounded slope, as Huber loss does.

Worked Example: Checking the Huber Join and Its Curvature

Let \(\delta=2\). At the positive join \(e=2\), the central formula gives \(\frac12(2)^2=2\), while the tail formula gives \(2(2-\frac12(2))=2\). At \(e=-2\), the central value is also \(2\), and the tail value is \(2(2-1)=2\). The pieces meet in value at both joins.

The derivative is \(e\) in the central region, so its values approaching the joins are \(-2\) and \(2\). In the tails it is respectively \(-2\) and \(2\). The derivative therefore also joins continuously. The curvature is \(1\) for \(-2<e<2\) and \(0\) for \(e<-2\) or \(e>2\); there is no second derivative at the joins. This checks both the smoothness of the slope and the change in curvature.

How the Threshold Changes the Penalty

A larger Huber threshold keeps more residuals in the quadratic region; a smaller one sends them into the linear tails sooner. This makes the threshold a substantive choice, not merely a label. The next example compares two thresholds for identical residuals.

Worked Example: Comparing Two Huber Thresholds

Consider residuals of magnitudes \(1\), \(2\), and \(3\). With \(\delta=1\), the Huber penalties are \(\frac12(1)^2=0.5\), \(1(2-0.5)=1.5\), and \(1(3-0.5)=2.5\). With \(\delta=2\), they are \(\frac12(1)^2=0.5\), \(\frac12(2)^2=2\), and \(2(3-1)=4\). At magnitude \(1\), both thresholds use the same quadratic formula. At magnitude \(2\), the threshold-\(2\) loss is still at the boundary of its quadratic region. At magnitude \(3\), it uses its linear formula.

The threshold-\(2\) penalties are larger at magnitudes \(2\) and \(3\) than the threshold-\(1\) penalties. In this convention, raising the threshold extends the quadratic region and increases the tail slope from \(1\) to \(2\). Comparisons should therefore state the threshold as well as the loss name.

Comparing Penalty Growth

When \(\delta=1\), Huber loss never exceeds absolute loss: for \(|e|\leq1\), \(\frac12e^2\leq|e|\), and for \(|e|>1\), \(|e|-\frac12\leq|e|\). It also never exceeds the squared loss. For \(|e|\leq\delta\), \(\frac12e^2\leq e^2\). For \(|e|>\delta\), the difference between squared loss and Huber loss is

e^2-\delta\left(|e|-\frac12\delta\right) = \frac12\left(|e|-\delta\right)^2 +\frac12 e^2 \geq 0.

Thus the Huber penalty in this convention is no larger than squared loss at any residual. These comparisons do not make the losses interchangeable: their slopes, curvature, and dependence on \(\delta\) differ, and absolute loss and Huber loss use different scales.

Common Mistake

Wrong reasoning: “Huber loss has linear tails, so it is bounded and is not convex.” Both conclusions are incorrect. A linear tail continues to increase without limit, and a change from quadratic to linear does not violate convexity if the slope remains nondecreasing.

Correction: Huber loss is unbounded above, but its derivative is bounded. Its derivative increases through the central region and stays constant in each tail, so it is nondecreasing and the loss is convex. Keep the value of the loss, its slope, and its curvature separate when comparing these properties.

Check Your Understanding

Use the definitions and derivative shapes to answer these questions.

For \(\delta=1\), calculate the Huber loss at residuals \(e=0.4\) and \(e=-3\).
Which of the three losses is not differentiable at zero, and which is differentiable but not twice differentiable at its threshold points?
Explain why an unbounded loss can still have a bounded derivative, using Huber loss as the example.
For fixed \(y\), why does convexity as a function of residual imply convexity as a function of prediction \(a\)?
At what residual magnitudes does Huber loss have positive curvature, and where is its curvature zero?