Huber Regression and Bounded Residual Influence

Why Use a Loss That Changes With Residual Size?

Least-squares regression penalizes a residual more heavily as its magnitude grows: its loss is proportional to the square of the residual. That steep growth can make a very large response discrepancy exert a strong pull on a fitted line or surface. Least absolute deviations uses a loss that grows linearly at every residual size. Huber regression takes a middle course: it is quadratic near zero and linear for large residuals.

The change in growth matters because the derivative of a loss describes how quickly its contribution changes when its residual changes. Huber loss has a derivative that stops increasing once the residual passes a chosen threshold. This limits an observation’s residual-based score, while still allowing the observation to contribute to the fit. It does not, by itself, guarantee that every kind of influence in a regression is bounded.

Model, Loss, and Tuning Threshold

Suppose there are $n$ observations and a regression model with response $y_i$ and predictor vector $x_i$. Write the fitted value as $x_i^\mathsf{T}\beta$, where $\beta$ is the vector of regression coefficients. For a proposed coefficient vector, the residual for observation $i$ is $r_i(\beta)=y_i-x_i^\mathsf{T}\beta$. Huber regression chooses coefficients by minimizing a sum of losses of these residuals.

Definition: For a threshold $c>0$, the Huber loss of a residual $r$ is $$ \rho_c(r)= \begin{cases} \frac{1}{2}r^2, & |r|\leq c,\\ c|r|-\frac{1}{2}c^2, & |r|>c. \end{cases} $$ The Huber regression objective is $Q_c(\beta)=\sum_{i=1}^n \rho_c(r_i(\beta))$. The threshold $c$ is the residual magnitude at which the loss changes from quadratic to linear.

Here $c$ has the same units as the response and residual. If the response is measured in millimeters, then the residual and $c$ are also measured in millimeters, and the loss is measured in squared millimeters. A threshold is not a cutoff for deleting observations: observations on either side of it remain in the objective.

This definition specifies an optimization criterion, not a probability model. Calculating a Huber fit does not require assuming normally distributed errors. Any later inferential claim would need its own assumptions and justification; the loss alone does not provide them.

How the Loss Changes Across the Threshold

For $|r|\leq c$, the loss is $\frac{1}{2}r^2$, so it has the familiar curved, quadratic shape around zero. For $|r|>c$, the loss is $c|r|-\frac{1}{2}c^2$, a straight line on each side of zero. The constant $-\frac{1}{2}c^2$ makes the two pieces meet at the threshold: both give a loss of $\frac{1}{2}c^2$ when $|r|=c$.

The derivative with respect to the residual is called the residual score function here. Differentiating the two pieces gives

\psi_c(r)=\rho_c'(r)= \begin{cases} r, & |r|\leq c,\\ c\,\operatorname{sign}(r), & |r|>c. \end{cases}

At the two threshold points, the derivative from either side agrees: it is $c$ at $r=c$ and $-c$ at $r=-c$. Thus the loss is continuous and differentiable everywhere. Its slope increases from $-c$ to $c$ in the quadratic region, then stays at $c$ on the positive tail and $-c$ on the negative tail.

Worked Example: Calculating Losses on Both Sides of the Threshold

Suppose residuals are measured in millimeters and the Huber threshold is $c=2$ mm. Calculate the loss for residuals of $1$ mm, $2$ mm, and $3$ mm.

For $r=1$ mm, the residual is inside the threshold, so $ \rho_2(1)=\frac{1}{2}(1)^2=0.5\ \text{mm}^2 $.

For $r=2$ mm, it is exactly at the threshold. Using the quadratic piece gives $ \rho_2(2)=\frac{1}{2}(2)^2=2\ \text{mm}^2 $. The linear piece gives the same value: $2|2|-\frac{1}{2}(2)^2=4-2=2\ \text{mm}^2$.

For $r=3$ mm, it is outside the threshold, so $ \rho_2(3)=2|3|-\frac{1}{2}(2)^2=6-2=4\ \text{mm}^2 $. The loss still grows as the residual grows, but beyond $2$ mm its slope is $2\ \text{mm}^2$ per millimeter of residual, rather than increasing with the residual.

Why the Residual Score Is Bounded

Within the threshold, the score equals the residual, so small residual changes still affect the score smoothly. Outside it, the score is fixed at one of two values. Directly from the piecewise definition,

|\psi_c(r)|\leq c\quad\text{for every residual }r.

This is the precise sense in which Huber loss has bounded residual influence: however large the residual becomes, its derivative with respect to that residual cannot exceed $c$ in magnitude. It does not mean that the loss itself is bounded. In the linear region, the loss continues to grow without limit as $|r|$ grows.

The regression objective also makes clear what this bound says about coefficients. Differentiating $Q_c(\beta)$ with respect to the coefficient vector gives

\nabla_\beta Q_c(\beta)=-\sum_{i=1}^n x_i\,\psi_c(r_i(\beta)).

For one observation, the score contribution to the coefficient objective is $-x_i\psi_c(r_i)$. With its predictor vector $x_i$ held fixed, the residual-dependent factor is bounded in magnitude by $c$. But the whole contribution also depends on $x_i$. A very large predictor value can therefore still produce a large coefficient-score contribution. Bounded residual influence is not a blanket guarantee against leverage or every form of observation influence.

Worked Example: Comparing the Effect of Moderate and Extreme Residuals

Let $c=1.5$ response units. Compare the Huber losses and residual scores for $r=2$ and $r=20$, both in response units.

Both residuals exceed the threshold, so the linear piece applies. For $r=2$, $ \rho_{1.5}(2)=1.5(2)-\frac{1}{2}(1.5)^2=3-1.125=1.875 $ squared response units, and $\psi_{1.5}(2)=1.5$ response units.

For $r=20$, $ \rho_{1.5}(20)=1.5(20)-\frac{1}{2}(1.5)^2=30-1.125=28.875 $ squared response units, and $\psi_{1.5}(20)=1.5$ response units. The extreme residual has a larger loss, but not a larger residual score.

For comparison, the half-squared loss $\frac{1}{2}r^2$ has derivative $r$: its derivatives at these residuals are $2$ and $20$. Huber loss caps that derivative at $1.5$ once the threshold is crossed. This comparison concerns the loss’s response to a residual; it does not show that the two observations would have identical effects on regression coefficients, since their predictor vectors may differ.

A Short Derivation of the Huber Score

The score follows by differentiating each part of the loss. For $-c<r<c$, differentiating $\frac{1}{2}r^2$ gives $r$. For $r>c$, the loss is $cr-\frac{1}{2}c^2$, whose derivative is $c$. For $r<-c$, it is $-cr-\frac{1}{2}c^2$, whose derivative is $-c$. At $r=\pm c$, the adjacent formulas give the same slope, so the derivative exists there as well.

This derivation also justifies the quadratic-to-linear description. The second derivative is $1$ inside the threshold and $0$ in the two tail regions; at the boundary it changes abruptly. The first derivative is nondecreasing across the whole real line, which means the loss is convex. Convexity is useful for optimization, but it does not mean every data set has a unique minimizing coefficient vector.

Worked Example: Finding a Huber Location Fit

Consider five hypothetical measurements, in millimeters: $0,1,1,2,10$. Fit a model with an intercept only, so the fitted value for every observation is a common location $\theta$, and let $c=2$ mm. The objective is $Q_2(\theta)=\sum_i \rho_2(y_i-\theta)$.

Because $r_i=y_i-\theta$, differentiating the objective with respect to $\theta$ gives $ Q_2'(\theta)=-\sum_i\psi_2(y_i-\theta) $. A candidate minimizer therefore satisfies $\sum_i\psi_2(y_i-\theta)=0$.

Try $\theta=1.5$ mm. The residuals are $-1.5,-0.5,-0.5,0.5,8.5$ mm. Their scores are the same for the first four residuals, which are inside the threshold, and $2$ mm for the last residual, which is above it. Thus $ \sum_i\psi_2(r_i)=-1.5-0.5-0.5+0.5+2=0 $. So $Q_2'(1.5)=0$. Since $Q_2$ is convex, this stationary point is a global minimizer.

The total loss at this fit is $1.125+0.125+0.125+0.125+15=16.5$ squared millimeters. The final contribution is $ \rho_2(8.5)=2(8.5)-\frac{1}{2}(2)^2=15\ \text{mm}^2 $. The ordinary mean of these five values is $14/5=2.8$ mm, while the Huber location fit is $1.5$ mm. The example illustrates how the large residual’s score is capped; it does not imply that the large observation has no effect on the fitted location.

Choosing and Interpreting the Threshold

The threshold determines where residuals stop receiving a score proportional to their size. A larger $c$ leaves more residuals in the quadratic region; a smaller $c$ moves more residuals into the linear region. If $c$ is at least as large as every residual magnitude at a proposed fit, the objective at that fit is the half-squared-error objective. In general, changing $c$ can change the minimizing coefficients as well as which residuals lie in each region.

The threshold must be considered in the response’s units. For example, a threshold of $2$ means something different if responses are recorded in millimeters rather than meters. Rescaling the response requires rescaling the threshold by the same factor to preserve the same division between the quadratic and linear regions. A threshold is a tuning choice, not a universal outlier boundary, and a residual beyond it is not automatically erroneous or removable.

Huber regression is useful when a squared-error fit would be highly responsive to a few large residuals, but the analyst still wants those observations to contribute. Its defining compromise is exact: small residuals retain a quadratic penalty, while large residuals retain a linear penalty and a capped residual score. Interpret that property narrowly. It limits the residual part of a score for a fixed predictor vector; it does not resolve unusual predictors, model misspecification, or the substantive question of whether an extreme case belongs in the analysis.

Common Mistakes

Calling the loss bounded. The loss grows linearly without limit in the tails. It is the derivative with respect to the residual that is bounded.
Treating the threshold as a deletion rule. A residual above $c$ still contributes $c|r|-\frac{1}{2}c^2$ to the objective. It is not discarded.
Claiming all influence is bounded. The residual score is capped, but a regression observation’s coefficient-score contribution also contains its predictor vector.
Using a threshold without its units. State the response or residual units for $c$, or explain the scale on which residuals were defined.

Key takeaway: Huber loss is quadratic when $|r|\leq c$ and linear when $|r|>c$. Its residual derivative equals $r$ near zero and is capped at $\pm c$ in the tails. The loss remains unbounded, and bounded residual influence is not the same as bounded influence from every aspect of a regression observation.

Check Your Understanding

Answer each question using the definition and score of Huber loss.

For $c=3$, calculate the Huber loss at $r=2$ and at $r=5$. Include the units if residuals are measured in seconds.
What is the residual score at $r=-8$ when $c=2$, and why does its magnitude not continue to increase?
Explain why the Huber loss is not bounded even though its residual score is bounded.
For a regression observation with a fixed predictor vector $x_i$, identify the part of its coefficient-score contribution that is bounded as the residual varies.
Why must a threshold be interpreted in the response or residual scale’s units?