Comparing Residual Losses and Observation Weights

Why Compare Losses and Weights?

Least-squares regression gives every case the same weight in its estimating equations, but the squared loss makes large residuals count much more than small ones. Robust regression changes how residual size contributes to a fit. Comparing the loss curves shows how rapidly each method penalizes residuals; comparing the weight curves shows how much each observation contributes to the regression equations at a given residual.

The comparison here concerns response residuals, not whether an observation has unusual predictor values. For a fixed fitted regression and a fixed positive residual scale, we can compare the methods by applying each loss to the same standardized residual. This isolates what the loss itself does.

Definition: For observation \(i\), let \(y_i\) be its response, let \(\mathbf{x}_i^T\boldsymbol{\beta}\) be its fitted value, and define its residual as \(r_i=y_i-\mathbf{x}_i^T\boldsymbol{\beta}\). For a common positive scale \(s\), the standardized residual is \(u_i=r_i/s\). A residual loss \(\rho(u)\) assigns a cost to that standardized residual. Its influence function is \(\psi(u)=\rho'(u)\), where the derivative exists. The corresponding residual weight is \(w(u)=\psi(u)/u\) for \(u\ne 0\); when appropriate, define its value at zero by continuity. These weights describe the estimating equations, not probabilities.

To compare methods fairly, use the same residuals and scale, and state the tuning constants. A loss convention may differ by a positive multiplicative constant without changing its minimizer, so the numerical heights of different loss curves depend partly on convention. Their shape and the resulting influence and weight patterns are the main comparison.

Four Losses and Their Weight Curves

The following definitions use a fixed scale \(s\), with tuning constants \(k>0\) for Huber loss and \(c>0\) for Tukey bisquare loss. Least squares and absolute loss do not require a tuning constant.

Definition: The four losses and their residual weights are:

Least squares: \(\rho(u)=\tfrac12u^2\), \(\psi(u)=u\), and \(w(u)=1\).

Absolute loss: \(\rho(u)=|u|\). For \(u\ne0\), \(\psi(u)=\operatorname{sign}(u)\) and \(w(u)=1/|u|\). At zero the loss is not differentiable; its subgradient ranges from \(-1\) to \(1\), so the weight formula is not defined there.

Huber loss: \(\rho(u)=\tfrac12u^2\) when \(|u|\le k\), and \(\rho(u)=k|u|-\tfrac12k^2\) when \(|u|>k\). Its weight is \(1\) when \(|u|\le k\), and \(k/|u|\) when \(|u|>k\).

Tukey bisquare loss: \(\rho(u)=\tfrac{c^2}{6}\left[1-\left\{1-(u/c)^2\right\}^3\right]\) when \(|u|\le c\), and \(\rho(u)=c^2/6\) when \(|u|>c\). Its weight is \(\left\{1-(u/c)^2\right\}^2\) when \(|u|\le c\), and \(0\) when \(|u|>c\).

Least squares keeps the same weight no matter how large the residual becomes, while its influence \(\psi(u)=u\) grows without bound. Absolute loss has constant-magnitude influence away from zero, but its equivalent weight is larger for smaller nonzero residuals and becomes very large near zero. Huber loss is quadratic near zero and grows linearly beyond \(k\); its influence is bounded in magnitude by \(k\), but it does not drop to zero for any finite residual. Tukey bisquare loss stops increasing beyond \(c\). Its influence decreases to zero at the cutoff, and larger residuals receive zero weight.

A weight is not the same as a loss or an influence. In particular, the weight \(w(u)\) is used with the residual in the estimating equations: \(\psi(u)=w(u)u\). Thus an observation with a small weight can still have a nonzero influence if its residual is large enough. The actual effect on coefficient estimates also depends on that observation’s predictor values.

Why the Weight Formula Appears

Suppose the regression coefficients are estimated by minimizing \(\sum_i \rho(u_i)\), with \(u_i=(y_i-\mathbf{x}_i^T\boldsymbol{\beta})/s\) and \(s\) held fixed. For a differentiable loss, differentiating with respect to the coefficient vector gives the estimating equation below. For absolute loss, the corresponding statement uses subgradients at zero.

\frac{\partial}{\partial\boldsymbol{\beta}}\sum_i\rho(u_i) =-\frac{1}{s}\sum_i\mathbf{x}_i\psi(u_i)=\mathbf{0} \quad\Longleftrightarrow\quad \sum_i\mathbf{x}_i w(u_i)r_i=\mathbf{0}.

The equality follows from \(\psi(u_i)=w(u_i)u_i\) and \(u_i=r_i/s\); multiplying the first equation by the positive constant \(s^2\) gives the second. In this sense, a robust loss can be expressed through weighted residual equations, with weights recalculated as the residuals change during fitting. This is the basis of iteratively reweighted least squares for losses with suitable weight formulas. The equation explains the comparison; it does not mean that one can calculate final regression coefficients by assigning weights once from the initial residuals.

Worked Example: Comparing Losses at Three Residual Sizes

Consider the same hypothetical standardized residuals \(u=0.5\), \(u=2\), and \(u=5\) under all four methods. For illustration, choose Huber \(k=1.5\) and Tukey \(c=4\). These are illustrative choices, not universal settings.

For \(u=0.5\), least-squares loss is \(\tfrac12(0.5)^2=0.125\), and its weight is \(1\). Absolute loss is \(|0.5|=0.5\), and its weight is \(1/|0.5|=2\). Huber loss is in its quadratic region, so its loss is \(0.125\) and its weight is \(1\). Tukey’s weight is \(\{1-(0.5/4)^2\}^2=(0.984375)^2\approx0.9690\); substituting in its loss gives \(\tfrac{16}{6}[1-(0.984375)^3]\approx0.1230\).

For \(u=2\), least-squares loss is \(\tfrac12(2)^2=2\), with weight \(1\), and absolute loss is \(2\), with weight \(1/2=0.5\). Huber is beyond its threshold, so its loss is \(1.5(2)-\tfrac12(1.5)^2=1.875\), and its weight is \(1.5/2=0.75\). Tukey’s weight is \(\{1-(2/4)^2\}^2=(0.75)^2=0.5625\), and its loss is \(\tfrac{16}{6}[1-(0.75)^3]\approx1.542\).

For \(u=5\), least-squares loss is \(\tfrac12(5)^2=12.5\), with weight \(1\), while absolute loss is \(5\), with weight \(1/5=0.2\). Huber loss is \(1.5(5)-\tfrac12(1.5)^2=6.375\), with weight \(1.5/5=0.3\). Since \(5>c=4\), Tukey loss has reached its ceiling, \(c^2/6=16/6\approx2.667\), and its weight is \(0\).

The sequence makes the different treatment visible: least squares continues to increase quadratically; absolute loss increases linearly; Huber becomes linear after its threshold; and Tukey stops increasing and rejects residuals beyond its cutoff from the estimating equations. Absolute-loss weights can exceed \(1\), as at \(u=0.5\); they are relative equation weights, not probabilities.

From a Residual Weight to an Equation Contribution

A residual weight alone does not give an observation’s whole contribution to coefficient estimation. In the estimating equation, the residual is multiplied by its weight and by the predictor vector. The product \(w(u)u=\psi(u)\) captures the residual-side contribution before the predictor values are considered.

Worked Example: Comparing Influence at a Large Residual

Suppose a hypothetical observation has standardized residual \(u=5\), and use the same illustrative choices \(k=1.5\) and \(c=4\). Compare its influence contribution \(\psi(u)=w(u)u\).

Least squares gives \(\psi(5)=5\). Absolute loss gives \(\psi(5)=1\). Huber gives \(w(5)=1.5/5=0.3\), so \(\psi(5)=0.3(5)=1.5\). Tukey gives \(w(5)=0\), so \(\psi(5)=0\).

Thus Huber limits the residual-side contribution to \(1.5\), rather than allowing it to grow to \(5\) as least squares does. Tukey removes the residual-side contribution entirely once the standardized residual exceeds \(4\). In the regression equation, each of these values is still multiplied by the observation’s predictor vector, so this calculation alone is not a complete assessment of its effect on the fitted coefficients.

Scale and Tuning Constants Change the Comparison

The rules operate on \(u=r/s\), not on the raw residual \(r\) alone. A residual of \(8\) response units is not necessarily extreme if the residual scale is large; a residual of \(2\) units can be extreme if the scale is small. Scale estimation and the choice of tuning constants therefore affect which observations fall into the quadratic, downweighted, or zero-weight regions. The constants set the transition points; they do not change the basic shapes of the four loss families.

Worked Example: A Cutoff Is Applied to the Standardized Residual

Imagine a hypothetical fitted regression with scale \(s=2\) response units and a residual \(r=8\) response units. Its standardized residual is \(u=r/s=8/2=4\). With Huber \(k=1.5\), the weight is \(1.5/4=0.375\), so the influence is \(0.375(4)=1.5\). With Tukey \(c=4\), the observation is exactly at the cutoff: its weight is \(\{1-(4/4)^2\}^2=0\), and its influence is \(0\).

If instead the scale were \(s=4\) response units, the same raw residual would give \(u=8/4=2\). The Huber weight would be \(1.5/2=0.75\), and the Tukey weight would be \(\{1-(2/4)^2\}^2=0.5625\). The raw residual did not change, but its status under each method did because its size relative to the scale changed.

Choosing What to Notice

These curves answer a focused question: how does a chosen loss respond to residual size, assuming the same fit and scale for comparison? Least squares gives the clearest penalty to large residuals, but that also lets them dominate the objective. Absolute loss and Huber loss reduce that growth. Tukey bisquare loss goes further by giving sufficiently large residuals no role in the coefficient estimating equations. That can be useful when severe response contamination is a concern, but a zero residual weight is a consequence of the selected cutoff, not proof that a case is erroneous or should be deleted.

Residual-based weights also do not automatically correct unusual predictor values. They are calculated from residual size; they do not directly measure how far an observation’s predictor values lie from the rest of the data. Interpreting a robust fit therefore requires separating the choice of residual loss from other questions about the observations and the model.

Common Mistakes

Calling every weight a measure of total influence. The weight multiplies a residual in the estimating equation; the predictor values also matter. Compare \(\psi(u)=w(u)u\) before drawing conclusions about a particular case.
Assuming every robust weight is between zero and one. Under the stated convention, absolute-loss weights are \(1/|u|\), so small nonzero residuals can have weights greater than one.
Thinking Huber discards extreme residuals. Huber influence is bounded, but its weight remains positive for every finite residual. Tukey bisquare, in contrast, assigns zero weight beyond its cutoff.
Applying cutoffs to raw residuals without checking scale. The cutoff is applied to \(|u|=|r|/s\). Always identify the scale and tuning constant before classifying a residual.
Treating zero weight as a finding that a case is invalid. It means only that the chosen Tukey estimating equation gives that residual no coefficient contribution at the current fit and scale.

Key takeaway: Least squares has constant residual weight but unbounded influence; absolute loss has constant-magnitude influence away from zero; Huber bounds influence while retaining nonzero weights; Tukey bisquare eventually assigns zero weight. Compare them using the same standardized residual, scale, and stated tuning constants.

Check Your Understanding

Use the definitions and weight curves to answer these questions.

For a Huber loss with \(k=2\), what are the weight and influence at \(u=6\)?
Why can an absolute-loss weight exceed \(1\), and what happens to its formula as a nonzero residual approaches zero?
For Tukey bisquare loss with \(c=3\), what weight is assigned at \(u=4\), and what does that imply for its residual-side contribution?
Two cases have the same raw residual but are standardized by different scales. Explain why their weights under a fixed Huber or Tukey rule can differ.
Why does a zero Tukey weight not by itself establish that an observation is erroneous or that it has no effect on every aspect of a regression analysis?