Why Compare Fits on the Same Data?
A regression method is not simply a way to draw a line through data. Each method defines what counts as a good fit, and that choice affects which observations have the greatest influence on the estimated coefficients. When a response value is contaminated, the methods can therefore produce different slopes, intercepts, and predictions—even though they use the same predictor values.
This tutorial compares least squares, least absolute deviations (LAD), Huber regression, and least trimmed squares (LTS). The examples keep the predictor values fixed and alter a response only. This isolates response contamination: an unusual or corrupted outcome at an otherwise unchanged predictor value. It does not address predictor contamination or the separate problem of high-leverage observations.
These methods answer closely related but not identical fitting questions. Least squares is especially sensitive to large residuals because squaring magnifies them. LAD penalizes residuals in proportion to their absolute size and, under suitable conditions, targets a conditional median. Huber loss combines a quadratic penalty near zero with a linear penalty in the tails. LTS uses only the \(h\) residuals that give the smallest squared-error total. Comparisons are most directly interpretable when the methods are applied to the same observations and their tuning choices are stated.
A Clean Dataset: Agreement When the Line Fits Exactly
Begin with five hypothetical observations that lie exactly on \(y=2+3x\). We will compare the four methods using these same points. For LTS, set \(h=4\); for Huber, use a fixed residual cutoff \(\delta=1\), in the response units of this example. Holding these choices fixed makes the comparison explicit.
| \(x\) | \(y\), clean |
|---|---|
| 0 | 2 |
| 1 | 5 |
| 2 | 8 |
| 3 | 11 |
| 4 | 14 |
Worked Example: Four Fits to an Exact Line
The least-squares slope is \(S_{xy}/S_{xx}\), where \(S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})\) and \(S_{xx}=\sum (x_i-\bar{x})^2\). Here \(\bar{x}=2\) and \(\bar{y}=8\), so
Thus \(b=30/10=3\), and \(a=\bar{y}-b\bar{x}=8-3(2)=2\). Every residual is zero, so the least-squares loss is \(0\).
The line \(a=2,\ b=3\) also gives zero LAD loss because every absolute residual is zero. Its Huber loss is zero as well, since each residual is zero. For LTS with \(h=4\), the four smallest squared residuals are all zero, so its objective is zero. No method can attain an objective below zero; moreover, the four distinct clean points determine this line. All four methods therefore agree on this dataset.
This example checks an important point: choosing a robust method does not require the coefficients to differ from least squares. When the observations conform exactly to the same straight-line relationship, the different objectives can have the same minimizer. The meaningful comparison begins when a response is far from the pattern of the other observations.
Response Contamination: Different Objectives, Different Fits
Now change only the response at \(x=4\), from \(14\) to \(34\). The increase of 20 response units is a deliberate hypothetical contamination; the predictor values and the other four responses remain unchanged.
| \(x\) | Clean \(y\) | Contaminated \(y\) |
|---|---|---|
| 0 | 2 | 2 |
| 1 | 5 | 5 |
| 2 | 8 | 8 |
| 3 | 11 | 11 |
| 4 | 14 | 34 |
Worked Example: Fitting the Contaminated Responses
For least squares, \(\bar{x}=2\) and the contaminated-response mean is \(\bar{y}=12\). The cross-product sum is
Therefore \(b=70/10=7\) and \(a=12-7(2)=-2\). The least-squares line is \(\hat{y}=-2+7x\). Its slope has moved from 3 to 7: the single large response has a substantial effect on the squared-error fit.
For LAD, consider the original line \(\hat{y}=2+3x\). Its four clean residuals are zero and the contaminated observation has residual \(20\), giving total absolute loss \(20\). To verify that this line is an LAD minimizer, use the subgradient condition for absolute loss: at a zero residual, its sign may take any value from \(-1\) to \(1\); the signed residual contributions must sum to zero, both without and with multiplication by \(x_i\). Assign contributions \(u_0=0,\ u_1=1,\ u_2=-1,\ u_3=-1\) to the four zero residuals, and \(u_4=1\) to the positive contaminated residual. Then
These conditions establish that \(\hat{y}=2+3x\) is an LAD minimizer. The argument verifies a minimizer; it does not require that the minimizer be unique.
For Huber regression, use the loss \(\rho_\delta(r)=r^2/2\) when \(|r|\leq\delta\), and \(\rho_\delta(r)=\delta(|r|-\delta/2)\) when \(|r|>\delta\). With \(\delta=1\), its derivative is \(\psi(r)=r\) for \(|r|\leq1\) and \(\psi(r)=\operatorname{sign}(r)\) for \(|r|>1\). At the proposed line \(\hat{y}=1.5+3.5x\), the residuals are \(0.5,\ 0,\ -0.5,\ -1,\ 18.5\). Their Huber scores are \(0.5,\ 0,\ -0.5,\ -1,\ 1\), respectively, so
These are the zero-gradient conditions for minimizing the Huber objective over the intercept and slope. Because Huber loss is convex, a point satisfying these conditions is a global minimizer. In this fit, the large residual contributes a bounded score of \(1\), rather than a score that grows in proportion to its size.
For LTS with \(h=4\), the line \(\hat{y}=2+3x\) has four zero residuals at \(x=0,1,2,3\). Its four smallest squared residuals sum to zero. Since the LTS objective cannot be negative, this line is a global minimizer. The four clean points determine it uniquely in this example. The contaminated fifth point is not among the four residuals used in the objective at this solution.
Compare Coefficients and Predictions
The resulting coefficients show the effect of the fitting rules. The “clean line” column is the relationship that generated the four uncontaminated responses; it is a reference for this hypothetical example, not a guarantee that the same line is correct in other data.
| Fit to contaminated data | Intercept \(a\) | Slope \(b\) | Prediction at \(x=2\) | Prediction at \(x=4\) |
|---|---|---|---|---|
| Clean line (reference) | 2 | 3 | 8 | 14 |
| Least squares | -2 | 7 | 12 | 26 |
| LAD | 2 | 3 | 8 | 14 |
| Huber, \(\delta=1\) | 1.5 | 3.5 | 8.5 | 15.5 |
| LTS, \(h=4\) | 2 | 3 | 8 | 14 |
Worked Example: What the Coefficients Mean for Prediction
At \(x=2\), which is inside the range of the observed predictor values, the least-squares prediction is \(-2+7(2)=12\). Relative to the reference value \(8\), its prediction is 4 response units higher. Huber predicts \(1.5+3.5(2)=8.5\), which is 0.5 response units higher; LAD and LTS each predict \(2+3(2)=8\).
At \(x=4\), the contaminated observation’s predictor value, the predictions are \(26\) for least squares, \(15.5\) for Huber, and \(14\) for LAD and LTS. Relative to the clean-line reference of \(14\), the least-squares prediction differs by 12 response units and the Huber prediction by 1.5 response units. This does not make prediction at \(x=4\) a fair independent test—the response there was the contaminated value \(34\). It simply illustrates how a single response changed the fitted predictions.
In this example, LAD and LTS recover the line followed by the four clean responses, while Huber moves partway toward the contaminated response and least squares moves further. That is an outcome of these particular data, tuning choice, and contamination location—not a universal ranking. LAD, Huber, and LTS can differ from one another, and their performance depends on the error pattern and on what feature of the response is scientifically relevant.
How to Read a Comparison
A fair comparison begins by stating the assumed data setting. Here, the predictor values are held fixed, the clean responses follow a straight line exactly, and only one response is changed. In applied work, the clean-data model may instead allow random errors around a conditional mean or another location. Least squares is naturally aligned with estimating a conditional mean under the usual linear-model conditions; LAD is aligned with a conditional median. Huber and trimmed procedures use different loss criteria, so their coefficients need not estimate exactly the same feature when the response distribution is asymmetric.
A robust fit limits the effect of some unusual residuals under its chosen objective, but it does not establish that an unusual observation is erroneous or irrelevant. An extreme case may be a valid member of the target population, and deleting or downweighting it can change the question being answered. Nor does reducing the influence of a large residual repair an unusual predictor value: this comparison changed responses only.
Common Mistakes
- Calling every difference “robustness.” Different fits can reflect different targets and loss functions, not simply a better or worse method. State what quantity the analysis aims to describe.
- Assuming a robust method must equal the clean-data line. In this example LAD and LTS have that fit, but the Huber solution is different. Results depend on the response pattern and tuning choices.
- Treating a small residual influence as proof that an observation should be ignored. A large residual may be genuine and substantively important. Investigate its origin and relevance before deciding how to handle it.
- Comparing predictions without naming the predictor value. A coefficient change has different prediction consequences at different \(x\) values. Calculate predictions at values relevant to the application and state their response units.
Check Your Understanding
Use the definitions and worked calculations above to answer these questions.
- Why does a single large residual have a different effect on least squares than on Huber regression with a fixed cutoff?
- For the contaminated data, verify the least-squares prediction at \(x=3\) and compare it with the clean-line value there.
- What does the LTS choice \(h=4\) mean in the five-observation example?
- Why is agreement among the four methods on the clean exact-line data unsurprising?
- Does the fact that LAD and LTS recover the clean line prove that either method is preferable for every response-contaminated dataset? Explain.