Least Absolute Deviations Regression

Why Fit a Line by Absolute Deviations?

A regression line summarizes how a quantitative response varies with a predictor. In ordinary least squares, the fitted line is chosen to minimize the sum of squared residuals. Squaring makes a very large residual contribute especially heavily to the objective. Least absolute deviations regression instead minimizes the sum of the residuals’ absolute values. A residual of 10 therefore contributes five times as much as a residual of 2, rather than 25 times as much.

This change in fitting criterion is useful when a few unusually large response residuals could strongly affect a squared-error fit. It does not automatically identify errors in the data, guarantee a stable fit, or make a fitted line appropriate. It provides a different rule for choosing the line, which must still be assessed in context.

Model and Definition

Suppose there are $n$ paired observations, with predictor values $x_i$ and quantitative responses $y_i$, for $i=1,\ldots,n$. A simple linear model writes the response as a line plus an error:

Y_i=\beta_0+\beta_1 x_i+\varepsilon_i

Here $\beta_0$ and $\beta_1$ describe the model’s intercept and slope, and $\varepsilon_i$ is the difference between the observed response and the model line. For a candidate fitted line with intercept $b_0$ and slope $b_1$, the fitted response for case $i$ is $b_0+b_1x_i$. Its residual is the observed response minus this fitted response.

Definition: A least absolute deviations (LAD) regression line is a line that minimizes the sum of the absolute residuals. For simple linear regression, the objective is $$ L(b_0,b_1)=\sum_{i=1}^{n}\left|y_i-(b_0+b_1x_i)\right|. $$ Any pair $(b_0,b_1)$ attaining the smallest possible value of $L$ gives a LAD fit. If more than one pair attains the minimum, the fit is not unique.

The optimization uses the observed pairs and this objective; it does not require a normal distribution for the errors. For the fitted line to be a useful summary, the observations must be correctly paired and measured, and a straight-line summary must be reasonable for the relationship of interest. The predictor must vary to estimate a slope. These are practical requirements for interpreting the fit, not a claim that minimizing absolute residuals alone verifies a model. No inference about population parameters is made by the calculations below.

How the Objective Changes the Weight of Residuals

A residual is signed: it is positive when an observed response exceeds the fitted response and negative when it falls below it. The LAD objective takes the absolute value, so positive and negative residuals both add a nonnegative amount. This differs from adding signed residuals, which could cancel, and from squaring residuals, which assigns an increasingly large penalty to each additional unit of residual magnitude.

Residual magnitude	Contribution to LAD loss	Contribution to squared-error loss
2	2	4
5	5	25
10	10	100

The table compares contributions, not fitted coefficients. The LAD contribution grows in direct proportion to residual magnitude. That limits how quickly the objective grows when one response is far from a candidate line, but the observation can still affect which line minimizes the total.

Worked Example: Finding a LAD Line From Three Points

Consider the hypothetical observations $(x,y)=(0,1),(1,2),(2,8)$. Find a line that minimizes the total absolute residual, and interpret its coefficients.

Write $a$ for the line’s fitted value at $x=0$, and $c$ for its fitted value at $x=2$. A straight line must have fitted value $(a+c)/2$ at $x=1$. Let $u=a-1$ and $v=c-8$, the endpoint fitted values’ differences from their observed responses. The absolute-residual total is therefore

L=|u|+\left|2-\frac{(1+u)+(8+v)}{2}\right|+|v| =|u|+|2.5+(u+v)/2|+|v|.

Set $t=u+v$. Since $|u|+|v|\geq |u+v|=|t|$, every line has loss at least $|t|+|2.5+t/2|$. This lower bound is at least 2.5: for $t\geq 0$, it is $2.5+1.5t$; for $-5\leq t<0$, it is $2.5-t/2$; and for $t<-5$, it is $-2.5-1.5t$. Each expression is at least 2.5 on its stated range.

The line through the two endpoint observations has $a=1$ and $c=8$, so $u=v=0$. Its middle fitted value is 4.5, and its absolute residuals are $0,2.5,0$, giving loss 2.5. It attains the lower bound and is a LAD fit. In the form $b_0+b_1x$, this line is

\widehat{y}=1+3.5x.

The intercept is 1 response unit at $x=0$; the slope is 3.5 response units per predictor unit. For example, if $x$ is measured in hours and $y$ in a measured quantity, the fitted line increases by 3.5 units of that quantity per additional hour. This is a description of the fitted association, not evidence that changing $x$ causes a change in $y$.

Nonuniqueness Is Possible

Unlike some smooth optimization problems, LAD objectives have corners where residuals are zero. A minimum can occur along a range of coefficient values, rather than at just one pair. This is not necessarily a computational mistake: the data and objective may genuinely support several minimizing lines.

Worked Example: Several Lines Can Minimize the Loss

Suppose four hypothetical observations are $(0,1),(0,3),(1,4),(1,6)$. Let $a=b_0$ be the fitted value at $x=0$, and $c=b_0+b_1$ the fitted value at $x=1$. The objective separates into the contributions at the two predictor values:

$$ L=|1-a|+|3-a|+|4-c|+|6-c|. $$

For any $a$, the triangle inequality gives $|1-a|+|3-a|\geq |3-1|=2$. Equality holds for every $a$ between 1 and 3. Likewise, $|4-c|+|6-c|\geq |6-4|=2$, with equality for every $c$ between 4 and 6. Thus every choice $1\leq a\leq 3$ and $4\leq c\leq 6$ attains the smallest possible total loss, 4.

For instance, choose $a=2$ and $c=5$. Then $b_0=2$, $b_1=c-a=3$, and the fitted line is $\widehat{y}=2+3x$. Its fitted values are 2 for the first two observations and 5 for the last two; the absolute residuals are $1,1,1,1$, for a total of 4. Other choices in the stated ranges give other minimizing lines, so reporting one such line should not suggest the data determined it uniquely.

What a Large Response Residual Can Do

Because absolute loss grows linearly, an extreme response does not dominate the objective as rapidly as it would under squared loss. The next example illustrates a particular data arrangement where changing one response substantially changes the minimized loss but leaves the selected fitted line unchanged. It is an illustration, not a guarantee that LAD fits are unaffected by response outliers.

Worked Example: A Large Response Changes the Loss, Not This Fit

Take the hypothetical observations $(0,1),(0,2),(0,2),(1,3),(1,4),(1,5)$. For a line, let $a$ be its fitted value at 0 and $c$ its fitted value at 1. The two fitted values can be chosen freely: they correspond to $b_0=a$ and $b_1=c-a$. At $x=0$, the contribution $|1-a|+2|2-a|$ is minimized at $a=2$, with loss 1. At $x=1$, the contribution $|3-c|+|4-c|+|5-c|$ is minimized at $c=4$, with loss 2. Thus the total minimum is 3, attained by $b_0=2$, $b_1=2$, or $\widehat{y}=2+2x$.

Now change only the last response from 5 to 100, making the observations at $x=1$ have responses 3, 4, and 100. Their contribution is $|3-c|+|4-c|+|100-c|$, minimized at $c=4$, where it equals $1+0+96=97$. To check the minimum, for $3\leq c\leq4$ the loss is $101-c$, which decreases to 97; for $4\leq c\leq100$ it is $c+93$, which increases from 97. Values outside $[3,100]$ cannot give a smaller value because moving toward that interval reduces all three distances. The fitted value at $x=0$ is still minimized at $a=2$, so the same line $\widehat{y}=2+2x$ minimizes the total, now with loss $1+97=98$.

In this example, the changed response increases the absolute loss from 3 to 98 but does not change the minimizing line. That occurs because there are repeated observations at each of only two predictor values, and the fitted values at those two locations can be chosen separately. Other arrangements can produce a different LAD line when a response changes. The example therefore illustrates the objective’s behavior in one setting, not immunity to influential observations.

Computing and Reporting a LAD Fit

For a small data set, the objective can sometimes be minimized by algebraic reasoning, as in the examples. For larger data sets, numerical optimization is generally used. The essential check is the same in either case: calculate the absolute residuals for the proposed line and add them. A lower total means a better fit according to the LAD criterion, though comparing a few candidate lines alone does not prove that one is globally optimal.

Specify the observations and line.
Record the paired values and write a candidate fitted value $b_0+b_1x_i$ for each case.

Calculate residuals.
For each case, subtract its fitted response from its observed response: $e_i=y_i-(b_0+b_1x_i)$.

Evaluate absolute loss.
Calculate $\sum |e_i|$. Compare candidate fits using this total, or use an optimization method that minimizes it.

Report the fit in context.
Give the equation, define the units of its coefficients, and make clear that the line summarizes the observed association rather than establishing causation.

LAD is most informative when its objective matches the analysis goal and when the fitted line is a defensible summary of the data. It should be considered alongside plots, residual patterns, and an audit of unusual observations. In particular, a small contribution from a large residual relative to squared loss does not by itself settle whether a case is an error, whether it should be retained, or whether the model is adequate.

Common Mistakes and Key Takeaway

Minimizing signed residuals. Positive and negative residuals can cancel. LAD minimizes the sum of their absolute values, not the sum of residuals.
Squaring the residuals by mistake. Squared residuals define the least-squares objective. In LAD, a residual of magnitude 10 contributes 10, not 100.
Assuming there must be one best line. Different coefficient pairs can attain the same minimum. Check whether the data and objective determine a unique fit.
Claiming LAD removes outlier influence. Absolute loss limits how quickly a large residual’s contribution grows, but the fitted line can still change when observations change.
Interpreting association as causation. A slope describes the fitted change in response per predictor unit; without an appropriate causal design, it does not show that changing the predictor causes the response to change.

Key takeaway: A LAD regression line minimizes the total absolute residual, $\sum |y_i-(b_0+b_1x_i)|$. Its linear penalty for large residuals differs from least squares’ squared penalty, but the fit still requires careful interpretation and assessment.

Check Your Understanding

Answer these questions by referring to the objective and the worked calculations.

For a residual of $-6$, what contribution does that observation make to the LAD objective?
A candidate line has absolute residuals 1, 2, and 8. What is its total LAD loss?
In the first worked example, what is the fitted response at $x=1$, and what is the residual for the observation $(1,2)$?
Why can more than one line minimize the objective in the repeated-predictor example?
Does minimizing absolute residuals guarantee that a fitted line is unaffected by a response outlier? Explain briefly.