Why Least Squares Is Not Absolute-Error Fitting

One Set of Residuals, Two Fitting Rules

A fitted line is a way to summarize how a response variable changes with a predictor. But how should we decide which candidate line fits best? A fitting rule answers this by assigning a total error to each line. Least squares uses squared residuals; another possible rule uses absolute residuals. These totals are not interchangeable: they can rank candidate lines differently and, when minimized over all lines, can select different fits.

The distinction matters because residuals can vary greatly in size. Squaring gives a large residual disproportionately more influence than a small one. Absolute error increases in direct proportion to a residual’s size. The choice between the two rules is therefore part of what “best-fitting line” means, not just a different way of reporting the same fit.

Definition: For observations \((x_i,y_i)\) and a candidate line \(\hat{y}_i=b_0+b_1x_i\), the residual is \(e_i=y_i-\hat{y}_i\). The squared-error total is \(\sum_{i=1}^n e_i^2\). The absolute-error total is \(\sum_{i=1}^n |e_i|\). The least-squares line is the line that minimizes the squared-error total over candidate straight lines. A line minimizing the absolute-error total is an absolute-error fit; it need not be the least-squares line.

Both totals measure vertical discrepancies between observed responses and a candidate line. Absolute errors are in the response variable’s units, while squared errors are in squared response units. Neither total is a probability or a percentage. For comparisons on the same data set, smaller totals mean a closer fit according to the chosen rule.

These are descriptive fitting criteria: comparing their totals does not require a probability model or a normality assumption. For a least-squares line with a freely chosen intercept and a predictor that varies, the minimizing line is given by the usual least-squares coefficients. The key point here is that minimizing a different total defines a different optimization problem.

Why Squaring Gives Large Residuals More Influence

A residual of magnitude 1 contributes \(1^2=1\) to the squared-error total, whereas a residual of magnitude 8 contributes \(8^2=64\). The same residuals contribute 1 and 8, respectively, to the absolute-error total. Thus the residual that is eight times as large contributes 64 times as much squared error, but only eight times as much absolute error.

More generally, multiplying a residual by a factor \(c\) multiplies its squared contribution by \(c^2\), but its absolute contribution by \(|c|\). This is the central difference between the two fitting rules. Squaring makes a line’s total error especially sensitive to large discrepancies.

Worked Example: One Large Residual and Three Small Ones

Suppose four hypothetical observations have predictor values \(0,1,2,3\). Compare the errors from the candidate horizontal line \(\hat{y}=0\) in two hypothetical response patterns: first \(y=(1,1,1,1)\), then \(y=(1,1,1,8)\).

For the first pattern, the residuals are \(1,1,1,1\). Therefore:

\text{Squared-error total}=1^2+1^2+1^2+1^2=4, \qquad \text{Absolute-error total}=|1|+|1|+|1|+|1|=4.

For the second pattern, the residuals are \(1,1,1,8\). Therefore:

\text{Squared-error total}=1^2+1^2+1^2+8^2=67, \qquad \text{Absolute-error total}=|1|+|1|+|1|+|8|=11.

Changing just one response from 1 to 8 raises the squared-error total from 4 to 67, an increase of 63. It raises the absolute-error total from 4 to 11, an increase of 7. The large residual contributes 64 units to the squared total but 8 units to the absolute total. These calculations show why squared-error fitting gives a large residual much more influence than each small residual.

A Worked Dataset Where the Best Lines Differ

The distinction is not limited to how totals are described: the two rules can lead to different minimizing lines. Consider three hypothetical observations with predictor values \(-1,0,1\) and responses \(0,0,10\). There is a large response at \(x=1\), while the other two responses are zero.

Worked Example: Least Squares and Absolute Error Select Different Lines

First find the least-squares line. The sample means are \(\bar{x}=0\) and \(\bar{y}=10/3\). The centered-data slope formula gives:

b_1= \frac{\sum_{i=1}^{3}(x_i-\bar{x})(y_i-\bar{y})} {\sum_{i=1}^{3}(x_i-\bar{x})^2} = \frac{(-1)(-10/3)+(0)(-10/3)+(1)(20/3)} {(-1)^2+0^2+1^2} = \frac{10}{2}=5.

The intercept is \(b_0=\bar{y}-b_1\bar{x}=10/3\). Thus the least-squares line is \(\hat{y}=10/3+5x\). Its fitted values at \(x=-1,0,1\) are \(-5/3,10/3,25/3\), so its residuals are \(5/3,-10/3,5/3\). Its totals are:

\text{Squared-error total} = (5/3)^2+(-10/3)^2+(5/3)^2 = 150/9 = 50/3 \approx 16.67,

\text{Absolute-error total} = |5/3|+|-10/3|+|5/3| = 20/3 \approx 6.667.

Now consider the different line \(\hat{y}=5+5x\). Its fitted values are \(0,5,10\), so its residuals are \(0,-5,0\). Its absolute-error total is \(0+5+0=5\), smaller than \(20/3\). Its squared-error total is \(0^2+(-5)^2+0^2=25\), larger than \(50/3\). Thus this line does better by absolute error, while the least-squares line does better by squared error.

We can also verify that the line \(\hat{y}=5+5x\) minimizes the absolute-error total for these observations. Write any candidate line as \(\hat{y}=a+bx\), and call its residuals at \(-1,0,1\) \(e_{-1},e_0,e_1\). Substituting the three data points gives:

e_{-1}=-a+b,\qquad e_0=-a,\qquad e_1=10-a-b.

The slope and intercept cancel in the following combination, leaving 10:

e_{-1}-2e_0+e_1=10.

By the triangle inequality, \(10\leq |e_{-1}|+2|e_0|+|e_1|\). Since \(2|e_0|\leq 2(|e_{-1}|+|e_0|+|e_1|)\), this implies \(10\leq 2(|e_{-1}|+|e_0|+|e_1|)\), so every line has absolute-error total at least 5. The line \(\hat{y}=5+5x\) attains that bound with residuals \(0,-5,0\). It is therefore an absolute-error minimizing line, but not the least-squares line.

Candidate-Line Rankings Can Reverse

Even before finding a minimizing line, we can compare two candidates and see how the error rule affects their ranking. The next example deliberately restricts the comparison to two lines. It illustrates a ranking reversal; it does not claim that either candidate is the best line among all possible lines.

Worked Example: Two Candidates Receive Opposite Rankings

Take hypothetical observations at \(x=-1,0,1\) with response \(y=0\) at all three predictor values. Compare candidate A, \(\hat{y}=-5x-5\), with candidate B, \(\hat{y}=-5.5\).

For candidate A, the fitted values are \(0,-5,-10\), giving residuals \(0,5,10\). Its totals are:

\text{Absolute-error total}=|0|+|5|+|10|=15, \qquad \text{Squared-error total}=0^2+5^2+10^2=125.

For candidate B, each fitted value is \(-5.5\), giving residuals \(5.5,5.5,5.5\). Its totals are:

\text{Absolute-error total}=3(5.5)=16.5, \qquad \text{Squared-error total}=3(5.5)^2=3(30.25)=90.75.

By absolute error, candidate A ranks better because \(15<16.5\). By squared error, candidate B ranks better because \(90.75<125\). Candidate A has one residual of magnitude 10, whose squared contribution is 100; that large contribution changes the squared-error comparison. In this data set, the line \(\hat{y}=0\) would fit all three observations exactly, so these two candidates are not global minimizers. Their purpose is to show that the two totals can order candidate lines differently.

What the Choice of Rule Means

Least squares is not a claim that absolute errors are unimportant, nor does it mean that squared residuals are the only possible measure of fit. It is a specific rule: choose the line that minimizes the sum of squared vertical residuals. An absolute-error rule asks a different question: which line minimizes the sum of the residual magnitudes? Since the objectives differ, there is no general reason for the answers to match.

Squared-error totals are useful for algebraic calculations and give large discrepancies substantial influence. That same feature means a few large residuals can have a strong effect on the least-squares fit. An absolute-error total grows more slowly as a residual gets large, but it still counts every discrepancy and does not make a large residual irrelevant. The appropriate description is about the fitting criterion, not a guarantee that one rule is universally better.

Key takeaway: Least squares minimizes \(\sum e_i^2\), not \(\sum |e_i|\). Squaring makes large residuals count disproportionately, so least-squares and absolute-error fitting can select different lines.

Common Mistake

A common mistake is to say, “The least-squares line minimizes the total error, so it must also minimize the absolute errors.” This reasoning treats “total error” as though it named only one quantity. It does not: the squared-error total and the absolute-error total apply different calculations to the same residuals. In the three-point example, the least-squares line has a smaller squared-error total, \(50/3\) rather than 25, but a larger absolute-error total, \(20/3\) rather than 5. Always state which total is being minimized before calling a line the best fit.

Check Your Understanding

Use the definitions and examples above to answer these questions.

A candidate line has residuals \(2,-2,5\). Calculate its squared-error and absolute-error totals.
If a residual changes from 3 to 6, by what factor does its squared contribution change? By what factor does its absolute contribution change?
In the three-point worked example, which line minimizes the absolute-error total, and what is that total?
Can two candidate lines be ranked differently by squared-error and absolute-error totals? Explain using the candidate-line comparison.
Why is it incorrect to say that a least-squares line must also minimize the absolute-error total?