Why Least Squares Squares the Errors

Why the Way We Add Errors Matters

A fitted line makes a prediction for each observed value of the response. The difference between an observed response and its prediction is a prediction error, also called a residual. When we compare possible fitted lines, we need a rule for combining their individual errors into one measure of overall fit. That rule matters: adding signed errors, adding absolute errors, and adding squared errors can give very different impressions of the same predictions.

Least-squares regression uses the sum of squared errors. Squaring prevents positive and negative errors from cancelling, and makes a large error count more than several small errors with the same combined absolute size. It also gives a smooth function of the line’s coefficients, which is convenient to minimize.

All three totals compare the same observations using the same predictions, but they answer different questions. A signed total preserves direction: errors above and below predictions have opposite signs. An absolute-error total measures the combined size of errors without their direction. A squared-error total also ignores direction, while assigning a greater penalty to larger errors. The squared total is expressed in squared response units; if responses are measured in centimeters, for example, the sum of squared errors is in square centimeters.

Signed Errors Can Cancel

A positive signed error means the observed response is above its prediction; a negative signed error means it is below. Adding the signed errors can therefore make a poor set of predictions look deceptively balanced. In particular, the total can be zero even though some predictions are wrong.

For a least-squares regression line with an intercept, the signed residuals sum to zero. This is a balance property of that fitted line, not evidence that all its predictions are exact. It is one reason the signed total is not the criterion used to choose the line: minimizing a signed total would not distinguish many sets of positive and negative errors at all.

Absolute Errors and Squared Errors Penalize Differently

Absolute errors and squared errors both prevent cancellation, but they do not give the same weight to a large miss. To see the difference, compare errors with the same total absolute size. If an error has magnitude \(a\), its absolute contribution is \(a\), while its squared contribution is \(a^2\). Doubling the error magnitude doubles its absolute contribution but quadruples its squared contribution.

This extra weight is a feature of the criterion, not a claim that every large error is especially harmful in every application. If a few unusually large errors should not dominate the assessment, a different criterion may be appropriate. Least squares is useful when penalizing large misses strongly is a reasonable choice for the problem.

Why Squaring Is Convenient to Minimize

A second reason for squaring is mathematical. Absolute value creates a corner at zero, while the square changes smoothly through zero. For a fixed observed response \(y_i\), the squared error as a function of a prediction \(\hat{y}_i\) is \((y_i-\hat{y}_i)^2\). Its derivative with respect to the prediction is \(-2(y_i-\hat{y}_i)\), which exists at every prediction, including an exact prediction. That smoothness makes calculus-based optimization of a regression line straightforward.

A simple constant-prediction problem shows how the square leads to a precise minimization result. Suppose a model must use the same prediction \(c\) for every response. The best value of \(c\) under squared error minimizes the sum of squared differences from the observations.

Differentiate with respect to \(c\), set the derivative equal to zero, and solve. The second derivative is positive, so the stationary value is the minimum.

Here \(\bar{x}_y\) denotes the arithmetic mean of the response values \(y_1,\ldots,y_n\). Equivalently, the minimizing constant is the mean response. To check that it is a minimum rather than a maximum, \(Q''(c)=2n>0\) for \(n>0\). This derivation illustrates how a squared-error objective can be minimized using a smooth function; in regression, the predictions depend on the line’s intercept and slope instead of one constant.

Comparing Candidate Regression Lines

To compare candidate lines using least squares, calculate the predictions for the same observations, find each signed error, square each error, and add. A candidate with the smaller sum of squared errors is preferred by this criterion. The comparison is about the observed data and the stated criterion; it does not establish that one line will predict every future response better.

What the Criterion Does—and Does Not—Tell You

Least squares gives a clear, calculable rule for choosing a line: minimize the sum of squared residuals. It does not make all residuals small, remove the effect of unusual observations, or guarantee that a straight line is a suitable description of the relationship. A large residual can have substantial influence on the squared total, while patterns or limitations in the data still require attention.

The three criteria can be summarized as follows:

A common mistake is to argue, “The signed errors add to zero, so the line has no prediction error.” The correction is that opposite-signed errors can cancel in a total even when individual predictions are wrong. Inspect the individual errors or use a nonnegative error criterion such as total absolute error or sum of squared errors. A related mistake is to treat the squared-error total as if it were measured in the original response units; it is measured in squared units.