Comparing Candidate Lines by Squared Error

Why Compare Lines by Squared Error?

Several straight lines can describe the same scatter of observed data, but they will generally make different predictions at the observed predictor values. A fitting rule needs a way to measure how far those predictions are from the observed responses. In least squares, that measure is the sum of squared residuals. Comparing this quantity shows which line fits a particular dataset better according to the least-squares rule.

For each observation, a line gives a fitted response. The difference between the observed response and that fitted response is its residual. Squaring each residual makes its contribution nonnegative, and adding those contributions gives one overall score for the line. A smaller score means a better fit by this criterion.

Definition: For observed pairs $(x_i,y_i)$, where $i=1,\ldots,n$, a proposed line has equation $\hat{y}=a+bx$. Its fitted response at $x_i$ is $\hat{y}_i=a+bx_i$. The residual is $e_i=y_i-\hat{y}_i$, and the sum of squared residuals is $$ \mathrm{SSE}=\sum_{i=1}^{n} e_i^2=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2. $$ The least-squares line is the straight line whose coefficients minimize this sum over all possible intercepts and slopes.

Here, $a$ is the candidate intercept and $b$ is the candidate slope. For the fitted line from simple linear regression, the intercept and slope are often written $b_0$ and $b_1$, giving $\hat{y}=b_0+b_1x$. In either notation, $\hat{y}_i$ is a fitted value, not an observed response.

The task in this tutorial is a calculation on observed data. It does not require errors to be normally distributed, independent, or to have constant variance in order to calculate and compare SSE values. Those conditions matter for other questions about a regression model; they are not part of the arithmetic definition of SSE. The observed pairs should, however, have quantitative predictor and response values, and each proposed line must be evaluated at the same observed predictor values for a fair comparison.

A Reliable Comparison Procedure

For every candidate line, use the same sequence: substitute each observed $x_i$ into the equation, subtract the fitted response from the observed response, square each residual, and add the squares. Keep residuals signed until after you calculate them. The sign records whether the observed response is above or below the line, but it does not change the squared contribution.

Calculate fitted responses.
For each observed predictor value, evaluate the candidate line to obtain $\hat{y}_i$.

Calculate residuals.
Subtract each fitted response from its observed response: $e_i=y_i-\hat{y}_i$.

Square and add.
Compute $e_i^2$ for each observation and add the results to get that line's SSE.

Compare scores.
The lowest SSE identifies the best-fitting line among those compared. It identifies the least-squares line overall only when that line is included or otherwise known to minimize SSE among all lines.

The last qualification matters. A small set of proposed lines is not automatically a search over every possible intercept and slope. If one line has the smallest score in a list, it is the least-squares choice from that list. To call it the least-squares line overall, confirm that the line is the least-squares fit, for example by using the coefficient results from earlier material.

Worked Comparisons

Worked Example: Finding the Least-Squares Line Among Three Candidates

Suppose a hypothetical dataset records $x=0,1,2$ and responses $y=1,2,2$. Compare the candidate lines $A:\hat{y}=1+x$, $B:\hat{y}=\frac{7}{6}+\frac{1}{2}x$, and $C:\hat{y}=\frac{1}{2}+\frac{3}{4}x$.

For line A, the fitted responses are $1,2,3$. The residuals, calculated as observed minus fitted, are $0,0,-1$. Therefore,

\mathrm{SSE}_A=0^2+0^2+(-1)^2=1.

For line B, the fitted responses are $\frac{7}{6},\frac{5}{3},\frac{13}{6}$. Its residuals are $-\frac{1}{6},\frac{1}{3},-\frac{1}{6}$, so

\mathrm{SSE}_B= \left(-\frac{1}{6}\right)^2+ \left(\frac{1}{3}\right)^2+ \left(-\frac{1}{6}\right)^2 =\frac{1}{36}+\frac{1}{9}+\frac{1}{36} =\frac{1}{6}\approx 0.1667.

For line C, the fitted responses are $\frac{1}{2},\frac{5}{4},2$. The residuals are $\frac{1}{2},\frac{3}{4},0$, giving

\mathrm{SSE}_C= \left(\frac{1}{2}\right)^2+ \left(\frac{3}{4}\right)^2+0^2 =\frac{1}{4}+\frac{9}{16} =\frac{13}{16}=0.8125.

Line B has the smallest SSE of the three. It is also the least-squares line overall for these data: $\bar{x}=1$, $\bar{y}=\frac{5}{3}$, and the centered-data sums are $S_{xx}=2$ and $S_{xy}=1$. Thus the least-squares slope is $S_{xy}/S_{xx}=\frac{1}{2}$, and the intercept is $\bar{y}-b_1\bar{x}=\frac{7}{6}$, exactly the coefficients of line B.

If the response is measured in, for example, centimetres, each residual is in centimetres and the SSE is in square centimetres. The lower value means line B has the smallest total squared discrepancy at these three observed predictor values.

Worked Example: A Candidate List Must Include the Least-Squares Fit

Consider a different hypothetical dataset: $x=1,2,3$ and $y=2,3,5$. Compare $A:\hat{y}=1+x$, $B:\hat{y}=\frac{1}{3}+\frac{3}{2}x$, and $C:\hat{y}=\frac{1}{2}+\frac{3}{2}x$.

For A, the fitted responses are $2,3,4$, so the residuals are $0,0,1$, and $\mathrm{SSE}_A=0^2+0^2+1^2=1$.

For B, the fitted responses are $\frac{11}{6},\frac{10}{3},\frac{29}{6}$. The residuals are $\frac{1}{6},-\frac{1}{3},\frac{1}{6}$, so

\mathrm{SSE}_B= \left(\frac{1}{6}\right)^2+ \left(-\frac{1}{3}\right)^2+ \left(\frac{1}{6}\right)^2 =\frac{1}{6}\approx 0.1667.

For C, the fitted responses are $2,\frac{7}{2},5$, with residuals $0,-\frac{1}{2},0$. Hence $\mathrm{SSE}_C=\frac{1}{4}=0.25$. Among the three candidates, B has the smallest SSE.

In this example B is the least-squares line overall, not just the best of the three candidates. To check, $\bar{x}=2$, $\bar{y}=\frac{10}{3}$, $S_{xx}=2$, and $S_{xy}=3$. Therefore the least-squares slope is $S_{xy}/S_{xx}=\frac{3}{2}$, and the intercept is $\bar{y}-b_1\bar{x}=\frac{1}{3}$. Those coefficients give line B.

Notice that line C has a smaller SSE than line A, but it is still not the least-squares fit. “Better than another candidate” and “best among all straight lines” are different claims; the coefficient check establishes the latter here.

Worked Example: Equal Scores Among Listed Lines

Suppose a hypothetical dataset has $x=0,1$ and $y=0,0$. Compare the horizontal candidates $A:\hat{y}=1$ and $B:\hat{y}=-1$.

For A, the fitted responses are $1,1$, giving residuals $-1,-1$ and $\mathrm{SSE}_A=(-1)^2+(-1)^2=2$. For B, the fitted responses are $-1,-1$, giving residuals $1,1$ and $\mathrm{SSE}_B=1^2+1^2=2$.

The two candidates tie under the squared-error criterion: neither fits these data better than the other. But the horizontal line $\hat{y}=0$, which was not in the list, has residuals $0,0$ and SSE $0$. So a tie between listed lines does not establish that either is the least-squares line overall.

Why Squared Residuals Give This Rule

For any fixed candidate line, each residual can be positive, negative, or zero. If signed residuals were added directly, positive and negative discrepancies could cancel. Squaring first prevents that cancellation because $e_i^2\geq 0$ for every observation. Consequently, the SSE is zero exactly when every observed response equals its fitted response; otherwise it is positive.

The least-squares rule compares lines by this total nonnegative discrepancy. If line $L_1$ has $\mathrm{SSE}_{L_1}=0.20$ and line $L_2$ has $\mathrm{SSE}_{L_2}=0.70$, then $L_1$ has the lower squared-error score for those observations. The rule does not say that every individual residual for $L_1$ must be smaller than the corresponding residual for $L_2$; it compares the totals.

Squaring also makes a large residual count more heavily than a small one. A residual of magnitude $2$ contributes $4$, while a residual of magnitude $1$ contributes $1$. Thus one large miss can have a substantial effect on a line's SSE. This is part of what the least-squares criterion does, not a calculation error.

Common Mistakes and Interpretation

Adding signed residuals instead of squared residuals. Positive and negative residuals can cancel, so their sum does not measure the total squared discrepancy. Square each residual before adding.
Squaring a sum instead of summing squares. In general, $\sum e_i^2$ is not the same as $(\sum e_i)^2$. Calculate the square for each observation separately, then add.
Choosing the largest SSE. A lower SSE means a smaller total squared discrepancy at the observed predictor values, so choose the smallest score when comparing candidates.
Calling the best listed line the overall least-squares line without checking. The smallest score in a finite list only guarantees the best fit in that list. The overall least-squares line minimizes SSE among all straight lines.
Forgetting squared units. If $y$ is measured in minutes, residuals are in minutes and SSE is in minutes squared.

SSE is useful because it gives a precise, reproducible way to compare candidate lines on the same data. Its value depends on the scale and units of the response, so it is most directly informative when comparing lines fitted to the same responses. It summarizes total squared discrepancy; by itself, it does not establish that a straight-line relationship is appropriate for a scientific question or that a predictor causes a response.

Check Your Understanding

For each question, focus on the calculation and on exactly what the comparison allows you to conclude.

For observed responses $3,4$, a candidate line gives fitted responses $2,5$. Calculate the two residuals and the SSE.
Two candidate lines have SSE values $4.2$ and $3.7$. Which fits better by the least-squares criterion, and what must be true before calling it the least-squares line overall?
Explain why residuals of $2$ and $-2$ contribute $8$, not $0$, to the SSE.
If a response is measured in kilograms, what are the units of a residual and of an SSE?
Two lines from a candidate list have identical SSE values. What can you conclude about those candidates, and what can you not conclude about all possible lines?