Residual Sum of Squares for Candidate Lines

Why Compare Candidate Lines?

A straight line can summarize how a response variable changes with a predictor, but different lines generally give different predictions for the observed data. To compare those lines, measure how far each observed response is from its prediction, then combine those discrepancies using a specified criterion. In ordinary least squares, that criterion is the sum of squared residuals. The line with the smallest such sum is the least-squares line. This tutorial develops that comparison in a form aligned with the SOA VEE Mathematical Statistics syllabus.

The comparison concerns the data points actually observed. It does not establish that a line describes a causal relationship, nor does a small residual sum by itself guarantee accurate predictions for new observations.

Definition: Given observed pairs $(x_i,y_i)$, for $i=1,\ldots,n$, and a candidate line $\hat{y}=a+bx$, the predicted response at $x_i$ is $\hat{y}_i=a+bx_i$. The residual is $e_i=y_i-\hat{y}_i$, the observed response minus the predicted response. The residual sum of squares (RSS) for this line is $$ \operatorname{RSS}(a,b)=\sum_{i=1}^{n}(y_i-a-bx_i)^2=\sum_{i=1}^{n}e_i^2. $$

The sign convention for a residual matters when describing whether a point is above or below the line: a positive residual means the observed response is above its predicted value, and a negative residual means it is below. But each residual is squared in RSS, so either sign contributes a nonnegative amount. A residual of $-3$ contributes $9$, not $-3$.

Unless stated otherwise, the calculations here use a fixed set of complete observations and compare ordinary, unweighted straight lines with an intercept. Every observation contributes one squared residual. The predictor values must not all be identical if we are to estimate a unique slope from the data. A weighted criterion or a line constrained to pass through the origin is a different fitting problem and may have a different minimizing line.

Calculating and Comparing RSS

For each candidate line, substitute each observed predictor value into the line to obtain a prediction. Subtract the prediction from the observed response to obtain the residual, square it, and add the squared residuals across all observations. The line with the smallest total wins under this criterion; the comparison is not based on visual closeness at selected points.

List the observations.
Retain each observed pair, including separate observations that share the same predictor value.

Calculate predictions and residuals.
For each candidate $(a,b)$, calculate $\hat{y}_i=a+bx_i$ and $e_i=y_i-\hat{y}_i$.

Square and sum.
Compute $e_i^2$ for every observation and add those values to obtain the candidate line's RSS.

Compare on the same data.
The smallest RSS identifies the best candidate under ordinary least squares. If the candidates use the same observations, dividing each RSS by $n$ would not change their ranking, but it would report a mean squared residual rather than RSS.

Worked Example: Comparing Three Lines for Four Observations

A hypothetical analyst records the following four paired observations of a predictor $x$ and a response $y$: $(0,1),(1,2),(2,2),(3,5)$. Compare the candidate lines $\hat{y}=1+x$, $\hat{y}=0.5+1.5x$, and the least-squares line. Use unweighted RSS.

Method. Calculate the four predictions and residuals for each line, then square and sum the residuals. The least-squares slope and intercept can be calculated from $b=S_{xy}/S_{xx}$ and $a=\bar{y}-b\bar{x}$, where $S_{xx}=\sum_i(x_i-\bar{x})^2$ and $S_{xy}=\sum_i(x_i-\bar{x})(y_i-\bar{y})$.

For $\hat{y}=1+x$, the predictions at $x=0,1,2,3$ are $1,2,3,4$. The observed-minus-predicted residuals are $0,0,-1,1$, so $\operatorname{RSS}=0^2+0^2+(-1)^2+1^2=2$.

For $\hat{y}=0.5+1.5x$, the predictions are $0.5,2,3.5,5$. The residuals are $0.5,0,-1.5,0$, so $\operatorname{RSS}=0.5^2+0^2+(-1.5)^2+0^2=0.25+2.25=2.5$.

Now calculate the least-squares line. The means are $\bar{x}=(0+1+2+3)/4=1.5$ and $\bar{y}=(1+2+2+5)/4=2.5$. The centered predictor values are $-1.5,-0.5,0.5,1.5$, so $S_{xx}=2.25+0.25+0.25+2.25=5$. The centered response values are $-1.5,-0.5,-0.5,2.5$, giving $S_{xy}=2.25+0.25-0.25+3.75=6$. Thus $b=6/5=1.2$ and $a=2.5-(1.2)(1.5)=0.7$.

The least-squares predictions are $0.7,1.9,3.1,4.3$. The residuals are $0.3,0.1,-1.1,0.7$, and $\operatorname{RSS}=0.09+0.01+1.21+0.49=1.80$.

The three RSS values are $2$, $2.5$, and $1.8$. The least-squares line has the smallest value, as expected. The common mistake avoided here is comparing the lines by how close they look at one or two observations rather than by adding all four squared residuals.

Why the Least-Squares Line Minimizes RSS

A minimizing line is not just one that happens to have a small RSS among a few candidates. It is the line that minimizes RSS over all choices of intercept and slope. The following derivation shows why. Suppose the observations are fixed, the line has an intercept, and the predictor values are not all equal, so $S_{xx}>0$.

For a fixed slope $b$, write the residual as $y_i-a-bx_i=(y_i-bx_i)-a$. The sum of squared differences from a collection of numbers is minimized when the subtracted constant is their mean. Here the numbers are $y_i-bx_i$, whose mean is $\bar{y}-b\bar{x}$. Therefore the best intercept for that fixed slope is $a=\bar{y}-b\bar{x}$. Substituting it into the residual gives $(y_i-\bar{y})-b(x_i-\bar{x})$.

The resulting RSS is a quadratic function of $b$. Expanding it gives

\begin{aligned} \operatorname{RSS}(b) &=\sum_{i=1}^{n}\left[(y_i-\bar{y})-b(x_i-\bar{x})\right]^2\\ &=\sum_{i=1}^{n}(y_i-\bar{y})^2-2bS_{xy}+b^2S_{xx}. \end{aligned}

Since $S_{xx}>0$, this quadratic opens upward. Its derivative with respect to $b$ is $-2S_{xy}+2bS_{xx}$. Setting the derivative equal to zero gives $b=S_{xy}/S_{xx}$, the unique minimizing slope. The corresponding intercept is $a=\bar{y}-b\bar{x}$. This proves that the line from these formulas has no larger RSS than any other straight line with an intercept on the same data.

The result assumes that the objective is the unweighted sum of squared vertical residuals in the response variable. If the fitting rule changes, for example by assigning different weights to observations or constraining the intercept to be zero, the minimizing coefficients need not be these values.

Worked Example: Repeated Predictor Values Still Give Separate Residuals

In a hypothetical quality study, six devices have predictor-response pairs $(1,2),(1,4),(2,4),(2,5),(3,6),(3,9)$. Compare the candidate lines $\hat{y}=1+2x$ and $\hat{y}=0.5+2.25x$. Then verify the least-squares line and identify which candidate has the smallest RSS.

Method. Compute a residual for every device, even when two devices have the same predictor value. First, for $\hat{y}=1+2x$, the predictions are $3,3,5,5,7,7$. The residuals are $-1,1,-1,0,-1,2$, so $\operatorname{RSS}=1+1+1+0+1+4=8$.

For $\hat{y}=0.5+2.25x$, the predictions are $2.75,2.75,5,5,7.25,7.25$. The residuals are $-0.75,1.25,-1,0,-1.25,1.75$. Their squares sum to $0.5625+1.5625+1+0+1.5625+3.0625=7.75$.

To verify the least-squares line, $\bar{x}=2$ and $\bar{y}=30/6=5$. The predictor deviations are $-1,-1,0,0,1,1$, so $S_{xx}=1+1+0+0+1+1=4$. The response deviations are $-3,-1,-1,0,1,4$, so $S_{xy}=3+1+0+0+1+4=9$. Consequently $b=9/4=2.25$ and $a=5-(2.25)(2)=0.5$. This is the second candidate line, with RSS $7.75$, which is smaller than $8$.

The common mistake avoided is treating the two observations at a shared predictor value as a single observation. Both devices contribute their own squared residual, even though they receive the same prediction from a given line.

Worked Example: Keep Full Precision When Comparing Lines

A hypothetical study records $(x,y)=(0,1),(1,2),(2,4)$. Compare the least-squares line with the rounded candidate $\hat{y}=0.8+1.5x$, and explain why rounding the fitted intercept too early can affect the comparison.

Method. Calculate the exact least-squares coefficients first. The means are $\bar{x}=1$ and $\bar{y}=7/3$. The predictor deviations are $-1,0,1$, so $S_{xx}=2$. The response deviations are $-4/3,-1/3,5/3$, so $S_{xy}=4/3+0+5/3=3$. Therefore $b=3/2$, and $a=7/3-(3/2)(1)=5/6$.

For the exact line $\hat{y}=5/6+(3/2)x$, the predictions are $5/6,7/3,23/6$. The residuals are $1/6,-1/3,1/6$, giving $\operatorname{RSS}=1/36+1/9+1/36=6/36=1/6$, or approximately $0.1667$.

For $\hat{y}=0.8+1.5x$, the predictions are $0.8,2.3,3.8$, and the residuals are $0.2,-0.3,0.2$. Thus $\operatorname{RSS}=0.04+0.09+0.04=0.17$. The exact least-squares line has the smaller RSS. The difference is small, but real: replacing $5/6$ by $0.8$ changes the line. The common mistake avoided is rounding a coefficient before calculating predictions and RSS. Retain exact values or sufficient precision until the final comparison.

Common Mistakes and Interpretation

RSS is a specific criterion, not a generic measure of discrepancy. Several shortcuts can produce the wrong comparison:

Adding signed residuals. Positive and negative residuals can cancel. In the first worked example, the line $\hat{y}=1+x$ has residuals $0,0,-1,1$, whose signed sum is $0$, but its RSS is $2$. A zero signed sum does not mean a perfect fit.
Using absolute residuals instead of squared residuals. The sum of absolute residuals is a different criterion and need not select the same line as RSS.
Dropping repeated predictor values. Every observed response is a separate data point and must contribute one squared residual.
Dividing by $n$ and calling the result RSS. Dividing by $n$ gives the mean squared residual. For candidates based on the same $n$ observations the ranking is unchanged, but the reported quantity is different.
Rounding too soon. Small coefficient changes can alter predictions and RSS, especially when candidate values are close. Keep adequate precision through the calculation.

A smaller RSS means the candidate's predictions are closer to the observed responses under squared-error scoring on this particular data set. Squaring gives larger residuals disproportionately more influence: a residual of $4$ contributes $16$, while a residual of $2$ contributes $4$. This is one reason a single unusual observation can materially affect a line's RSS.

Key takeaway: For a candidate line, calculate every residual as observed response minus predicted response, square each residual, and add them. Among ordinary straight lines with an intercept fitted to the same observations, the line with the smallest RSS is the least-squares line.

Check Your Understanding

For each problem, use the stated observations and compare lines using ordinary, unweighted RSS. Keep all observations in the calculation.

For observations $(0,1),(1,2),(2,4)$, calculate RSS for $\hat{y}=1+x$, $\hat{y}=0.5+1.5x$, and the least-squares line. Which has the smallest RSS? (Answer: The RSS values are $1$, $0.5$, and $1/6$, respectively; the least-squares line is $\hat{y}=5/6+1.5x$, with the smallest RSS.)
For observations $(0,1),(0,3),(2,4),(2,6)$, calculate RSS for $\hat{y}=2+2x$, $\hat{y}=1+2x$, and the least-squares line. (Answer: The first two candidate RSS values are $6$ and $6$; the least-squares line is $\hat{y}=2+1.5x$, with RSS $4$.)
For observations $(1,3),(2,2),(3,5)$, calculate the least-squares line and its RSS, then compare it with $\hat{y}=1+x$ and $\hat{y}=2+x$. (Answer: The least-squares line is $\hat{y}=4/3+x$, with RSS $8/3$; the other RSS values are $3$ and $4$, so the least-squares line is smallest.)
For observations $(1,4),(2,4),(3,7),(4,9)$, calculate RSS for $\hat{y}=2+1.5x$ and $\hat{y}=1+2x$. Also calculate the mean squared residual for each. (Answer: The RSS values are $2.5$ and $2$, respectively; the mean squared residuals are $0.625$ and $0.5$, so $\hat{y}=1+2x$ is better by both criteria.)

What You'll Learn