What Is Being Minimized?
A regression line produces a predicted response for each observed explanatory value. The prediction will generally differ from the response that was actually recorded. Least squares chooses a line by adding up the squared vertical prediction errors for all the observations and seeking the smallest total.
Here, “vertical” means in the response-variable direction: at each observed \(x\)-value, compare the recorded \(y\)-value with the line’s predicted \(y\)-value. The method does not minimize distances measured along the line or horizontal distances in \(x\).
The sign convention matters when calculating each residual: observed response minus predicted response. But the residual is squared before it contributes to RSS, so a residual of \(-2\) and a residual of \(2\) each contribute \(4\). The RSS is never negative. A smaller RSS means a smaller total of squared vertical errors for those observations.
A line is not judged by its RSS in isolation. Compare candidate lines using the same data, with the same response variable and units. Changing the data changes the quantities being added, so the numerical RSS values are no longer a like-for-like comparison.
Calculating RSS for One Candidate Line
A useful calculation table keeps the observed response, the prediction, the signed residual, and the squared residual separate. This helps prevent a sign error or an accidental comparison of predictions to the wrong observations.
Worked Example: Finding the RSS of a Candidate Line
Suppose a hypothetical data set records practice time \(x\), in hours, and quiz score \(y\), in points. The four observations are \((1,3)\), \((2,4)\), \((3,4)\), and \((4,7)\). Calculate the RSS for the candidate line \(\hat{y}=2+1.1x\).
For each observed \(x_i\), substitute into the candidate line to get the prediction. Then subtract the prediction from the observed score and square that residual.
| \(x_i\) (hours) | \(y_i\) (points) | \(\hat{y}_i=2+1.1x_i\) (points) | \(e_i=y_i-\hat{y}_i\) (points) | \(e_i^2\) (points squared) |
|---|---|---|---|---|
| 1 | 3 | 3.1 | \(-0.1\) | 0.01 |
| 2 | 4 | 4.2 | \(-0.2\) | 0.04 |
| 3 | 4 | 5.3 | \(-1.3\) | 1.69 |
| 4 | 7 | 6.4 | 0.6 | 0.36 |
For example, at \(x=3\), the predicted score is \(2+1.1(3)=5.3\) points. The residual is \(4-5.3=-1.3\) points, and its contribution to RSS is \((-1.3)^2=1.69\) points squared.
This is the total squared vertical prediction error for this candidate line on these four observations. It is not a sum of signed residuals, and its units are points squared because each residual in points was squared.
Comparing Candidate Lines
To compare candidate lines, calculate an RSS for each one on the same observations. The line with the smallest value among the candidates has the smallest total squared vertical error among those candidates. To call a line a least-squares line, however, it must have the smallest RSS among all possible straight lines, not just a short list chosen for comparison.
Worked Example: Comparing Four Lines and Verifying the Minimum
Consider the hypothetical observations \((0,1)\), \((1,2)\), and \((2,2)\), where \(x\) is measured in hours and \(y\) in response units. Compare four candidate lines by calculating their RSS values.
For the line \(\hat{y}=1+x\), the predictions at \(x=0,1,2\) are \(1,2,3\). The residuals are \(0,0,-1\), so
For \(\hat{y}=0.5+0.75x\), the predictions are \(0.5,1.25,2\), and the residuals are \(0.5,0.75,0\). Thus,
For \(\hat{y}=1.25+0.5x\), the predictions are \(1.25,1.75,2.25\), and the residuals are \(-0.25,0.25,-0.25\). Therefore,
A fourth line is \(\hat{y}=\frac{7}{6}+\frac{1}{2}x\). Its predictions are \(\frac{7}{6},\frac{10}{6},\frac{13}{6}\); subtracting them from the observed responses gives residuals \(-\frac{1}{6},\frac{1}{3},-\frac{1}{6}\). Its RSS is
Among the four lines, this last line has the smallest RSS. We can go further and verify that it minimizes RSS among all straight lines for these observations. Let \(f(x)=\frac{7}{6}+\frac{1}{2}x\) be this line, and let \(g(x)=a+bx\) be any other candidate. Write \(u=a-\frac{7}{6}\) and \(v=b-\frac{1}{2}\). Then \(g(x)=f(x)+u+vx\).
The residuals from \(f\) are \(e_1=-\frac{1}{6}\), \(e_2=\frac{1}{3}\), and \(e_3=-\frac{1}{6}\), at \(x=0,1,2\). They satisfy
For line \(g\), each residual is \(e_i-(u+vx_i)\). Expanding its RSS shows exactly how it compares with the RSS of \(f\):
The middle terms vanish because both residual sums are zero. The remaining added term is a sum of squares, so it cannot be negative. Therefore no other straight line has a smaller RSS for these data. This verifies, rather than merely assumes from a short list of candidates, that \(f\) is a least-squares line here.
What the Comparison Tells You
RSS gives one overall measure of squared vertical errors. It does not show whether every point is close to the line: a small number of large residuals can contribute substantially to the total. Nor does a smaller RSS establish that a relationship is causal or that a line is appropriate for every purpose. It answers a focused question: for these observations and this response variable, which candidate line has the smaller total of squared residuals?
The value also depends on the units of the response. If residuals are measured in points, RSS is in points squared; if the response is measured in centimeters, RSS is in centimeters squared. Consequently, RSS values from different response units should not be compared as if they measured the same quantity.
Squaring has a practical consequence worth noticing: large residuals can have a strong effect on the total. A residual of \(5\) contributes \(25\), while a residual of \(1\) contributes only \(1\). A line that reduces a large miss can therefore have a lower RSS even if it has somewhat larger errors at other observations. The RSS summarizes all observations through the sum; it does not require every individual error to decrease.
Worked Example: One Large Residual Can Dominate
Suppose another hypothetical data set has observations \((0,1)\), \((1,2)\), and \((2,10)\). Compare \(\hat{y}=1+2x\) and \(\hat{y}=4x\) using these same three observations.
For \(\hat{y}=1+2x\), the predictions are \(1,3,5\). The residuals are \(0,-1,5\), so
For \(\hat{y}=4x\), the predictions are \(0,4,8\). The residuals are \(1,-2,2\), so
The second line has the smaller RSS of these two candidates, \(9\) rather than \(26\). For the first line, the single residual of \(5\) contributes \(25\) of the total \(26\). This comparison does not prove that the second line is the least-squares line among all possible lines; it establishes only that it is better by RSS than the first line on these observations.
Common Mistakes
- Adding residuals instead of squared residuals. Signed residuals can cancel. RSS squares each residual before adding.
- Comparing different data sets. A valid candidate-line comparison uses the same observed \(x\)- and \(y\)-values for every line.
- Forgetting the prediction step. Substitute each observed \(x_i\) into the candidate equation to find \(\hat{y}_i\) before calculating \(y_i-\hat{y}_i\).
- Calling the best of a few options the least-squares line without qualification. It is best among the options checked; a least-squares line minimizes RSS across all straight lines.
- Reporting RSS in the original response units. Squared residuals have squared response units, such as points squared.
Check Your Understanding
Answer each question using the definition and calculations in this tutorial.
- For one observation, the recorded response is \(8\) and a candidate line predicts \(6.5\). What is the residual, and how much does this observation contribute to RSS?
- Why can two candidate lines be compared fairly by RSS only when they are evaluated on the same observations?
- A candidate line has residuals \(-2\), \(1\), and \(3\). Calculate its RSS and state the units if the response was measured in kilograms.
- If one of four candidate lines has the smallest RSS, what additional comparison is needed before calling it a least-squares line?
- Explain why a residual of \(5\) contributes more to RSS than five residuals each equal to \(1\), even though both situations have a total absolute error of \(5\).