From Coefficients to Residual Sum of Squares
Once a multiple-regression coefficient vector is available, the calculations for each observation follow a fixed sequence: use its predictor values to calculate a fitted value, subtract that fitted value from the observed response to obtain a residual, and square and add the residuals to obtain the residual sum of squares. Keeping these operations in order matters. In particular, a residual is signed, while its contribution to the residual sum of squares is nonnegative.
The coefficient vector must also be read correctly. If it contains an intercept, that intercept is included in every fitted value. If the specified model has no intercept, one must not add one. These calculations describe how well the fitted model matches the observations in the data table; by themselves, they do not establish whether the model is appropriate or whether its coefficients are statistically significant.
A Reliable Calculation Sequence
Use one row at a time. First record the observed response and predictor values. Then calculate the fitted value by multiplying each predictor by its matching coefficient and adding the terms, including the intercept only if one is specified. Next calculate \(e_i=y_i-\hat{y}_i\), in that order. Finally square each residual and add those squares.
Identify the response, the predictors, the order in which the predictors correspond to coefficients, and whether an intercept is present.
Substitute one observation's predictor values into the specified regression equation. Keep enough intermediate arithmetic to check each term.
Subtract the fitted value from the observed response. A positive residual means the observation is above its fitted value; a negative residual means it is below.
Square each signed residual, then sum those squared values. Do not add the residuals first or leave any observation out.
In matrix notation, if \(X\) is the design matrix, \(\hat{\boldsymbol{\beta}}\) is the estimated coefficient vector, and \(\mathbf{y}\) is the vector of observed responses, the fitted-value and residual vectors are \(\hat{\mathbf{y}}=X\hat{\boldsymbol{\beta}}\) and \(\mathbf{e}=\mathbf{y}-\hat{\mathbf{y}}\). Thus the same calculation can be written compactly as:
The superscript \(\mathsf T\) denotes transpose. This expression is the sum of the squared entries of the residual vector, not the sum of its entries.
Worked Calculations
Worked Example: A Model Without an Intercept
A hypothetical study records response \(y\) and two predictors \(x_1\) and \(x_2\) for four observations. The specified model has no intercept, with coefficient vector \((1.5,-0.5)\), so \(\hat{y}_i=1.5x_{i1}-0.5x_{i2}\). Calculate all fitted values and residuals, then find RSS.
| Observation | \(x_1\) | \(x_2\) | \(y\) |
|---|---|---|---|
| 1 | 2 | 1 | 4 |
| 2 | 0 | 2 | 1 |
| 3 | 1 | 4 | 2 |
| 4 | 3 | 0 | 6 |
Method. Apply the two stated coefficients to each row, without adding an intercept. Then subtract each fitted value from its observed response.
Observation 1. \(\hat{y}_1=1.5(2)-0.5(1)=3-0.5=2.5\). Thus \(e_1=4-2.5=1.5\), and \(e_1^2=1.5^2=2.25\).
Observation 2. \(\hat{y}_2=1.5(0)-0.5(2)=0-1=-1\). Thus \(e_2=1-(-1)=2\), and \(e_2^2=2^2=4\).
Observation 3. \(\hat{y}_3=1.5(1)-0.5(4)=1.5-2=-0.5\). Thus \(e_3=2-(-0.5)=2.5\), and \(e_3^2=2.5^2=6.25\).
Observation 4. \(\hat{y}_4=1.5(3)-0.5(0)=4.5-0=4.5\). Thus \(e_4=6-4.5=1.5\), and \(e_4^2=1.5^2=2.25\).
Therefore \(\mathrm{RSS}=2.25+4+6.25+2.25=14.75\). The check is that the four residuals are \(1.5,2,2.5,1.5\), whose squares sum to \(14.75\). The common mistake avoided here is adding an intercept that the model does not contain; also, the second residual is \(2\), not \(0.5\), because its fitted value is \(-1\).
Worked Example: Including the Intercept in Every Row
A hypothetical service operation models weekly processing hours \(y\) using two workload measures \(x_1\) and \(x_2\). The fitted model is \(\hat{y}=10+2x_1-3x_2\). For the following four observations, calculate fitted values, residuals, and RSS.
| Observation | \(x_1\) | \(x_2\) | \(y\) |
|---|---|---|---|
| A | 1 | 1 | 12 |
| B | 2 | 0 | 15 |
| C | 0 | 2 | 3 |
| D | 1 | 2 | 7 |
For A, \(\hat{y}_A=10+2(1)-3(1)=10+2-3=9\), so \(e_A=12-9=3\) and \(e_A^2=9\). For B, \(\hat{y}_B=10+2(2)-3(0)=10+4=14\), so \(e_B=15-14=1\) and \(e_B^2=1\).
For C, \(\hat{y}_C=10+2(0)-3(2)=10-6=4\), so \(e_C=3-4=-1\) and \(e_C^2=1\). For D, \(\hat{y}_D=10+2(1)-3(2)=10+2-6=6\), so \(e_D=7-6=1\) and \(e_D^2=1\).
The residuals are \(3,1,-1,1\). Therefore \(\mathrm{RSS}=9+1+1+1=12\) hours squared. As a check, the only negative residual is for C, but its contribution to RSS is positive because it is squared. The common mistake avoided is dropping the intercept when substituting predictor values.
Worked Example: Tracking Units and Contributions to RSS
A hypothetical insurer records a response \(y\) in thousands of dollars, \(x_1\) in hundreds of policies, and \(x_2\) in risk-score points. The specified fitted model is \(\hat{y}=20+0.5x_1-2x_2\). Calculate the residuals and RSS for the observations below, and identify the largest individual contributions to RSS.
| Observation | \(x_1\) | \(x_2\) | \(y\), in thousands of dollars |
|---|---|---|---|
| A | 4 | 1 | 20 |
| B | 8 | 2 | 22 |
| C | 6 | 0 | 24 |
| D | 2 | 3 | 13 |
For A, \(\hat{y}_A=20+0.5(4)-2(1)=20+2-2=20\), so \(e_A=20-20=0\) and \(e_A^2=0\). For B, \(\hat{y}_B=20+0.5(8)-2(2)=20+4-4=20\), so \(e_B=22-20=2\) and \(e_B^2=4\).
For C, \(\hat{y}_C=20+0.5(6)-2(0)=20+3=23\), so \(e_C=24-23=1\) and \(e_C^2=1\). For D, \(\hat{y}_D=20+0.5(2)-2(3)=20+1-6=15\), so \(e_D=13-15=-2\) and \(e_D^2=4\).
Thus the residuals are \(0,2,1,-2\), and \(\mathrm{RSS}=0+4+1+4=9\) in squared thousands of dollars. B and D tie for the largest contribution, each contributing \(4\). The unit check is important: the residuals are measured in thousands of dollars, so their squares and RSS are measured in squared thousands of dollars. The common mistake avoided is treating a negative residual as a negative contribution to RSS.
Why RSS Uses Squared Residuals
For any proposed coefficient vector, least squares measures the total discrepancy between observed and fitted responses by adding the squared discrepancies. If \(b_0,\ldots,b_p\) are proposed coefficients, the corresponding objective is:
When the coefficients are the fitted ordinary least squares coefficients \(\hat{\beta}_0,\ldots,\hat{\beta}_p\), substituting them into this objective gives \(\sum_{i=1}^{n}(y_i-\hat{y}_i)^2\), which is exactly RSS. This is the defining least-squares criterion: the fitted coefficients minimize this sum over the permitted coefficient values. Squaring prevents positive and negative discrepancies from canceling and gives larger discrepancies more weight than smaller ones. For example, a residual of \(2\) contributes \(4\), whereas a residual of \(1\) contributes \(1\).
This justification does not mean that RSS is itself an estimate of predictive accuracy on new observations. It summarizes squared errors for the observations used in the calculation. Nor is RSS the same as the residual standard error: the latter involves an additional scaling by degrees of freedom and a square root. For the calculations in this tutorial, the requested quantity is the unscaled sum of squared residuals.
Common Calculation Traps
- Reversing the residual subtraction. The convention is observed minus fitted: \(e_i=y_i-\hat{y}_i\). Reversing the subtraction changes the sign, even though the square for that individual row stays the same. Keep the stated convention when reporting residuals.
- Forgetting or inventing an intercept. Check the model equation or coefficient vector before calculating. A model with an intercept uses it in every row; a no-intercept model does not acquire one by default.
- Pairing the wrong coefficients and predictors. Follow the specified order of the predictor columns. Swapping coefficients can change every fitted value and the resulting RSS.
- Adding residuals instead of their squares. A signed residual sum can cancel positive and negative values. RSS is calculated by squaring each residual first, then adding.
- Squaring incorrectly or omitting a row. Write a separate squared contribution for each observation. For example, \((-2)^2=4\), and every row in the data table contributes to RSS.
Check Your Understanding
For each problem, calculate the requested fitted values or residual sum of squares from the stated model. Keep the intercept specification and residual sign convention explicit.
- A model is \(\hat{y}=3+2x\). Two observations have \((x,y)=(0,4)\) and \((2,6)\). Calculate the fitted values, residuals, and RSS. (Answer: fitted values \(3,7\); residuals \(1,-1\); RSS \(2\).)
- A no-intercept model is \(\hat{y}=4x_1-x_2\). The observations \((x_1,x_2,y)\) are \((1,2,1)\), \((2,1,6)\), and \((0,3,-2)\). Calculate the fitted values and RSS. (Answer: fitted values \(2,7,-3\); residuals \(-1,-1,1\); RSS \(3\).)
- A model is \(\hat{y}=5+x_1+2x_2\). The observations \((x_1,x_2,y)\) are \((1,0,7)\), \((0,2,8)\), and \((2,1,9)\). Calculate the residuals and RSS. (Answer: residuals \(1,-1,0\); RSS \(2\).)
- A model is \(\hat{y}=12-0.5x_1+3x_2\). For \((x_1,x_2,y)=(4,1,15),(2,0,10),(0,2,20)\), calculate the fitted values and identify each squared residual contribution. (Answer: fitted values \(13,11,18\); squared contributions \(4,1,4\); RSS \(9\).)
- The coefficient vector for a no-intercept model is \((2,-1)\), so \(\hat{y}=2x_1-x_2\). The observations \((x_1,x_2,y)\) are \((3,1,5),(0,4,-2),(2,2,1)\). Calculate all residuals and RSS. (Answer: fitted values \(5,-4,2\); residuals \(0,2,-1\); RSS \(5\).)