Residual Sum of Squares

Why Summarize Residuals?

A regression model gives a fitted outcome for each observed case. Comparing each fitted outcome with what was actually observed shows where the model misses: some fitted values are too high, and others are too low. A summary is useful because a list of signed differences can conceal the overall size of those discrepancies. Residual sum of squares provides one such summary by squaring each difference before adding them.

Residual sum of squares is a measure of fit for a particular model and data set. It is not a test of whether the model is true, a measure of causation, or by itself a percentage of variation explained. Its units and the data used to calculate it matter when interpreting or comparing its value.

Definition and Model Setup

Suppose there are \(n\) observed outcomes \(y_1,\ldots,y_n\). Once a regression model has been fitted, let \(\hat{y}_i\) be the fitted outcome for case \(i\), calculated using that case’s predictor values. The residual is the observed outcome minus the fitted outcome.

For example, in a simple linear regression, a common model is \(y_i=\beta_0+\beta_1x_i+\varepsilon_i\). Here \(x_i\) is the predictor, \(\beta_0+\beta_1x_i\) describes the model’s mean outcome at that predictor value, and \(\varepsilon_i\) is the deviation of the observed outcome from that mean. After estimating the coefficients, the fitted value is \(\hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_i\). A residual is calculated from the observed and fitted values; it is not the unobserved error \(\varepsilon_i\).

The calculation of RSS requires observed and fitted outcomes but does not require a normal distribution or any other distributional assumption. In the usual linear regression model, assumptions such as a correctly specified linear mean and independent errors with mean zero and constant variance matter for regression inference. They are not needed to perform the arithmetic that defines RSS.

Squaring makes every contribution nonnegative, so residuals of opposite signs cannot cancel. The square also gives larger discrepancies more weight: a residual of 4 contributes \(16\), whereas a residual of 2 contributes \(4\). If outcomes are measured in points, residuals are in points and RSS is in points squared.

RSS and Least-Squares Fitting

In least-squares regression, the coefficients are chosen to make RSS as small as possible among the coefficient values allowed by the model. Thus RSS is both a summary of the remaining discrepancies after fitting and the quantity that least squares minimizes. It is not generally zero: a fitted line or other model may not pass through every observed outcome.

A short derivation illustrates why fitting a least-squares line with an intercept produces residuals whose sum is zero. Fix a slope \(b\) and choose an intercept \(a\). The sum of squared residuals is \(Q(a)=\sum_{i=1}^{n}(y_i-a-bx_i)^2\). Differentiating with respect to \(a\) gives \(-2\sum_{i=1}^{n}(y_i-a-bx_i)\). At a minimum this derivative is zero, so the residuals sum to zero. This condition concerns the signed residuals; it does not imply that their squares sum to zero.

Interpreting the Value

RSS combines the discrepancies for all included cases into one number. A smaller RSS means smaller total squared residuals for the particular observations and response scale used. It does not describe how the residuals are distributed among cases: many moderate discrepancies and one relatively large discrepancy can contribute quite differently while producing the same total.

A residual has the same units as the outcome, but RSS has squared outcome units. If the response is recorded in kilograms, RSS is in kilograms squared. The numerical value alone is therefore not an average error in the original units. To summarize a typical discrepancy in the outcome’s units, other summaries can be used, but RSS itself remains the sum of squared discrepancies.

A value of zero means every observed outcome equals its fitted value on the included cases. That is a perfect fit to those observations, not proof that the model captures the underlying process or will predict new outcomes perfectly. Conversely, a nonzero RSS is expected whenever at least one observed outcome differs from its fitted value.

Comparisons and a Common Pitfall

RSS can help describe and compare model fit, but a comparison is meaningful only when the quantities are calculated on comparable data and outcomes. In particular, the cases included and the response scale should be the same. RSS also depends on the number of observations: adding cases adds more squared terms, so values calculated from different sets or counts of cases are not automatically comparable.

A common mistake is to say that the model with the smaller raw RSS is always better. That conclusion may be unjustified if the models were assessed on different observations or outcome scales. For example, changing the outcome from grams to kilograms changes the numerical RSS without changing the fit in any substantive sense. Compare RSS values only after checking that the response and cases match, and interpret the result in squared response units.

Another mistake is to treat RSS as a percentage. It is a sum of squared discrepancies, not a fraction of outcomes or a percentage of variation explained. A percentage-based fit summary requires additional quantities and definitions; RSS alone does not supply one.