Why Squared Error Amplifies Extreme Responses

Why Square the Residual?

A fitted regression model predicts a response for each observed case. The difference between an observed response and its fitted value is that case’s residual. Least squares chooses model coefficients to make the overall squared residuals small. This choice has an important consequence: a case with a very large residual can contribute much more to the objective than a case with a modest residual.

The reason is arithmetic, not a special rule about outliers. A residual $r$ contributes $r^2$, so doubling its size multiplies its squared contribution by four. This tutorial focuses on that casewise calculation and on what it does—and does not—tell us about a fitted regression.

Definition: For case $i$, let $y_i$ be the observed response and $\hat{y}_i$ its fitted value. The residual is $e_i=y_i-\hat{y}_i$. Its squared-error contribution is $e_i^2$. For $n$ cases, the least-squares objective, also called the residual sum of squares, is $$ SSE=\sum_{i=1}^{n}e_i^2=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2. $$

In a linear regression, the fitted values depend on the model’s estimated coefficients and the predictor values. Least squares selects coefficients that minimize $SSE$ for the observed data and the specified model. The algebraic statement about squaring requires no probability distribution for the errors: it follows directly from the definition of the objective. The examples below use hypothetical observations and fitted values, and focus on the squared-error calculation rather than statistical inference.

The Fourfold Result

A squared contribution depends on the magnitude of a residual, not its sign. A residual of $+r$ and one of $-r$ each contribute $r^2$. If a residual changes from $r$ to $2r$, its contribution changes from $r^2$ to $(2r)^2$:

\frac{(2r)^2}{r^2}=4 \quad \text{when } r\ne 0.

Thus, doubling a nonzero residual’s magnitude quadruples its own squared-error contribution. More generally, multiplying a residual by a factor $c$ multiplies its contribution by $c^2$, because $(cr)^2=c^2r^2$. This is why the squared-error objective places disproportionate weight on large residuals compared with small ones.

The result concerns one residual’s contribution. It does not automatically mean that the total $SSE$ quadruples, or that a fitted coefficient changes by a factor of four. The total is a sum of contributions from all cases, and when a regression is refitted, its coefficients and fitted values can change together. The fourfold calculation isolates the effect of changing a residual while holding the comparison fixed.

Worked Example: Comparing One Case With the Rest

Suppose a hypothetical data set contains five response values measured in points: 16, 21, 21, 21, and 21. Fit an intercept-only model, which predicts the sample mean for every case. What is the squared-error contribution of the first case, and how does it compare with the other four combined?

The mean is $$ \bar{x}=\frac{16+21+21+21+21}{5}=\frac{100}{5}=20 \text{ points}. $$

The model therefore predicts 20 points for each case. The first residual is $16-20=-4$ points, so its squared contribution is $$ (-4)^2=16 \text{ points}^2. $$

Each of the other four residuals is $21-20=1$ point, so each contributes $1^2=1\text{ point}^2$. Together they contribute $4(1)=4\text{ points}^2$. The total is $SSE=16+4=20\text{ points}^2$. The first case contributes four times as much as the other four cases combined: $16/4=4$.

The example uses one consistent response unit and a specified fitted model. It shows why a residual that is four times as large in magnitude as each of the others can dominate the sum: its squared contribution is 16 times each individual contribution.

Worked Calculations and Units

Squaring also changes the units. If the response is measured in milligrams per deciliter, a residual is measured in milligrams per deciliter, while its squared contribution is measured in $(\text{mg/dL})^2$. The $SSE$ has the same squared units. This is mathematically appropriate for comparing candidate fits to the same response, although the squared units may be less intuitive than the original response units.

Worked Example: Doubling a Residual in Response Units

A hypothetical regression predicts a response of 50 mg/dL for a particular case. For this casewise calculation, hold that fitted value fixed. If the observed response is 56 mg/dL, calculate its residual contribution; then calculate the contribution if the observed response were 62 mg/dL instead.

With an observed response of 56 mg/dL, the residual is $$ e=56-50=6\text{ mg/dL}, $$ and its squared contribution is $$ e^2=6^2=36\;(\text{mg/dL})^2. $$

With an observed response of 62 mg/dL and the same fitted value, the residual is $$ e=62-50=12\text{ mg/dL}, $$ and its squared contribution is $$ e^2=12^2=144\;(\text{mg/dL})^2. $$

The residual doubled from 6 to 12 mg/dL, while the contribution increased from 36 to 144 $(\text{mg/dL})^2$, exactly four times as much. This is an isolated calculation with the fitted value held fixed; it does not claim that refitting a regression after changing an observed response would leave its prediction unchanged.

A useful way to understand the growth is to compare squared contributions at different residual sizes. A residual twice as large contributes four times as much; one three times as large contributes nine times as much. The pattern applies to positive and negative residuals alike. Squaring removes the sign, so the objective treats an overprediction and an underprediction of the same magnitude equally.

Worked Example: Finding a Case’s Share of the Total

Suppose a hypothetical intercept-only fit has five residuals, in response units, of $3,-3,1,-1,$ and $0$. These sum to zero, as required for residuals from an intercept-only least-squares fit. Find the total $SSE$, the contribution from the case with residual $-3$, and that case’s share of the total.

Square each residual and add: $$ SSE=3^2+(-3)^2+1^2+(-1)^2+0^2=9+9+1+1+0=20. $$

The case with residual $-3$ contributes $(-3)^2=9$ squared response units. Its share of the total is $$ \frac{9}{20}=0.45=45\%. $$

The negative sign indicates that the observed response is below its fitted value, but it has no effect on this squared contribution. Also, the case’s contribution is 9, while the other four cases contribute $9+1+1+0=11$ combined. A large individual contribution can be important without being most of the total.

What the Calculation Does—and Does Not—Establish

The least-squares objective sums the casewise penalties. Consequently, a large residual can have a much larger effect on the objective than several smaller residuals. This is a property of the loss being minimized; it is not, by itself, a conclusion that a case is erroneous, that it should be removed, or that it has a particular effect on the estimated coefficients.

There is an important distinction between calculating a contribution and refitting a model. If one residual is changed while all fitted values and other residuals are held fixed, its squared contribution follows the fourfold rule exactly. But a refit changes the coefficients to minimize the new total $SSE$. That can alter multiple fitted values and residuals, so the new total need not be four times the old total. The casewise arithmetic remains true; the overall refitting outcome is a separate question.

A squared contribution is also not a measure of whether the response value was recorded correctly or whether the regression model is appropriate. It quantifies discrepancy from a particular fitted value under a particular model, on a squared scale. Understanding why the objective weights large discrepancies heavily helps explain why an extreme response can matter to a least-squares fit, but further checks are needed to understand the case and its role.

Common Mistake

Wrong reasoning: “If one residual doubles, the total $SSE$ must quadruple, so the fitted coefficients must also become four times larger.” This confuses one case’s contribution with the sum of all contributions, and confuses an objective value with estimated coefficients.

Correction: If that residual changes from $r$ to $2r$, its own contribution changes from $r^2$ to $4r^2$, holding the comparison fixed. The total $SSE$ also includes every other case’s contribution. A refit can change several residuals and the coefficients, but neither the total’s multiplier nor the coefficients’ changes are determined by the fourfold rule alone.

Key takeaway: Least squares adds the squared residuals. Doubling one residual’s magnitude multiplies that case’s contribution by four, because $(2r)^2=4r^2$. Keep that exact casewise result separate from claims about the total objective or the fitted coefficients.

Check Your Understanding

Use the definition of a residual and its squared-error contribution to answer these questions.

A residual is $-5$ minutes. What is its squared contribution, including units? What contribution would a residual of $+5$ minutes have?
A residual changes in magnitude from 2 to 6 response units. By what factor does its squared contribution change?
One case contributes 25 squared units to an $SSE$ of 100 squared units. What percentage of the total does that case contribute?
Why does doubling one residual not imply that the total $SSE$ will double or quadruple after a model is refitted?
Explain why a large squared-error contribution alone does not establish that a case is recorded incorrectly or should be removed.