Why a Fitted Line Need Not Hit Every Point
In real datasets, measurements often show an approximately linear pattern without falling exactly on one straight line. A least-squares line summarizes that pattern by choosing the line with the smallest possible sum of squared residuals. Its fitted values describe the line; the differences between observed and fitted values describe how far individual observations sit from it.
This tutorial follows a hypothetical calibration exercise. At each input setting, a technician records an instrument's output. The goal is to calculate a line that summarizes the association in these five observations, then examine what remains after fitting. The data are deliberately not perfectly linear: the exercise shows how the method works when observations depart from a straight-line pattern.
The fitted value \(\hat{y}_i\) is on the line at predictor value \(x_i\); it is not the observed response \(y_i\). A positive residual means the observation is above the line, and a negative residual means it is below. Residuals use the response's units, while the squared residuals used to select the line have squared response units.
The Hypothetical Calibration Data
The input setting \(x\) is recorded in volts (V), and the output \(y\) is recorded in millivolts (mV). These are hypothetical readings, not results from a real study.
| Observation | Input, \(x\) (V) | Output, \(y\) (mV) |
|---|---|---|
| 1 | 1 | 2 |
| 2 | 2 | 4 |
| 3 | 3 | 5 |
| 4 | 4 | 8 |
| 5 | 5 | 8 |
As the input increases, the output generally increases, but the changes are not perfectly uniform. For example, the output increases by 2 mV from the first to the second setting, by 1 mV from the second to the third, and by 0 mV from the fourth to the fifth. A fitted line gives a compact summary of this overall pattern; it does not require every successive change to be equal.
Worked Example: Calculating the Least-Squares Line
Use the five observations to calculate the slope and intercept. The predictor values vary, so the line's slope can be determined: in particular, \(S_{xx}=\sum (x_i-\bar{x})^2\) is positive.
1. Calculate the means. The predictor mean and response mean are
2. Calculate the centered sums. The least-squares slope is \(b_1=S_{xy}/S_{xx}\), where \(S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})\) and \(S_{xx}=\sum (x_i-\bar{x})^2\). The deviations from the means give
3. Find the slope and intercept. Therefore,
The fitted line is \(\hat{y}=0.6+1.6x\), with \(x\) in volts and \(\hat{y}\) in millivolts. The slope describes an increase of 1.6 mV in fitted output for each 1 V increase in input. The intercept is the fitted output at \(x=0\), namely 0.6 mV; because zero is outside the observed input range of 1 to 5 V, interpreting that intercept as a meaningful calibration reading would require caution.
Why These Coefficients Minimize Squared Residuals
The coefficients are not selected by drawing a line that looks close by eye. They minimize the total squared discrepancy between observed and fitted responses. The calculation can be justified by starting with an arbitrary line \(a+bx\) and its sum of squared residuals:
For any fixed slope \(b\), the value of \(a\) that minimizes \(Q(a,b)\) is \(\bar{y}-b\bar{x}\). To see this, differentiate with respect to \(a\) and set the derivative equal to zero:
Rearranging gives \(\sum y_i-na-b\sum x_i=0\), so \(a=\bar{y}-b\bar{x}\). Substituting this intercept makes each residual \(y_i-\bar{y}-b(x_i-\bar{x})\). Minimizing the resulting sum of squares with respect to \(b\) gives
This calculation explains why the slope formula used in the worked example produces the least-squares line. It also shows why \(S_{xx}>0\) is needed: if all predictor values were identical, the denominator would be zero and this slope formula would not define a unique line.
Fitted Values and Residuals
Once the line is fitted, substitute each observed predictor value into \(\hat{y}=0.6+1.6x\). Then subtract each fitted value from its corresponding observed response to obtain its residual. Keep the row pairing intact: each residual compares the observed and fitted response at the same input setting.
Worked Example: Finding Every Fitted Value and Residual
For observation 1, \(x_1=1\) V and \(y_1=2\) mV. The fitted value is \(\hat{y}_1=0.6+1.6(1)=2.2\) mV, so \(e_1=y_1-\hat{y}_1=2-2.2=-0.2\) mV. Applying the same calculation to all five observations gives:
| Observation | \(x_i\) (V) | \(y_i\) (mV) | \(\hat{y}_i=0.6+1.6x_i\) (mV) | \(e_i=y_i-\hat{y}_i\) (mV) |
|---|---|---|---|---|
| 1 | 1 | 2 | 0.6 + 1.6(1) = 2.2 | 2 - 2.2 = -0.2 |
| 2 | 2 | 4 | 0.6 + 1.6(2) = 3.8 | 4 - 3.8 = 0.2 |
| 3 | 3 | 5 | 0.6 + 1.6(3) = 5.4 | 5 - 5.4 = -0.4 |
| 4 | 4 | 8 | 0.6 + 1.6(4) = 7.0 | 8 - 7.0 = 1.0 |
| 5 | 5 | 8 | 0.6 + 1.6(5) = 8.6 | 8 - 8.6 = -0.6 |
For instance, at 4 V the line fits an output of 7.0 mV, while the observed output is 8 mV. The residual is \(+1.0\) mV, so this observation is 1.0 mV above the line. At 5 V, the fitted value is 8.6 mV and the observed value is 8 mV; the residual of \(-0.6\) mV places that observation below the line.
The residuals do not all equal zero, so the fitted line does not pass through every observation. That is expected for noisy data. In this example, the residuals sum to \(-0.2+0.2-0.4+1.0-0.6=0\), a standard property of a least-squares line with an intercept. A zero sum does not mean that each residual is zero: positive and negative differences can cancel.
Reading the Fit Without Overstating It
The line summarizes the overall association in these five paired measurements. Its slope is positive, and the fitted response increases by 1.6 mV per additional volt of input. The residuals show that the actual observations differ from this straight-line summary by amounts ranging from \(-0.6\) mV to \(1.0\) mV. The line is a compromise across the observations, not a rule that predicts every recorded value exactly.
Least squares squares residuals before adding them. For these data, the sum of squared residuals is
This quantity is the squared-error total for the fitted line, not a typical error in millivolts: its units are mV\(^2\). The largest contribution here comes from the \(1.0\) mV residual, whose square is \(1.00\) mV\(^2\). Squaring makes larger residuals count more heavily in the fitting criterion.
The calculation is a descriptive fit to the observations provided. It does not by itself show that changing the input causes a change in output, establish that the same pattern will hold outside the observed range, or give a probability model for future readings. Those claims require further design or modeling considerations. No distributional assumption is needed just to calculate this least-squares line and its residuals.
A Common Mistake
Wrong reasoning: “The fitted line gives 7.0 mV at 4 V, so the observed output at 4 V must be 7.0 mV.” This confuses a fitted value with an observation. The fitted value comes from the line; the observed value is the recorded data point. Here, \(y=8\) mV at 4 V, so its residual is \(8-7=1\) mV.
Correction: Keep the three quantities distinct. The line gives \(\hat{y}\), the dataset supplies \(y\), and their difference \(e=y-\hat{y}\) tells whether and how far the observation lies above or below the line. A least-squares fit describes a pattern in the data; it does not erase the departures from that pattern.
Check Your Understanding
Use the calibration example and the definitions above to answer these questions.
- What does the fitted slope of 1.6 mV per V mean in the context of the hypothetical calibration data?
- At \(x=3\) V, calculate the fitted value and residual. Is the observed point above or below the fitted line?
- Why does the fact that the residuals sum to zero not imply that all five observations lie exactly on the line?
- What is the squared-error total for the fitted line, and what are its units?
- Explain why the fitted output at \(x=0\) V should be interpreted cautiously for these data.