From Observations to a Straight-Line Model
A scatter of measurements may suggest an overall trend without lying exactly on a line. A straight-line regression chooses a line that summarizes that trend by making the total squared vertical discrepancy between the observations and the line as small as possible. The calculation produces a fitted line; the residuals, which are the discrepancies it leaves behind, help us decide whether that line is an adequate description.
Suppose there are \(n\) observed predictor-response pairs \((x_i,y_i)\), where \(x_i,y_i\in\mathbb{R}\). A straight-line model with an intercept is \(\widehat{y}_i=a+bx_i\). Here \(a\) is the intercept and \(b\) is the slope. The hat indicates a fitted or predicted response, not a new observation. The residual at observation \(i\) is \(e_i=y_i-\widehat{y}_i\): a positive residual means the observed response is above the fitted line, and a negative one means it is below.
The first column of \(\mathbf{X}\) represents the intercept: it gives the coefficient \(a\) a contribution at every observation. The second column holds the predictor values and gives the slope coefficient \(b\) its contribution. This is an instance of a design matrix: its columns encode the terms included in the model. The fitted line is a particular case of fitting a model by least squares, but here the two coefficients can be interpreted directly.
For a straight line with an intercept, the coefficients satisfy the normal equations \(\mathbf{X}^{\top}\mathbf{X}\boldsymbol{\beta}=\mathbf{X}^{\top}\mathbf{y}\). If at least two predictor values differ, the two columns of \(\mathbf{X}\) are linearly independent: a relation \(c_1+c_2x_i=0\) at two distinct \(x_i\) forces \(c_2=0\), then \(c_1=0\). In that case the least-squares coefficients are unique.
Proof. First, \(\mathbf{X}^{\top}\mathbf{X}\) is invertible. For any nonzero \(\mathbf{z}\in\mathbb{R}^p\), \(\mathbf{z}^{\top}\mathbf{X}^{\top}\mathbf{X}\mathbf{z}=(\mathbf{X}\mathbf{z})^{\top}(\mathbf{X}\mathbf{z})=\|\mathbf{X}\mathbf{z}\|^2>0\), because independent columns imply \(\mathbf{X}\mathbf{z}\ne\mathbf{0}\). Thus the null space of the square matrix \(\mathbf{X}^{\top}\mathbf{X}\) contains only \(\mathbf{0}\), so it is invertible. Set \(\boldsymbol{\beta}=(\mathbf{X}^{\top}\mathbf{X})^{-1}\mathbf{X}^{\top}\mathbf{y}\); this vector satisfies the normal equations. Its residual has \(\mathbf{X}^{\top}\mathbf{e}=\mathbf{X}^{\top}\mathbf{y}-\mathbf{X}^{\top}\mathbf{X}\boldsymbol{\beta}=\mathbf{0}\). For any other coefficient vector \(\boldsymbol{\beta}+\mathbf{h}\), the residual is \(\mathbf{e}-\mathbf{X}\mathbf{h}\), so \(\|\mathbf{e}-\mathbf{X}\mathbf{h}\|^2=\|\mathbf{e}\|^2-2\mathbf{h}^{\top}\mathbf{X}^{\top}\mathbf{e}+\|\mathbf{X}\mathbf{h}\|^2=\|\mathbf{e}\|^2+\|\mathbf{X}\mathbf{h}\|^2\ge\|\mathbf{e}\|^2\). Equality requires \(\mathbf{X}\mathbf{h}=\mathbf{0}\), which, by column independence, requires \(\mathbf{h}=\mathbf{0}\). The specified vector is therefore the unique minimizer, and its residual is orthogonal to every column of \(\mathbf{X}\). \(\square\)
For the line model, the last condition says two things: the residuals sum to zero, and their predictor-weighted sum is zero. These are useful arithmetic checks on a fit with an intercept. They are not by themselves evidence that the line is a good model; the order and shape of the residuals still matter.
Worked Fits and Checks
Worked Example: Fitting a Line to Four Observations
Fit a straight line with an intercept to the observations \((x_i,y_i)=(0,1),(1,2),(2,2),(3,5)\). Then calculate the fitted values and residuals.
The design matrix and response vector are \(\mathbf{X}=\begin{bmatrix}1&0\\1&1\\1&2\\1&3\end{bmatrix}\) and \(\mathbf{y}=\begin{bmatrix}1\\2\\2\\5\end{bmatrix}\). The products in the normal equations are
Thus \(4a+6b=10\) and \(6a+14b=21\). Multiply the first equation by \(3\) and the second by \(2\): \(12a+18b=30\) and \(12a+28b=42\). Subtraction gives \(10b=12\), so \(b=6/5\). Substitution into the first equation gives \(4a+36/5=10\), hence \(4a=14/5\) and \(a=7/10\). The fitted line is \(\widehat{y}=7/10+(6/5)x\).
At \(x=0,1,2,3\), the fitted values are \(7/10,19/10,31/10,43/10\). Subtract each fitted value from its observation to obtain the residuals \(3/10,1/10,-11/10,7/10\). Thus \(\widehat{\mathbf{y}}=\begin{bmatrix}7/10\\19/10\\31/10\\43/10\end{bmatrix}\) and \(\mathbf{e}=\begin{bmatrix}3/10\\1/10\\-11/10\\7/10\end{bmatrix}\).
Check: Adding fitted values and residuals gives \(\begin{bmatrix}1\\2\\2\\5\end{bmatrix}=\mathbf{y}\). Also, \(3/10+1/10-11/10+7/10=0\), and \(0(3/10)+1(1/10)+2(-11/10)+3(7/10)=0\). These verify \(\mathbf{X}^{\top}\mathbf{e}=\mathbf{0}\), as required by the normal equations.
A residual plot places residuals against the predictor, or sometimes against the fitted values. The horizontal line \(e=0\) marks exact agreement with the fitted line. A useful plot has residuals scattered on both sides of zero without a pronounced systematic shape. A bend, wave, or steady increase or decrease in the residuals can indicate that a straight-line trend misses structure. With just a few observations, any such diagnosis is tentative: a plot can flag a concern, but it cannot establish a detailed explanation for it.
Worked Example: A Curved Pattern the Line Leaves Behind
Consider \(x=-2,-1,0,1,2\) and \(y=4,1,0,1,4\). Fit a line and use the residuals to assess whether a linear trend captures the pattern.
The design matrix has first column all ones and second column \(-2,-1,0,1,2\). Direct multiplication gives \(\mathbf{X}^{\top}\mathbf{X}=\begin{bmatrix}5&0\\0&10\end{bmatrix}\) and \(\mathbf{X}^{\top}\mathbf{y}=\begin{bmatrix}10\\0\end{bmatrix}\). The normal equations are \(5a=10\) and \(10b=0\), so \(a=2\) and \(b=0\). The fitted value is \(2\) at every predictor value.
The residuals, in the order \(x=-2,-1,0,1,2\), are \(2,-1,-2,-1,2\). The fitted and residual vectors are \(\widehat{\mathbf{y}}=\begin{bmatrix}2\\2\\2\\2\\2\end{bmatrix}\) and \(\mathbf{e}=\begin{bmatrix}2\\-1\\-2\\-1\\2\end{bmatrix}\).
Check: Adding the vectors gives \(\begin{bmatrix}4\\1\\0\\1\\4\end{bmatrix}=\mathbf{y}\). The residuals sum to \(2-1-2-1+2=0\), and their predictor-weighted sum is \((-2)(2)+(-1)(-1)+0(-2)+1(-1)+2(2)=0\). The calculation is consistent with a least-squares fit. But plotted against \(x\), the residuals are positive at both ends and negative in the middle. That pronounced U-shaped pattern shows that the fitted straight line leaves a curved structure in these data; the small residuals in the middle do not erase the pattern.
Worked Example: Small Residuals Around a Trend
For \(x=1,2,3,4\) and \(y=2,4,5,7\), calculate the fitted line and inspect the residuals.
Here \(\mathbf{X}=\begin{bmatrix}1&1\\1&2\\1&3\\1&4\end{bmatrix}\). Multiplication gives \(\mathbf{X}^{\top}\mathbf{X}=\begin{bmatrix}4&10\\10&30\end{bmatrix}\) and \(\mathbf{X}^{\top}\mathbf{y}=\begin{bmatrix}18\\53\end{bmatrix}\). The equations are \(4a+10b=18\) and \(10a+30b=53\). Multiply the first by \(5\) and the second by \(2\), obtaining \(20a+50b=90\) and \(20a+60b=106\). Subtraction gives \(10b=16\), so \(b=8/5\); then \(4a+16=18\), so \(a=1/2\). The fitted line is \(\widehat{y}=1/2+(8/5)x\).
The fitted values are \(21/10,37/10,53/10,69/10\), and the residuals are \(-1/10,3/10,-3/10,1/10\). Thus \(\widehat{\mathbf{y}}=\begin{bmatrix}21/10\\37/10\\53/10\\69/10\end{bmatrix}\) and \(\mathbf{e}=\begin{bmatrix}-1/10\\3/10\\-3/10\\1/10\end{bmatrix}\).
Check: Adding the fitted and residual vectors recovers \(\mathbf{y}=\begin{bmatrix}2\\4\\5\\7\end{bmatrix}\). The residual sum is \((-1+3-3+1)/10=0\), and the predictor-weighted sum is \((-1+6-9+4)/10=0\). The residuals are relatively small and alternate around zero, with no obvious bend in this four-point sequence. This is compatible with a straight-line summary, but four observations provide limited evidence; the plot should be interpreted cautiously.
What to Look For in Residual Plots
Fitted values and residuals together make the model’s successes and limitations visible. Look for patterns rather than treating the residuals as an unordered list.
- Curvature: residuals that are mostly positive at one end and mostly negative in the middle, or the reverse, suggest that a straight line may miss a bend.
- Changing spread: residuals that fan outward as the predictor or fitted value increases indicate that the size of the discrepancies changes across the range.
- One unusually distant residual: a single point far from the zero line deserves inspection. It may reflect a data issue or an unusual observation; the plot alone does not determine why.
- Order in time or sequence: if observations have a meaningful order, plotting residuals in that order may reveal runs or cycles that a predictor-only plot does not show.
A residual plot diagnoses the adequacy of the fitted trend, not the cause of a pattern. A curved sequence tells us that the straight line has not captured all the visible structure; it does not, by itself, tell us which alternative model or explanation is correct. Nor does a tidy-looking plot prove that every statistical assumption is satisfied. The plot is a practical check, to be interpreted alongside how the measurements were collected and what the predictor and response represent.
Common Pitfall: Dropping the Intercept
Removing the first column of ones changes the model to \(\widehat{y}=bx\). This forces the fitted line through the origin, so it is appropriate only when that restriction is justified by the application. It is not simply a quicker way to fit the same line.
For the observations \((1,2),(2,4),(3,5),(4,7)\) above, a through-origin fit would use \(b=(\sum_i x_i y_i)/(\sum_i x_i^2)=53/30\), since \(\sum_i x_i y_i=2+8+15+28=53\) and \(\sum_i x_i^2=1+4+9+16=30\). That gives a different model from the fitted line \(\widehat{y}=1/2+(8/5)x\). Presenting \(53/30\) as the slope of the intercept model is incorrect: it solves a different least-squares problem with the intercept fixed at zero. In the original model, both \(a\) and \(b\) must be estimated.
The practical workflow is therefore: put the observations into the design matrix for the model you intend to fit, solve the normal equations, calculate fitted values and residuals, check the reconstruction \(\mathbf{y}=\widehat{\mathbf{y}}+\mathbf{e}\), and inspect residuals for structure. The calculation produces a line; the residual plot helps judge whether that line is a plausible summary of the observed trend.
Check Your Understanding
For each question, distinguish the arithmetic of fitting from the interpretation of the residual pattern.
- For \(x=0,1,2\) and \(y=1,2,4\), write the design matrix and response vector for a line with an intercept.
- What two scalar checks follow from \(\mathbf{X}^{\top}\mathbf{e}=\mathbf{0}\) in a straight-line model with an intercept?
- Why do two distinct predictor values ensure that the intercept and predictor columns are independent?
- Residuals are positive at small and large predictor values but negative in the middle. What feature of the fitted line might this suggest is missing?
- Explain why fitting through the origin is a different model from fitting a line with an intercept.
