What Regression Coefficients Describe
A fitted regression model summarizes how a response varies with one or more predictors. Its coefficients describe the fitted values, not a guarantee that each observation will lie exactly on the model. To interpret the coefficients and assess the fit, keep the fitted response and the residual separate.
For \(n\) observations and \(p\) predictors, write the model in matrix form as \(\mathbf{y}\approx\mathbf{X}\boldsymbol{\beta}\), where \(\mathbf{y}\in\mathbb{R}^n\) is the column of observed responses, \(\mathbf{X}\in\mathbb{R}^{n\times(p+1)}\) is the design matrix, and \(\boldsymbol{\beta}\in\mathbb{R}^{p+1}\) is the coefficient column. In a model with an intercept, the first column of \(\mathbf{X}\) consists of ones. For observation \(i\), the fitted response is \(\widehat{y}_i=\widehat{\beta}_0+\widehat{\beta}_1x_{i1}+\cdots+\widehat{\beta}_px_{ip}\), using the least-squares coefficient estimates \(\widehat{\beta}_j\).
A positive residual means the observed response is above the fitted value; a negative residual means it is below. Residuals are measured in the same units as the response, while \(\operatorname{SSE}\) is measured in squared response units. A residual is calculated from the observed data and fitted model. It is not the same thing as an unobserved random error posited in a statistical model.
Proof. Let the original predictor values be \(x_1,\dots,x_p\), and replace \(x_j\) by \(x_j+\Delta\), leaving all other values unchanged. Subtract the original fitted response from the new one. Every term other than the \(j\)th predictor term cancels, leaving \(\widehat{\beta}_j(x_j+\Delta)-\widehat{\beta}_jx_j=\widehat{\beta}_j\Delta\). Setting every predictor to zero in the fitted equation leaves \(\widehat{y}=\widehat{\beta}_0\), which proves the intercept statement. \(\square
Thus a coefficient is a conditional description within the fitted equation: it compares fitted values with the other predictors held fixed. Its units are response units per predictor unit. The intercept may have a useful interpretation when zero is meaningful and relevant for all predictors. Otherwise, it is still the equation's fitted value at zero, but that value may lie outside the observed data and may not have a useful practical meaning.
Interpreting and Checking a Multiple-Predictor Fit
Worked Example: Reading Coefficients and Calculating Residuals
Consider the fitted model \(\widehat{y}=2+3x_1-x_2\), where \(x_1\) and \(x_2\) are two predictors. Interpret its coefficients. For four observations, calculate the fitted responses, residuals, and \(\operatorname{SSE}\).
The intercept is \(2\): the fitted response is \(2\) when \(x_1=x_2=0\). Holding \(x_2\) fixed, increasing \(x_1\) by one unit increases the fitted response by \(3\). Holding \(x_1\) fixed, increasing \(x_2\) by one unit decreases the fitted response by \(1\).
Take the design matrix and response column to be
Multiply each design-matrix row by the coefficient column to find the fitted responses:
Subtract fitted responses from observed responses, observation by observation:
Check: Adding the residuals back to the fitted responses gives \((2+1,5-1,1-1,4+1)=(3,4,0,5)\), the observed responses. We can also check that these residuals are orthogonal to the three columns of \(\mathbf{X}\): their sum is \(1-1-1+1=0\); their products with the \(x_1\) column sum to \((-1)+1=0\); and their products with the \(x_2\) column sum to \((-1)+1=0\). This verifies \(\mathbf{X}^{\top}\mathbf{e}=\mathbf{0}\), as required for a least-squares fit.
Why Least-Squares Residuals Have Useful Checks
The least-squares coefficient column minimizes the squared length of the residual column. The normal-equations characterization says that \(\widehat{\boldsymbol{\beta}}\) is a least-squares solution exactly when \(\mathbf{X}^{\top}(\mathbf{y}-\mathbf{X}\widehat{\boldsymbol{\beta}})=\mathbf{0}\), or equivalently \(\mathbf{X}^{\top}\mathbf{e}=\mathbf{0}\). Here is a proof of this characterization that also explains what orthogonality establishes.
Proof. Set \(\mathbf{r}=\mathbf{y}-\mathbf{X}\mathbf{b}\). First suppose \(\mathbf{X}^{\top}\mathbf{r}=\mathbf{0}\). For any other coefficient column \(\mathbf{c}\), write \(\mathbf{d}=\mathbf{c}-\mathbf{b}\). Then \(\mathbf{y}-\mathbf{X}\mathbf{c}=\mathbf{r}-\mathbf{X}\mathbf{d}\). Expanding the squared length gives \(\|\mathbf{r}-\mathbf{X}\mathbf{d}\|^2=\|\mathbf{r}\|^2-2\mathbf{r}^{\top}\mathbf{X}\mathbf{d}+\|\mathbf{X}\mathbf{d}\|^2\). Since \(\mathbf{r}^{\top}\mathbf{X}\mathbf{d}=(\mathbf{X}^{\top}\mathbf{r})^{\top}\mathbf{d}=0\), the result is at least \(\|\mathbf{r}\|^2\). Thus \(\mathbf{b}\) minimizes the squared residual length.
Conversely, suppose \(\mathbf{b}\) is a least-squares solution. For any \(\mathbf{h}\in\mathbb{R}^q\) and any real \(t\), \(\mathbf{b}+t\mathbf{h}\) is another possible coefficient column. Its squared residual length is \(\|\mathbf{r}-t\mathbf{X}\mathbf{h}\|^2=\|\mathbf{r}\|^2-2t\mathbf{r}^{\top}\mathbf{X}\mathbf{h}+t^2\|\mathbf{X}\mathbf{h}\|^2\). If \(\mathbf{r}^{\top}\mathbf{X}\mathbf{h}\ne0\), choosing \(t\) with the same sign as that inner product and sufficiently small nonzero magnitude makes the linear term negative with magnitude greater than the quadratic term. The squared length would then be smaller than \(\|\mathbf{r}\|^2\), contradicting minimality. Therefore \(\mathbf{r}^{\top}\mathbf{X}\mathbf{h}=0\) for every \(\mathbf{h}\). Taking \(\mathbf{h}\) to be each standard basis vector of \(\mathbb{R}^q\) shows every entry of \(\mathbf{X}^{\top}\mathbf{r}\) is zero. This proves the other direction. \(\square
This result says that the residual column is orthogonal to every column of the design matrix. In particular, an intercept in the model contributes a column of ones, so the residuals sum to zero.
Proof. The product of the transpose of the ones column with \(\mathbf{e}\) is \(1e_1+\cdots+1e_n=\sum_{i=1}^{n}e_i\). By the theorem, this product is zero. \(\square
The residuals also have zero inner product with each predictor column included in the model. This is an algebraic property of the least-squares fit, not evidence that the fitted model describes every feature of the data. A model can have residuals that satisfy all these equalities and still leave a visible pattern when plotted.
Residual Patterns and Basic Model Mismatches
A residual plot places residuals on the vertical axis and a predictor or fitted response on the horizontal axis. A horizontal reference line at zero helps show whether the residuals lie on both sides of zero. The plot is a diagnostic: it can reveal patterns worth investigating, but it does not by itself prove why a pattern occurs.
- A curved trend in residuals against a predictor or fitted response can suggest that a straight-line relationship is inadequate.
- A spread that grows or shrinks across the horizontal axis can suggest that the size of typical deviations changes with the predictor or fitted response.
- A residual far from the rest can flag an observation whose response is poorly represented by the fitted values. Check the data and context before drawing conclusions.
- Separate bands or clusters can suggest that an important grouping or predictor is not represented in the model.
Worked Example: A Curved Residual Pattern
Fit a straight-line model with an intercept to the five observations \((x,y)=(-2,4),(-1,1),(0,0),(1,1),(2,4)\). Calculate its residuals and interpret their pattern.
The mean predictor is \(\overline{x}=(-2-1+0+1+2)/5=0\), and the mean response is \(\overline{y}=(4+1+0+1+4)/5=2\). For the least-squares line, the slope is \(\sum_i(x_i-\overline{x})(y_i-\overline{y})/\sum_i(x_i-\overline{x})^2\). The numerator is \((-2)(2)+(-1)(-1)+0(-2)+1(-1)+2(2)=-4+1+0-1+4=0\). The denominator is \(4+1+0+1+4=10\). Thus the slope is \(0/10=0\), and the intercept is \(\overline{y}-0\overline{x}=2\).
The fitted value at every observation is therefore \(2\). Subtracting these fitted values from the observed responses gives
Check: The residuals sum to \(2-1-2-1+2=0\), and their products with the predictor values sum to \((-2)(2)+(-1)(-1)+0(-2)+1(-1)+2(2)=0\). The residuals are positive at both ends and negative in the middle, making a curved pattern rather than an unstructured band around zero. This suggests that a straight line may miss curvature in these data; it is a diagnostic clue, not a proof of the cause.
When comparing fits to the same response measurements, a smaller \(\operatorname{SSE}\) means a smaller total squared residual length for those observations. It does not, by itself, establish that the model is appropriate: adding predictors can reduce \(\operatorname{SSE}\) while leaving an important pattern. The residual plot adds information about where the fitted model misses, while the sum of squares gives one overall measure of the residual size.
Common Mistakes
Reversing the residual sign. If an observed response is \(5\) and its fitted response is \(4\), the residual is \(5-4=1\), not \(4-5=-1\). The positive sign records that the observation lies above its fitted value.
Interpreting a coefficient without holding other predictors fixed. In the model \(\widehat{y}=2+3x_1-x_2\), the coefficient \(3\) describes a one-unit change in \(x_1\) with \(x_2\) unchanged. It does not state that every observation whose \(x_1\) increases by one must have an observed response increase of exactly \(3\): the other predictor and the residual may also change.
Treating zero residual sum as proof of a good fit. With an intercept, \(\sum_i e_i=0\) is a least-squares property even when residuals form a clear curve, as the previous example shows. Check the residual pattern as well as the total squared size.
Check Your Understanding
Use the definitions and results in this tutorial to answer the following questions.
- In the fitted model \(\widehat{y}=5-2x_1+4x_2\), how does the fitted response change if \(x_2\) increases by \(3\) while \(x_1\) is fixed?
- An observation has response \(12\) and fitted response \(15\). What is its residual, and what does its sign indicate?
- Why do least-squares residuals sum to zero when the design matrix contains a column of ones?
- What might a U-shaped pattern in a residual plot suggest, and why is it not by itself proof of a particular cause?
- For residuals \((-2,1,1)\), calculate the residual sum of squares and state its units relative to the response.
