A Regression Fit as a Sequence of Decisions
A regression calculation is not just a matrix operation. It begins by deciding what response to predict and which predictors to include; those choices determine the design matrix and therefore what the fitted coefficients mean. After solving, the residuals help check the calculation and show which parts of the observed response the model does not capture. A complete workflow keeps these stages connected: specify the model, build the matrix, solve, verify, and interpret with appropriate caution.
Suppose there are \(m\) observations and \(n\) unknown coefficients. Write the observed responses as a column \(\mathbf{y}\in\mathbb{R}^m\), and put the model coefficients in \(\boldsymbol{\beta}\in\mathbb{R}^n\). The design matrix \(\mathbf{A}\in\mathbb{R}^{m\times n}\) records the model terms at each observation. The model predicts \(\mathbf{A}\boldsymbol{\beta}\). When no coefficient vector fits every response exactly, least squares chooses one that minimizes the squared length of the residual vector.
For a linear model with an intercept and two predictors \(x\) and \(z\), the prediction at observation \(i\) is \(\beta_0+\beta_1x_i+\beta_2z_i\). Thus row \(i\) of the design matrix is \((1,x_i,z_i)\), and the coefficient vector is \((\beta_0,\beta_1,\beta_2)^{\top}\). The first column represents the intercept; each later column represents one predictor. Changing the model changes the columns, and consequently changes the meaning of the coefficients.
Worked Example: A Two-Predictor Fit Using QR
Worked Example: A Two-Predictor Fit Using QR
Suppose four observations have predictor pairs \((1,1),(0,1),(-1,-1),(0,-1)\), with responses \(5,0,1,2\). Use the model \(\beta_0+\beta_1x+\beta_2z\), solve by reduced QR, and check the residuals.
Following the model, the design matrix and response vector are
For this full-column-rank matrix, take the reduced QR factorization \(\mathbf{A}=\mathbf{Q}\mathbf{R}\), where the columns of \(\mathbf{Q}\) are orthonormal and \(\mathbf{R}\) is upper triangular:
Multiplying \(\mathbf{Q}\mathbf{R}\) recovers \(\mathbf{A}\): its columns are respectively \((1,1,1,1)^{\top}\), \((1,0,-1,0)^{\top}\), and \((1,1,-1,-1)^{\top}\). The reduced-QR least-squares calculation solves \(\mathbf{R}\widehat{\boldsymbol{\beta}}=\mathbf{Q}^{\top}\mathbf{y}\). First compute the right-hand side:
The triangular equations are \(2\widehat{\beta}_0=4\), \(\sqrt2\widehat{\beta}_1+\sqrt2\widehat{\beta}_2=2\sqrt2\), and \(\sqrt2\widehat{\beta}_2=-\sqrt2\). Solving from the bottom gives \(\widehat{\beta}_2=-1\), then \(\widehat{\beta}_1=3\), and finally \(\widehat{\beta}_0=2\). The fitted model is therefore \(2+3x-z\).
Check the fit. Multiplying the design matrix by the coefficients gives
So \(\mathbf{r}=\mathbf{y}-\mathbf{A}\widehat{\boldsymbol{\beta}}=(1,-1,1,-1)^{\top}\). To check the residual against each design column, compute
The residual is orthogonal to every column of the design matrix, as a least-squares residual should be. The sum of squared residuals is \(1^2+(-1)^2+1^2+(-1)^2=4\). This completes the numerical check: the coefficients reproduce the fitted values, and the residual satisfies the orthogonality test.
Why the QR Calculation Gives a Least-Squares Fit
The calculation above uses a standard fact about reduced QR: if \(\mathbf{A}\in\mathbb{R}^{m\times n}\) has full column rank, it can be written \(\mathbf{A}=\mathbf{Q}\mathbf{R}\), with \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}_n\) and \(\mathbf{R}\) an invertible upper-triangular matrix. Here, full column rank means that the columns of \(\mathbf{A}\) are linearly independent. The equation \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}_n\) says that distinct columns of \(\mathbf{Q}\) are orthogonal unit vectors. These properties are what make the solve and the residual check work.
Proof. Since \(\mathbf{R}\) is invertible, there is exactly one vector \(\widehat{\boldsymbol{\beta}}\) satisfying \(\mathbf{R}\widehat{\boldsymbol{\beta}}=\mathbf{Q}^{\top}\mathbf{y}\). Define \(\mathbf{r}=\mathbf{y}-\mathbf{A}\widehat{\boldsymbol{\beta}}\). Using \(\mathbf{A}=\mathbf{Q}\mathbf{R}\), we obtain \(\mathbf{Q}^{\top}\mathbf{r}=\mathbf{Q}^{\top}\mathbf{y}-\mathbf{Q}^{\top}\mathbf{Q}\mathbf{R}\widehat{\boldsymbol{\beta}}=\mathbf{Q}^{\top}\mathbf{y}-\mathbf{R}\widehat{\boldsymbol{\beta}}=\mathbf{0}\). Hence \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{R}^{\top}\mathbf{Q}^{\top}\mathbf{r}=\mathbf{0}\).
For any other coefficient vector \(\boldsymbol{\beta}=\widehat{\boldsymbol{\beta}}+\mathbf{d}\), where \(\mathbf{d}\in\mathbb{R}^n\), its residual is \(\mathbf{y}-\mathbf{A}\boldsymbol{\beta}=\mathbf{r}-\mathbf{A}\mathbf{d}\). The two terms are orthogonal because \(\mathbf{r}^{\top}\mathbf{A}\mathbf{d}=(\mathbf{A}^{\top}\mathbf{r})^{\top}\mathbf{d}=0\). Expanding the squared norm therefore gives \(\|\mathbf{r}-\mathbf{A}\mathbf{d}\|^2=\|\mathbf{r}\|^2+\|\mathbf{A}\mathbf{d}\|^2\geq\|\mathbf{r}\|^2\). If \(\mathbf{d}\ne\mathbf{0}\), full column rank implies \(\mathbf{A}\mathbf{d}\ne\mathbf{0}\), so the inequality is strict. Thus \(\widehat{\boldsymbol{\beta}}\) is the unique minimizer, and its residual is orthogonal to the columns of \(\mathbf{A}\). \(\square\)
The proof gives the meaning of the residual check: the residual cannot be made shorter by adjusting the coefficients, because every change \(\mathbf{A}\mathbf{d}\) lies in the column space of \(\mathbf{A}\), perpendicular to the residual. It also identifies an important condition. If the columns are dependent, then a nonzero \(\mathbf{d}\) can satisfy \(\mathbf{A}\mathbf{d}=\mathbf{0}\); the fitted response may still be unique, but the coefficients need not be.
Worked Example: A One-Predictor Workflow
Worked Example: A One-Predictor Workflow
For observations \(x=0,1,2\) and responses \(y=1,2,2\), fit the model \(\beta_0+\beta_1x\). The intercept is included, so the design matrix is
A reduced QR factorization is given by \(\mathbf{Q}\) with columns \((1,1,1)^{\top}/\sqrt3\) and \((-1,0,1)^{\top}/\sqrt2\), and \(\mathbf{R}=\begin{bmatrix}\sqrt3&\sqrt3\\0&\sqrt2\end{bmatrix}\). Indeed, the first column of \(\mathbf{Q}\) times \(\sqrt3\) gives the first column of \(\mathbf{A}\); the second column of \(\mathbf{Q}\) times \(\sqrt2\), plus the first column times \(\sqrt3\), gives \((0,1,2)^{\top}\).
Now \(\mathbf{Q}^{\top}\mathbf{y}=(5/\sqrt3,1/\sqrt2)^{\top}\). Solving \(\mathbf{R}\widehat{\boldsymbol{\beta}}=\mathbf{Q}^{\top}\mathbf{y}\), the second row gives \(\sqrt2\widehat{\beta}_1=1/\sqrt2\), so \(\widehat{\beta}_1=1/2\). The first row gives \(\sqrt3(\widehat{\beta}_0+\widehat{\beta}_1)=5/\sqrt3\), hence \(\widehat{\beta}_0+\widehat{\beta}_1=5/3\), and \(\widehat{\beta}_0=7/6\).
Check: the fitted responses are \(7/6,5/3,13/6\), so \(\mathbf{r}=(-1/6,1/3,-1/6)^{\top}\). The residual sums to \(-1/6+1/3-1/6=0\), and its dot product with the predictor column is \(0(-1/6)+1(1/3)+2(-1/6)=0\). Thus \(\mathbf{A}^{\top}\mathbf{r}=\mathbf{0}\), verifying the fit.
Interpreting Coefficients and Predictions
Each coefficient must be interpreted relative to the model and the units in which its predictors were recorded. In the two-predictor fit, \(\widehat{\beta}_0=2\) is the model's predicted response when \(x=0\) and \(z=0\). The coefficient \(\widehat{\beta}_1=3\) is the change in predicted response for a one-unit increase in \(x\), holding \(z\) fixed. The coefficient \(\widehat{\beta}_2=-1\) is the change for a one-unit increase in \(z\), holding \(x\) fixed. These are descriptions of the fitted formula; they do not, by themselves, establish why the response changes.
Worked Example: Use the Fit, Then Check the Prediction's Scope
Use the fitted model \(2+3x-z\) from the four-observation example to predict at \(x=2,z=1\). Substitution gives \(2+3(2)-1=7\). This is an exact evaluation of the fitted formula.
The observed \(x\)-values were \(1,0,-1,0\), so \(x=2\) lies outside their observed range. The numerical prediction is therefore an extrapolation in \(x\), not a claim that the data directly confirm a response of \(7\) at that predictor pair. A fitted equation can produce predictions beyond the data, but the calculation alone does not show that its pattern remains appropriate there.
Regression coefficients are also conditional on which columns were included. An omitted predictor, a nonlinear relationship, measurement error, or observations with unusual influence can limit the fit's usefulness. Residual orthogonality is a mathematical property of a least-squares solution; it does not prove that the chosen model captures the underlying process or that a coefficient has a causal interpretation.
Common Mistakes and a Rank Check
Mistake: treating \(\mathbf{Q}^{\top}\mathbf{y}\) as the coefficient vector. In the one-predictor example, that vector is \((5/\sqrt3,1/\sqrt2)^{\top}\), not \((\widehat{\beta}_0,\widehat{\beta}_1)^{\top}\). The triangular matrix \(\mathbf{R}\) still has to be accounted for. The correct calculation solves \(\mathbf{R}\widehat{\boldsymbol{\beta}}=\mathbf{Q}^{\top}\mathbf{y}\), giving \((7/6,1/2)^{\top}\). Substituting these coefficients recovers the fitted responses \(7/6,5/3,13/6\); using \(\mathbf{Q}^{\top}\mathbf{y}\) directly would not.
Mistake: assuming a QR least-squares workflow always identifies every coefficient. Consider a model with an intercept and two identical predictor columns, with predictor values \(0,1,2\) and responses \(1,3,5\). Its design matrix is \(\begin{bmatrix}1&0&0\\1&1&1\\1&2&2\end{bmatrix}\). The coefficient vectors \((1,2,0)^{\top}\) and \((1,0,1)^{\top}\) both give fitted responses \((1,3,5)^{\top}\), as direct multiplication verifies. The two predictor columns are dependent, so the individual slopes are not uniquely determined. The fitted response is the same because only the sum of the two slope coefficients affects it.
In practice, inspect the design matrix before interpreting coefficients. Confirm that its columns correspond to the intended model terms, check whether its columns are independent when unique coefficients are needed, and verify the result by computing both fitted responses and residuals. A correct solution to the matrix problem is only as informative as the model encoded by that matrix.
Check Your Understanding
For each question, connect the matrix calculation to the model it represents.
- For a model \(\beta_0+\beta_1x+\beta_2z\) and observations \((x,z)=(2,0)\) and \((x,z)=(-1,3)\), write the design matrix.
- Why does the reduced-QR least-squares calculation solve \(\mathbf{R}\widehat{\boldsymbol{\beta}}=\mathbf{Q}^{\top}\mathbf{y}\)? State the required QR properties.
- For the one-predictor fit above, verify that the residual is orthogonal to both columns of the design matrix.
- What does the coefficient of \(z\) describe in a two-predictor model, and what does it not establish on its own?
- If two design-matrix columns are identical, can the corresponding two coefficients necessarily be interpreted separately? Explain using the prediction formula.
