From a Regression Equation to a Matrix
In multiple regression, each observation has a response and several predictor values. A design matrix organizes those predictor values so that one matrix equation represents all the observation-level regression equations at once. Constructing it correctly is a bookkeeping task with important consequences: a misplaced row or mismatched coefficient can describe a different model from the one intended.
Suppose there are \(n\) observations and \(p\) numeric predictors. For observation \(i\), let \(Y_i\) be the response and let \(x_{ij}\) be the value of predictor \(j\). With an intercept, the multiple regression model is
Here \(\beta_0\) is the intercept, \(\beta_j\) is the coefficient paired with predictor \(j\), and \(\varepsilon_i\) is the error for observation \(i\). The design matrix records the observed predictor values; it does not contain the response or the errors.
In this notation the model is \(\mathbf{Y}=X\boldsymbol{\beta}+\boldsymbol{\varepsilon}\), where \(\boldsymbol{\varepsilon}=(\varepsilon_1,\ldots,\varepsilon_n)^\mathsf{T}\). The dimensions provide an immediate check: \(X\) has \(n\) rows and \(p+1\) columns, \(\boldsymbol{\beta}\) has \(p+1\) entries, and \(X\boldsymbol{\beta}\) has \(n\) entries. The intercept column is necessary because multiplying a row of ones by \(\beta_0\) contributes \(\beta_0\) to each observation’s linear predictor.
Why the Matrix Equation Represents the Model
The key is the rule for matrix multiplication: the \(i\)th entry of \(X\boldsymbol{\beta}\) is the sum of the products of row \(i\) of \(X\) with the entries of \(\boldsymbol{\beta}\). If row \(i\) is \((1,x_{i1},\ldots,x_{ip})\), that sum is exactly the intercept plus the predictor terms in observation \(i\)’s regression equation.
For observation \(i\), the systematic part is \(\beta_0+\sum_{j=1}^{p}\beta_jx_{ij}\).
Write \((1,x_{i1},\ldots,x_{ip})\). The 1 supplies the intercept term.
The row product \((1,x_{i1},\ldots,x_{ip})\boldsymbol{\beta}\) equals the systematic part of the equation.
Putting the rows together gives \(X\); stacking responses and errors in that same order gives \(\mathbf{Y}\) and \(\boldsymbol{\varepsilon}\).
This justification is about representation, not estimation. Writing \(X\boldsymbol{\beta}\) does not by itself determine coefficient values or say how to estimate them. It only encodes the model equations compactly.
Worked Example: Building a Matrix for Two Claim Predictors
A hypothetical insurer records three claims. The response \(Y_i\) is paid amount in thousands of dollars, \(x_{i1}\) is exposure duration in months, and \(x_{i2}\) is claimant age in years. The observations, in their recorded order, are \((x_{i1},x_{i2},Y_i)=(2,30,2.1),(5,50,2.4),(3,40,2.2)\). Use the model \(Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i\), with \(\boldsymbol{\beta}=(2,0.4,-0.03)^\mathsf{T}\). For this illustration, assume independent errors with \(\varepsilon_i\sim N(0,0.25)\), so the error mean is zero and its variance is \(0.25\) in squared response units. Construct \(X\) and \(\mathbf{Y}\), then calculate \(X\boldsymbol{\beta}\).
Method. Use the column order intercept, exposure duration, claimant age. Insert a 1 in the first position of every row, and keep the observations in their given order. The design matrix and response vector are
The matrix has three rows, one per claim, and three columns: one intercept column and two predictor columns. Multiply each row by the coefficient vector:
Thus the three linear predictor values are \(1.9\), \(2.5\), and \(2.0\) thousand dollars, in the same order as the claims. The observed responses are not required to construct \(X\); they form the separate vector \(\mathbf{Y}\). The error vector under the model is \(\mathbf{Y}-X\boldsymbol{\beta}=(0.2,-0.1,0.2)^\mathsf{T}\). This check confirms that each component of the difference uses the matching row and response. The common mistake this setup avoids is treating the response column as an additional predictor column.
Column Order and Units
The columns of \(X\) and entries of \(\boldsymbol{\beta}\) must follow the same order. The order itself is a convention: the model is unchanged if predictor columns and their coefficients are permuted together. But changing only one of them changes the calculation. Naming the columns before writing the matrix is a reliable way to prevent this error.
Each coefficient also has a meaning tied to its predictor’s units. If the response is measured in hours and a predictor in kilometers, its coefficient is in hours per kilometer. This does not change how the matrix is built, but it is a useful check that a coefficient has been paired with the intended column.
Worked Example: Reordering Predictor Columns Correctly
A hypothetical repair center models repair time \(Y_i\), in hours, using complexity score \(x_{i1}\) and travel distance \(x_{i2}\), in kilometers. The two recorded cases are \((x_{i1},x_{i2},Y_i)=(2,100,9.2)\) and \((4,50,8.8)\). Use \(Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i\) with \(\boldsymbol{\beta}=(6,0.5,0.02)^\mathsf{T}\). Assume independent errors with \(\varepsilon_i\sim N(0,1)\), mean zero and variance \(1\) hour squared. Construct the representation in the stated order, then show an equivalent representation with the two predictor columns reversed.
Method. In the stated order, use intercept, complexity score, distance. This gives
The linear predictor for case 1 is \(6+2(0.5)+100(0.02)=6+1+2=9\) hours. For case 2 it is \(6+4(0.5)+50(0.02)=6+2+1=9\) hours. Therefore
If the columns are instead ordered intercept, distance, complexity score, the matrix must be changed to \(\begin{pmatrix}1&100&2\\1&50&4\end{pmatrix}\), and the coefficient vector must become \((6,0.02,0.5)^\mathsf{T}\). The row products remain \(9\) and \(9\). This verifies that a simultaneous permutation preserves the model. Reversing the matrix columns without reversing the matching coefficients would not.
Keeping Rows Aligned
A row represents one observation’s complete predictor profile, not one predictor and not one response category. The order of the rows is arbitrary as long as the same reordering is applied to the response vector and error vector. In practical calculations, the safest rule is to choose an observation order once and maintain it in every object.
For a new case, the predictor values form a new row with a leading 1. That row has \(p+1\) entries when the model includes an intercept. It can be multiplied by the coefficient vector in the same way as any existing row. A row for a new case is not itself a new coefficient or a new response.
Worked Example: Checking a New Observation Row
A hypothetical service plan study records monthly service calls \(x_{i1}\) and account age \(x_{i2}\) in years for three accounts. The responses \(Y_i\) are monthly service cost in dollars. The data, in order, are \((1,2,40),(3,1,50),(2,4,55)\), where each tuple is \((x_{i1},x_{i2},Y_i)\). Use the model \(Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i\), with \(\boldsymbol{\beta}=(20,5,3)^\mathsf{T}\). Assume independent errors with \(\varepsilon_i\sim N(0,16)\), mean zero and variance \(16\) dollars squared. Write \(X\), \(\mathbf{Y}\), and \(X\boldsymbol{\beta}\), then add the design row for a fourth account with 4 calls and 2 years of account age.
Method. Preserve the listed observation order, with columns intercept, monthly calls, and account age. The first three rows and response vector are
Compute each row product separately. For account 1, \(20+5(1)+3(2)=31\). For account 2, \(20+5(3)+3(1)=38\). For account 3, \(20+5(2)+3(4)=42\). Hence
The fourth account’s design row is \((1,4,2)\): the 1 represents the intercept, followed by its two predictor values in the established order. Its linear predictor is \(20+5(4)+3(2)=46\) dollars. The row does not change the first three rows or reorder their responses. The common mistake avoided here is putting the fourth account’s data in a predictor column rather than adding one complete observation row.
Common Mistakes and Checks
A small matrix can be checked in several independent ways before using it in later calculations. Count its rows and columns; label every column; verify that its first column is all ones when the model has an intercept; and expand one row product back into its regression equation. Also confirm that the response in position \(i\) belongs to the predictors in row \(i\).
For example, in the claim illustration, an incorrect matrix that omits the intercept column would be \(\begin{pmatrix}2&30\\5&50\\3&40\end{pmatrix}\). Multiplying this \(3\)-by-2 matrix by the three-entry vector \((2,0.4,-0.03)^\mathsf{T}\) is not defined: the inner dimensions, 2 and 3, do not match. Even if someone instead drops \(\beta_0\) and uses \((0.4,-0.03)^\mathsf{T}\), that computes a model with no intercept, not the stated model. The correction is to include the column of ones and retain the full coefficient vector \((\beta_0,\beta_1,\beta_2)^\mathsf{T}\).
- Omitting the intercept column: include a leading 1 for every observation when an intercept is in the model.
- Mislabeling columns: write the predictor order explicitly and use it for both \(X\) and \(\boldsymbol{\beta}\).
- Mixing observation orders: keep each response aligned with the predictor row from the same observation.
- Confusing data with parameters: observed predictor values go in \(X\); model coefficients go in \(\boldsymbol{\beta}\); observed responses go in \(\mathbf{Y}\).
Check Your Understanding
For each problem, write the requested representation or calculation and check that rows and coefficients remain aligned.
- A model has an intercept and two predictors \(x_1,x_2\). Three observations have predictor pairs \((2,5),(0,3),(4,1)\), in that order. Write the design matrix. (Answer: \(X=\begin{pmatrix}1&2&5\\1&0&3\\1&4&1\end{pmatrix}\).)
- For \(X=\begin{pmatrix}1&3&2\\1&1&4\end{pmatrix}\) and \(\boldsymbol{\beta}=(5,2,-1)^\mathsf{T}\), calculate \(X\boldsymbol{\beta}\). (Answer: \((9,3)^\mathsf{T}\).)
- A model uses columns in the order intercept, distance \(d\), then load \(l\), with coefficient vector \((10,0.2,3)^\mathsf{T}\). What is the correct design row for a case with \(l=4\) and \(d=15\), and what is its linear predictor? (Answer: row \((1,15,4)\); linear predictor \(25\).)
- Three responses are recorded in order \(Y_1,Y_2,Y_3\), and the corresponding predictor pairs are \((1,2),(3,0),(2,5)\). Write the response vector symbolically and the design matrix for a model with an intercept. (Answer: \(\mathbf{Y}=(Y_1,Y_2,Y_3)^\mathsf{T}\), \(X=\begin{pmatrix}1&1&2\\1&3&0\\1&2&5\end{pmatrix}\).)
- A design matrix has columns ordered as intercept, \(x_1\), \(x_2\), and coefficient vector \((4,-2,0.5)^\mathsf{T}\). If the columns are reordered to intercept, \(x_2\), \(x_1\), what coefficient vector preserves the same model? (Answer: \((4,0.5,-2)^\mathsf{T}\).)