Regression Design Matrices for Linear Models

From a Regression Formula to a Matrix

A linear regression model describes a response by combining predictor values with coefficients. Writing the model as a matrix-vector product makes the bookkeeping explicit: each row records one observation, each column records one model term, and the coefficient vector gives the weight assigned to each column. This representation also makes it possible to calculate all predicted responses at once.

Suppose there are \(m\) observations and \(p\) numerical predictors. For observation \(i\), write \(y_i\) for its observed response and \(x_{ij}\) for the value of predictor \(j\). A model with an intercept and a separate coefficient for each predictor has the form

Here \(\hat y_i\) is the response predicted by the model, while \(y_i\) is the response actually observed. The coefficients \(\beta_0,\beta_1,\dots,\beta_p\) are real numbers. In this tutorial, we take the coefficients as given in order to construct and interpret predictions; deciding how to estimate them from data is a separate task.

The first column consists entirely of ones because multiplying it by \(\beta_0\) supplies the intercept in every observation’s prediction. The next column contains the values of the first predictor, the following column contains the values of the second predictor, and so on. The dimensions agree: an \(m\times(p+1)\) matrix times a \((p+1)\times1\) vector produces an \(m\times1\) vector.

Why Each Row Gives One Prediction

The matrix equation is not a new model added on top of the scalar equations. It is a compact way to write those equations together.

Proof. By the definition of matrix-vector multiplication, entry \(i\) of \(\mathbf{X}\boldsymbol{\beta}\) is the dot product of row \(i\) of \(\mathbf{X}\) with \(\boldsymbol{\beta}\). Substituting the stated row and coefficient vector gives \((1,x_{i1},\dots,x_{ip})(\beta_0,\beta_1,\dots,\beta_p)^{\top} =1\beta_0+x_{i1}\beta_1+\cdots+x_{ip}\beta_p\). This is precisely the prediction specified for observation \(i\). Since this holds for each \(i=1,\dots,m\), the full product lists all \(m\) predictions in observation order. \(\square\)

There is also a useful column-wise reading. If \(\mathbf{x}_j\) denotes the column of values for predictor \(j\), then the product adds the intercept column weighted by \(\beta_0\) to the predictor columns weighted by their respective coefficients:

This equation shows how each model term contributes to the prediction vector. The coefficient \(\beta_j\) scales the entire column of observed values for predictor \(j\), while \(\beta_0\) adds the same amount to every predicted response.

Worked Examples: Constructing and Using Design Matrices

What the Coefficients Mean

In a model with an intercept, \(\beta_0\) is the predicted response when all predictors equal zero, because substituting \(x_{i1}=\cdots=x_{ip}=0\) leaves \(\hat y_i=\beta_0\). Whether that zero-predictor case is meaningful depends on the context and on the values the predictors can take.

For predictor \(j\), the coefficient \(\beta_j\) describes how the prediction changes when that predictor changes while the other predictor values remain fixed. More precisely, increasing \(x_{ij}\) by an amount \(d\), with all other predictors unchanged, changes the prediction by \(d\beta_j\).

Proof. Let the original prediction be \(\hat y_i=\beta_0+\sum_{k=1}^{p}\beta_kx_{ik}\). After the stated change, the new prediction is \(\hat y_i'=\beta_0+\sum_{k\ne j}\beta_kx_{ik}+\beta_j(x_{ij}+d)\). Expanding the final term gives \(\hat y_i'=\beta_0+\sum_{k\ne j}\beta_kx_{ik}+\beta_jx_{ij}+\beta_jd=\hat y_i+\beta_jd\). Therefore \(\hat y_i'-\hat y_i=d\beta_j\), as claimed. The argument applies for every predictor index \(j\), including when \(d=0\). \(\square\)

For example, in the two-predictor model above, increasing the first predictor by \(1\) while holding the second fixed changes the prediction by \(3(1)=3\). Increasing the second predictor by \(1\) while holding the first fixed changes it by \((-1)(1)=-1\). These are statements about the model’s predictions, not claims about cause and effect.

A Reliable Construction and a Common Mistake

Common mistake: putting observations in columns. In the two-predictor example, a mistaken transposition would use \(\mathbf{X}^{\top}\) instead of \(\mathbf{X}\). Since that particular matrix is square, the multiplication is still possible, which can hide the error:

These are not the predictions for the three observations: the correct fitted vector is \((7,2,9)^{\top}\). The first entry of the transposed product, for instance, combines the coefficients as \(1(4)+1(3)+1(-1)=6\), rather than combining the first observation’s row with the coefficients. The correction is to put observations in rows and predictors in columns, then verify the first and last predictions by direct substitution.

Dimensions are a useful warning, although not a complete safeguard. With \(m\) observations and \(p\) predictors in a model with an intercept, \(\mathbf{X}\) must be \(m\times(p+1)\), \(\boldsymbol{\beta}\) must be \((p+1)\times1\), and \(\widehat{\mathbf{y}}\) must be \(m\times1\). When the matrix is square, as in the mistaken calculation, checking dimensions alone will not catch a transposition, so the row-by-row meaning still matters.