Why Constructing the Design Matrix Matters
A regression model written in words or as an equation must be translated into a numerical form before its terms can be used in a calculation. The design matrix records the value of every model term for every observation. Constructing it carefully is especially important when a model includes categorical predictors and interactions: the names of the terms alone do not guarantee that their columns have been coded correctly.
The task is mechanical, but it is not merely clerical. A missing interaction column represents a different model; an incorrectly coded indicator can make a category’s observations look as though they belong to the reference group. A reliable approach is to fix the model’s term list and column order first, then calculate each row from that observation’s predictor values.
Each observation supplies one row, and each term in the specified model supplies one column. If the model has an intercept, its column consists of ones. A numeric predictor contributes its observed numeric value. A categorical predictor is represented by indicator columns under a chosen coding scheme. An interaction contributes a column formed by multiplying the values of its component terms for each observation.
For constructing \(X\), no normality assumption or particular error distribution is needed. The model specifies which columns to build. If the model is used for ordinary least squares inference, additional assumptions may be relevant, but they do not change how the entries of the design matrix are calculated.
A Row-by-Row Construction Method
Before doing arithmetic, write the model with every included term visible. Then list the columns in a fixed order. For each observation, record the intercept value, the numeric values, the indicator values, and the products required by the model. Keeping that order unchanged from row to row is essential: a column cannot mean one thing in one row and another thing in the next.
Include the intercept if specified, all main effects, and only the interactions included in the model.
For example, list the intercept, numeric predictors, indicator variables, and then interaction terms.
Enter its numeric values, set each indicator to zero or one according to the stated reference category, and calculate each product.
Confirm that every product is calculated from that row and that the row has exactly one entry for every model term.
The matrix notation follows directly from the row equations. If the \(i\)th row is \((x_{i1},\ldots,x_{ip})\) and the coefficients are \((\beta_1,\ldots,\beta_p)\), their product is \(x_{i1}\beta_1+\cdots+x_{ip}\beta_p\). Thus, each row of \(X\beta\) gives the model’s linear part for the corresponding observation. Stacking those \(n\) row equations gives \(X\beta\); adding the error vector gives \(y=X\beta+\varepsilon\). An interaction is a column like any other: its entry is the specified product of predictor values for that observation.
Worked Example: A Binary Group and a Numeric Predictor
A hypothetical insurer studies a response for four policy records using a numeric risk score \(x\) and a binary indicator \(D\), where \(D=1\) identifies the enhanced-coverage group and \(D=0\) identifies the standard-coverage reference group. The specified model is \(y_i=\beta_0+\beta_1x_i+\beta_2D_i+\beta_3x_iD_i+\varepsilon_i\). Construct the design matrix for records \((x,D)=(0,0),(2,0),(1,1),(3,1)\).
Method. Use the column order \(1,x,D,xD\). The first column is the intercept; the last is the product of the numeric value and the indicator.
For record 1, \(x=0\) and \(D=0\), so \(xD=0 \times 0=0\). Its row is \((1,0,0,0)\). For record 2, \(x=2\) and \(D=0\), so \(xD=2 \times 0=0\), giving \((1,2,0,0)\). For record 3, \(x=1\) and \(D=1\), so \(xD=1 \times 1=1\), giving \((1,1,1,1)\). For record 4, \(x=3\) and \(D=1\), so \(xD=3 \times 1=3\), giving \((1,3,1,3)\).
Check. The indicator is zero in the first two rows, so the interaction column is also zero there. In each of the last two rows, the indicator is one, so the interaction entry equals \(x\). This verifies all four interaction entries. A common mistake avoided here is to enter \(D\) in the interaction column instead of multiplying \(x\) by \(D\).
Multiple Categories and Their Interactions
With a categorical predictor having \(K\) levels, a common reference-category coding uses \(K-1\) indicators. The reference level has zero in all of these columns. Each nonreference level has one in its own indicator column and zero in the other indicator columns. If a numeric predictor interacts with the categorical predictor, the model may include a separate product of that predictor with each nonreference indicator. The product columns are constructed observation by observation just like a binary interaction.
The model determines which interaction columns belong in \(X\). Do not add a product just because two predictors appear in the same model, and do not omit a specified product because it seems redundant with a main-effect column. Construct exactly the listed terms.
Worked Example: Three Categories With a Numeric Interaction
A hypothetical study records exposure duration \(t\) and service area, with areas North, Central, and South. North is the reference category. Define \(C=1\) for Central and \(0\) otherwise, and \(S=1\) for South and \(0\) otherwise. The specified model includes \(1,t,C,S,tC,tS\). Construct \(X\) for observations (North, 2), (Central, 1), (Central, 4), and (South, 3), where each pair gives (area, \(t\)).
Method. Fix the column order as \(1,t,C,S,tC,tS\). In the North row, both indicators are zero, so both products are zero: \((1,2,0,0,0,0)\). For Central with \(t=1\), \(C=1\), \(S=0\), \(tC=1 \times 1=1\), and \(tS=1 \times 0=0\), giving \((1,1,1,0,1,0)\). For Central with \(t=4\), the products are \(4 \times 1=4\) and \(4 \times 0=0\), giving \((1,4,1,0,4,0)\). For South with \(t=3\), \(C=0\), \(S=1\), \(tC=3 \times 0=0\), and \(tS=3 \times 1=3\), giving \((1,3,0,1,0,3)\).
Check. The Central rows have \(C=1\), so their \(tC\) entries equal their respective \(t\) values, while their \(tS\) entries are zero. The South row has the reverse pattern, and the North row has both indicators and both products equal to zero. The common mistake avoided is assigning an indicator to the reference category; North must have \(C=S=0\) under this coding.
Combining Numeric and Categorical Products
A model can include more than one numeric predictor and more than one interaction. In that case, calculate every product separately from the row’s original predictor values. For example, \(xz\) uses the numeric values \(x\) and \(z\); \(xD\) uses \(x\) and the indicator \(D\). These are distinct columns, even if some observations happen to give them the same numerical value.
Worked Example: Two Numeric Predictors and a Group Indicator
A hypothetical portfolio analysis uses numeric predictors \(x\) and \(z\), plus an indicator \(D\) for a specialist service group (\(D=1\)); the comparison group has \(D=0\). The model includes all terms \(1,x,z,D,xz,xD,zD\). For observations \((x,z,D)=(1,2,0),(2,1,1),(0,3,1),(3,2,0),(1,4,1)\), construct the design matrix.
Method. Use columns \(1,x,z,D,xz,xD,zD\). Calculate the three products independently for each row:
- For \((1,2,0)\): \(xz=1 \times 2=2\), \(xD=1 \times 0=0\), and \(zD=2 \times 0=0\). The row is \((1,1,2,0,2,0,0)\).
- For \((2,1,1)\): \(xz=2 \times 1=2\), \(xD=2 \times 1=2\), and \(zD=1 \times 1=1\). The row is \((1,2,1,1,2,2,1)\).
- For \((0,3,1)\): \(xz=0 \times 3=0\), \(xD=0 \times 1=0\), and \(zD=3 \times 1=3\). The row is \((1,0,3,1,0,0,3)\).
- For \((3,2,0)\): \(xz=3 \times 2=6\), \(xD=3 \times 0=0\), and \(zD=2 \times 0=0\). The row is \((1,3,2,0,6,0,0)\).
- For \((1,4,1)\): \(xz=1 \times 4=4\), \(xD=1 \times 1=1\), and \(zD=4 \times 1=4\). The row is \((1,1,4,1,4,1,4)\).
Check. The \(xz\) entries are \(2,2,0,6,4\), matching the five products \(1 \times 2,2 \times 1,0 \times 3,3 \times 2,1 \times 4\). The \(xD\) and \(zD\) columns are zero whenever \(D=0\), as required. A common mistake avoided is setting every interaction to zero for a group-reference observation: \(xz\) does not contain \(D\), so it remains \(6\) in the fourth row even though \(D=0\).
Checks, Errors, and What the Matrix Represents
A completed matrix should pass several simple checks. Its number of rows must equal the number of observations; its number of columns must equal the number of specified model terms. Every intercept entry should be one if the model includes an intercept. Each indicator should match the stated coding rule, and each product column should equal the product of the relevant columns in that same row.
One exam trap is to confuse an interaction column with an indicator or a main effect. Suppose the specified terms are \(1,x,D,xD\), and an observation has \(x=3,D=1\). A wrong row is \((1,3,1,1)\), because it copies \(D\) into the \(xD\) column. The correction is \(xD=3 \times 1=3\), so the row is \((1,3,1,3)\). Conversely, when \(D=0\), \(xD=0\), but another product such as \(xz\) need not be zero.
A second check is to read each row back into the model-term list. If a row has seven entries but the model has six terms, a column has been added accidentally; if it has five entries, a term has been omitted. Also ensure that the response is not inserted as a predictor column: the design matrix contains the explanatory terms, while the observed responses form \(y\).
Check Your Understanding
For each problem, write the columns in the stated order and check the products within each observation.
- A model uses \(1,x,D,xD\), where \(D=1\) denotes the mobile-service group. Construct \(X\) for \((x,D)=(2,0),(1,1),(4,1)\). (Answer: \(\begin{bmatrix}1&2&0&0\\1&1&1&1\\1&4&1&4\end{bmatrix}\))
- A three-level predictor has levels East, West, and Central, with East as reference. Let \(W\) and \(C\) be the West and Central indicators. For the model columns \(1,W,C\), construct rows for West, East, Central, Central. (Answer: \(\begin{bmatrix}1&1&0\\1&0&0\\1&0&1\\1&0&1\end{bmatrix}\))
- Using the model terms \(1,t,C,tC\), where \(C=1\) for the Central group and zero for the reference group, construct the matrix for (reference, \(t=3\)), (Central, \(t=2\)), and (Central, \(t=5\)). (Answer: \(\begin{bmatrix}1&3&0&0\\1&2&1&2\\1&5&1&5\end{bmatrix}\))
- A model includes \(1,x,z,D,xz,xD\). Construct \(X\) for \((x,z,D)=(2,3,0),(1,4,1),(3,0,1)\), using exactly that column order. (Answer: \(\begin{bmatrix}1&2&3&0&6&0\\1&1&4&1&4&1\\1&3&0&1&0&3\end{bmatrix}\))