Design Matrices For Category-Specific Predictors

From Predictors to Logit Design Matrices

In a baseline-category multinomial logit model, each nonbaseline response category has its own linear predictor. The design matrix records which predictor values enter each predictor, while the coefficient vector records the matching parameters. Building the matrix carefully matters: a correct collection of predictor values paired with coefficients in the wrong order does not represent the intended model.

Let the response have \(K\) categories, with category \(K\) chosen as the response baseline. For observation \(i\), write \(\eta_{ij}\) for the linear predictor of category \(j\) relative to category \(K\), for \(j=1,\ldots,K-1\). A general category-specific specification is \(\eta_{ij}=z_{ij}^{T}\beta_j\), where \(z_{ij}\) is a vector of \(q_j\) known predictor values and \(\beta_j\) is its coefficient vector. The predictors, and even their number, may differ across logits.

Definition: For logit \(j\), the design matrix \(Z_j\) is the \(n\times q_j\) matrix whose \(i\)th row is \(z_{ij}^{T}\). The category-specific parameter block \(\beta_j\) is ordered to match the columns of \(Z_j\), so the vector of linear predictors for that logit is \(Z_j\beta_j\). A factor’s coding determines its design columns; it does not change which response category is the baseline.

A common model uses the same \(p\) predictor columns for every logit, but gives each logit its own coefficients. In that case \(Z_j=X\) and \(q_j=p\) for all \(j\). “The same predictors” does not mean “the same coefficients”: the parameter blocks remain distinct unless the model explicitly constrains them to be equal.

Stacking by Logit

One convenient convention places all \(n\) linear predictors for logit 1 first, then all \(n\) predictors for logit 2, and so on. Use the coefficient order \(\beta=(\beta_1^T,\ldots,\beta_{K-1}^T)^T\), and define \(\eta_{\mathrm{block}}=(\eta_{11},\ldots,\eta_{n1},\eta_{12},\ldots,\eta_{n,K-1})^T\). Then

\eta_{\mathrm{block}}= \begin{pmatrix} Z_1&0&\cdots&0\\ 0&Z_2&\cdots&0\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\cdots&Z_{K-1} \end{pmatrix} \begin{pmatrix} \beta_1\\ \beta_2\\ \vdots\\ \beta_{K-1} \end{pmatrix}.

The block-diagonal matrix has \(n(K-1)\) rows and \(\sum_{j=1}^{K-1}q_j\) columns. Its first block of rows gives all observations for logit 1; its second block gives all observations for logit 2. This row convention is useful precisely because each block multiplies the coefficient block with the same index.

Worked Example: Two Logits With a Shared Numeric Predictor

There are \(n=2\) observations and \(K=3\) response categories, with category 3 as the baseline. Each logit has an intercept and the same numeric predictor \(x\), but separate coefficients. Let \(x_1=0\), \(x_2=2\), and use the design matrix \(X\) with columns ordered as intercept, then \(x\):

X=\begin{pmatrix}1&0\\1&2\end{pmatrix},\qquad \beta_1=\begin{pmatrix}0.2\\0.5\end{pmatrix},\qquad \beta_2=\begin{pmatrix}-0.4\\-0.1\end{pmatrix}.

With logit-major stacking, the full design matrix and coefficient vector are

D_{\mathrm{block}}= \begin{pmatrix} 1&0&0&0\\ 1&2&0&0\\ 0&0&1&0\\ 0&0&1&2 \end{pmatrix},\qquad \beta= \begin{pmatrix}0.2\\0.5\\-0.4\\-0.1\end{pmatrix}.

The four rows give both observations for logit 1, followed by both observations for logit 2. Multiplication gives

D_{\mathrm{block}}\beta = \begin{pmatrix} 0.2\\ 0.2+2(0.5)\\ -0.4\\ -0.4+2(-0.1) \end{pmatrix} = \begin{pmatrix}0.2\\1.2\\-0.4\\-0.6\end{pmatrix}.

Thus the logit-1 predictors are \(0.2,1.2\), and the logit-2 predictors are \(-0.4,-0.6\). The row order follows the blocks of \(D_{\mathrm{block}}\); it is not observation-major.

Stacking by Observation

Sometimes it is preferable to keep the predictors for one observation together. Define \(\eta_{\mathrm{obs}}=(\eta_{11},\ldots,\eta_{1,K-1},\eta_{21},\ldots,\eta_{n,K-1})^T\), so that the entries for observation 1 come first, then those for observation 2, and so forth. Keep the same logit-major coefficient vector \(\beta\). The corresponding design matrix \(D_{\mathrm{obs}}\) has one row for each pair \((i,j)\), in this observation-major order. That row contains \(z_{ij}^T\) in the columns for \(\beta_j\), and zeros in the columns for every other coefficient block. In particular, for two logits and two observations its rows are ordered \((1,1),(1,2),(2,1),(2,2)\): observation 1 for logit 1, observation 1 for logit 2, observation 2 for logit 1, observation 2 for logit 2.

Result: The block-stacked and observation-stacked representations contain the same linear predictors in different row orders. If \(P\) is the permutation matrix that reorders logit-major entries into observation-major entries, then \(D_{\mathrm{obs}}=P D_{\mathrm{block}}\) and \(\eta_{\mathrm{obs}}=P\eta_{\mathrm{block}}\).

Proof. In logit-major order, the entry for observation \(i\), logit \(j\), is in position \((j-1)n+i\). In observation-major order, the same entry is in position \((i-1)(K-1)+j\). Define \(P\) to move the coordinate at the first position to the second, for every \(i\) and \(j\). This reorders coordinates only; it does not change their values or the coefficient vector. Since \(\eta_{\mathrm{block}}=D_{\mathrm{block}}\beta\),

\eta_{\mathrm{obs}} =P\eta_{\mathrm{block}} =P D_{\mathrm{block}}\beta.

The row of \(P D_{\mathrm{block}}\) at observation-major position \((i-1)(K-1)+j\) is exactly the row containing \(z_{ij}^T\) in block \(j\) and zeros elsewhere. That is the defining row of \(D_{\mathrm{obs}}\). Therefore \(D_{\mathrm{obs}}=P D_{\mathrm{block}}\), as claimed. \(\square\)

Worked Example: Checking Observation-Major Ordering

Continue with the two-observation, two-logit model above. With the coefficient order \((\beta_{10},\beta_{11},\beta_{20},\beta_{21})^T\), observation-major stacking requires the rows in the order: observation 1 for logit 1; observation 1 for logit 2; observation 2 for logit 1; observation 2 for logit 2. Thus

D_{\mathrm{obs}}= \begin{pmatrix} 1&0&0&0\\ 0&0&1&0\\ 1&2&0&0\\ 0&0&1&2 \end{pmatrix}.

Multiplying by the same \(\beta\) gives

D_{\mathrm{obs}}\beta = \begin{pmatrix} 0.2\\ -0.4\\ 1.2\\ -0.6 \end{pmatrix}.

This is the same four calculated predictors as before, now ordered by observation. In contrast, the earlier block-diagonal matrix placed the two logit-1 predictors first. Either convention is valid, but the row order must be stated and used consistently.

Factors and Chosen Contrasts

A factor with \(L\) levels is represented by numeric columns chosen through a coding scheme. With an intercept and \(L-1\) contrast columns, a row of the design matrix consists of the intercept followed by the contrast values for that observation’s factor level. The same factor coding can be used in every logit, or a model can specify different design columns for different logits. A factor’s reference level in treatment coding is a predictor-coding choice; it is not the response baseline category.

For treatment coding, choose one factor level as the reference and make one indicator column for each other level. For a factor with levels A, B, and C and A as the reference, the rows, in columns (intercept, indicator for B, indicator for C), are \((1,0,0)\), \((1,1,0)\), and \((1,0,1)\). Each nonbaseline logit has its own coefficient block for these columns.

Worked Example: Treatment-Coded Factor in Two Logits

Suppose three observations have factor levels A, C, and B, in that order. There are three response categories, with category 3 as the baseline. Use A as the factor’s treatment reference level, and include an intercept in both logits. The design matrix is

X= \begin{pmatrix} 1&0&0\\ 1&0&1\\ 1&1&0 \end{pmatrix}.

The coefficient columns are ordered as intercept, B indicator, C indicator in both logits. Choose \(\beta_1=(0.1,0.7,-0.2)^T\) and \(\beta_2=(-0.3,0.4,0.6)^T\). The matrix products are

X\beta_1= \begin{pmatrix} 0.1\\ 0.1-0.2\\ 0.1+0.7 \end{pmatrix} = \begin{pmatrix}0.1\\-0.1\\0.8\end{pmatrix},\qquad X\beta_2= \begin{pmatrix} -0.3\\ -0.3+0.6\\ -0.3+0.4 \end{pmatrix} = \begin{pmatrix}-0.3\\0.3\\0.1\end{pmatrix}.

The first and second entries of each product correspond to observations at A and C, respectively, and the third corresponds to B. If these logits are assembled in logit-major order, the full design matrix is \(\operatorname{diag}(X,X)\), and the coefficient vector is \((\beta_1^T,\beta_2^T)^T\).

Other contrast systems are also valid. For example, for levels A, B, and C, choose two sum contrasts \(c_1=(1,0,-1)^T\) and \(c_2=(0,1,-1)^T\), with entries indexed by level. The factor design rows, in columns (intercept, \(c_1\), \(c_2\)), are \((1,1,0)\), \((1,0,1)\), and \((1,-1,-1)\). The coding should be stated explicitly so that the design columns and coefficient ordering are unambiguous.

Result: If two full-rank contrast codings include the same intercept and span the same predictor-column space, they produce the same set of possible linear predictors after a one-to-one coefficient reparameterization.

Proof. Let \(Z_A\) and \(Z_B\) be the two \(n\times p\) design matrices. Because their column spaces are equal and both have full column rank \(p\), every column of \(Z_B\) is a unique linear combination of columns of \(Z_A\). Thus \(Z_B=Z_A A\) for a \(p\times p\) matrix \(A\). Full column rank of \(Z_B\) implies \(A\) is nonsingular: if \(Av=0\), then \(Z_Bv=Z_AAv=0\), so \(v=0\). For any coefficient vector \(\gamma\) under coding \(B\), take \(\beta=A\gamma\); then \(Z_B\gamma=Z_AA\gamma=Z_A\beta\). Conversely, nonsingularity gives \(\gamma=A^{-1}\beta\) for every \(\beta\). Hence the two codings yield exactly the same possible predictor vectors. This argument applies separately to each logit’s block when the same full-rank recoding is used across logits. \(\square\)

Worked Example: Relating Sum and Treatment Contrasts

For levels A, B, and C, let \(X_T\) be the treatment-coded matrix with columns (intercept, B indicator, C indicator), and let \(X_S\) use the sum contrasts above. For rows ordered A, B, C,

X_T= \begin{pmatrix}1&0&0\\1&1&0\\1&0&1\end{pmatrix}, \qquad X_S= \begin{pmatrix}1&1&0\\1&0&1\\1&-1&-1\end{pmatrix}.

The sum-contrast column \(c_1\) equals the intercept column minus the B indicator column minus twice the C indicator column; \(c_2\) equals the B indicator column minus the C indicator column. Therefore

X_S=X_T A,\qquad A= \begin{pmatrix} 1&0&0\\ -1&1&0\\ -2&0&-1 \end{pmatrix}.

For example, direct multiplication gives rows \((1,1,0)\), \((1,0,1)\), and \((1,-1,-1)\), as required. The determinant of \(A\) is \(-1\), so \(A\) is nonsingular. If a logit has sum-coded coefficients \(\gamma\), its equivalent treatment-coded coefficients are \(\beta=A\gamma\), because \(X_S\gamma=X_TA\gamma\). This is a change of coordinates, not a change in the set of fitted linear predictors.

Category-Specific Predictor Blocks

The block-diagonal construction also handles logits with different predictor sets. For example, one logit might contain an intercept, a numeric predictor, and a factor; another might contain only an intercept and the numeric predictor. Write the two corresponding design matrices as \(Z_1\) and \(Z_2\), with their own column counts \(q_1\) and \(q_2\). Then the full parameter vector has length \(q_1+q_2\), and the block design has \(n(K-1)\) rows and \(q_1+q_2\) columns. It is not necessary to insert a coefficient for a predictor that is absent from a logit; the relevant block simply has fewer columns.

When a predictor is shared across logits but its effect is constrained to be identical, the model no longer has separate unconstrained coefficients for that predictor in each block. Such constraints can be represented by a lower-dimensional parameterization, but the unconstrained block-diagonal formula above is specifically for separate coefficient blocks. State the parameterization before assembling the matrix; otherwise, the matrix dimensions alone can conceal an unintended restriction or duplication.

Common Mistakes and a Construction Checklist

The most frequent error is mixing row orderings. A block-diagonal matrix has rows grouped by logit, not by observation. Writing its result as though the rows were grouped by observation silently assigns predictors to the wrong observations. A second error is confusing the response baseline with a predictor factor’s reference level: category \(K\) defines the logits, whereas a level such as A defines one particular factor coding. Neither choice determines the other.

Before using a design matrix in a likelihood or fitting calculation, check these items:

State which response category is the baseline and which logits are included.
For each logit, list the predictor columns in order and the matching coefficient block.
Specify factor levels, contrast columns, and any factor reference level.
State whether rows are ordered by logit or by observation.
Check dimensions and multiply at least one row by its coefficient block to verify the intended predictor.

These checks are not cosmetic. The design matrix is the exact map from parameters to linear predictors. Keeping its blocks and ordering explicit makes category-specific models easier to specify, audit, and extend.

Check Your Understanding

Use the stated coefficient and row-order conventions in your answers.

With \(n=4\) and \(K=3\), how many rows does a logit-major block design matrix have, and which observations appear in its first block of rows?
For a factor with levels A, B, and C under treatment coding with A as reference, write the three design rows when columns are ordered as intercept, B indicator, C indicator.
Explain why choosing A as a predictor factor’s reference level does not make A the response baseline category.
Suppose the coefficient vector is ordered by logit, but the design rows are ordered by observation. What must be changed or specified to ensure the matrix product has the intended entries?
What conditions on two contrast design matrices allow a one-to-one reparameterization that preserves all possible linear predictors?