Deriving the Matrix Form eta = X beta

From observation-level equations to one matrix equation

A generalized linear model assigns a linear predictor to each observation. With several observations, it is useful to collect these separate equations into one expression. The matrix equation \(\eta=X\beta\) does exactly that: each row of \(X\) contains the covariate values for one observation, and multiplying by the coefficient vector gives that observation’s predictor.

The goal here is to derive this equation carefully, including the dimensions that make the multiplication valid. The result is a compact way to write the same observation-by-observation model, not a different model. No assumption about the rank of \(X\) is needed for this derivation.

Definition: For \(n\) observations and \(p\) coefficients, let \(x_i=(x_{i1},\ldots,x_{ip})^T\) be the \(p\)-by-1 column of design values for observation \(i\), and let \(\beta=(\beta_1,\ldots,\beta_p)^T\) be the \(p\)-by-1 coefficient vector. The observation-level linear predictor is the scalar \(\eta_i=x_i^T\beta=\sum_{j=1}^p x_{ij}\beta_j\). Define \(\eta=(\eta_1,\ldots,\eta_n)^T\) and let \(X\) be the \(n\)-by-\(p\) matrix whose \(i\)th row is \(x_i^T\). Then the stacked predictor equation is \(\eta=X\beta\).

Dimensions and the row convention

The transpose in \(x_i^T\beta\) is essential. The covariate vector \(x_i\) is a column with \(p\) entries, so \(x_i^T\) is a row with \(p\) entries. Its product with the \(p\)-by-1 column \(\beta\) is a \(1\)-by-1 scalar, which is \(\eta_i\).

\underbrace{x_i^T}_{1\times p} \underbrace{\beta}_{p\times 1} = \underbrace{\eta_i}_{1\times 1}

Stack the \(n\) row vectors \(x_i^T\), in the same order as the observations, to form \(X\). Stack the corresponding scalar predictors in that order to form \(\eta\). The dimensions then give

\underbrace{X}_{n\times p} \underbrace{\beta}_{p\times 1} = \underbrace{\eta}_{n\times 1}.

The observation index labels rows; the coefficient or predictor index labels columns. In particular, \(X_{ij}=x_{ij}\), so the entry in row \(i\), column \(j\) is the value of the \(j\)th design variable for observation \(i\). The order of rows in \(X\) must match the order of entries in \(\eta\).

Worked Example: Three Poisson Predictors with One Covariate

Suppose \(Y_i\) are independent Poisson responses with means \(\mu_i\), and use the log-link model \(\log(\mu_i)=\eta_i\). For three observations, suppose the model has an intercept and one numerical covariate \(z\):

\eta_i=\beta_0+\beta_1z_i,\qquad z_1=0,\quad z_2=2,\quad z_3=-1.

There are \(n=3\) rows and \(p=2\) coefficients. Include the intercept by giving its design column the value 1 for each observation. Thus

x_1=\begin{pmatrix}1\\0\end{pmatrix},\quad x_2=\begin{pmatrix}1\\2\end{pmatrix},\quad x_3=\begin{pmatrix}1\\-1\end{pmatrix},\qquad \beta=\begin{pmatrix}\beta_0\\\beta_1\end{pmatrix}.

Transposing each \(x_i\) makes it a row, and stacking those rows gives a \(3\)-by-\(2\) matrix. Stacking the three predictors gives a \(3\)-by-\(1\) vector:

X= \begin{pmatrix} 1&0\\ 1&2\\ 1&-1 \end{pmatrix}, \qquad \eta= \begin{pmatrix}\eta_1\\\eta_2\\\eta_3\end{pmatrix}, \qquad \eta=X\beta.

The first row of the product represents \(1\beta_0+0\beta_1=\eta_1\); the second represents \(1\beta_0+2\beta_1=\eta_2\); and the third represents \(1\beta_0-1\beta_1=\eta_3\). These are exactly the three observation-level equations. The dimensions check as \((3\times2)(2\times1)=3\times1\). The response distribution and link specify the GLM context; the matrix equation stacks its linear predictors.

The stacking result

Theorem: Suppose that for every \(i=1,\ldots,n\), the predictor is defined by \(\eta_i=\sum_{j=1}^p x_{ij}\beta_j\), using the same ordered coefficient vector \(\beta\). If \(X\) is the \(n\)-by-\(p\) matrix with entries \(X_{ij}=x_{ij}\), and \(\eta\) is the \(n\)-by-1 vector with entries \(\eta_i\), then \(\eta=X\beta\).

Proof. By the definition of matrix multiplication, the \(i\)th entry of \(X\beta\) is the dot product of row \(i\) of \(X\) and \(\beta\):

(X\beta)_i=\sum_{j=1}^p X_{ij}\beta_j =\sum_{j=1}^p x_{ij}\beta_j =\eta_i.

This equality holds for every \(i=1,\ldots,n\). Therefore \(X\beta\) and \(\eta\) have the same entries in the same order, so they are equal as vectors. The dimensions are valid because \(X\) is \(n\)-by-\(p\) and \(\beta\) is \(p\)-by-1. \(\square\)

The theorem shows why the matrix equation is equivalent to the observation-level equations: its \(i\)th component is precisely the equation for observation \(i\). It does not say that the coefficients can necessarily be recovered uniquely from \(X\); that is a separate question. Here, \(X\beta\) is well-defined for every \(n\)-by-\(p\) matrix \(X\) and \(p\)-by-1 vector \(\beta\).

Worked Example: Bernoulli Model with Two Numerical Predictors

Suppose \(Y_i\) are independent Bernoulli responses with success probabilities \(\pi_i\), and the logit model has an intercept and two numerical predictors:

\log\!\left(\frac{\pi_i}{1-\pi_i}\right) =\eta_i=\beta_0+\beta_1a_i+\beta_2b_i.

For two observations, let \((a_1,b_1)=(4,1)\) and \((a_2,b_2)=(3,5)\). The coefficient vector has three entries, so \(p=3\). The design vectors, written as columns, are

x_1=\begin{pmatrix}1\\4\\1\end{pmatrix}, \qquad x_2=\begin{pmatrix}1\\3\\5\end{pmatrix}, \qquad \beta=\begin{pmatrix}\beta_0\\\beta_1\\\beta_2\end{pmatrix}.

The intercept is the first component of both design vectors. Transposing the vectors and stacking them in observation order yields

X= \begin{pmatrix} 1&4&1\\ 1&3&5 \end{pmatrix}, \qquad \eta= \begin{pmatrix}\eta_1\\\eta_2\end{pmatrix}, \qquad \eta=X\beta.

The row-by-row expansion confirms the correspondence:

(X\beta)_1=1\beta_0+4\beta_1+1\beta_2=\eta_1, \qquad (X\beta)_2=1\beta_0+3\beta_1+5\beta_2=\eta_2.

The matrix is \(2\)-by-\(3\), the coefficient vector is \(3\)-by-\(1\), and the result is \(2\)-by-\(1\), as required. This construction organizes the linear predictors; it does not require evaluating the predictors at particular coefficient values.

How additive terms appear in matrix form

An observation-level equation may be written as a sum of contributions, with each contribution involving one or more coefficients. In matrix form, those contributions occupy corresponding blocks of columns in \(X\), while the matching coefficients occupy blocks of \(\beta\). The full product adds the blockwise contributions for each observation.

Result: Suppose \(X\) is partitioned into column blocks \(X_1,\ldots,X_K\), and \(\beta\) is partitioned into matching coefficient blocks \(\beta_1,\ldots,\beta_K\). If each \(X_k\) has \(n\) rows and \(\beta_k\) has as many entries as \(X_k\) has columns, then \(\eta=X\beta=\sum_{k=1}^K X_k\beta_k\).

Proof. Write the horizontal block partition as \(X=[X_1\ \cdots\ X_K]\) and the corresponding vertical partition as \(\beta=(\beta_1^T,\ldots,\beta_K^T)^T\). By the definition of block matrix multiplication, the product is the sum of the products of matching blocks:

X\beta=[X_1\ \cdots\ X_K] \begin{pmatrix}\beta_1\\\vdots\\\beta_K\end{pmatrix} =\sum_{k=1}^K X_k\beta_k.

Each \(X_k\beta_k\) has \(n\) entries, so the sum is an \(n\)-by-\(1\) vector. Its \(i\)th entry is the sum of the \(K\) contributions for observation \(i\). This is the matrix version of adding the corresponding observation-level terms. \(\square\)

Worked Example: Intercept-Only Normal Model

Suppose \(Y_i\) are independent with \(Y_i\sim N(\eta_i,\sigma^2)\), where \(\sigma^2\) is a common variance, and the model has an intercept only: \(\eta_i=\beta_0\) for \(i=1,\ldots,4\). There is one coefficient, so \(p=1\), and each observation has the same one-entry design vector \(x_i=(1)\). Stacking the four rows gives

X= \begin{pmatrix} 1\\1\\1\\1 \end{pmatrix}, \qquad \beta= \begin{pmatrix}\beta_0\end{pmatrix}, \qquad \eta= \begin{pmatrix}\eta_1\\\eta_2\\\eta_3\\\eta_4\end{pmatrix} =X\beta.

The dimensions are \((4\times1)(1\times1)=4\times1\). The \(i\)th row gives \((X\beta)_i=1\beta_0=\eta_i\), for each of the four observations. Thus one coefficient contributes the same constant predictor to every row, and the matrix equation records all four equations at once.

Common mistakes and why the notation matters

Putting design vectors in columns instead of rows. With the convention here, \(x_i\) is a column vector but \(x_i^T\) is the \(i\)th row of \(X\). Stacking the untransposed \(x_i\) as rows would not follow this definition and can change the intended product.
Multiplying in the wrong order. \(X\beta\) is \(n\)-by-\(1\). The product \(\beta X\) is generally not defined, and \(\beta X^T\) is not the stated representation of the stacked predictors.
Forgetting the intercept column. If the observation-level equation includes \(\beta_0\), the matrix representation needs a corresponding coefficient and design value for each observation, conventionally a column of ones.
Mixing row order. If rows of \(X\) are rearranged, entries of \(\eta\) must be rearranged in the same way. Otherwise the \(i\)th matrix-product entry no longer corresponds to the intended observation.

The dimension check is a quick safeguard: an \(n\)-entry predictor vector requires \(n\) rows in \(X\), and the number of columns in \(X\) must equal the number of entries in \(\beta\). But dimensions alone do not catch every mistake; the \(i\)th row must also contain the design values for observation \(i\), in the same coefficient order used in \(\beta\).

Key takeaway: Write each equation as \(\eta_i=x_i^T\beta\); make row \(i\) of \(X\) equal to \(x_i^T\); and stack the \(\eta_i\)'s in the same observation order. Then the complete collection of equations is exactly \(\eta=X\beta\), with dimensions \(n\times1=(n\times p)(p\times1)\).

Check Your Understanding

Use the observation-level equations and dimensions to answer these questions.

If there are \(n=12\) observations and \(p=4\) coefficients, what are the dimensions of \(X\), \(\beta\), and \(\eta\)?
For a model \(\eta_i=\beta_0+\beta_1u_i+\beta_2v_i\), how many columns does \(X\) have, and what are the three entries in its \(i\)th row?
Explain why the \(i\)th entry of \(X\beta\) equals the observation-level predictor \(\eta_i\).
If two predictor terms are stored in separate blocks \(X_1\) and \(X_2\), how does their combined matrix product relate to \(X_1\beta_1\) and \(X_2\beta_2\)?
What must be changed in the predictor vector if the rows of \(X\) are reordered?