Linear Predictors With Transformed Covariates

Why Transform a Covariate?

A linear predictor combines design-matrix columns with coefficients. A column need not be a raw covariate: it can be a specified function of one. For example, a model may include the square of a measurement to allow its contribution to change with the measurement’s level, or the logarithm of a positive dose. The construction is straightforward, but it is important to distinguish two claims: the predictor can be linear in its coefficients while being nonlinear in the original covariate.

Here the focus is on how to encode such a model. The transformation is part of the model specification: it determines the entries of a design-matrix column before the coefficients are applied. It does not change the meaning of the matrix product \(\eta=X\beta\).

Definition: Let \(x_i\) denote covariate values for observation \(i\), and let \(g_j\) be a specified function of those values. A transformed-covariate column has entries \(g_j(x_i)\), for \(i=1,\ldots,n\). A predictor that includes this column has the form \(\eta_i=\sum_{j=1}^{p}\beta_j g_j(x_i)\). It is linear in the coefficients \(\beta_j\), even when one or more functions \(g_j\) are nonlinear in \(x_i\).

For an intercept, one of the functions is the constant function \(g_1(x_i)=1\). Common choices for numerical covariates include \(g(x_i)=x_i^2\) and \(g(x_i)=\log(x_i)\). A logarithm requires \(x_i>0\) for every observation to which that column is applied. Every transformed column must be defined for the data used in the model.

From a Transformation to a Matrix Column

Suppose there are \(n\) observations and \(p\) specified columns. The \(i\)th row of the design matrix records all the functions evaluated at observation \(i\), in the same order used for the coefficient vector. Thus a squared-covariate term contributes \(x_i^2\), not \(x_i\), to its column. A logged term contributes \(\log(x_i)\), not the logarithm of a coefficient or of the entire predictor.

X_{ij}=g_j(x_i), \qquad \eta_i=\sum_{j=1}^{p}X_{ij}\beta_j, \qquad \eta=X\beta.

The row and coefficient order must match. If the second column is \(x_i^2\), its coefficient is the second component of \(\beta\); if the columns are reordered, their coefficients must be reordered in the same way. This is the same row-by-column construction used for raw numerical covariates, with the chosen transformation evaluated before the matrix multiplication.

Specify the predictor terms.
Write each intended contribution explicitly, including any intercept and the function applied to each covariate.

Check transformation domains.
For example, verify that every input to a logarithm is positive.

Evaluate each function observation by observation.
These values, in the stated term order, form the design-matrix rows.

Multiply by the matching coefficients.
Compute each \(\eta_i\) as the sum of the row entries times their corresponding coefficients.

Worked Example: Adding a Squared Covariate

Suppose four observations have covariate values \(x=(0,1,2,3)\), and the specified predictor is \(\eta_i=\beta_0+\beta_1x_i+\beta_2x_i^2\). Construct its design matrix and calculate the predictors for \(\beta=(0.4,-0.3,0.2)^\mathsf{T}\).

The columns, in order, are the intercept, \(x_i\), and \(x_i^2\). Squaring each observed value gives \(0,1,4,9\), so

X= \begin{pmatrix} 1&0&0\\ 1&1&1\\ 1&2&4\\ 1&3&9 \end{pmatrix}, \qquad \eta=X\beta= \begin{pmatrix} 1&0&0\\ 1&1&1\\ 1&2&4\\ 1&3&9 \end{pmatrix} \begin{pmatrix} 0.4\\-0.3\\0.2 \end{pmatrix}.

Row by row, the calculations are \(\eta_1=0.4+0(-0.3)+0(0.2)=0.4\), \(\eta_2=0.4-0.3+0.2=0.3\), \(\eta_3=0.4+2(-0.3)+4(0.2)=0.6\), and \(\eta_4=0.4+3(-0.3)+9(0.2)=1.3\). Therefore \(\eta=(0.4,0.3,0.6,1.3)^\mathsf{T}\). In particular, the last column contains \(x_i^2\); the coefficient \(\beta_2\) multiplies those squared values.

Worked Example: Combining a Logarithm and a Square

For three observations, let age be \(40,50,60\) years and dose be \(2,1,4\), respectively. Consider \(\eta_i=\beta_0+\beta_1\mathrm{age}_i+\beta_2\log(\mathrm{dose}_i)+\beta_3\mathrm{dose}_i^2\), with \(\beta=(1,0.02,-0.5,0.01)^\mathsf{T}\). Construct the matrix in that term order and calculate each predictor. All doses are positive, so the logarithm is defined.

The age column is \((40,50,60)^\mathsf{T}\), the logged-dose column is \((\log 2,\log 1,\log 4)^\mathsf{T}\), and the squared-dose column is \((4,1,16)^\mathsf{T}\). Hence

X= \begin{pmatrix} 1&40&\log 2&4\\ 1&50&\log 1&1\\ 1&60&\log 4&16 \end{pmatrix}.

Using \(\log 1=0\), \(\log 2\approx0.6931471806\), and \(\log 4\approx1.3862943611\), the first predictor is \(\eta_1=1+0.02(40)-0.5\log 2+0.01(4)=1.84-0.5\log 2\approx1.4934\). The second is \(\eta_2=1+0.02(50)-0.5\log 1+0.01(1)=2.01\). The third is \(\eta_3=1+0.02(60)-0.5\log 4+0.01(16)=2.36-0.5\log 4\approx1.6669\). The displayed decimals are rounded from the exact logarithmic expressions, not from prematurely rounded intermediate results.

Linearity in Coefficients Is Not Linearity in Covariates

In this setting, “linear predictor” refers to the way coefficients enter: each coefficient multiplies a specified column, and the contributions are added. It does not require every column to be a raw covariate. A square or a logarithm changes the shape of the predictor as a function of the original covariate, but it does not make the predictor nonlinear in its coefficients.

Theorem: For fixed covariate values and fixed specified functions \(g_1,\ldots,g_p\), the predictor vector \(\eta\) is a linear function of the coefficient vector \(\beta\). In particular, for any coefficient vectors \(\beta,\gamma\) and scalars \(a,b\), \(\eta(a\beta+b\gamma)=a\eta(\beta)+b\eta(\gamma)\).

Proof. The design matrix has entries \(X_{ij}=g_j(x_i)\), so it is fixed when the covariate values and functions are fixed. By the definition of the predictor, \(\eta(\beta)=X\beta\). For any \(a,b,\beta,\gamma\), distributivity of matrix multiplication gives

\eta(a\beta+b\gamma) =X(a\beta+b\gamma) =aX\beta+bX\gamma =a\eta(\beta)+b\eta(\gamma).

Thus the predictor is linear in \(\beta\), regardless of whether a column is \(x_i\), \(x_i^2\), or \(\log(x_i)\), provided the specified column is defined. \(\square\)

The same expression can be nonlinear in the raw covariate. For a single covariate, consider \(\eta(x)=\beta_0+\beta_1x+\beta_2x^2\). Holding the coefficients fixed and differentiating with respect to \(x\) gives

\frac{d\eta}{dx}=\beta_1+2\beta_2x, \qquad \frac{d^2\eta}{dx^2}=2\beta_2.

When \(\beta_2\ne0\), the second derivative is nonzero, so this predictor is not a straight-line function of \(x\), even though it remains linear in \((\beta_0,\beta_1,\beta_2)\). A log term also generally creates a nonlinear relationship in the raw covariate: for \(x>0\), the contribution \(\beta\log x\) has derivative \(\beta/x\), which depends on \(x\) when \(\beta\ne0\). These calculations concern the predictor as a function of the covariate; they do not change the coefficient-linearity result.

Worked Example: A Logged Covariate Is Linear in Its Own Column

Suppose \(x=(1,e,e^2)\) and the specified predictor is \(\eta_i=0.7-0.4\log(x_i)\). Construct the design matrix and calculate the predictor vector. Then compare the values with a straight-line relationship in raw \(x\).

The columns are the intercept and \(\log(x_i)\). Since \(\log 1=0\), \(\log e=1\), and \(\log(e^2)=2\),

X= \begin{pmatrix} 1&0\\ 1&1\\ 1&2 \end{pmatrix}, \qquad \beta= \begin{pmatrix} 0.7\\-0.4 \end{pmatrix}, \qquad \eta=X\beta= \begin{pmatrix} 0.7\\0.3\\-0.1 \end{pmatrix}.

The predictor decreases by \(0.4\) each time \(\log(x)\) increases by one. But the corresponding raw-\(x\) gaps are \(e-1\) and \(e^2-e\), which are unequal. The equal predictor changes therefore occur over unequal changes in \(x\): this is linear in the logged covariate column, not a straight line in raw \(x\).

What the Transformation Does—and Does Not—Specify

A transformed column is a modeling choice, not a consequence of observing a numerical covariate. Including \(x_i^2\) specifies a squared contribution; including \(\log(x_i)\) specifies a logged contribution. The choice of columns says which functions are included in the predictor. It does not, by itself, assert that the transformation is useful or that the fitted relationship has a particular shape in every data set.

It also matters whether a model includes both a raw and transformed covariate or only the transformed one. For example, \(\beta_0+\beta_1x+\beta_2x^2\) includes both columns, whereas \(\beta_0+\beta_2x^2\) does not include a separate raw-\(x\) contribution. These are different specifications and produce different design matrices. The order of columns is arbitrary only if each column remains paired with its corresponding coefficient.

Common Mistakes

Calling a model nonlinear in its coefficients because it contains \(x^2\). The raw covariate enters through a nonlinear function, but the term is still \(\beta_2x^2\), linear in the coefficient \(\beta_2\).
Putting the raw value in a transformed column. If the term is \(x_i^2\), the column entry is \(x_i^2\), not \(x_i\). Evaluate the stated function separately for each observation.
Transforming a coefficient instead of the covariate. A term \(\beta\log(x_i)\) means the logarithm of the covariate value is multiplied by \(\beta\); it does not mean \(\log(\beta)x_i\).
Using a logarithm outside its domain. The real-valued \(\log(x_i)\) requires \(x_i>0\). Check this condition before constructing the column.
Treating “linear” as a claim about raw \(x\). State which sense is intended. A predictor can be linear in coefficients and nonlinear in the original covariate.

Key takeaway: Write each model term as a coefficient times a specified function of the covariates. Evaluate those functions to form the design-matrix columns, check their domains, and remember that linearity in coefficients does not imply linearity in raw covariates.

Check Your Understanding

Use the specified terms and their order when answering.

For the predictor \(\eta_i=\beta_0+\beta_1x_i+\beta_2x_i^2\), what entries appear in its third design-matrix column?
A model includes \(\log(z_i)\). What condition must the observed \(z_i\) values satisfy for this real-valued column to be defined?
Explain how a predictor containing \(x_i^2\) can be linear in its coefficients but nonlinear in \(x_i\).
For \(x=(1,e,e^2)\) and \(\eta_i=2+3\log(x_i)\), construct the design matrix and calculate \(\eta\).
How does the design matrix differ between \(\beta_0+\beta_1x_i+\beta_2x_i^2\) and \(\beta_0+\beta_2x_i^2\)?