Reparameterization Under Predictor Coding Changes

Why Predictor Coding Changes the Coefficients

A regression coefficient is defined relative to the predictor columns used to represent a model. Changing the origin or units of a predictor, or choosing a different reference level for a factor, can therefore change the numerical coefficients. That change does not necessarily alter the model’s fitted responses: when the new columns span the same predictor space, the same fitted values can be expressed using new coefficients. This distinction is important in multivariate regression, where one predictor design is paired with several response variables.

Write the multivariate linear model as \(Y=XB+E\), where \(Y\) is an \(n\times q\) response matrix, \(X\) is an \(n\times p\) design matrix, \(B\) is a \(p\times q\) coefficient matrix, and \(E\) is the error matrix. The fitted response matrix is \(XB\). A row of \(B\) gives the coefficients for one design column across all \(q\) responses. The discussion below assumes \(X\) has full column rank when referring to unique coefficient estimates; the fitted-value identity itself does not require that assumption.

The General Invariance Result

The key algebra is short, but its consequence is broad: a design recoding and the inverse coefficient transformation cancel one another.

Proof. By associativity of matrix multiplication,

For the least-squares statement, let \(\hat B\) minimize \(\lVert Y-XB\rVert_F^2\), where \(\lVert\cdot\rVert_F\) is the Frobenius norm. Each candidate coefficient matrix \(C\) for the recoded design corresponds to \(AC\) for the original design, because \(X^*C=XAC\). Since \(A\) is invertible, this correspondence is one-to-one and covers all original coefficient matrices. Thus both minimization problems have the same possible fitted matrices and the same minimum residual sum of squares. Under full column rank, each has a unique minimizer, so \(\hat B^*=A^{-1}\hat B\), and the fitted matrices agree. \(\square\)

This result concerns the representation of the regression, not a change in the response variables. Every column of the coefficient matrix is transformed by the same \(A^{-1}\), so the argument applies to all responses at once. If \(X\) is rank deficient, the fitted-value identity still holds for corresponding coefficient matrices, but coefficients need not be unique; the full-rank uniqueness claim then does not apply.

Centering and Rescaling Quantitative Predictors

Suppose a model includes an intercept and one quantitative predictor \(x\). For each response, write the intercept and slope as \(\beta_0\) and \(\beta_1\). Centering at a constant \(c\) replaces \(x\) by \(x_c=x-c\). To keep the same fitted mean, the intercept must absorb the contribution from the shift:

For \(q\) responses, let \(B\) have rows \(b_0\) and \(b_1\), each a \(1\times q\) row vector. The centered coefficient rows are \(b_0^*=b_0+cb_1\) and \(b_1^*=b_1\). Thus the slope is unchanged by centering, while the intercept now gives the fitted response at \(x=c\), not at \(x=0\).

If the recoded predictor is \(z=(x-c)/s\), with \(s\ne 0\), then \(x=c+sz\), so

The intercept remains the fitted response at the original predictor value \(c\), while the new slope is \(s\) times the old slope. In particular, if \(s\) is the original measurement units per new unit, the coefficient is interpreted as change per one new unit. When \(s>0\), the direction of the slope is unchanged; a negative \(s\) also reverses the predictor’s direction.

Changing a Categorical Reference Level

Treatment coding represents a categorical predictor by indicator columns for all but one reference category. The intercept is the fitted response for the reference category, and each indicator coefficient is the difference between its category’s fitted response and the reference category’s fitted response, holding other predictors fixed. If the reference category changes, these interpretations and coefficient values change, although the fitted category means need not change.

For three categories A, B, and C, use A as the original reference and let the design columns be \(1,I_B,I_C\), where \(I_B\) and \(I_C\) indicate membership in B and C. With B as the new reference, the columns are \(1,I_A,I_C\). Since \(I_A=1-I_B-I_C\), the new design is an invertible recoding of the old one. For each response, if the old coefficients are \(\beta_0,\beta_B,\beta_C\), the new coefficients are

The new intercept is the fitted response for category B. The new A and C coefficients are contrasts against B. These identities apply response by response, or simultaneously to the rows of a multivariate coefficient matrix.

What Depends on Coding, and What Does Not

A fitted response at a specified predictor configuration does not depend on which invertible coding is used to represent the same model. Coefficients, however, describe changes along particular design columns and therefore depend on the chosen origins, units, and contrasts. This distinction also applies to statements about coefficients: a claim about one coded slope is not automatically the same claim after recoding. The corresponding linear combination of coefficients must be transformed as well.

In particular, centering changes the reference point for the intercept, scaling changes the unit associated with a slope, and changing a categorical reference changes the comparator for its contrasts. These are interpretive changes, not evidence that the fitted relationship changed. A practical analysis should report the coding and units so that the intercepts and coefficients have a precise meaning.

Common Pitfall

A frequent mistake is to center a predictor but leave the intercept unchanged, then conclude that centering changes the fitted model. For example, replacing \(x\) by \(x-c\) while keeping \(\beta_0\) and \(\beta_1\) gives \(\beta_0+\beta_1(x-c)\), which differs from \(\beta_0+\beta_1x\) by \(-c\beta_1\). The correction is to transform the intercept to \(\beta_0+c\beta_1\) at the same time. Another mistake is to read a treatment-coded coefficient as an absolute category mean: it is a difference from the chosen reference, and changing the reference changes that difference.

A recoding must also preserve the design space for the invariance argument to apply. If columns are dropped, constrained, or otherwise changed in a way that is not an invertible transformation, the same coefficient transformation cannot be assumed. In such cases, first establish whether the models represent the same set of fitted values rather than relying on the formulas for centering or releveling.