Predictor Scaling for Better Conditioning

Why Predictor Scales Matter

A GLM can include predictors measured on very different numerical scales: age in years, income in thousands of dollars, or a laboratory value recorded in small units. These units do not change the scientific model, but they can change the numerical behavior of the information matrix used in an iterative fit. When columns of the design matrix have very different magnitudes, solving the information system can be less well conditioned.

Centering and scaling replace continuous predictors with more convenient coordinates. The operation is a change of parameterization, not a change to the fitted model: coefficients must be converted consistently so that every linear predictor, and hence every fitted mean, remains the same. Scaling can improve numerical conditioning, but it does not guarantee improvement in every model or at every iteration.

Definition and Coefficient Conversion

For clarity, consider a model with an intercept and \(p\) continuous predictors. Write \(x_i=(x_{i1},\ldots,x_{ip})^\top\), \(z_i=(z_{i1},\ldots,z_{ip})^\top\), and \(\eta_i=\beta_0+x_i^\top\beta\). Let \(c=(c_1,\ldots,c_p)^\top\), and let \(D\) be the diagonal matrix with entries \(s_1,\ldots,s_p\). Since \(x_i=c+Dz_i\), the same linear predictor can be written \(\eta_i=\gamma_0+z_i^\top\gamma\), where \(\gamma_0=\beta_0+c^\top\beta\) and \(\gamma=D\beta\). Equivalently, \(\beta=D^{-1}\gamma\) and \(\beta_0=\gamma_0-c^\top D^{-1}\gamma\).

The intercept therefore has a different interpretation after centering: it is the linear predictor at \(x=c\), rather than at \(x=0\). A slope also changes its numerical value when its predictor's unit changes. These changes are coordinate changes; they do not imply that the underlying fitted relationship has changed.

How the Information Matrix Changes

Let \(X\) be the design matrix with first column equal to one and remaining columns the original predictors. Let \(X_*\) be the design matrix with the centered and scaled predictors. For fixed values of the GLM working weights collected in a diagonal matrix \(W\), write the information matrix in the original coordinates as \(I_\beta=X^\top W X\). This is the usual weighted information form at the current fit. The same transformation applies to observed information when the log-likelihood is expressed in the transformed coordinates.

Define the block matrix \(A=\begin{pmatrix}1&c^\top\\0&D\end{pmatrix}\). Then \(X=X_*A\), and the transformed coefficient vector is \(\gamma=A\beta\). Because every \(s_j>0\), \(A\) is invertible. The information matrix for \(\gamma\) is therefore \(I_\gamma=A^{-\top}I_\beta A^{-1}\). This relation makes the numerical effect explicit: changing predictor units changes the matrix representation of the same local likelihood geometry.

Proof. The linear predictors satisfy \(X_*\gamma=X_*A\beta=X\beta\), so they agree observation by observation. A GLM log-likelihood depends on the coefficients through the linear predictors (with the observations, offsets, and other fixed quantities unchanged). Thus the corresponding coefficient values give the same log-likelihood. More explicitly, \(\ell_\gamma(\gamma)=\ell_\beta(A^{-1}\gamma)\). Since \(A^{-1}\) is constant, the chain rule gives \(\nabla_\gamma\ell_\gamma=A^{-\top}\nabla_\beta\ell_\beta\), and differentiating again gives \(\nabla_\gamma^2\ell_\gamma=A^{-\top}(\nabla_\beta^2\ell_\beta)A^{-1}\). Taking the negative Hessian gives the stated information transformation. The same first-derivative calculation gives the weighted-information identity for \(X^\top W X\), since \(X=X_*A\). \(\square\)

Weighted Centering and the Intercept–Slope Term

Ordinary centering uses the unweighted mean of a predictor. In a GLM information matrix, however, the cross-term between the intercept and a predictor is weighted by the current working weights. Consequently, centering at the ordinary mean does not generally make that cross-term zero. Centering at the weighted mean does, provided the weights are positive and have a positive sum.

Proof. The intercept–predictor entry is \(\sum_i w_i(x_i-c_w)\). Substituting the definition of \(c_w\) gives \[ \sum_i w_ix_i-c_w\sum_i w_i =\sum_i w_ix_i- \frac{\sum_i w_ix_i}{\sum_i w_i}\sum_i w_i =0. \] The denominator is nonzero because the weights are positive. \(\square\)

In a fitted GLM, working weights can change as the fitted means change. Weighted centering at one iteration need not remove the cross-term at a later iteration. The result describes the information matrix for the weights at which the center was calculated; it is not a permanent orthogonality guarantee.

Practical Use and a Common Pitfall

A common practical choice is to center each continuous predictor at a representative value, often its sample mean, and divide by a positive scale such as its standard deviation or a scientifically meaningful unit. This can make coefficient magnitudes easier to interpret and can reduce avoidable numerical disparity among design-matrix columns. It is often useful to leave indicator variables and the intercept unchanged unless there is a specific reason to transform them.

Common mistake: “I centered and scaled a predictor, so the GLM fit must be different—or I can keep the old coefficients.” Both conclusions are wrong. If the coefficients are transformed by the stated formulas, each linear predictor is exactly preserved, and so are the fitted means and likelihood. If the predictor is transformed but the old numerical coefficients are used unchanged, the linear predictor generally changes. Another overclaim is that standardizing every predictor guarantees a well-conditioned information matrix: it does not. Correlation between predictors, the current working weights, and other features of the design still matter. Scaling changes coordinates; it does not make every numerical difficulty disappear.

For a numerical implementation, choose and record the centers and scales, fit using the transformed predictors, and convert coefficient estimates back when original-unit interpretation is needed. For a single predictor, the conversion is \(\beta_1=\gamma_1/s\) and \(\beta_0=\gamma_0-c\gamma_1/s\). For multiple predictors, use \(\beta=D^{-1}\gamma\) and \(\beta_0=\gamma_0-c^\top D^{-1}\gamma\). Verify the conversion by comparing linear predictors at selected observations; this is a direct check that the parameter coordinates were handled consistently.