Centering and Scaling Variables Before Fitting

Why Center and Scale Data Columns?

A data matrix records measurements for several variables across a collection of observations. Its columns can use very different units: one might record distance in metres, another income in thousands of dollars, and another temperature in degrees. These differences matter when comparing numerical coefficient sizes or when a calculation treats columns according to their lengths. Centering removes each column’s average level; scaling adjusts its size. These are separate operations, and keeping track of each one makes the result interpretable.

Suppose there are \(m\) observations and \(n\) measured variables, arranged as rows and columns. Write the data matrix as \(\mathbf{X}\in\mathbb{R}^{m\times n}\), and denote its \(j\)th column by \(\mathbf{x}_j\in\mathbb{R}^m\). Thus each entry in \(\mathbf{x}_j\) is one observation of variable \(j\). We will use the Euclidean norm \(\|\mathbf{v}\|=\sqrt{v_1^2+\cdots+v_m^2}\) for a real column vector \(\mathbf{v}\).

Subtracting the mean changes the column’s level but not the differences between its observations. Scaling then changes the column’s length. If the goal is unit Euclidean norm, the scaling factor must be calculated from the centered column. A constant column, whose observations are all equal, becomes the zero vector when centered; it cannot be scaled to unit norm by division.

Centering and Unit-Norm Scaling

Proof. By the definition of \(\mu_j\), the entries of \(\widetilde{\mathbf{x}}_j\) sum to \(\sum_{i=1}^m(x_{ij}-\mu_j)=\sum_{i=1}^m x_{ij}-m\mu_j=0\). So the centered column has mean zero. If \(\widetilde{\mathbf{x}}_j\ne\mathbf{0}\), its norm is positive, and dividing each entry by that same number preserves a zero sum. By the definition of Euclidean norm and because the divisor is positive, \(\|\mathbf{z}_j\|=\|\widetilde{\mathbf{x}}_j\|/\|\widetilde{\mathbf{x}}_j\|=1\). If the centered column is zero, the division is undefined, which is why the nonzero condition is required. \(\square\)

The calculation also illustrates the effect of units. If the second variable is the first measured in units one hundred times smaller, its raw deviations and its centered norm are one hundred times larger. Dividing by its own centered norm removes that numerical size difference. It does not, however, make two unrelated variables equivalent: scaling changes length, not the pattern of variation across observations.

Unit Norm and Standard Deviation Are Different Choices

“Standardize” is sometimes used for a different scaling convention: divide a centered column by its sample standard deviation. For \(m\ge 2\), define the sample standard deviation of a column by \(s_j=\sqrt{\frac{1}{m-1}\sum_{i=1}^m(x_{ij}-\mu_j)^2}\). When the centered column is nonzero, this is positive. Since the sum of squares inside the square root is \(\|\widetilde{\mathbf{x}}_j\|^2\), we have \(s_j=\|\widetilde{\mathbf{x}}_j\|/\sqrt{m-1}\). Consequently, dividing by \(s_j\) gives a centered column of norm \(\sqrt{m-1}\), not \(1\). Both conventions are useful, but they produce different lengths.

Why Coefficient Sizes Depend on Units

A coefficient multiplying a variable must compensate for that variable’s units. This can be seen without assuming a particular fitting method. Suppose a linear expression contains the term \(\beta x\), and a new coordinate is defined by \(z=(x-\mu)/s\), where \(s\ne0\). Solving for the original variable gives \(x=\mu+sz\). Therefore

The coefficient of \(z\) is \(\beta s\), while the constant contribution changes by \(\beta\mu\). Thus centering shifts the constant term, and scaling multiplies the coefficient by the scale factor. The numerical coefficient is not an intrinsic measure of importance unless the variable scales and units are taken into account.

Proof. Substitute \(x_j=\mu_j+s_jz_j\) into each term: \(\beta_jx_j=\beta_j(\mu_j+s_jz_j)=\beta_j\mu_j+\beta_js_jz_j\). Adding these equalities over \(j=1,\dots,n\) gives the stated expression. \(\square\)

When centered, scaled variables are used in a later calculation, the coefficient in the transformed coordinates can be useful for comparing variables put on a common scale. But the comparison has a precise scope: it refers to the chosen centering and scaling convention. It does not by itself account for relationships between variables, measurement uncertainty, or the way a model is fitted. To interpret a transformed coefficient in the original units, undo the scaling using the substitution above.

Common Pitfall: Scale the Centered Column, Not the Raw One

A tempting shortcut is to divide a raw column by its norm and assume that the result is both centered and unit norm. Those properties do not generally hold together. For \(\mathbf{x}=(1,2,3)^\top\), the raw norm is \(\sqrt{1+4+9}=\sqrt{14}\). Dividing the raw column by \(\sqrt{14}\) gives a vector of norm \(1\), but its mean is \(6/(3\sqrt{14})=2/\sqrt{14}\), which is not zero. It has not been centered.

Reversing the order without recalculating the scaling factor also fails. The mean is \(2\), so the centered column is \((-1,0,1)^\top\). If we divide this centered column by the raw norm \(\sqrt{14}\), its norm becomes \(\sqrt{2/14}=1/\sqrt{7}\), not \(1\). The correction is to calculate the norm after centering: \(\sqrt{2}\). Dividing \((-1,0,1)^\top\) by \(\sqrt{2}\) gives mean zero and norm one.

A second failure occurs with a constant column. If \(\mathbf{x}=(4,4,4)^\top\), then \(\mu=4\) and \(\widetilde{\mathbf{x}}=(0,0,0)^\top\). Its norm is zero, so the proposed unit-norm division has no meaning. Such a column contains no variation across the observations to rescale; it should be identified as constant rather than divided by zero.

A Reliable Column-by-Column Procedure

The checks are part of the calculation, not a cosmetic final step. A zero mean confirms centering; a norm of \(1\) confirms unit-norm scaling; and a norm of \(\sqrt{m-1}\) confirms sample-standard-deviation scaling. Recording which convention was used is essential when interpreting coefficients or comparing results across data sets.