Standardizing Variables Correctly

Why Standardization Must Be Done by Variable

In correlation-based principal components analysis (PCA), the variables are put on a common scale before their relationships are summarized. This is important when the variables use different units or have very different spreads: without standardization, a variable with numerically large values or a large variance can dominate the analysis. The key operation is performed down each column, across the observations of one variable. Standardizing across the variables within each observation is a different operation and does not produce correlation-based PCA.

The earlier material on centering the data matrix and building the sample covariance matrix supplies two useful ideas: subtracting a variable’s mean centers its column, and sample covariance uses the divisor $n-1$. Here those ideas are combined with scaling by each column’s own sample standard deviation.

Definition: Let $X$ be a data matrix with $n$ observations in rows and $p$ variables in columns. For variable $j$, its sample mean and sample standard deviation are $$ \bar{x}_j=\frac{1}{n}\sum_{i=1}^{n}x_{ij}, \qquad s_j=\sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_{ij}-\bar{x}_j)^2}. $$ When $s_j>0$, the standardized value for observation $i$ and variable $j$ is $$ z_{ij}=\frac{x_{ij}-\bar{x}_j}{s_j}. $$ Thus, each column is centered and scaled using summaries computed from that column across the $n$ observations.

The standardization convention above uses the sample standard deviation, with denominator $n-1$. For each standardized column, the sample mean is zero and the sample variance is one. Small differences from zero or one in decimal calculations are rounding effects.

From Standardized Columns to the Correlation Matrix

Let $Z$ be the matrix of standardized values. Its sample covariance matrix is $Z^\mathsf{T}Z/(n-1)$. The diagonal entries are one, and each off-diagonal entry is the sample correlation between the corresponding original variables.

Result: If every variable has positive sample standard deviation and the same $n$ observations are used for all columns, then standardizing each column by its sample mean and sample standard deviation gives $$ \frac{Z^\mathsf{T}Z}{n-1}=R, $$ where $R$ is the sample correlation matrix of the original variables. Correlation-based PCA can therefore be carried out by forming the sample covariance matrix of the standardized data, or equivalently by using the sample correlation matrix directly.

The identity follows directly from the definitions. For standardized columns $j$ and $k$, their sample covariance is

\frac{1}{n-1}\sum_{i=1}^{n}z_{ij}z_{ik} = \frac{\sum_{i=1}^{n}(x_{ij}-\bar{x}_j)(x_{ik}-\bar{x}_k)} {(n-1)s_js_k}.

The expression on the right is the sample covariance of variables $j$ and $k$, divided by their sample standard deviations. That is the definition of their sample correlation. When $j=k$, the numerator is $(n-1)s_j^2$, so the entry is exactly one.

Worked Example: Standardizing One Variable

A hypothetical portfolio review records a risk score for four exposures: $2,4,6,8$. Standardize the scores using the sample mean and sample standard deviation, and verify the result. The method is column-wise sample standardization; the trap to avoid is using $n$ instead of $n-1$ in the sample variance.

Step 1: Calculate the sample mean.

\bar{x}=\frac{2+4+6+8}{4}=5.

Step 2: Calculate the sample standard deviation. The deviations from 5 are $-3,-1,1,3$, and their squared sum is $9+1+1+9=20$. Therefore,

s^2=\frac{20}{4-1}=\frac{20}{3}, \qquad s=\sqrt{\frac{20}{3}}\approx 2.5820.

Step 3: Standardize each value. Divide each deviation by $s$:

z_1=\frac{-3}{\sqrt{20/3}}\approx-1.1619,\quad z_2=\frac{-1}{\sqrt{20/3}}\approx-0.3873,\quad z_3\approx0.3873,\quad z_4\approx1.1619.

Check. The deviations sum to zero, so the standardized values sum to zero. Their squared sum is $20/(20/3)=3$, giving sample variance $3/(4-1)=1$. The displayed values are rounded to four decimal places; the checks use the unrounded values.

Worked Example: Building a Correlation Matrix from Standardized Data

Worked Example: Three Variables on Different Scales

A hypothetical analyst records three measurements on four assets. Standardize each column and obtain the pairwise sample correlations. Use sample standard deviations throughout; the trap to avoid is treating the observations as rows to standardize individually.

Observation	Variable X	Variable Y	Variable W
1	1	10	100
2	2	10	80
3	3	20	100
4	4	20	120

Step 1: Find the column means and sample standard deviations. For $X$, the mean is $2.5$; its deviations are $-1.5,-0.5,0.5,1.5$, whose squared sum is $5$. Thus $s_X=\sqrt{5/3}$. For $Y$, the mean is $15$; its deviations are $-5,-5,5,5$, whose squared sum is $100$. Thus $s_Y=\sqrt{100/3}$. For $W$, the mean is $100$; its deviations are $0,-20,0,20$, whose squared sum is $800$. Thus $s_W=\sqrt{800/3}$.

Step 2: Standardize by column. The centered values divided by the relevant sample standard deviation give:

Z= \begin{pmatrix} -1.1619 & -0.8660 & 0\\ -0.3873 & -0.8660 & -1.2247\\ 0.3873 & 0.8660 & 0\\ 1.1619 & 0.8660 & 1.2247 \end{pmatrix}.

For example, the first standardized $X$ value is $(1-2.5)/\sqrt{5/3}\approx-1.1619$, while the second standardized $W$ value is $(80-100)/\sqrt{800/3}\approx-1.2247$.

Step 3: Calculate the correlations. It is convenient to use the raw centered cross-product sums and divide by the square root of the two centered sums of squares. For $X$ and $Y$, the cross-product sum is $7.5+2.5+2.5+7.5=20$, giving

r_{XY}=\frac{20}{\sqrt{5(100)}}=\frac{2}{\sqrt{5}}\approx0.8944.

For $X$ and $W$, the cross-product sum is $0+10+0+30=40$, so $r_{XW}=40/\sqrt{5(800)}\approx0.6325$. For $Y$ and $W$, it is $0+100+0+100=200$, so $r_{YW}=200/\sqrt{100(800)}\approx0.7071$. Therefore the sample correlation matrix is

R= \begin{pmatrix} 1 & 0.8944 & 0.6325\\ 0.8944 & 1 & 0.7071\\ 0.6325 & 0.7071 & 1 \end{pmatrix}.

Check. Each diagonal entry is one, as required for a correlation matrix. The entries are consistent with the cross-product calculations; for instance, the sample covariance of the standardized $X$ and $Y$ columns is $20/3$ divided by $s_Xs_Y$, which equals $2/\sqrt{5}$. Values in the displayed matrix are rounded to four decimal places.

Standardizing Rows Is Not a Substitute

A row represents one observation, and its entries are measurements of different variables. Subtracting a row mean and dividing by a row standard deviation compares the variables within that observation. It does not center and scale each variable across the population of observations. Consequently, row standardization generally changes the data structure needed for correlation-based PCA and can even remove all variation from the columns.

Worked Example: Detecting the Row-Standardization Error

Three hypothetical locations have measurements $(1,2)$, $(10,11)$, and $(100,101)$ on variables $A$ and $B$. Compare the correct column standardization with standardizing each row. The trap is to assume that any transformation yielding values near zero and one must be appropriate for PCA.

Correct method: standardize columns. The means are $\bar{x}_A=37$ and $\bar{x}_B=38$. Both columns have deviations $-36,-27,63$, with squared sum $5994$, so both sample standard deviations equal $\sqrt{5994/2}=\sqrt{2997}$. Thus the standardized columns are identical, and their sample correlation is $1$. There is genuine variation across locations, and the two measurements move together.

Incorrect method: standardize rows. For the first row, the row mean is $1.5$, and its sample standard deviation is $\sqrt{0.5}=1/\sqrt{2}$. Its two row-standardized values are $-1/\sqrt{2}$ and $1/\sqrt{2}$. The other rows have the same pattern: their entries differ by 1, so each row-standardized pair is also $(-1/\sqrt{2},1/\sqrt{2})$.

After this row-wise operation, the column for $A$ is constant at $-1/\sqrt{2}$, and the column for $B$ is constant at $1/\sqrt{2}$. Both have zero sample variance, so their correlation cannot be calculated. The row operation has erased the between-location variation rather than producing the intended correlation matrix.

Check. Column standardization preserves the perfect positive association in this example. Row standardization leaves no variation across observations. The error is not a small numerical discrepancy; it changes what is being compared.

Sample and Population Scaling Conventions

A common source of inconsistent calculations is mixing sample and population standard deviations. For the convention used here, both the standard deviation and the covariance matrix use the sample divisor $n-1$. If instead each column is divided by its population standard deviation, which uses divisor $n$, the standardized values are larger by $\sqrt{n/(n-1)}$. That alternative can still produce the correlation matrix, but only when paired consistently with a covariance calculation using divisor $n$.

Worked Example: Keeping the Divisors Consistent

For three observations on variable $Q$, the values are $1,2,3$. Compare the sample- and population-standard-deviation conventions, and identify what happens if they are mixed. The trap is to use population scaling and then calculate a sample covariance as though the column had sample variance one.

The mean is $2$, and the deviations are $-1,0,1$, with squared sum $2$. The sample standard deviation is $s=\sqrt{2/(3-1)}=1$, giving sample-standardized values $(-1,0,1)$. Their sample variance is $2/2=1$.

The population standard deviation is $\sqrt{2/3}$, giving population-standardized values $(-\sqrt{3/2},0,\sqrt{3/2})$. Their squared sum is $3$. Using divisor $n=3$ gives variance $3/3=1$, as intended. But using the sample divisor $n-1=2$ gives $3/2=1.5$, not one.

Check. The two conventions are related by the factor $\sqrt{3/2}$. A covariance matrix based on population-standardized columns and divisor $n$ has the same correlation entries as the sample-based construction. Mixing population scaling with divisor $n-1$ instead multiplies the covariance matrix by $n/(n-1)=3/2$. That common factor changes eigenvalues and component variances, even though it does not change the eigenvectors.

Practical Checks Before Correlation-Based PCA

Before forming the correlation matrix, check that every column has been treated consistently and that the transformation is defined. If $s_j=0$, every observed value in variable $j$ is identical. Division by $s_j$ is impossible, and the variable has no variation to contribute to a correlation-based analysis. It must be addressed before standardization; do not replace the zero standard deviation with an arbitrary small number.

The same observations should also be used to compute each column’s mean, standard deviation, and correlations. If missing values cause different pairs of variables to use different subsets of observations, the resulting correlations may not correspond to one common standardized data matrix. In that case, the calculation method and its implications need to be made explicit.

Use columns: compute one mean and one standard deviation per variable across observations.
Use a stated divisor: sample standardization uses $n-1$ for the variance, consistent with sample covariance.
Check the result: each standardized column should have sample mean zero and sample variance one.
Check the matrix: the resulting sample covariance matrix should have ones on the diagonal and correlations off the diagonal.
Check for zero variance: a constant column cannot be standardized.

These checks help distinguish an intentional choice between covariance-based and correlation-based PCA from an accidental scaling error. In an application aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus, a clear solution states which variables are being standardized, which convention is used, and why that convention matches the matrix being analyzed.

Check Your Understanding

For each problem, state the standardization convention where relevant and verify the role of the column summaries.

A variable has observations $3,5,7,9$. Find its sample mean, sample standard deviation, and standardized values. (Answer: mean $6$, sample standard deviation $\sqrt{20/3}$, standardized values approximately $-1.1619,-0.3873,0.3873,1.1619$.)
Two columns have centered cross-product sum $12$, and their centered sums of squares are $9$ and $16$. What is their sample correlation, and what is the sample covariance of their standardized columns? (Answer: correlation $12/\sqrt{9\cdot16}=1$; standardized-column sample covariance $1$.)
Each of three observations has two measurements, with the second exactly 5 greater than the first. What happens to the columns after standardizing each row across its two measurements? (Answer: every row becomes $(-1/\sqrt{2},1/\sqrt{2})$; each resulting column is constant, so its sample variance is zero.)
There are $n=5$ observations. If columns are divided by their population standard deviations, by what factor are their standardized values multiplied relative to sample-standardized values? If their covariance is then computed with divisor $n-1$, what is the diagonal entry? (Answer: factor $\sqrt{5/4}$; diagonal entry $5/4$.)
A variable has values $8,8,8,8$. Can it be standardized for correlation-based PCA, and why? (Answer: no; its sample standard deviation is zero, so the standardization denominator is zero and the variable has no observed variation.)

What You'll Learn