Correlation Matrices and Standardization

Why Standardize a Random Vector?

The components of a random vector can be measured in very different units. A laboratory value might be measured in milligrams per liter, while a second component is recorded in years. Their covariance depends on both units, so its numerical size can be difficult to interpret directly. Standardization puts each coordinate on a common scale: it subtracts that coordinate's mean and divides by its standard deviation. The resulting pairwise covariances are correlations, which are dimensionless measures of linear association.

Standardization is a change of location and scale, not a way to make variables independent or to capture every feature of their joint distribution. The correlation matrix summarizes pairwise, second-order information; its interpretation depends on that limitation.

Standardized Coordinates and the Correlation Matrix

Let \(X=(X_1,\ldots,X_p)^\mathsf{T}\) be a random vector, where every coordinate has a finite second moment and strictly positive variance. Write \(\mu_i=E[X_i]\), \(\sigma_i^2=\operatorname{Var}(X_i)>0\), and \(\Sigma\) for the covariance matrix, whose \((i,j)\) entry is \(\operatorname{Cov}(X_i,X_j)\).

Definition: The standardized coordinate corresponding to \(X_i\) is \(Z_i=(X_i-\mu_i)/\sigma_i\), where \(\sigma_i=\sqrt{\operatorname{Var}(X_i)}\) is the positive standard deviation. The correlation of \(X_i\) and \(X_j\) is \(\operatorname{Corr}(X_i,X_j)=\operatorname{Cov}(X_i,X_j)/(\sigma_i\sigma_j)\). The correlation matrix \(R\) has entries \(R_{ij}=\operatorname{Corr}(X_i,X_j)\).

Let \(D\) be the diagonal matrix with diagonal entries \(\sigma_1,\ldots,\sigma_p\). The vector \(Z=(Z_1,\ldots,Z_p)^\mathsf{T}\) is \(D^{-1}(X-\mu)\), where \(\mu=E[X]\). Its covariance matrix is the correlation matrix:

R=\operatorname{Cov}(Z)=D^{-1}\Sigma D^{-1}, \qquad R_{ij}=\frac{\Sigma_{ij}}{\sqrt{\Sigma_{ii}\Sigma_{jj}}}.

The condition \(\sigma_i>0\) is essential: if a coordinate has zero variance, it is constant with probability one and cannot be standardized by dividing by its standard deviation. The correlation matrix is symmetric and has ones on its diagonal, since each standardized coordinate has variance one.

Worked Example: Standardizing a Discrete Pair

Let \(U,V\) take values in \(\{-1,1\}\), with probabilities \(P(U=1,V=1)=0.4\), \(P(U=-1,V=-1)=0.4\), and \(P(U=1,V=-1)=P(U=-1,V=1)=0.1\). Define \(X=10+2U\) and \(Y=-4+5V\). Find their means, standard deviations, covariance, and correlation.

For each variable, the probabilities of \(1\) and \(-1\) are both \(0.5\). Thus \(E[U]=E[V]=0\) and \(E[U^2]=E[V^2]=1\). Also, \(E[UV]=0.4+0.4-0.1-0.1=0.6\).

It follows that \(E[X]=10+2E[U]=10\), \(E[Y]=-4+5E[V]=-4\), \(\operatorname{Var}(X)=4\operatorname{Var}(U)=4\), and \(\operatorname{Var}(Y)=25\operatorname{Var}(V)=25\). Their standard deviations are \(2\) and \(5\).

Using the constant-shift and scaling rules for covariance, \(\operatorname{Cov}(X,Y)=\operatorname{Cov}(10+2U,-4+5V)=10\operatorname{Cov}(U,V)=10(0.6)=6\). Therefore \(\operatorname{Corr}(X,Y)=6/(2\cdot5)=0.6\). The standardized coordinates are \(Z_X=(X-10)/2=U\) and \(Z_Y=(Y+4)/5=V\), so their covariance matrix is

R= \begin{pmatrix} 1&0.6\\ 0.6&1 \end{pmatrix}.

The original units and offsets disappear from the standardized coordinates, but the pairwise correlation remains.

Properties of Every Correlation Matrix

The first important structural property is positive semidefiniteness. A real symmetric matrix \(A\) is positive semidefinite if \(a^\mathsf{T}Aa\geq0\) for every real vector \(a\). This condition is necessary because the matrix represents variances of linear combinations.

Theorem: If the coordinates of \(X\) have finite second moments and strictly positive variances, its correlation matrix \(R\) is symmetric, has \(R_{ii}=1\) for every \(i\), and is positive semidefinite.

Proof. Covariance is symmetric, so \(R_{ij}=R_{ji}\). Also, \(R_{ii}=\operatorname{Var}(Z_i)=1\). For any fixed \(a\in\mathbb{R}^p\), the linear combination \(a^\mathsf{T}Z\) has a finite second moment, and

a^\mathsf{T}Ra =a^\mathsf{T}\operatorname{Cov}(Z)a =\operatorname{Var}(a^\mathsf{T}Z) \geq 0.

Thus \(R\) is positive semidefinite. \(\square\)

A consequence is that every pairwise correlation lies between \(-1\) and \(1\). For a pair \(Z_i,Z_j\), nonnegativity of the variance of \(Z_i-tZ_j\) holds for every real \(t\). Since both variances equal one, this gives

0\leq \operatorname{Var}(Z_i-tZ_j) =1-2tR_{ij}+t^2.

The quadratic on the right cannot have two distinct real roots: otherwise it would be negative between them. Its discriminant must therefore be nonpositive, so \(4R_{ij}^2-4\leq0\), or \(|R_{ij}|\leq1\).

Positive semidefiniteness also matters when specifying a matrix of proposed correlations. Unit diagonal and entries between \(-1\) and \(1\) are not, by themselves, enough to guarantee that a proposed matrix is a valid correlation matrix. The whole matrix must satisfy the quadratic-form condition.

Worked Example: Forming a Correlation Matrix From Covariances

Suppose \(U,V,W\) are independent random variables, each taking the values \(-1\) and \(1\) with equal probability. Define \(X_1=2U\), \(X_2=U+2V\), and \(X_3=-U+V+2W\). Find the covariance and correlation matrices.

Each of \(U,V,W\) has mean zero and variance one. Independence gives zero covariance between distinct variables. Thus all three \(X_i\) have mean zero. Their variances are \(\operatorname{Var}(X_1)=4\), \(\operatorname{Var}(X_2)=1+4=5\), and \(\operatorname{Var}(X_3)=1+1+4=6\).

For the off-diagonal entries, expanding covariance and dropping the zero cross-covariances gives \(\operatorname{Cov}(X_1,X_2)=2\), \(\operatorname{Cov}(X_1,X_3)=-2\), and \(\operatorname{Cov}(X_2,X_3)=-1+2=1\). Therefore

\Sigma= \begin{pmatrix} 4&2&-2\\ 2&5&1\\ -2&1&6 \end{pmatrix}, \qquad D= \begin{pmatrix} 2&0&0\\ 0&\sqrt{5}&0\\ 0&0&\sqrt{6} \end{pmatrix}.

Divide each covariance by the product of the corresponding standard deviations. The three distinct off-diagonal correlations are \(2/(2\sqrt{5})=1/\sqrt{5}\), \(-2/(2\sqrt{6})=-1/\sqrt{6}\), and \(1/(\sqrt{5}\sqrt{6})=1/\sqrt{30}\). Hence

R= \begin{pmatrix} 1&1/\sqrt{5}&-1/\sqrt{6}\\ 1/\sqrt{5}&1&1/\sqrt{30}\\ -1/\sqrt{6}&1/\sqrt{30}&1 \end{pmatrix}.

This matrix is positive semidefinite because it is the covariance matrix of the standardized vector. Its validity follows from the construction, not merely from checking each entry separately.

What Standardization Changes—and What It Leaves

Standardization subtracts a constant and divides by a positive constant in each coordinate. It gives the coordinate mean zero and variance one, but it does not remove general dependence. In fact, each original coordinate can be recovered from its standardized version by \(X_i=\mu_i+\sigma_iZ_i\). Each transformation is one-to-one, so the standardized vector retains the same information about the coordinates' joint behavior.

Result: Coordinatewise standardization preserves whether the coordinates are independent. More precisely, the random variables \(X_1,\ldots,X_p\) are independent if and only if \(Z_1,\ldots,Z_p\) are independent.

Proof. Each \(Z_i\) is a measurable function of \(X_i\), and each \(X_i\) is a measurable function of \(Z_i\). If the \(X_i\) are independent, measurable coordinatewise functions of them are independent, so the \(Z_i\) are independent. Conversely, if the \(Z_i\) are independent, applying the measurable coordinatewise maps \(z\mapsto\mu_i+\sigma_i z\) shows that the \(X_i\) are independent. \(\square\)

Worked Example: A Nonlinear Relationship After Standardization

Let \(X\) be uniform on \((-1,1)\), and set \(Y=X+X^2\). Calculate their correlation, then express the standardized \(Y\) as a function of the standardized \(X\).

For this uniform distribution, \(E[X]=E[X^3]=0\), \(E[X^2]=1/3\), and \(E[X^4]=1/5\). Hence \(E[Y]=E[X+X^2]=1/3\), and \(\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]=E[X^2+X^3]=1/3\).

The variance of \(Y\) is \(\operatorname{Var}(Y)=E[(X+X^2)^2]-(E[Y])^2 =E[X^2+2X^3+X^4]-1/9 =1/3+0+1/5-1/9=19/45\). Since \(\operatorname{Var}(X)=1/3\), the correlation is \(\operatorname{Corr}(X,Y)=(1/3)/\sqrt{(1/3)(19/45)}=\sqrt{15/19}\).

Here \(Z_X=X/\sqrt{1/3}=\sqrt{3}X\), so \(X=Z_X/\sqrt{3}\), and \(Z_Y=(Y-1/3)/\sqrt{19/45}\). Substitution gives

Z_Y= \frac{Z_X/\sqrt{3}+Z_X^2/3-1/3}{\sqrt{19/45}}.

Thus the standardized coordinates still have an exact nonlinear relationship. Standardizing puts them on unit scales; it does not erase their joint structure or turn a nonlinear relationship into a linear one.

Common Pitfall and Practical Use

Wrong reasoning: “After standardization, every variable has variance one, so the coordinates are on the same scale and their dependence has been removed.” Equal marginal variances do not imply independence, and the standardization formula changes each coordinate separately without disrupting its relationship to the others. The correction is to read \(R\) as a summary of pairwise linear association on standardized scales, not as a complete description of the joint law.

Another frequent error is to compare covariances without considering units. For example, changing a measurement from meters to centimeters multiplies its covariance with another variable by 100, while its correlation is unchanged. Correlation offers a scale-free comparison, but it does not tell the whole story: distinct joint distributions can have the same means and correlation matrix.

In applications, the matrix is useful for summarizing pairwise associations and preparing variables measured in different units for procedures that use standardized coordinates. Before interpreting it, check that each coordinate has positive finite variance and remember that the entries describe second-order relationships, not every feature of dependence.

Check Your Understanding

Use the definitions and properties in this tutorial to answer the following.

What condition on \(\operatorname{Var}(X_i)\) is required to standardize coordinate \(X_i\), and why?
If \(\operatorname{Cov}(X_1,X_2)=-6\), \(\operatorname{Var}(X_1)=9\), and \(\operatorname{Var}(X_2)=16\), what is their correlation?
Why must the correlation matrix be positive semidefinite?
Does standardizing each coordinate change whether the coordinates are independent? Explain.
Why do unit diagonal and pairwise entries between \(-1\) and \(1\) not, by themselves, establish that a proposed matrix is a valid correlation matrix?