Why Covariance Divisors Do Not Change Sample Directions

Why the Divisor Changes Scale, Not Direction

Sample principal component analysis uses eigenvectors of a covariance matrix to describe directions of variation. Two common sample covariance conventions divide the same centered cross-product matrix by \(n\) or by \(n-1\). The divisor changes the numerical scale of the covariance matrix, so its eigenvalues change too. But, except for the zero-variation boundary case, multiplying a matrix by a positive scalar does not change its eigenspaces. That is why the two conventions give the same sample principal directions and the same explained-variance proportions.

The claim is algebraic: it does not require the observations to follow a particular probability distribution. The important qualifications are that there must be at least two observations to use the \(n-1\) divisor, and that explained-variance proportions require positive total variation.

Two Covariance Conventions

Let \(x_1,\ldots,x_n\) be observed vectors in \(\mathbb{R}^p\), with \(n\geq 2\). Write their sample mean as \(\bar{x}=n^{-1}\sum_{i=1}^n x_i\), and let \(X_c\) be the \(n\) by \(p\) matrix whose \(i\)th row is \((x_i-\bar{x})^\mathsf{T}\). The centered cross-product matrix is \(X_c^\mathsf{T}X_c\). Its entries are sums of products of centered coordinates.

Definition: The two sample covariance matrices considered here are

S_n=\frac{1}{n}X_c^\mathsf{T}X_c, \qquad S_{n-1}=\frac{1}{n-1}X_c^\mathsf{T}X_c.

The first uses divisor \(n\); the second uses divisor \(n-1\). Both are symmetric positive semidefinite matrices.

The matrices contain exactly the same centered cross-products. Their relation follows by comparing the two scalar divisors:

S_n=\frac{n-1}{n}S_{n-1}.

Since \(n\geq2\), the factor \((n-1)/n\) is strictly positive. If \(X_c\neq 0\), the matrices are nonzero and numerically different; their nonzero eigenvalues have the same positive rescaling factor. If \(X_c=0\), both matrices are zero, so in that boundary case they are numerically equal.

Eigenvalues Rescale; Eigenspaces Do Not

Theorem: Let \(A\) be a real symmetric matrix and \(c>0\). Then \(A\) and \(cA\) have the same eigenspaces. If \(\lambda\) is an eigenvalue of \(A\), the corresponding eigenvalue of \(cA\) is \(c\lambda\), with the same multiplicity.

Proof. If \(v\neq 0\) and \(Av=\lambda v\), then

(cA)v=c(Av)=c\lambda v.

Thus every eigenvector of \(A\) is an eigenvector of \(cA\), with its eigenvalue multiplied by \(c\). Conversely, if \((cA)v=\mu v\), then \(Av=(\mu/c)v\), because \(c>0\); so \(v\) lies in an eigenspace of \(A\) as well. The spectral theorem gives an orthonormal eigenbasis for \(A\), and the same basis diagonalizes \(cA\), with each diagonal eigenvalue multiplied by \(c\). This also preserves the dimensions of the eigenspaces and hence eigenvalue multiplicities. \(\square\)

Applying the theorem with \(A=S_{n-1}\) and \(c=(n-1)/n\) proves the sample PCA direction claim. An eigenvector basis for either covariance matrix is an eigenvector basis for the other, and corresponding eigenvalues satisfy

\lambda_j(S_n)=\frac{n-1}{n}\lambda_j(S_{n-1}).

This statement concerns eigenspaces, not a uniquely determined list of signed vectors. An eigenvector and its negative describe the same axis; if an eigenvalue is repeated, its eigenspace can have multiple orthonormal bases. Positive rescaling preserves the eigenspace itself, including in those cases.

Explained-Variance Proportions

For a covariance matrix \(S\) with ordered eigenvalues \(\lambda_1,\ldots,\lambda_p\), the explained-variance proportion of component \(j\) is \(\lambda_j/\sum_{k=1}^p\lambda_k\), provided the denominator is positive. For a covariance matrix, the eigenvalues are nonnegative and their sum equals its trace, the total sample variance across the variables.

Result: If \(\operatorname{tr}(S_{n-1})>0\), then \(S_n\) and \(S_{n-1}\) have identical explained-variance proportions. If the total variance is zero, those proportions are undefined.

Derivation. Put \(c=(n-1)/n\), so \(S_n=cS_{n-1}\). The eigenvalue result gives \(\lambda_j(S_n)=c\lambda_j(S_{n-1})\). The trace also scales by \(c\), since the trace is the sum of the eigenvalues. Therefore, when the total is positive,

\frac{\lambda_j(S_n)}{\sum_{k=1}^p\lambda_k(S_n)} = \frac{c\lambda_j(S_{n-1})}{\sum_{k=1}^p c\lambda_k(S_{n-1})} = \frac{\lambda_j(S_{n-1})}{\sum_{k=1}^p\lambda_k(S_{n-1})}.

The cancellation is valid because \(c>0\) and the denominator is positive. If the total is zero, the ratio has a zero denominator and is not defined. \(\square\)

Worked Calculations

Worked Example: A Two-Variable Sample

Take three centered observations \(x_1=(1,0)^\mathsf{T}\), \(x_2=(0,1)^\mathsf{T}\), and \(x_3=(-1,-1)^\mathsf{T}\). They are centered because their coordinate-wise sums are both zero. The centered cross-product matrix is

X_c^\mathsf{T}X_c = \begin{pmatrix} 1&0&-1\\ 0&1&-1 \end{pmatrix} \begin{pmatrix} 1&0\\ 0&1\\ -1&-1 \end{pmatrix} = \begin{pmatrix} 2&1\\ 1&2 \end{pmatrix}.

The vectors \((1,1)^\mathsf{T}\) and \((1,-1)^\mathsf{T}\) are eigenvectors of this cross-product matrix: multiplication gives \(3(1,1)^\mathsf{T}\) and \(1(1,-1)^\mathsf{T}\), respectively. Thus the covariance eigenvalues are \(3\) and \(1\), scaled by the chosen divisor.

With divisor \(n=3\), the eigenvalues are \(1\) and \(1/3\). With divisor \(n-1=2\), they are \(3/2\) and \(1/2\). In both cases the first direction is proportional to \((1,1)^\mathsf{T}\), the second to \((1,-1)^\mathsf{T}\), and the explained-variance proportions are \(3/4\) and \(1/4\). For example, under divisor \(n\), the total is \(1+1/3=4/3\), giving \(1/(4/3)=3/4\) and \((1/3)/(4/3)=1/4\).

Worked Example: Three Variables and a Correct Total

Consider four centered observations in \(\mathbb{R}^3\):

(2,\sqrt{2},1),\quad (-2,\sqrt{2},-1),\quad (2,-\sqrt{2},-1),\quad (-2,-\sqrt{2},1).

Each column sums to zero, so these observations are centered. The column cross-products are \(16\), \(8\), and \(4\): for example, the second column has four squared entries equal to \(2\), so its squared length is \(2+2+2+2=8\). The cross-products between distinct columns are zero: the first-second products sum to \(2\sqrt{2}-2\sqrt{2}-2\sqrt{2}+2\sqrt{2}=0\); the first-third products sum to \(2+2-2-2=0\); and the second-third products sum to \(\sqrt{2}-\sqrt{2}+\sqrt{2}-\sqrt{2}=0\). Hence

X_c^\mathsf{T}X_c= \begin{pmatrix} 16&0&0\\ 0&8&0\\ 0&0&4 \end{pmatrix}.

With divisor \(n=4\), the covariance eigenvalues are \(4,2,1\), and their total is \(4+2+1=7\). With divisor \(n-1=3\), they are \(16/3,8/3,4/3\), and their total is \(16/3+8/3+4/3=28/3\). The respective proportions agree: \(4/7=(16/3)/(28/3)\), \(2/7=(8/3)/(28/3)\), and \(1/7=(4/3)/(28/3)\). The directions are the three coordinate axes in either convention.

Worked Example: No Variation at All

Suppose \(n=3\) observations in \(\mathbb{R}^2\) are all \((5,-2)^\mathsf{T}\). Their sample mean is \((5,-2)^\mathsf{T}\), so every centered observation is \((0,0)^\mathsf{T}\). Consequently \(X_c\) is the zero matrix and

S_3=\frac{1}{3}X_c^\mathsf{T}X_c = S_2=\frac{1}{2}X_c^\mathsf{T}X_c = \begin{pmatrix} 0&0\\ 0&0 \end{pmatrix}.

Both eigenvalues are zero under either convention. Every vector is an eigenvector of the zero matrix, but the data contain no variation from which to select a meaningful principal direction. The explained-variance proportions would require dividing each zero eigenvalue by the zero total, so they are undefined—not zero and not one.

Common Pitfall and Practical Meaning

A common mistake is to say that the \(n\) and \(n-1\) covariance matrices always differ, then conclude that their eigenvalues always differ. That reasoning overlooks the possibility that the centered data matrix is zero. The precise statement is: when \(X_c\neq0\), the two matrices differ by a positive scalar factor, and their nonzero eigenvalues are rescaled by that factor; when \(X_c=0\), both matrices are zero. In either case, the same eigenspaces are available, although with no variation the data do not identify a useful direction.

A second mistake is to compare raw eigenvalues and infer that the component directions or relative importance changed. Raw eigenvalues measure variance in the units induced by the chosen divisor, so they do change when the data have nonzero variation. Relative importance is measured by the proportion of total variance: that ratio is unchanged because the same factor multiplies both its numerator and denominator.

This distinction is useful when reading software output or comparing calculations made under different conventions. A disagreement in eigenvalue magnitudes alone may reflect only the divisor. A disagreement in principal eigenspaces or explained-variance proportions cannot be explained by switching between \(n\) and \(n-1\) for the same centered data; it indicates some other difference, such as different data or a different preprocessing choice.

Key takeaway: For \(n\geq2\), \(S_n=((n-1)/n)S_{n-1}\). The positive scalar changes eigenvalue magnitudes but preserves eigenspaces, and it cancels from explained-variance proportions whenever total variance is positive. If all centered observations are zero, both covariance matrices are zero and the proportions are undefined.

Check Your Understanding

Use the definitions and calculations above to check each conclusion.

For \(n=8\), what is the scalar relating \(S_n\) to \(S_{n-1}\)?
If a nonzero covariance eigenvalue under the \(n-1\) convention is \(6\), what is its value under the \(n\) convention when \(n=5\)?
Why does multiplying every eigenvalue by the same positive constant leave the explained-variance proportions unchanged?
Can a covariance matrix have zero total variance but a nonzero eigenvalue? Explain.
When every observation equals the sample mean, what are the two covariance matrices, and are explained-variance proportions defined?