PCA With More Variables Than Observations

Why PCA Changes When There Are More Variables Than Observations

In some studies, each observation has many recorded measurements, but only a small number of observations are available. A hypothetical insurer, for example, might have a short pilot sample with many measured claim characteristics. In this setting, the sample covariance matrix is large, and its eigenvalues include zeros whether or not any individual variable has zero variance. The reason is a limit on how many independent directions the observations can support.

The key computational idea is to work with an observation-by-observation matrix instead of the variable-by-variable covariance matrix. This smaller matrix has the same positive eigenvalues after the appropriate scaling. Its eigenvectors can then be mapped back to loadings for the original variables.

Definition: Let \(X\) be the \(n\)-by-\(p\) data matrix whose rows are observations and whose columns are variables. Assume \(n\ge 2\), and that each column has been centered by subtracting its sample mean. The sample covariance matrix is \(S=X^TX/(n-1)\), a \(p\)-by-\(p\) matrix. The observation-by-observation matrix is \(G=XX^T\), an \(n\)-by-\(n\) matrix.

Rank and the Number of Zero Eigenvalues

Centering imposes a linear constraint on the rows of \(X\): their sum is the zero vector. Equivalently, the \(n\)-vector of ones is in the null space of \(X^T\), because \(X^T\mathbf{1}_n=0\). Therefore, the rank of \(X\) is at most \(n-1\). It is also at most \(p\), so \(\operatorname{rank}(X)\le \min(n-1,p)\).

The covariance matrix has the same rank as \(X\). To see why, for any vector \(z\) of length \(p\), \(z^TSz=\|Xz\|^2/(n-1)\). This quantity is zero exactly when \(Xz=0\), so \(S\) and \(X\) have the same null space and rank. Thus, if \(p>n\), at most \(n-1\) covariance eigenvalues can be positive, and at least \(p-n+1\) must be zero. Some additional eigenvalues may be zero if the centered data have further linear dependencies.

Key result: For centered data with \(p>n\), the \(p\)-by-\(p\) sample covariance matrix has at least \(p-n+1\) zero eigenvalues. If \(X\) has its maximum possible rank, exactly \(n-1\) eigenvalues are positive and \(p-n+1\) are zero.

Computing Nonzero Components From the Smaller Matrix

The matrices \(X^TX\) and \(XX^T\) are not the same size, but they share their positive eigenvalues. Here is the basic argument. Suppose \(u\) is a unit eigenvector of \(G=XX^T\) with positive eigenvalue \(\mu\). Define \(v=X^Tu/\sqrt{\mu}\). Then \(v\) has length one, and \(v\) is an eigenvector of \(X^TX\) with eigenvalue \(\mu\). Since the sample covariance is \(S=X^TX/(n-1)\), its corresponding eigenvalue is \(\lambda=\mu/(n-1)\).

X^TXv =\frac{X^TXX^Tu}{\sqrt{\mu}} =\frac{X^TGu}{\sqrt{\mu}} =\frac{\mu X^Tu}{\sqrt{\mu}} =\mu v.

The normalization also follows from the eigenvector equation:

\|X^Tu\|^2 =u^TXX^Tu =u^TGu =\mu u^Tu =\mu.

So dividing \(X^Tu\) by \(\sqrt{\mu}\) produces a unit loading. This construction works only for \(\mu>0\); a zero eigenvalue of \(G\) cannot be used as a denominator. The method finds all positive covariance eigenvalues and their loadings, while the remaining covariance eigenvalues are zero.

Center the data.
Check that the columns of \(X\) sum to zero. If using standardized variables, standardize first and confirm the resulting columns are centered.

Form the smaller matrix.
Calculate \(G=XX^T\), which is \(n\)-by-\(n\), and find its positive eigenvalues \(\mu_j\) and unit eigenvectors \(u_j\).

Convert to covariance eigenpairs.
For each positive \(\mu_j\), use \(\lambda_j=\mu_j/(n-1)\) and \(v_j=X^Tu_j/\sqrt{\mu_j}\).

Verify.
Check that \(v_j^Tv_j=1\) and \(Sv_j=\lambda_jv_j\). Count the remaining zero eigenvalues using the rank bound.

Worked Calculations

Worked Example: Counting Zero Eigenvalues From a Two-Observation Sample

A hypothetical pilot study records four centered variables on two observations. Its data matrix is

X= \begin{pmatrix} 2&-1&0&3\\ -2&1&0&-3 \end{pmatrix}.

Find the rank and eigenvalues of the sample covariance matrix, and identify the number of zero eigenvalues.

Step 1: Check centering. The second row is the negative of the first, so every column sums to zero. Here \(n=2\), \(p=4\), and \(n-1=1\). The centered-rank bound is \(\operatorname{rank}(X)\le 1\).

Step 2: Calculate the covariance matrix. Since \(n-1=1\), \(S=X^TX\). Writing the first row as \(w=(2,-1,0,3)^T\), the two rows are \(w^T\) and \(-w^T\), so \(S=2ww^T\). Also, \(w^Tw=2^2+(-1)^2+0^2+3^2=14\).

Step 3: Find its eigenvalues. The vector \(w\) is an eigenvector of \(2ww^T\) with eigenvalue \(2w^Tw=28\). Every vector orthogonal to \(w\) is mapped to zero, because \(2ww^Tz=2w(w^Tz)=0\) whenever \(w^Tz=0\). Thus the eigenvalues are \(28,0,0,0\).

Check. The rank is one, consistent with the centered bound \(n-1=1\), and there are \(p-1=3\) zero eigenvalues. The trace is \(28\), equal to the sum of the eigenvalues. A common mistake is to expect \(n=2\) positive components; centering reduces the maximum to \(n-1=1\).

Worked Example: Recovering Loadings From the Observation Matrix

For a hypothetical study with three observations and four centered variables, let

X= \begin{pmatrix} 1&2&1&0\\ -1&-2&1&0\\ 0&0&-2&0 \end{pmatrix}.

Find the positive covariance eigenvalues and their unit loadings using \(G=XX^T\).

Step 1: Form \(G\). The three rows of \(X\) are \(r_1=(1,2,1,0)\), \(r_2=(-1,-2,1,0)\), and \(r_3=(0,0,-2,0)\). Their squared lengths are \(6,6,4\). Their pairwise inner products are \(r_1\cdot r_2=-1-4+1=-4\), \(r_1\cdot r_3=-2\), and \(r_2\cdot r_3=-2\). Therefore,

G= \begin{pmatrix} 6&-4&-2\\ -4&6&-2\\ -2&-2&4 \end{pmatrix}.

Step 2: Find eigenpairs. Let \(a=(1,-1,0)^T\) and \(b=(1,1,-2)^T\). Direct multiplication gives \(Ga=(10,-10,0)^T=10a\) and \(Gb=(6,6,-12)^T=6b\). The vector \(\mathbf{1}_3=(1,1,1)^T\) satisfies \(G\mathbf{1}_3=0\). The vectors \(a\) and \(b\) are orthogonal, with squared lengths \(2\) and \(6\), respectively. Thus the unit eigenvectors are \(u_1=a/\sqrt{2}\) and \(u_2=b/\sqrt{6}\), with positive eigenvalues \(\mu_1=10\) and \(\mu_2=6\).

Step 3: Convert eigenvalues. Here \(n-1=2\), so the positive sample covariance eigenvalues are \(\lambda_1=10/2=5\) and \(\lambda_2=6/2=3\).

Step 4: Convert the first eigenvector. Multiplication gives \(X^Ta=(2,4,0,0)^T\), so \(X^Tu_1=(2,4,0,0)^T/\sqrt{2}\). Therefore,

v_1=\frac{X^Tu_1}{\sqrt{10}} =\frac{(2,4,0,0)^T}{\sqrt{20}} =\left(\frac1{\sqrt5},\frac2{\sqrt5},0,0\right)^T.

Its squared length is \(1/5+4/5=1\). As a direct check, \(Xv_1=(\sqrt5,-\sqrt5,0)^T\), and \(Sv_1=5v_1\), as required.

Step 5: Convert the second eigenvector. Here \(X^Tb=(0,0,6,0)^T\), so \(X^Tu_2=(0,0,6,0)^T/\sqrt6\). Dividing by \(\sqrt{\mu_2}=\sqrt6\) gives \(v_2=(0,0,1,0)^T\). It has unit length; \(Xv_2=(1,1,-2)^T\), and \(Sv_2=3v_2\).

Check. The centered rank is two, so the four-by-four covariance matrix has two positive eigenvalues and two zeros. The eigenvalues \(5\) and \(3\) sum to \(8\); this equals \(\operatorname{tr}(S)\), since the diagonal entries of \(G\) sum to \(16\) and \(S= X^TX/2\). The frequent scaling error is to report \(10\) and \(6\), the eigenvalues of \(G\), as covariance eigenvalues without dividing by \(n-1=2\).

Worked Example: Detecting an Additional Zero Direction

A hypothetical three-observation data set has five centered variables whose columns are \(c_1=(1,-1,0)^T\), \(c_2=(1,1,-2)^T\), \(c_3=3c_1\), and \(c_4=c_5=0\). Find its positive covariance eigenvalues and corresponding loadings.

Step 1: Use the column inner products. The vectors \(c_1\) and \(c_2\) are orthogonal, since \(1-1+0=0\). Their squared lengths are \(c_1^Tc_1=2\) and \(c_2^Tc_2=6\). The first and third columns contribute the block

\begin{pmatrix} c_1^Tc_1&c_1^Tc_3\\ c_3^Tc_1&c_3^Tc_3 \end{pmatrix} = \begin{pmatrix}2&6\\6&18\end{pmatrix}.

This block has eigenvalues \(20\) and \(0\): its trace is \(20\), and its determinant is \(36-36=0\). The \(c_2\) column contributes the separate eigenvalue \(6\); the two zero columns contribute zero eigenvalues.

Step 2: Scale for covariance. Since \(n-1=2\), the positive covariance eigenvalues are \(20/2=10\) and \(6/2=3\).

Step 3: Give unit loadings and verify. The loading for eigenvalue \(10\) is \(v_1=(1,0,3,0,0)^T/\sqrt{10}\); its squared length is \((1+9)/10=1\). Multiplying by \(X\) gives \(Xv_1=(\sqrt{10},-\sqrt{10},0)^T\), so \(Sv_1=10v_1\). For eigenvalue \(3\), the loading is \(v_2=(0,1,0,0,0)^T\), since its variable column is \(c_2\); \(Sv_2=3v_2\).

Check. The rank is two, so there are three zero covariance eigenvalues. One zero direction comes from the exact redundancy \(c_3=3c_1\); the other two come from the zero columns. This distinction matters: a zero eigenvalue can arise from having too few centered observations, from redundant variables, or from both.

Practical Checks and Common Traps

The smaller-matrix method is useful when \(p\) is much larger than \(n\): it replaces a \(p\)-by-\(p\) eigenvalue problem with an \(n\)-by-\(n\) one. It does not remove the need to center the data, use the correct covariance scaling, or verify the resulting loading vectors.

Forgetting the centering constraint. Centered data have rank at most \(n-1\), not merely at most \(n\). Thus the guaranteed zero-eigenvalue count for \(p>n\) is at least \(p-n+1\).
Confusing \(G\) and \(S\). The positive eigenvalues of \(G=XX^T\) are \((n-1)\) times the sample covariance eigenvalues. Divide by \(n-1\) when reporting covariance eigenvalues.
Using the wrong conversion factor for loadings. For a unit eigenvector \(u\) of \(G\) with eigenvalue \(\mu>0\), the unit loading is \(X^Tu/\sqrt{\mu}\), not \(X^Tu/\mu\).
Treating tiny numerical values as genuine components. Floating-point calculations may return values close to zero rather than exactly zero. Compare their size with the scale of the positive eigenvalues and with the rank bound before interpreting them.

If the variables are standardized before PCA, the same rank argument applies as long as each standard deviation used is nonzero: multiplying centered columns by nonzero scale factors does not increase the rank. If a variable has zero variance, it cannot be standardized by dividing by its standard deviation and must be handled separately.

Key takeaway: With more variables than centered observations, zero covariance eigenvalues are unavoidable. Find the positive eigenpairs from \(XX^T\), divide its eigenvalues by \(n-1\), and map each unit observation-space eigenvector \(u\) to the loading \(X^Tu/\sqrt{\mu}\).

Check Your Understanding

For each problem, use sample covariance scaling unless stated otherwise.

A centered data matrix has \(n=4\) observations and \(p=7\) variables, and its rank is three. How many zero covariance eigenvalues does it have? (Answer: 4 zero eigenvalues.)
A centered data matrix has \(n=3\), \(p=5\), and \(XX^T\) has eigenvalues \(12,4,0\). Find the covariance eigenvalues, including zeros. (Answer: \(6,2,0,0,0\).)
An eigenvalue of \(XX^T\) is \(\mu=18\) for a centered data set with \(n=7\). What is the corresponding sample covariance eigenvalue? (Answer: \(3\).)
A centered matrix has \(n=4\), \(p=6\), and rank two. How many covariance eigenvalues are positive and how many are zero? (Answer: 2 positive and 4 zero eigenvalues.)
For a centered data matrix with \(n=4\), \(XX^T u=9u\), and \(u^Tu=1\), state the covariance eigenvalue and the formula for its unit loading. (Answer: Eigenvalue \(3\); loading \(v=X^Tu/3\).)

What You'll Learn