Why Rank Determines the Number of Nonzero Components
Principal component analysis (PCA) replaces a set of measured variables with orthogonal component scores. The number of variables is not always the number of components that carry variation: some columns may be constant, or one column may be an exact linear combination of others. In those cases, the data occupy fewer independent directions than the number of recorded variables.
This connection is useful when interpreting a PCA output or checking a calculation. The rank of the centered data matrix gives the number of components with positive variance. It does not tell us which directions are substantively useful, and it does not make a merely strong relationship exact.
For covariance PCA, assume \(n\ge 2\), the observations are complete numerical vectors, and each column is centered before analysis. The sample covariance matrix is \(S=X_c^{\mathsf T}X_c/(n-1)\). Its eigenvalues are nonnegative; each eigenvalue is the variance of the scores in its associated unit eigenvector direction. A component is called nonzero here when its eigenvalue is positive.
Why the Rank and Eigenvalue Counts Agree
The key fact is that a linear combination of centered columns has zero variance exactly when that combination is zero for every observation. For any coefficient vector \(z\), the vector \(X_cz\) contains the scores obtained by combining the centered variables with coefficients \(z\). Its squared length is the sum of squared scores.
For any \(z\), \(z^{\mathsf T}Sz=z^{\mathsf T}X_c^{\mathsf T}X_cz/(n-1)\).
Since \(z^{\mathsf T}X_c^{\mathsf T}X_cz=(X_cz)^{\mathsf T}(X_cz)\), we have \(z^{\mathsf T}Sz=\lVert X_cz\rVert^2/(n-1)\).
Because \(n-1>0\), this quantity is zero exactly when \(X_cz=0\). Therefore \(S\) and \(X_c\) have the same coefficient vectors in their null spaces.
The covariance matrix is symmetric and nonnegative definite, so its rank is the number of its positive eigenvalues. The equality of the null spaces gives \(\operatorname{rank}(S)=\operatorname{rank}(X_c)\).
The divisor \(n-1\) sets the sample-variance scale but does not change the rank. Using \(n\) instead would multiply the covariance matrix by a positive constant and leave the number of zero and positive eigenvalues unchanged. The same rank logic applies to correlation PCA when every variable has positive sample variance: standardizing columns only rescales them by nonzero constants, which does not change the number of independent columns. A constant column is an exception for correlation PCA because its standard deviation is zero, so its correlation is undefined.
Constant Variables and Exact Redundancy
A constant variable has no variation to contribute after centering: its entire column becomes zero. It contributes no rank and no positive-variance component. In covariance PCA, it appears as a zero row and column in the covariance matrix. In correlation PCA, it cannot be standardized, so it must be identified and handled before constructing a correlation matrix.
Exact redundancy occurs when a centered column is a linear combination of other centered columns. For example, if \(C=A+B\) in every observation, then after centering \(C_c=A_c+B_c\). The three columns cannot span three independent directions, so their rank is at most two. An affine relationship in the original variables also becomes a linear relationship after centering: if \(C=a+bA+cB\) for constants \(a,b,c\), then \(C_c=bA_c+cB_c\).
By contrast, a high but imperfect correlation does not make columns exactly redundant. It may produce a small positive eigenvalue, but not a zero eigenvalue. Rank is an exact algebraic property of the observed values; it is not a threshold based on whether a relationship looks close.
Worked Example: A Constant Feature in a Covariance PCA
A hypothetical analyst records two centered measurement patterns and one fixed account-category code for four observations. The raw columns are \(A=(9,9,11,11)\), \(B=(19,21,19,21)\), and \(C=(5,5,5,5)\). Find the centered matrix, its rank, the covariance eigenvalues, and the number of nonzero components.
Method. Center each column, then form the sample covariance matrix. The trap to avoid is treating the constant column as an additional direction of variation merely because it is a recorded variable.
The column means are \(\bar A=10\), \(\bar B=20\), and \(\bar C=5\). Therefore \(A_c=(-1,-1,1,1)\), \(B_c=(-1,1,-1,1)\), and \(C_c=(0,0,0,0)\), giving
The first two columns are nonzero and are not multiples of each other; their dot product is \(1-1-1+1=0\). The third column is zero. Thus \(\operatorname{rank}(X_c)=2\). The first column's sum of squares is \(4\), the second column's sum of squares is \(4\), and their cross-product sum is \(0\). With \(n-1=3\),
Its eigenvalues are \(4/3,4/3,0\). There are two positive eigenvalues, matching the rank. The constant code contributes a zero eigenvalue. For correlation PCA, that code cannot be standardized because its sample standard deviation is zero; it must not be assigned an ordinary correlation with the other variables.
Check. The trace of \(S\) is \(8/3\), which equals the sum \(4/3+4/3+0\) of the eigenvalues. The count of positive eigenvalues is two, as required by the centered-matrix rank.
Worked Example: Three Variables but Only Two Directions
For three observations, suppose a hypothetical monitoring file contains \(A=(9,10,11)\), \(B=(19,16,19)\), and \(C=(28,26,30)\). Determine the exact redundancy, rank, and covariance eigenvalues.
Method. Center the variables, verify the column relationship, and use the covariance matrix. The trap is to count three recorded columns as three independent components without checking whether one is redundant.
The means are \(\bar A=10\), \(\bar B=18\), and \(\bar C=28\). Hence the centered columns are \(a=(-1,0,1)\), \(b=(1,-2,1)\), and \(c=(0,-2,2)\). Entry by entry, \(c=a+b\), so the rank is at most two. Since \(a\) and \(b\) are nonzero and not multiples, the rank is exactly two.
The column cross-products are \(a^{\mathsf T}a=2\), \(b^{\mathsf T}b=6\), \(c^{\mathsf T}c=8\), \(a^{\mathsf T}b=0\), \(a^{\mathsf T}c=2\), and \(b^{\mathsf T}c=6\). Dividing by \(n-1=2\) gives
The trace is \(1+3+4=8\). The determinant is zero because the third centered column is the sum of the first two. The sum of the three principal \(2\times2\) minors is \(3+3+3=9\). Therefore the characteristic polynomial is \(\lambda(\lambda^2-8\lambda+9)\). The eigenvalues are \(0\) and \((8\pm\sqrt{64-36})/2=4\pm\sqrt7\). Both \(4-\sqrt7\) and \(4+\sqrt7\) are positive, so exactly two components have positive variance.
Check. The positive eigenvalues sum to \(8\), matching the trace, and their product is \((4-\sqrt7)(4+\sqrt7)=16-7=9\), matching the sum of principal minors. The zero eigenvalue agrees with the exact relation \(c=a+b\).
Worked Example: Redundancy After Standardization
A hypothetical portfolio study records \(U=(1,2,3,4)\) and \(V=(12,14,16,18)\), where \(V=2U+10\) for each observation. Compare covariance and correlation PCA, and verify the positive component's sample variance.
Method. Center the variables, calculate the sample covariance, and then standardize each nonconstant column for correlation PCA. The trap is to assume that different units or a nonzero intercept remove exact redundancy.
The means are \(\bar U=2.5\) and \(\bar V=15\). The centered columns are \(x=(-1.5,-0.5,0.5,1.5)\) and \(y=(-3,-1,1,3)=2x\). Thus the rank is one. The sums of squares are \(x^{\mathsf T}x=5\) and \(y^{\mathsf T}y=20\); the cross-product sum is \(x^{\mathsf T}y=10\). With \(n-1=3\),
Its determinant is \((5/3)(20/3)-(10/3)^2=100/9-100/9=0\), and its trace is \(25/3\). The eigenvalues are therefore \(25/3\) and \(0\). A unit eigenvector for the positive eigenvalue is \(q=(1,2)^{\mathsf T}/\sqrt5\). The scores are \(X_cq=\sqrt5x\), so their sum of squares is \(5(x^{\mathsf T}x)=25\). Their sample variance is \(25/3\), matching the positive eigenvalue \(25/3\).
For correlation PCA, the two standardized columns are identical because \(y=2x\) and both variables have positive standard deviation. The correlation matrix is
with eigenvalues \(2\) and \(0\). Standardization changes the positive eigenvalue's scale, but the rank remains one and there is still exactly one nonzero component.
Check. The covariance eigenvalues sum to \(25/3\), the trace of \(S\); the correlation eigenvalues sum to \(2\), the trace of \(R\). Both analyses identify one positive-variance direction and one redundant direction.
Common Mistakes and Practical Checks
- Counting variables instead of independent centered columns. Three columns can have rank one or two if some are constant or exactly redundant.
- Confusing a small eigenvalue with a zero eigenvalue. Only an exactly zero eigenvalue indicates no variation in that direction in the observed data. Rounding a small positive value to zero can change the reported component count.
- Using correlation PCA on a constant variable. A zero standard deviation makes its correlation undefined. Remove or separately handle that variable before forming a correlation matrix.
- Ignoring the divisor when checking score variance. For sample covariance PCA, score variance uses divisor \(n-1\), just as the entries of \(S\) do. A score sum of squares of \(25\) with \(n=4\) gives sample variance \(25/3\), not \(5\).
A reliable check is to compare the number of positive eigenvalues with the rank of the centered columns and to compare the sum of all eigenvalues with the trace of the covariance or correlation matrix. For covariance PCA, that trace is the total sample variance of the included variables. These checks do not replace the calculations, but they often reveal an omitted constant column, a missed exact relation, or an arithmetic error.
The rank result is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus: it connects the algebra of the data matrix to the interpretation of PCA output. Keep the distinction between exact redundancy and strong association explicit whenever component counts are requested.
Check Your Understanding
For each problem, determine the centered-column rank and the number of nonzero components; use the stated PCA scale where relevant.
- A data matrix has centered columns \(u=(-1,0,1)\), \(v=(2,0,-2)\), and \(w=(0,1,0)\). What is its rank, and how many covariance-PCA eigenvalues are positive? (Answer: Rank \(2\); two positive eigenvalues.)
- Three recorded variables have centered columns \(a=(-1,0,1)\), \(b=(1,-2,1)\), and \(c=(0,-2,2)\). State the exact relationship and the number of positive covariance-PCA eigenvalues. (Answer: \(c=a+b\); rank \(2\), so two positive eigenvalues.)
- A four-observation data set has variables \(P=(4,5,6,7)\), \(Q=(9,9,9,9)\), and \(R=(3,5,7,9)\). How many positive covariance-PCA eigenvalues are there, and can all three variables be used directly in correlation PCA? (Answer: The centered columns of \(P\) and \(R\) are proportional and \(Q\) is zero, so rank \(1\) and one positive eigenvalue; no, \(Q\) has zero standard deviation.)
- Two centered columns \(x\) and \(y\) satisfy \(y=-3x\), and \(x^{\mathsf T}x=12\) for \(n=5\). Find the covariance eigenvalues. (Answer: The rank is \(1\); the sole positive eigenvalue is \((x^{\mathsf T}x+y^{\mathsf T}y)/(n-1)=(12+108)/4=30\), and the other is \(0\).)
- A correlation matrix for two nonconstant variables is \(\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\). Is either component exactly zero-variance? Explain using its eigenvalues. (Answer: No. The eigenvalues are \(1.8\) and \(0.2\), both positive; correlation \(0.8\) is strong but not exact redundancy.)