Why the Covariance Matrix Matters in PCA
After centering data for PCA, each feature has sample mean zero. The next step is to describe how much each feature varies and how pairs of features vary together. A single variance summarizes one feature; a covariance matrix collects the variances and covariances for every feature pair in one object. In covariance-based PCA, this matrix is the summary of variation that the analysis uses as its input.
The entries depend on the features’ original measurement units. A diagonal entry is a variance, while an off-diagonal entry is a covariance. Their sizes are therefore not directly comparable unless the units are taken into account. How to standardize features for correlation-based PCA is a separate choice; here the focus is on constructing and interpreting the covariance matrix itself.
The denominator is \(n-1\), not \(n\), for the usual sample covariance matrix. This convention makes each sample variance an unbiased estimator of the corresponding population variance when the observations are independent draws from a population with finite second moments. The denominator does not change the sign of a covariance, but it does change every entry’s magnitude.
Constructing the Matrix
For two features, the matrix has two variances on its diagonal and the same covariance in both off-diagonal positions. More generally, the matrix can be calculated by multiplying the transpose of the centered data matrix by the centered data matrix, then dividing by \(n-1\):
The product \(Z^\mathsf{T}Z\) has one row and one column for each feature. Its entry in position \((j,k)\) is the sum of the products \(z_{ij}z_{ik}\) over observations. Dividing by \(n-1\) gives the sample covariance in that position. This calculation is available directly in R: if the rows of a numeric matrix `Z` are centered observations, `crossprod(Z)/(n - 1)` returns the sample covariance matrix. The base R function `cov()` also uses the sample-covariance denominator \(n-1\).
Worked Example: A Two-Feature Covariance Matrix
A hypothetical portfolio team records two centered features for four simulated portfolios: deviation in a projected expense measure, \(x\), and deviation in a projected service-time measure, \(y\). The centered observations are \((-2,-1)\), \((-1,1)\), \((1,-1)\), and \((2,1)\). Construct the sample covariance matrix and interpret its entries.
Method. Calculate each feature’s sum of squared deviations and the cross-product sum, then divide all three sums by \(n-1=4-1=3\). The observations are already centered: the \(x\) values sum to \(-2-1+1+2=0\), and the \(y\) values sum to \(-1+1-1+1=0\).
For \(x\), the sum of squares is \((-2)^2+(-1)^2+1^2+2^2=4+1+1+4=10\). For \(y\), it is \((-1)^2+1^2+(-1)^2+1^2=1+1+1+1=4\). The cross-product sum is \((-2)(-1)+(-1)(1)+(1)(-1)+(2)(1)=2-1-1+2=2\).
The sample variances are \(s_{xx}=10/3\) and \(s_{yy}=4/3\). The sample covariance is \(s_{xy}=2/3\). Because covariance is symmetric, \(s_{yx}=s_{xy}\). Thus
The diagonal values are the sample variances in the squared units of their respective features. The off-diagonal value is positive, so these four observations show that above-average \(x\) values tend to occur with above-average \(y\) values, and below-average values tend to occur together. Its units are the product of the \(x\) and \(y\) units. The matrix is symmetric, as required. A common mistake would be to divide by \(n=4\); the sample covariance convention uses \(n-1=3\), so that calculation would give the wrong matrix.
Why the Formula Uses \(n-1\)
The centered product sum has a useful justification. Suppose paired observations \((X_i,Y_i)\) are independent draws from a population with means \(\mu_X,\mu_Y\), finite variances, and population covariance \(\operatorname{Cov}(X,Y)\). Let \(\bar X\) and \(\bar Y\) be their sample means. Expanding around the population means gives
To see the identity, write \(X_i-\bar X=(X_i-\mu_X)-(\bar X-\mu_X)\), and similarly for \(Y\). On multiplying and summing, the two terms involving one sample-mean deviation simplify because \(\sum_i(X_i-\mu_X)=n(\bar X-\mu_X)\) and \(\sum_i(Y_i-\mu_Y)=n(\bar Y-\mu_Y)\). The resulting terms combine to the single subtraction shown above.
Taking expectations, the first sum on the right has expectation \(n\operatorname{Cov}(X,Y)\). The second term has expectation \(n\operatorname{Cov}(\bar X,\bar Y)\). Independence across observations implies \(\operatorname{Cov}(\bar X,\bar Y)=\operatorname{Cov}(X,Y)/n\), so the expectation of that second term is \(\operatorname{Cov}(X,Y)\). Therefore
Dividing the centered product sum by \(n-1\) consequently gives an unbiased sample covariance under these assumptions. This justification concerns estimation from independent, identically distributed observations; the algebraic construction of \(S\) still defines a matrix for any centered data set with \(n>1\), but the unbiasedness interpretation need not hold for data generated under a different sampling design.
Interpreting the Entries and Checking the Matrix
A covariance describes the direction of paired variation, not a causal relationship. A positive sample covariance indicates that paired deviations tend to have the same sign; a negative sample covariance indicates that they tend to have opposite signs. A covariance of zero means the observed centered cross-products sum to zero. It does not, by itself, establish that the features are independent or that there is no nonlinear relationship.
The units matter. If one feature is measured in dollars and another in years, their covariance is in dollar-years. A variance is in squared units, such as dollars squared. Multiplying a feature by a constant changes its variance and its covariances with other features. Thus covariance-based PCA retains the effects of the original units; a feature measured on a much larger numerical scale can contribute large covariance entries. This is one reason the analyst must know how the data were measured before interpreting the matrix.
Two structural checks are useful before using a covariance matrix in PCA:
- Symmetry. \(s_{jk}=s_{kj}\), since \(z_{ij}z_{ik}=z_{ik}z_{ij}\). A matrix with unequal corresponding off-diagonal entries has been calculated or transcribed incorrectly.
- Nonnegative variances. Each diagonal entry is a sum of squared centered values divided by \(n-1\), so it cannot be negative.
- Positive semidefiniteness. For any \(p\)-component vector \(a\), the quadratic form \(a^\mathsf{T}Sa\) is nonnegative. Indeed, \(Za\) is a vector of projected centered values, so \(a^\mathsf{T}Sa=(Za)^\mathsf{T}(Za)/(n-1)=\sum_i ((Za)_i)^2/(n-1)\geq 0\). A negative quadratic form signals an invalid covariance matrix, apart from numerical rounding effects in computation.
This last property is a check on the whole matrix, not just its diagonal. In practice, a matrix constructed as \(Z^\mathsf{T}Z/(n-1)\) has this property by construction. For instance, exact linear dependence between features may make a covariance matrix singular, but that does not make its variances or covariances invalid.
Worked Example: Computing from Uncentered Observations
A hypothetical underwriting team records two quantitative summaries for five simulated risks. The first feature \(x\) has values \(2,4,6,8,10\); the second feature \(y\) has values \(1,3,2,5,4\). Construct the sample covariance matrix. A reviewer has proposed using the raw product sum \(\sum_i x_i y_i\) as the cross-product sum; check that proposal.
Method. First calculate the means, then subtract them observation by observation. Covariances use products of centered values, not products of the original values. The means are \(\bar x=(2+4+6+8+10)/5=30/5=6\) and \(\bar y=(1+3+2+5+4)/5=15/5=3\).
The centered values are \(x-\bar x=(-4,-2,0,2,4)\) and \(y-\bar y=(-2,0,-1,2,1)\). Both centered columns sum to zero. The sum of squared \(x\) deviations is \(16+4+0+4+16=40\). The sum of squared \(y\) deviations is \(4+0+1+4+1=10\). The centered cross-products are \(8,0,0,4,4\), so their sum is \(16\).
There are \(n=5\) observations, so the denominator is \(n-1=4\). Hence \(s_{xx}=40/4=10\), \(s_{yy}=10/4=2.5\), and \(s_{xy}=16/4=4\). The matrix is
The proposed raw product sum is not the centered cross-product sum: \(\sum_i x_i y_i=2+12+12+40+40=106\). The correction is to subtract \(n\bar x\bar y=5(6)(3)=90\), giving \(106-90=16\), which agrees with the centered calculation. Dividing the incorrect raw sum by 4 would give \(26.5\), not the sample covariance \(4\). The common mistake is to omit centering; using the raw cross-products measures a different quantity.
Worked Example: A Three-Feature Matrix and Its Structure
For four hypothetical policy records, suppose the centered feature columns are \(a=(-1,-1,1,1)\), \(b=(-2,0,0,2)\), and \(c=(1,-1,-1,1)\). Calculate the complete sample covariance matrix and describe what its entries show.
Method. Compute the three sums of squares and three distinct cross-product sums. Since \(n=4\), divide each by \(n-1=3\). The columns are centered: their sums are \(0\), \(0\), and \(0\), respectively.
The sums of squares are \(\sum a_i^2=1+1+1+1=4\), \(\sum b_i^2=4+0+0+4=8\), and \(\sum c_i^2=1+1+1+1=4\). The cross-product sum for \(a,b\) is \(2+0+0+2=4\). For \(a,c\), it is \(-1+1-1+1=0\). For \(b,c\), it is \(-2+0+0+2=0\).
After dividing by 3, the variances are \(s_{aa}=4/3\), \(s_{bb}=8/3\), and \(s_{cc}=4/3\). The covariances are \(s_{ab}=4/3\), \(s_{ac}=0\), and \(s_{bc}=0\). Placing each covariance symmetrically gives
The positive \(a,b\) covariance indicates same-direction paired variation in this sample. The zero covariances involving \(c\) mean that the corresponding centered cross-products cancel; they do not prove independence. The matrix is symmetric and has nonnegative diagonal entries. As another check, its first two features have a positive covariance, which is consistent with the matching signs of their deviations in the four records.
Common Exam Traps
The most reliable way to avoid errors is to identify the rows, the feature columns, and the centering status before calculating. Then calculate sums of squares and cross-products explicitly, and only afterward apply the denominator. The following mistakes can materially change a PCA input:
- Using \(n\) instead of \(n-1\). For the usual sample covariance matrix, divide by \(n-1\). Be alert to the distinction between a sample covariance and a population covariance calculated from a complete population or a specified probability distribution.
- Using uncentered products. The formula uses \(z_{ij}z_{ik}\), where each \(z_{ij}\) is measured relative to its feature’s mean. Raw products \(x_{ij}x_{ik}\) are not a substitute unless the relevant columns are already centered.
- Putting covariances in only one triangle. A covariance matrix must be symmetric. The covariance between feature \(j\) and feature \(k\) appears in both positions \((j,k)\) and \((k,j)\).
- Calling an off-diagonal covariance a correlation. Covariance is not confined to the interval from \(-1\) to \(1\), and it has units. Correlation is a different, unit-free quantity.
- Overinterpreting zero covariance. A zero sample covariance says that the centered products sum to zero in this data set. It does not establish independence or rule out nonlinear dependence.
For covariance-based PCA, the covariance matrix is the map of the observed sample’s feature variation: diagonal entries summarize individual-feature variation, and off-diagonal entries summarize paired variation. Its construction should be checked before moving on to directions or components. The matrix alone does not justify causal claims, and its entries reflect the measurement scales chosen for the features.
Check Your Understanding
For each problem, show the centered sums or explain the relevant matrix property before checking the answer.
- Three centered observations have feature columns \(x=(-1,0,1)\) and \(y=(1,-2,1)\). Calculate the sample covariance matrix. (Answer: \(S=\begin{pmatrix}1&0\\0&3\end{pmatrix}\).)
- Four centered observations have \(a=(-3,-1,1,3)\) and \(b=(-1,1,-1,1)\). Calculate the sample covariance matrix using the correct denominator. (Answer: \(S=\begin{pmatrix}20/3&4/3\\4/3&4/3\end{pmatrix}\).)
- For seven centered pairs, the cross-product sum is 18. A candidate divides by 7 and reports covariance \(18/7\). Identify the error and give the sample covariance. (Answer: the denominator should be \(7-1=6\); the sample covariance is \(18/6=3\).)
- Four observations have \(x=(1,2,3,4)\) and \(y=(4,2,3,1)\). Calculate the sample covariance, showing the means and centered cross-product sum. (Answer: both means are \(2.5\); the centered cross-product sum is \(-4\), so the sample covariance is \(-4/3\).)
- A feature is measured in dollars and another in years. What are the units of their covariance, and what does a negative covariance indicate about paired deviations? (Answer: dollar-years; negative covariance indicates that above-mean deviations in one feature tend to pair with below-mean deviations in the other, and vice versa.)