Why the Sample Covariance Matrix Matters
Principal components analysis (PCA) begins with a matrix that summarizes how measured variables vary on their own and together. The diagonal entries of a sample covariance matrix record the sample variances; the off-diagonal entries record the sample covariances. For covariance-based PCA, this matrix is the starting summary of the centered data. A mistake in its construction—such as using the wrong divisor or pairing values from different observations—changes that summary.
Suppose a data set contains observations on several variables. Each row represents one observation and each column represents one variable. To build a covariance matrix, first center each column by subtracting its sample mean. Then calculate a sample covariance for every pair of columns. This tutorial develops that calculation, including how to check the result and what its entries do and do not tell us.
The condition \(n\geq 2\) matters: with one observation, \(n-1=0\), so this sample covariance matrix is undefined. The entries \(s_{jj}\) are sample variances. For \(j\neq k\), \(s_{jk}\) is the sample covariance between variables \(j\) and \(k\). The ordering of the variables in \(S\) follows the ordering of the columns in \(Z\).
Calculating Entries and Reading the Matrix
For any pair of variables, multiply their centered values observation by observation, add those products, and divide by \(n-1\). For a variance, the two variables in the pair are the same, so each product is a squared centered value. The same observations must be paired: multiplying a value from one row by a value from another row does not calculate the sample covariance.
Worked Example: Two Hypothetical Risk Measures
A hypothetical analyst records two measures for three territories: annual claim frequency per 100 policy-years and average claim severity in units of $10,000. The observations are \((1,2)\), \((2,4)\), and \((3,6)\), in that variable order. Calculate the sample covariance matrix. State what the off-diagonal entry says and check the result.
Method. Calculate the column means, center each column, then use the pairwise-product formula with \(n-1=2\).
The means are \(\bar{x}_1=(1+2+3)/3=2\) and \(\bar{x}_2=(2+4+6)/3=4\). Thus the centered rows are \((-1,-2)\), \((0,0)\), and \((1,2)\). Both centered columns sum to zero, as required.
For the first diagonal entry, the sum of squared centered values is \((-1)^2+0^2+1^2=2\), so \(s_{11}=2/2=1\). For the second diagonal entry, it is \((-2)^2+0^2+2^2=8\), so \(s_{22}=8/2=4\). For the off-diagonal entry, the paired products sum to \((-1)(-2)+(0)(0)+(1)(2)=4\), so \(s_{12}=4/2=2\). Symmetry gives \(s_{21}=2\).
The sample covariance is positive: in these three observations, above-average frequency is paired with above-average severity, and below-average frequency with below-average severity. The check is that the matrix is symmetric, its diagonal entries are the calculated variances, and its off-diagonal entries agree. Common mistake avoided: dividing by \(n=3\) would give a different matrix; the stated sample covariance uses \(n-1=2\).
A compact way to perform all of the pairwise calculations is matrix multiplication. The \((j,k)\) entry of \(Z^{\mathsf T}Z\) is the inner product of centered columns \(j\) and \(k\), which is exactly \(\sum_{i=1}^n z_{ij}z_{ik}\). Dividing every entry by \(n-1\) therefore gives the stated covariance matrix. This also explains why its dimensions are \(p\) by \(p\), even though \(Z\) has \(n\) rows and \(p\) columns.
Worked Example: Constructing a Three-Variable Matrix
For four hypothetical portfolios, an analyst has already centered three risk indicators. The columns are, in order, a frequency measure, a volatility measure, and a severity measure. The centered data matrix is
Calculate the sample covariance matrix, and check that the centered columns and the resulting entries are consistent. Here \(n=4\), so every sum of products is divided by \(n-1=3\).
Method. First confirm centering. The column sums are \(1-1+0+0=0\), \(0+2-1-1=0\), and \(2+0-1-1=0\). Next calculate the six distinct entries; symmetry supplies the other three.
The first variance uses \(1^2+(-1)^2+0^2+0^2=2\), giving \(s_{11}=2/3\). The second uses \(0^2+2^2+(-1)^2+(-1)^2=6\), giving \(s_{22}=2\). The third uses \(2^2+0^2+(-1)^2+(-1)^2=6\), giving \(s_{33}=2\).
For the first and second columns, the paired products sum to \((1)(0)+(-1)(2)+(0)(-1)+(0)(-1)=-2\), so \(s_{12}=-2/3\). For the first and third columns, they sum to \((1)(2)+(-1)(0)+(0)(-1)+(0)(-1)=2\), so \(s_{13}=2/3\). For the second and third columns, they sum to \((0)(2)+(2)(0)+(-1)(-1)+(-1)(-1)=2\), so \(s_{23}=2/3\).
The negative \(s_{12}\) records opposite-direction sample variation between the first two indicators; the positive \(s_{13}\) and \(s_{23}\) record same-direction sample variation for those pairs. The result is symmetric, and its diagonal entries are the three sample variances. Common mistake avoided: checking only the diagonal misses the off-diagonal pairings, which are essential to the joint summary.
Why the Divisor Is \(n-1\)
The divisor \(n-1\) defines the usual sample covariance. It is also the divisor that makes this matrix an unbiased estimator of a population covariance matrix under independent, identically distributed sampling with finite second moments. This unbiasedness is a statement about repeated samples from that model; it is not needed merely to calculate the descriptive covariance matrix of a particular data set.
The identity in this justification follows by writing \(X_i-\bar{X}=(X_i-\mu)-(\bar{X}-\mu)\), expanding the outer products, and using \(\sum_i(X_i-\mu)=n(\bar{X}-\mu)\). The resulting terms combine to subtract \(n(\bar{X}-\mu)(\bar{X}-\mu)^{\mathsf T}\). The expectation calculation relies on the stated independent, identical sampling assumptions and finite second moments. Do not treat unbiasedness as a claim that each sample entry equals its population counterpart.
Worked Example: Detecting the Divisor Trap
Four hypothetical observations have already been centered on two claim indicators. In the stated variable order, their centered rows are \((2,1)\), \((0,-1)\), \((-1,0)\), and \((-1,0)\). Calculate the sample covariance matrix, then identify the result that would come from mistakenly dividing by \(n\).
Method. Verify the column sums, calculate the three distinct sums of products, and compare the two divisors. This problem is designed to separate the sample covariance convention from the uncorrected average of products.
The first column sums to \(2+0-1-1=0\), and the second sums to \(1-1+0+0=0\). For the first variance, the sum of squares is \(2^2+0^2+(-1)^2+(-1)^2=6\). For the second, it is \(1^2+(-1)^2+0^2+0^2=2\). The cross-product sum is \((2)(1)+(0)(-1)+(-1)(0)+(-1)(0)=2\).
Since \(n-1=3\), the sample covariance matrix is
If the sums were incorrectly divided by \(n=4\), the result would instead be \(\begin{pmatrix}6/4&2/4\\2/4&2/4\end{pmatrix}=\begin{pmatrix}1.5&0.5\\0.5&0.5\end{pmatrix}\). Common mistake avoided: that second matrix uses a different divisor and is not the sample covariance matrix defined here. The sample result is checked by confirming the centered columns, the three product sums, and the symmetry of \(S\).
Checks and Interpretation for PCA
Before using a calculated matrix as the input to covariance-based PCA, check the construction and interpret each entry in context. The matrix does not discard the units of the original variables: a variance has squared units, and a covariance has the product of the two variables’ units. Thus, changing the measurement scale of a variable changes the corresponding covariance entries. The size of a covariance should not be compared across unrelated pairs without accounting for their units and scales.
Several structural checks are useful:
- Dimensions: with \(p\) variables, \(S\) must have \(p\) rows and \(p\) columns.
- Symmetry: \(s_{jk}=s_{kj}\), because multiplication of real numbers is commutative.
- Diagonal: each \(s_{jj}\) is nonnegative, since it is a sum of squared centered values divided by a positive number.
- Centering: each column of \(Z\) should sum to zero, apart from any small rounding discrepancies if rounded values were supplied.
- Pairing: each product in \(s_{jk}\) must use variables \(j\) and \(k\) from the same observation.
More generally, \(S\) is positive semidefinite: for any \(p\)-component vector \(a\), \(a^{\mathsf T}Sa=\frac{1}{n-1}(Za)^{\mathsf T}(Za)\geq 0\), because the right-hand side is a sum of squares divided by a positive number. This property is a useful consistency condition, although checking symmetry and nonnegative diagonal entries alone does not establish it for an arbitrary proposed matrix.
A covariance’s sign describes the direction of linear co-variation in the observed sample, not a causal relationship. A covariance of zero means the centered cross-products sum to zero; it does not, in general, establish independence. In PCA, the covariance matrix retains both individual variation and pairwise co-variation, so errors in scale, centering, or column order can affect the analysis that follows. The choice between covariance-based PCA and other approaches depends on the modeling question and measurement scales; constructing \(S\) correctly does not by itself make that choice.
Check Your Understanding
For each problem, use the usual sample covariance divisor and keep the variable order given.
- A single centered variable has values \(-2,0,2\) across three observations. Calculate its sample variance. (Answer: \(4\))
- Two centered variables have rows \((1,1)\), \((-1,-1)\), and \((0,0)\). Calculate their sample covariance matrix and state the off-diagonal value. (Answer: \(S=\begin{pmatrix}1&1\\1&1\end{pmatrix}\); covariance \(=1\))
- Four observations have centered rows \((1,1)\), \((-1,1)\), \((1,-1)\), and \((-1,-1)\). Calculate the sample covariance matrix. (Answer: \(S=\begin{pmatrix}4/3&0\\0&4/3\end{pmatrix}\))
- Three centered observations on variables \(A\) and \(B\) are \((1,2)\), \((-1,0)\), and \((0,-2)\). Calculate \(S\), and identify whether the sample covariance is positive, negative, or zero. (Answer: \(S=\begin{pmatrix}1&1\\1&4\end{pmatrix}\); positive)
- Four centered observations have rows \((2,0)\), \((0,2)\), \((-1,-1)\), and \((-1,-1)\). Calculate the sample covariance matrix and explain what using \(n\) instead of \(n-1\) would change. (Answer: \(S=\begin{pmatrix}2&2/3\\2/3&2\end{pmatrix}\); using \(n\) gives \(\begin{pmatrix}1.5&0.5\\0.5&1.5\end{pmatrix}\))