Reconstruction Error And Discarded Eigenvalues

Why Discarded Components Determine Reconstruction Error

A principal component reconstruction replaces each centered observation with its projection onto a selected subspace. Retaining fewer components generally makes the representation more compact, but some of the original variation is then lost. The central question is whether a particular choice of subspace loses as little as possible, and how to calculate that loss.

For PCA based on the sample covariance matrix, the answer is especially direct: among all rank-\(k\) orthogonal projections, the subspace spanned by the first \(k\) principal component directions has the smallest total squared reconstruction error. The minimum equals the sample-size factor \(n-1\) multiplied by the sum of the eigenvalues that are discarded.

Definition: Let \(\mathbf X\) be an \(n\times p\) centered data matrix, with \(n>1\), and let \(S=\mathbf X^\mathsf{T}\mathbf X/(n-1)\) be its sample covariance matrix. A rank-\(k\) orthogonal projection in variable space is a matrix \(P\) satisfying \(P^\mathsf{T}=P\), \(P^2=P\), and \(\operatorname{rank}(P)=k\). The reconstruction is \(\widehat{\mathbf X}=\mathbf X P\), and its total squared reconstruction error is \(\|\mathbf X-\widehat{\mathbf X}\|_F^2\), the sum of the squared entries of the residual matrix.

Write the eigenvalues of \(S\) in nonincreasing order as \(\lambda_1\ge\lambda_2\ge\cdots\ge\lambda_p\ge0\), with corresponding orthonormal eigenvectors \(v_1,\ldots,v_p\). Nonnegativity follows because a covariance matrix is positive semidefinite. The projection onto the first \(k\) eigenvectors is \(P_k=\sum_{j=1}^k v_jv_j^\mathsf{T}\). For \(k=0\), this sum is empty and \(P_0=0\).

The Minimum-Error Result

Theorem: For \(0\le k\le p\), the minimum total squared reconstruction error over all rank-\(k\) orthogonal projections is

\min_{\operatorname{rank}(P)=k,\;P^\mathsf{T}=P,\;P^2=P} \|\mathbf X-\mathbf X P\|_F^2 =(n-1)\sum_{j=k+1}^{p}\lambda_j.

The minimum is attained by projecting onto the span of \(v_1,\ldots,v_k\). An empty sum, as occurs for \(k=p\), is zero.

The eigenvalues here are sample variances in the principal component directions. The sum of discarded eigenvalues is therefore the residual variance per observation when measured as squared Euclidean distance in the variables used for PCA. Multiplication by \(n-1\) converts that sample-variance sum into the total squared error across the \(n\) rows. This is a total squared error, not an average per observation; the average is obtained by dividing the total by \(n\).

Why the First Components Are Optimal

The argument connects reconstruction error to the amount of variance captured. Since \(P\) is a symmetric projection, \(I-P\) is also symmetric and idempotent. Using the identity \(\|A\|_F^2=\operatorname{tr}(A^\mathsf{T}A)\), the reconstruction error is

\begin{aligned} \|\mathbf X-\mathbf X P\|_F^2 &=\|\mathbf X(I-P)\|_F^2\\ &=\operatorname{tr}\!\left((I-P)\mathbf X^\mathsf{T}\mathbf X(I-P)\right)\\ &=\operatorname{tr}\!\left((I-P)\mathbf X^\mathsf{T}\mathbf X\right)\\ &=(n-1)\left\{\operatorname{tr}(S)-\operatorname{tr}(PS)\right\}. \end{aligned}

The third line uses \( (I-P)^2=I-P \) and the cyclic property of the trace. Thus, minimizing residual error is equivalent to maximizing \(\operatorname{tr}(PS)\), the total covariance variation captured by the projection.

To compare any rank-\(k\) projection with the PCA projection, express it in the eigenvector basis. Define \(a_j=v_j^\mathsf{T}Pv_j=\|Pv_j\|^2\). An orthogonal projection cannot increase vector length, so \(0\le a_j\le1\). Also,

\sum_{j=1}^p a_j=\operatorname{tr}(P)=k, \qquad \operatorname{tr}(PS)=\sum_{j=1}^p\lambda_j a_j.

For \(1\le k\le p\), compare the captured variation with that from the first \(k\) eigenvectors. Since \(\sum_{j=1}^p a_j=k\), the amount of projection weight missing from the first \(k\) directions equals the weight assigned to later directions:

\sum_{j=1}^{k}(1-a_j)=\sum_{j=k+1}^{p}a_j.

The first \(k\) eigenvalues are at least \(\lambda_k\), while all later eigenvalues are at most \(\lambda_k\). Therefore,

\begin{aligned} \sum_{j=1}^{k}\lambda_j-\sum_{j=1}^{p}\lambda_j a_j &=\sum_{j=1}^{k}\lambda_j(1-a_j)-\sum_{j=k+1}^{p}\lambda_j a_j\\ &\ge \lambda_k\sum_{j=1}^{k}(1-a_j) -\lambda_k\sum_{j=k+1}^{p}a_j\\ &=0. \end{aligned}

Thus no rank-\(k\) orthogonal projection captures more variation than the first \(k\) principal directions. Those directions capture \(\sum_{j=1}^k\lambda_j\), leaving \(\sum_{j=k+1}^p\lambda_j\). Substitution into the error identity gives the stated minimum.

The case \(k=0\) must be handled separately because \(\lambda_k\) is not defined. The only rank-zero projection is \(P=0\), so the reconstruction is zero and

\|\mathbf X-\mathbf X P\|_F^2 =\|\mathbf X\|_F^2 =\operatorname{tr}(\mathbf X^\mathsf{T}\mathbf X) =(n-1)\operatorname{tr}(S) =(n-1)\sum_{j=1}^{p}\lambda_j.

This is exactly the theorem's formula when no components are retained. At the other endpoint, \(k=p\) retains the entire variable space, so \(P=I\) and the reconstruction error is zero.

Worked Calculations

Worked Example: Error From a Given Eigenvalue Spectrum

A hypothetical centered panel has \(n=8\) observations and \(p=4\) variables. Its sample covariance eigenvalues are \(6,3,2,1\). A reconstruction retains the first \(k=2\) principal components. Find the minimum total squared reconstruction error and the mean squared error per observation, where the latter is the total squared error divided by \(n\).

Method. The first two eigenvalues are retained, so the discarded eigenvalues are \(2\) and \(1\). There are \(n-1=7\) degrees of freedom in the sample covariance scaling.

Total squared error. The sum of discarded eigenvalues is \(2+1=3\). Therefore,

\text{total error}=(n-1)(\lambda_3+\lambda_4)=7(2+1)=21.

Mean squared error per observation. Divide the total by \(n=8\), not by \(n-1\):

\text{mean squared error per observation}=\frac{21}{8}=2.625.

Check. The total sample variance is \(6+3+2+1=12\), so the total squared distance before reducing dimension is \(7(12)=84\). The retained directions account for \(7(6+3)=63\); the remaining \(84-63=21\) agrees with the discarded-eigenvalue calculation. The common mistake avoided here is reporting \(3\) as the total error: it is the discarded variance sum, before multiplying by \(n-1\).

Worked Example: Comparing PCA With Another Rank-One Projection

A hypothetical pricing study has \(n=6\) centered observations and two variables. In the eigenvector coordinate system, the covariance matrix is \(S=\begin{pmatrix}7&0\\0&2\end{pmatrix}\). Compare the total squared error for the PCA rank-one projection with the error from the rank-one projection onto \(u=(3/5,4/5)^\mathsf{T}\).

Method. For a unit vector \(u\), the projection is \(P=uu^\mathsf{T}\). Its captured variation is \(\operatorname{tr}(PS)=u^\mathsf{T}Su\), and its total squared error is \((n-1)\{\operatorname{tr}(S)-u^\mathsf{T}Su\}\).

First calculate the arbitrary projection:

P= \begin{pmatrix} 9/25&12/25\\ 12/25&16/25 \end{pmatrix}, \qquad u^\mathsf{T}Su =7\left(\frac{9}{25}\right)+2\left(\frac{16}{25}\right) =\frac{63+32}{25} =\frac{19}{5}.

The total covariance trace is \(7+2=9\), and \(n-1=5\). Thus this projection has error

5\left(9-\frac{19}{5}\right) =5\left(\frac{26}{5}\right)=26.

PCA retains the eigenvector associated with eigenvalue \(7\). It captures \(7\), leaving \(2\), so its error is \(5(2)=10\). The PCA projection has the smaller error, as the theorem guarantees.

Check. The arbitrary projection captures \(19/5=3.8\), which is less than \(7\), and its error \(26\) exceeds the PCA error \(10\). The common mistake avoided is assuming that every rank-one projection retains the largest eigenvalue; only a projection onto the corresponding eigenvector does so.

Worked Example: Checking the Error From Observation Residuals

A hypothetical centered data matrix, already expressed in its principal component coordinates, is

\mathbf X= \begin{pmatrix} 1&1\\ 1&-1\\ -2&0 \end{pmatrix}.

Retain only the first component. Compute the reconstruction and its total squared error, then verify the eigenvalue formula.

Method. The column vectors are \(c_1=(1,1,-2)^\mathsf{T}\) and \(c_2=(1,-1,0)^\mathsf{T}\). Their squared lengths are \(1+1+4=6\) and \(1+1+0=2\), and their inner product is \(1-1+0=0\). Consequently,

S=\frac{\mathbf X^\mathsf{T}\mathbf X}{n-1} =\frac{1}{2} \begin{pmatrix}6&0\\0&2\end{pmatrix} = \begin{pmatrix}3&0\\0&1\end{pmatrix}.

The eigenvalues are \(3\) and \(1\). The rank-one projection onto the first coordinate is \(P=\begin{pmatrix}1&0\\0&0\end{pmatrix}\), giving

\widehat{\mathbf X}=\mathbf X P= \begin{pmatrix} 1&0\\ 1&0\\ -2&0 \end{pmatrix}, \qquad \mathbf X-\widehat{\mathbf X}= \begin{pmatrix} 0&1\\ 0&-1\\ 0&0 \end{pmatrix}.

The row-wise squared errors are \(1^2=1\), \((-1)^2=1\), and \(0^2=0\), so the total is \(1+1+0=2\). The eigenvalue result gives \((n-1)\lambda_2=2(1)=2\), confirming the direct calculation.

Check. The residual contains only the second component's scores, whose squared sum is \(2\). The common mistake avoided is summing the omitted eigenvalue alone and calling \(1\) the total error; the covariance eigenvalue must be multiplied by \(n-1=2\).

Interpretation and Common Traps

The result is about squared Euclidean reconstruction error in the coordinate system used to form the covariance matrix. If variables were standardized before PCA, the error is measured in those standardized coordinates; if PCA used unstandardized variables, units with larger numerical scales can contribute more to the squared error. The result does not say that every individual observation has the same error. It identifies the minimum sum over all observations.

Do not omit the sample-size factor. The sum of discarded eigenvalues is the residual variance sum; total squared error is \((n-1)\) times that sum.
Do not confuse total and average error. Total error is divided by \(n\), not \(n-1\), to obtain the mean squared reconstruction error per row.
Do not apply the result to any rank-\(k\) map without checking its form. The theorem compares orthogonal projections. A general rank-\(k\) transformation need not satisfy the projection conditions used in the error identity.
Handle \(k=0\) directly. There is no \(\lambda_0\); the only rank-zero projection is zero, and its error is the full squared norm of the centered data matrix.

Key takeaway: For a centered data matrix with sample covariance eigenvalues ordered from largest to smallest, keeping the first \(k\) principal components minimizes total squared reconstruction error among rank-\(k\) orthogonal projections. The minimum is \((n-1)\) times the sum of the omitted eigenvalues.

Check Your Understanding

For each problem, use the sample covariance convention \(S=\mathbf X^\mathsf{T}\mathbf X/(n-1)\), and distinguish total squared error from average error where requested.

A centered data set has \(n=10\) observations and eigenvalues \(5,2,1\). Find the minimum total squared error when retaining two components. (Answer: \(9\))
A centered data set has \(n=5\) observations and eigenvalues \(8,3,0\). Find the minimum total squared error when retaining one component, and state why the zero eigenvalue contributes nothing. (Answer: \(12\); the discarded-eigenvalue sum is \(3+0=3\), so \(4(3)=12\))
A covariance matrix has eigenvalues \(6\) and \(2\), and \(n=4\). A rank-one projection uses \(u=(1/\sqrt{2},1/\sqrt{2})^\mathsf{T}\) in the eigenvector coordinate system. Find its total squared error and compare it with the PCA minimum. (Answer: arbitrary projection error \(12\); PCA minimum \(6\))
A centered data set has \(n=7\) observations and eigenvalues \(4,2,1\). Find the error when no components are retained, using the rank-zero case. (Answer: \(42\))
A centered data set has \(n=8\) observations and eigenvalues \(10,5,1\). Compare the minimum total errors for retaining one and two components; how much does the error decrease when the second component is added? (Answer: errors \(42\) and \(7\), respectively; decrease \(35\))

What You'll Learn