Reconstruction Error and Residual Variance

Why Reconstruction Error Is a Variance Quantity

Principal component analysis (PCA) represents a centered random vector by its coordinates along orthogonal covariance directions. Retaining only the first $K$ directions gives a lower-dimensional representation, so some variation is omitted. The size of that omission can be measured by the squared Euclidean distance between the original vector and its reconstruction.

This measure is useful because it puts the reconstruction question on the same scale as covariance: directions with large eigenvalues contribute more expected squared error when discarded, while directions with zero variance contribute none. The key population identity is exact, not an approximation: the expected squared residual is the sum of the eigenvalues belonging to the omitted directions.

Definition: Let $X$ be a random vector in $\mathbb{R}^p$ with finite second moments, mean $\mu=E[X]$, and covariance matrix $\Sigma$. Let $q_1,\ldots,q_p$ be an orthonormal eigenbasis of $\Sigma$, with eigenvalues ordered as $\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\geq 0$. For an integer $K$ with $0\leq K\leq p$, define the first-$K$ reconstruction and its residual by $$ \widehat X_K=\mu+\sum_{j=1}^{K}q_jq_j^{T}(X-\mu), \qquad R_K=X-\widehat X_K. $$ An empty sum is zero, so $K=0$ gives $\widehat X_0=\mu$. The population mean squared reconstruction error is $E[\|R_K\|^2]$, where $\|\cdot\|$ is the Euclidean norm.

The eigenvectors are taken to be orthonormal, and the eigenvalues are nonnegative because a covariance matrix is positive semidefinite. The formula for $\widehat X_K$ says that the centered vector is projected onto the span of the first $K$ principal directions, then the mean is added back. In particular, $K=p$ retains every direction and reconstructs $X$ exactly.

Deriving the Residual Covariance and Error

Write $Z=X-\mu$ for the centered vector, and let $T_j=q_j^T Z$ be its score in direction $q_j$. Since the eigenvectors form an orthonormal basis, $Z=\sum_{j=1}^p T_jq_j$. The residual is therefore the part of this expansion in the directions not retained:

R_K=Z-\sum_{j=1}^{K}T_jq_j=\sum_{j=K+1}^{p}T_jq_j.

Each score has mean zero, and the covariance between two scores is determined by the covariance eigenstructure:

E[T_j]=0, \qquad E[T_jT_\ell]=q_j^T\Sigma q_\ell =\lambda_\ell q_j^Tq_\ell = \begin{cases} \lambda_j,&j=\ell,\\ 0,&j\ne\ell. \end{cases}

Thus the omitted score coordinates are uncorrelated, with variances equal to their corresponding eigenvalues. The next result gives the complete residual covariance matrix and the expected squared error.

Theorem: Under the conditions in the definition, the residual has mean zero and covariance $$ \operatorname{Cov}(R_K)=\sum_{j=K+1}^{p}\lambda_jq_jq_j^T. $$ Consequently, $$ E[\|R_K\|^2]=\sum_{j=K+1}^{p}\lambda_j. $$ For $K=p$, both expressions are zero.

Proof. The expression for $R_K$ is a finite sum, so its expectation is the sum of the score expectations multiplied by their directions. Each score has mean zero; hence $E[R_K]=0$. Expanding its outer product and taking expectations gives

\begin{aligned} \operatorname{Cov}(R_K) &=E[R_KR_K^T]\\ &=E\left[\left(\sum_{j=K+1}^{p}T_jq_j\right) \left(\sum_{\ell=K+1}^{p}T_\ell q_\ell^T\right)\right]\\ &=\sum_{j=K+1}^{p}\sum_{\ell=K+1}^{p} E[T_jT_\ell]q_jq_\ell^T\\ &=\sum_{j=K+1}^{p}\lambda_jq_jq_j^T. \end{aligned}

The double sum is finite, so taking expectations term by term requires no interchange of an infinite sum and an integral. To obtain the scalar error, use $\|R_K\|^2=\operatorname{tr}(R_KR_K^T)$. Linearity of expectation and trace then yields

\begin{aligned} E[\|R_K\|^2] &=\operatorname{tr}\{E[R_KR_K^T]\}\\ &=\operatorname{tr}\left(\sum_{j=K+1}^{p}\lambda_jq_jq_j^T\right)\\ &=\sum_{j=K+1}^{p}\lambda_j\operatorname{tr}(q_jq_j^T)\\ &=\sum_{j=K+1}^{p}\lambda_j, \end{aligned}

because $\operatorname{tr}(q_jq_j^T)=q_j^Tq_j=1$. If $K=p$, the sum has no terms and equals zero, as it must because $R_p=0$. $\square$

Total Variance Minus Retained Variance

The same error can be understood as the difference between the total variation in the centered vector and the variation captured by the retained scores. This connection provides a useful check on calculations and makes clear why the discarded eigenvalues, rather than the retained ones, determine the residual.

Result: The total variance, measured as $E[\|X-\mu\|^2]$, is $\sum_{j=1}^{p}\lambda_j$. The expected squared reconstruction error after retaining $K$ components is the total variance minus the variance captured by those components: $$ E[\|R_K\|^2] =E[\|X-\mu\|^2]-\sum_{j=1}^{K}\lambda_j =\sum_{j=K+1}^{p}\lambda_j. $$

Derivation. Expand the centered vector in the complete orthonormal eigenbasis. By orthonormality, the squared norm of its expansion is the sum of the squared scores. Taking expectations, and using $E[T_j^2]=\lambda_j$, gives

E[\|X-\mu\|^2] =E\left[\sum_{j=1}^{p}T_j^2\right] =\sum_{j=1}^{p}E[T_j^2] =\sum_{j=1}^{p}\lambda_j.

The same argument applied to the reconstruction $\widehat X_K-\mu=\sum_{j=1}^{K}T_jq_j$ gives expected squared length $\sum_{j=1}^{K}\lambda_j$. Subtracting this captured amount from the total leaves precisely $\sum_{j=K+1}^{p}\lambda_j$. These are finite sums, and orthogonality ensures that the squared lengths split into retained and omitted parts without cross terms.

Worked Calculations From Eigenvalue Spectra

Worked Example: Two Components Retained From Four

Suppose a four-dimensional population has ordered covariance eigenvalues $9,4,1,0.5$. Find the expected squared error when the first two principal components are retained.

Here $p=4$ and $K=2$, so the discarded directions are the third and fourth. The theorem gives

$E[\|R_2\|^2]=\lambda_3+\lambda_4=1+0.5=1.5.$

As a check, total variance is $9+4+1+0.5=14.5$, while the retained components capture $9+4=13$. Their difference is $14.5-13=1.5$, agreeing with the direct sum. The error is measured in squared units of the original variables.

Worked Example: Residual Error and Average Omitted Variance

Let the ordered spectrum be $6,3,2,1$, and retain only the first component. Calculate the expected squared residual and the average variance per discarded direction.

There are $p-K=4-1=3$ discarded directions. Their eigenvalues are $3,2,1$, so the expected squared residual is

$E[\|R_1\|^2]=3+2+1=6.$

Because $K<p$, there are discarded directions over which to average. Their average variance is $6/3=2$. This average is not the total residual error: the total is $6$, while $2$ is the average contribution per omitted direction. The calculation also checks against the total variance $6+3+2+1=12$: the first component captures $6$, leaving $12-6=6$.

Worked Example: Retaining Every Direction

A three-dimensional population has eigenvalues $5,2,0$. What is the expected squared reconstruction error when $K=3$? Is an average variance per discarded direction defined?

Since $K=p=3$, every principal direction is retained. The discarded-eigenvalue sum is empty:

$E[\|R_3\|^2]=\sum_{j=4}^{3}\lambda_j=0.$

Equivalently, the reconstruction uses the complete orthonormal basis, so $R_3=0$ for every realization of $X$. The zero eigenvalue does not change this conclusion: it represents a direction with no population variation, and it is retained along with the other directions. Here $p-K=0$, so there are no discarded directions and no average variance per discarded direction. Dividing the error by $p-K$ would be undefined and would not produce a meaningful average.

Interpreting the Error Carefully

The scalar $E[\|R_K\|^2]$ is an aggregate error across the population distribution. It is the expected squared Euclidean distance from a random observation to its rank-$K$ reconstruction. It is not the error for every individual observation, and it is not the square root of that error. If a root-mean-square distance is desired, it is $\sqrt{E[\|R_K\|^2]}$, which is a different summary.

The residual covariance contains more detail than its trace. Specifically, $\operatorname{Cov}(R_K)$ identifies which directions contain the omitted variation, while its trace $E[\|R_K\|^2]$ adds that variation across all coordinates. Since the eigenvalues are ordered, the first $K$ directions capture the largest individual variance contributions, and the tail sum quantifies what remains after this truncation.

When $K<p$, dividing the total residual error by $p-K$ gives the average variance per discarded principal direction. It does not give the residual error itself. At $K=p$, this average is not defined: there are zero discarded directions. The total error remains well-defined and equals zero.

Common Pitfall

Using retained eigenvalues as the residual error. The first $K$ eigenvalues quantify variance captured by the reconstruction. Residual error is the sum of the eigenvalues with indices $K+1$ through $p$.
Calling the average discarded variance the total error. The sum measures aggregate squared error; only when $K<p$ can that sum be divided by $p-K$ to describe the average per discarded direction.
Dividing by zero when all directions are retained. For $K=p$, the residual is exactly zero, but there is no average per discarded direction because none are discarded.
Forgetting that the error is squared. The eigenvalues and their sum have units of squared measurement units. A distance-scale summary requires taking a square root after calculating the expected squared error.

Key takeaway: For a centered population vector with covariance eigenvalues ordered from largest to smallest, retaining the first $K$ principal directions leaves expected squared reconstruction error $\sum_{j=K+1}^{p}\lambda_j$. This sum is zero when $K=p$; an average per omitted direction is defined only when $K<p$.

Check Your Understanding

Use the eigenvalue-tail rule and distinguish total error from average omitted variance.

A covariance spectrum is $8,3,2,1$. What is the expected squared residual error for $K=2$?
For the same spectrum and $K=2$, what is the average variance per discarded direction, and why does it differ from the total error?
Explain why the expected squared residual is zero when $K=p$, even if one of the eigenvalues is zero.
If the total variance is $20$ and the first three eigenvalues sum to $17$, what is the expected squared error after retaining three components?
Is the average variance per discarded direction defined when $K=p$? Explain using the number of discarded directions.