Why PCA Reconstruction Error Matters
PCA is often used to represent a data set with fewer dimensions. Keeping \(k\) principal components means replacing each centered observation by its projection onto a \(k\)-dimensional subspace. The resulting reconstruction is generally not exact: the difference between the original and reconstructed data is the reconstruction residual. The sum of squared residuals measures how much variation is lost by using only \(k\) components.
A central result is that the first \(k\) principal components do more than capture the largest possible variance among \(k\)-dimensional projections: they also produce the smallest possible squared reconstruction error among rank-\(k\) linear approximations. This gives a precise reason for using the leading components when a low-dimensional approximation is the objective.
The reconstruction is the projection of each row of \(X\) onto the subspace spanned by the retained directions. It is a rank-at-most-\(k\) matrix. The results below concern squared error in the coordinates of \(X\): if \(X\) contains centered raw measurements, the error is in their original measurement units; if \(X\) contains standardized measurements, the error is in standardized units.
From Eigenvalues to Reconstruction Error
Let the eigenvalues of \(S\) be \(\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0\). With the sample covariance convention \(S=X^\mathsf{T}X/(n-1)\), the total squared error from retaining the first \(k\) components is
Thus the discarded eigenvalues give the residual error, but the covariance divisor matters: with sample covariance, multiply their sum by \(n-1\) to obtain the total squared error across the \(n\) observations. Dividing that total by \(n\) gives mean squared error per observation, where each observation’s squared error is summed over its \(p\) variables. Dividing by \(np\) instead gives mean squared error per matrix entry.
Why the Leading Directions Minimize the Error
A direct argument first considers approximations formed by orthogonal projection onto a \(k\)-dimensional subspace. Let \(U\) be a \(p \times k\) matrix with orthonormal columns spanning that subspace. The projection of the data is \(XUU^\mathsf{T}\). The squared residual is the total squared size of \(X\) minus the amount captured by the projection:
The first term is fixed, so minimizing residual error is equivalent to maximizing \(\operatorname{tr}(U^\mathsf{T}X^\mathsf{T}XU)\), the squared size of the projected data. Write \(X^\mathsf{T}X\) in an orthonormal eigenbasis with eigenvalues \((n-1)\lambda_1,\ldots,(n-1)\lambda_p\). The maximum captured amount for \(k\) orthonormal directions is the sum of the \(k\) largest eigenvalues of \(X^\mathsf{T}X\). It is attained by choosing those eigenvectors. This is the variational characterization of the leading eigenspace.
The conclusion also applies to arbitrary rank-at-most-\(k\) matrix approximations, not only to a projection chosen in advance. For any proposed approximation, its columns lie in some subspace of dimension at most \(k\). Among matrices whose columns lie in that fixed subspace, orthogonal projection gives the smallest squared distance from \(X\). Optimizing over the subspace therefore gives the same leading PCA directions. Equivalently, the singular value decomposition of \(X\) gives the same result: retaining its first \(k\) singular values and directions is the best rank-\(k\) approximation in squared Frobenius error.
Worked Example: Calculating the Error From a Centered Data Matrix
A hypothetical data set has four observations and three centered variables, with the following data matrix. Find the rank-two PCA reconstruction error, verify it from the residuals, and state the mean squared error per observation.
Method. Compute the sample covariance eigenvalues and sum the discarded eigenvalues, multiplying by \(n-1\). This uses the covariance convention \(S=X^\mathsf{T}X/(n-1)\).
Each column sums to zero, so the matrix is centered. The columns are mutually orthogonal. Their squared lengths are \(16\), \(4\), and \(1\), respectively; for example, the third is \(0.5^2+(-0.5)^2+(-0.5)^2+0.5^2=1\). Therefore
For \(k=2\), only the third direction is discarded. The total squared reconstruction error is
As a check, the first two principal directions are the first and second coordinate axes. The rank-two reconstruction therefore sets the third column to zero. Its residuals are \(0.5,-0.5,-0.5,0.5\), whose squared values sum to \(0.25+0.25+0.25+0.25=1\). The mean squared error per observation is \(1/4=0.25\), summing squared error over the three variables for each observation.
Common mistake avoided: The discarded eigenvalue is \(1/3\), not the total squared error. It is a sample variance; multiplying by \(n-1=3\) gives the sum of squared residuals, which the direct check confirms.
Working With Eigenvalues and Error Summaries
Eigenvalues can answer several related questions, but the requested error summary must be identified before calculating. The discarded eigenvalue sum is residual sample variance on the PCA coordinate scale. Multiplying it by \(n-1\) gives total squared reconstruction error. The following table summarizes the distinctions for centered data using sample covariance.
| Quantity | Calculation | Interpretation |
|---|---|---|
| Discarded sample variance | \(\sum_{j=k+1}^{p}\lambda_j\) | Residual variance summed across variables |
| Total squared error | \((n-1)\sum_{j=k+1}^{p}\lambda_j\) | Squared residuals summed over observations and variables |
| Mean squared error per observation | \(\frac{n-1}{n}\sum_{j=k+1}^{p}\lambda_j\) | Average, over observations, of each observation’s summed squared residual |
| Mean squared error per entry | \(\frac{n-1}{np}\sum_{j=k+1}^{p}\lambda_j\) | Average squared residual for one observation-variable entry |
Worked Example: Comparing Two Retention Choices
A hypothetical centered matrix has \(n=6\) observations and sample covariance eigenvalues \(9,4,1\). Compare the total squared reconstruction error for \(k=1\) and \(k=2\), then calculate the corresponding mean squared errors per matrix entry. Do not round until the final step; report the final mean squared errors to four decimal places.
Method. Apply the discarded-eigenvalue formula with \(n-1=5\). There are \(np=6(3)=18\) entries.
With \(k=1\), the discarded eigenvalues sum to \(4+1=5\). Hence the total squared error is \(5(5)=25\), and the mean squared error per entry is \(25/18=1.3889\) to four decimal places.
With \(k=2\), only the eigenvalue \(1\) is discarded. The total squared error is \(5(1)=5\), and the mean squared error per entry is \(5/18=0.2778\) to four decimal places.
The reduction in total squared error is \(25-5=20\), which equals \(5(4)\), the factor \(n-1\) times the eigenvalue \(4\) newly retained. The result is consistent: retaining an additional component cannot increase the optimal reconstruction error.
Common mistake avoided: The number of entries, \(18\), is used only when calculating mean squared error per entry. It does not replace \(n-1=5\) in the total squared error formula.
Scaling Changes the Meaning of the Error
The eigenvalue formula measures squared error in the coordinates on which PCA was performed. If variables were standardized before PCA, the residual error computed from the eigenvalues is on the standardized scale. It is not automatically the squared error in the original units.
Suppose the standardized variable \(z_j\) was formed from an original variable \(x_j\) using \(z_j=(x_j-\bar{x}_j)/s_j\), where \(\bar{x}_j\) and \(s_j\) are its training mean and standard deviation. If \(e_{ij}\) is the standardized residual for observation \(i\) and variable \(j\), the corresponding original-unit residual is \(s_je_{ij}\). Thus original-unit total squared error is \(\sum_j s_j^2\sum_i e_{ij}^2\). The discarded eigenvalue sum alone gives the unweighted standardized error, not this scale-weighted original-unit error.
Worked Example: Translating Standardized Error to Original Units
A hypothetical four-observation, two-variable data set is standardized before PCA. The standardized variables have sample standard deviations \(1\), and their sample correlation is \(0.6\). The original variables have standard deviations \(s_1=10\) units and \(s_2=2\) units. Retain one component. Find the standardized total squared error and the original-unit total squared error.
Method. The standardized covariance matrix is the correlation matrix. Its eigenvalues are \(1+0.6=1.6\) and \(1-0.6=0.4\), with the discarded direction \(q_2=(1,-1)^\mathsf{T}/\sqrt{2}\). Since \(n-1=3\), the squared length of the discarded standardized score vector \(t\) is
The standardized residual matrix has the form \(E_z=tq_2^\mathsf{T}\). Its first residual column is \(t/\sqrt{2}\), and its second is \(-t/\sqrt{2}\). Therefore the total squared standardized error is
To convert to original units, multiply the first column’s squared residual sum by \(10^2\) and the second column’s by \(2^2\). Each standardized column has squared residual sum \(1.2/2=0.6\). Thus
The two error totals differ because a residual of one standardized unit corresponds to different distances in the two original measurement scales. The calculation uses the stated scales and the residual direction; the discarded eigenvalue alone does not determine original-unit error after standardization.
Common mistake avoided: Reporting \(1.2\) as the original-unit squared error would confuse the standardized PCA coordinates with the original variables. The value \(1.2\) is correct on the standardized scale; \(62.4\) is the original-unit total for this residual direction and these scales.
Interpretation and Common Traps
The eigenvalue result is exact for squared Euclidean reconstruction error on the data matrix used to fit PCA. It does not say that a low-rank reconstruction preserves every feature of the observations, nor that it is optimal for a different objective such as prediction. It says that, among rank-at-most-\(k\) linear approximations, the leading principal directions minimize this particular sum of squared residuals.
The data must be centered if the intended approximation is around the data mean. When the original observations are not centered, the usual PCA reconstruction first subtracts the fitted column means, projects the centered observations, and then adds the means back. This is an affine approximation in the original coordinates. If centering is omitted, the origin is treated as a special reference point and the usual covariance-eigenvalue interpretation does not apply as stated.
For \(k=0\), the centered reconstruction is the zero matrix and the total squared error is \((n-1)\sum_{j=1}^{p}\lambda_j\). For \(k\) at least the rank of \(X\), all nonzero directions are retained and the residual error is zero. If eigenvalues tie at a retention boundary, different choices of directions within the tied eigenspace can give different reconstructions, but they have the same optimal squared error because the tied eigenvalues contribute equally.
- Do not report an eigenvalue as total squared error. With sample covariance, multiply the discarded eigenvalue sum by \(n-1\).
- Do not confuse total error with an average. Divide by \(n\) for mean summed squared error per observation, or by \(np\) for mean squared error per entry.
- Do not mix scales. Eigenvalues from standardized data quantify error on the standardized scale unless a separate conversion to original units is made.
- Do not claim a different objective. Minimum squared reconstruction error is not, by itself, proof of best predictive performance.
These distinctions make the PCA result useful in practice: the discarded eigenvalues quantify what the chosen low-rank representation leaves behind, while the centering, scaling, covariance convention, and requested error summary determine how to interpret the number.
Check Your Understanding
For each problem, use the stated covariance convention and distinguish total squared error from other summaries.
- A centered matrix has \(n=8\) observations and sample covariance eigenvalues \(6,3,1\). If \(k=2\), calculate the total squared reconstruction error. (Answer: \(7\))
- A centered matrix has \(n=11\) observations and sample covariance eigenvalues \(4,2,1\). For \(k=1\), calculate the total squared error and the mean squared error per observation. (Answer: total \(30\); mean per observation \(30/11 \approx 2.7273\))
- A standardized data matrix has \(n=9\) observations, and the discarded eigenvalues sum to \(0.6\). Calculate the total squared error on the standardized scale. (Answer: \(8(0.6)=4.8\))
- For a two-variable standardized PCA reconstruction, the only residual direction is \((1,-1)^\mathsf{T}/\sqrt{2}\), the discarded eigenvalue is \(0.25\), and \(n=5\). The original variable standard deviations are \(3\) and \(1\). Calculate the original-unit total squared error. (Answer: standardized residual sum \(=4(0.25)=1\); original-unit error \(=\frac{3^2+1^2}{2}(1)=5\))
- A centered matrix has \(n=6\) and eigenvalues \(5,3,3,1\). For \(k=2\), calculate the total squared error and explain whether choosing either tied direction as the second component changes it. (Answer: \(5(3+1)=20\); the choice within the tied eigenspace does not change the optimal error.)