Choosing Truncation by Reconstruction Error

Why Truncation Is a Choice

Principal component analysis orders directions by the variance they capture. Keeping only the first \(K\) directions compresses the representation, but necessarily leaves some variation unreconstructed. Choosing \(K\) is therefore a decision about acceptable loss: a smaller \(K\) is more compact, while a larger \(K\) preserves more of the original variation.

The eigenvalues make this tradeoff measurable. Their sum is the total variance, and the sum of the eigenvalues omitted after the first \(K\) is the squared reconstruction error for the corresponding principal subspace. This gives a direct way to compare candidate values of \(K\), rather than choosing by visual impression alone.

Reconstruction Error and Explained Variance

Let \(X\) be a centered \(p\)-dimensional random vector with finite second moments and covariance matrix \(\Sigma\). Write its eigenvalues in nonincreasing order, \(\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\geq0\), with corresponding orthonormal eigenvectors \(v_1,\ldots,v_p\). For \(K\) between 0 and \(p\), the leading principal subspace is spanned by \(v_1,\ldots,v_K\). The orthogonal projection onto that subspace is denoted by \(P_K\). For \(K=0\), it is the zero projection; for \(K=p\), it is the identity.

The error is measured in squared units of the variables being analyzed. For example, when variables are measured in centimetres, a squared reconstruction error is expressed in squared centimetres under the relevant Euclidean metric. The explained-variance proportion has no units. It is also the fraction of total variance not left in the residual: the two quantities sum to one after the error is divided by total variance.

For sample PCA, the same comparison uses the eigenvalues of the chosen sample covariance matrix. The divisor used in that covariance convention scales all eigenvalues by the same factor, so it changes the error's numerical scale but not cumulative explained-variance proportions or comparisons based on those proportions. Use one covariance convention consistently when applying an absolute error tolerance.

Why the Leading Components Minimize Error

The discarded-eigenvalue formula is useful because it is not merely an error calculation for one convenient subspace. Among all \(K\)-dimensional linear subspaces, the leading principal subspace has the smallest expected squared reconstruction error.

Proof. Since \(P\) is an orthogonal projection, \(I-P\) is also an orthogonal projection. Using \(E[XX^\mathsf{T}]=\Sigma\), the expected squared residual is

For each eigenvector define \(a_j=v_j^\mathsf{T}Pv_j=\|Pv_j\|^2\). Because \(P\) is an orthogonal projection, \(0\leq a_j\leq1\), and \(\sum_{j=1}^{p}a_j=\operatorname{tr}(P)=K\). In the eigenvector basis, \(\operatorname{tr}(P\Sigma)=\sum_{j=1}^{p}\lambda_j a_j\). We show this sum cannot exceed \(\sum_{j=1}^{K}\lambda_j\). For \(0<K<p\), let \(t=\sum_{j=1}^{K}(1-a_j)\). The identity \(\sum_j a_j=K\) implies \(t=\sum_{j=K+1}^{p}a_j\). Ordering the eigenvalues gives

Therefore \(\operatorname{tr}(P\Sigma)\leq\sum_{j=1}^{K}\lambda_j\), so the residual error is at least \(\operatorname{tr}(\Sigma)-\sum_{j=1}^{K}\lambda_j=\sum_{j=K+1}^{p}\lambda_j\). Projection onto the leading eigenvectors attains equality, because it captures exactly the first \(K\) eigenvalues. For \(K=0\) the only subspace is the zero subspace; for \(K=p\) the full space has zero error. This also covers the endpoints and completes the proof. \(\square\)

If eigenvalues tie at the cutoff, the minimizing subspace need not be unique. The minimum error remains the same: different choices within a tied eigenspace retain the same total variance.

How Error Changes as Components Are Added

Derivation. Subtracting the two tail sums cancels every term except \(\lambda_K\):

Likewise, subtracting the cumulative sums gives \(V_K-V_{K-1}=\lambda_K\). Since every eigenvalue is nonnegative, \(E_K\leq E_{K-1}\). Since the eigenvalues are ordered, the absolute reduction in error from successive components is nonincreasing. These facts describe the tradeoff precisely: an additional component never worsens squared reconstruction fidelity, but its variance contribution is no larger than that of the preceding component. \(\square\)

Choosing a Rule and Avoiding Common Mistakes

A practical choice begins by stating what “enough fidelity” means. One option is an absolute tolerance, requiring \(E_K\leq\varepsilon\) for a specified error budget \(\varepsilon\). Another is a relative tolerance, requiring \(E_K/\operatorname{tr}(\Sigma)\leq\delta\), equivalently cumulative explained variance of at least \(1-\delta\). In either case, because the error decreases with \(K\), choose the smallest \(K\) satisfying the requirement when compactness is also desired.

A third situation is a fixed component budget: choose the largest permissible \(K\) when the goal is to minimize reconstruction error subject to that budget. In practice, the acceptable target or budget depends on how the representation will be used. A variance-retention rule is a descriptive criterion, not a guarantee that the retained components preserve every feature important for a later scientific or predictive task.

• Mistake: treating a popular percentage as a theorem. A rule such as “retain 90%” is a chosen convention, not a mathematical fact that 90% is always sufficient. State the target and explain why it suits the application.
• Mistake: confusing explained variance with individual-variable accuracy. The discarded-eigenvalue sum is an expected squared error across the analyzed coordinates. It does not say that every observation, or every variable, has the same reconstruction error.
• Mistake: comparing absolute errors across different scales. A common change of units multiplies squared errors by the square of the conversion factor. Check that the units and covariance convention match before applying an absolute tolerance.
• Mistake: assuming more components can hurt reconstruction. Under the same metric and nested leading principal subspaces, adding a component reduces error by its nonnegative eigenvalue. The real tradeoff is lower compression in exchange for no worse reconstruction.