Trace Variance of a Spectral Filter

Why Trace Variance Counts Singular Directions

A spectral estimator treats different singular directions of the forward operator differently. Its total variance in the unknown parameter space is therefore obtained by adding the variance contributed along each right singular vector. The trace of the estimator’s covariance matrix is exactly this total: it measures expected squared deviation from the estimator’s mean, in the parameter-space norm.

This tutorial works in finite dimensions, where the observation noise and all traces are ordinary random vectors and matrix traces. The same sum formula applies to a countable sequence of singular directions when the sum of the variance contributions is finite. Throughout, noise is assumed to have mean zero and covariance \(\sigma^2 I\); Gaussianity is not needed.

The equality in this definition follows by writing the squared norm as the sum of squared coordinates in an orthonormal basis. The total variance is not the mean squared error: the latter also includes squared bias. The bias–variance decomposition separates these contributions.

The Spectral Sum Formula

Let \(A\) be an \(m\)-by-\(d\) matrix of rank \(r\). Write its singular value decomposition on its nonzero singular directions as \(A v_j=s_j u_j\), where \(s_j>0\), \(u_1,\ldots,u_r\) are orthonormal observation-space vectors, and \(v_1,\ldots,v_r\) are orthonormal parameter-space vectors. Consider observations \(y=Ax^\dagger+\varepsilon\), where \(E[\varepsilon]=0\) and \(\operatorname{Cov}(\varepsilon)=\sigma^2 I_m\).

A spectral estimator with scalar filter \(g_\lambda\) is

Here \(s_jg_\lambda(s_j^2)\) is the response applied to the observed coefficient in direction \(u_j\). To isolate the noise contribution, let \(\bar{x}_\lambda=E[\hat{x}_\lambda]\). Then

Proof. Because the \(v_j\) are orthonormal, the squared norm of the centered estimator is the sum of the squared coefficients:

The covariance assumption gives \(E[\langle\varepsilon,u_j\rangle^2]=u_j^\top(\sigma^2 I_m)u_j=\sigma^2\), since \(\|u_j\|=1\). Taking expectations in the finite sum yields

For any random vector \(Z\) with finite second moments, expanding its squared norm in an orthonormal basis gives \(E[\|Z-E[Z]\|^2]=\sum_k\operatorname{Var}(Z_k)=\operatorname{tr}(\operatorname{Cov}(Z))\). Apply this to \(Z=\hat{x}_\lambda\) to obtain the trace identity. \(\square\)

The formula shows where amplification enters: a direction contributes in proportion to \(s_j^2g_\lambda(s_j^2)^2\). It is often useful to define the response \(q_\lambda(t)=t g_\lambda(t)\). The estimator’s coefficient multiplying \(\langle y,u_j\rangle\) is then \(q_\lambda(s_j^2)/s_j\), so the same variance can be written as

Truncated SVD Variance

For truncated singular value decomposition (TSVD), a direction is retained if its squared singular value meets the chosen threshold \(\tau>0\), and discarded otherwise. In filter notation, \(g_\tau(t)=1/t\) when \(t\geq\tau\) and \(g_\tau(t)=0\) when \(t<\tau\). Consequently, \(q_\tau(t)=1\) on retained directions and \(0\) on discarded directions.

This follows directly from the spectral sum formula: a retained term is \(\sigma^2s_j^2(1/s_j^2)^2=\sigma^2/s_j^2\), and a discarded term is zero. Thus each retained direction contributes more variance when its singular value is smaller. The threshold controls which terms appear, while the inverse squared singular values determine their sizes.

Tikhonov Variance

For Tikhonov regularization with parameter \(\lambda>0\), the filter is \(g_\lambda(t)=1/(t+\lambda)\). The estimator’s response in a direction with squared singular value \(t\) is \(q_\lambda(t)=t/(t+\lambda)\). Substitution in the trace formula gives the variance functional.

For a fixed finite spectrum, this variance is nonincreasing as \(\lambda\) increases. More precisely, it is strictly decreasing when at least one singular direction is present. To verify this, differentiate the finite sum:

Every summand in the derivative is negative because \(\sigma^2>0\), \(s_j^2>0\), and \(s_j^2+\lambda>0\). If \(\sigma^2=0\) or \(r=0\), the variance is identically zero instead. This establishes how Tikhonov’s parameter changes the variance functional; it does not, by itself, determine the total estimation error, which also depends on bias.

What the Trace Does and Does Not Measure

The trace here measures variability of the estimate in parameter space. If the quantity of interest is instead the fitted observation \(A\hat{x}_\lambda\), its noise coefficient in direction \(u_j\) is \(q_\lambda(s_j^2)\), not \(q_\lambda(s_j^2)/s_j\). Its total variance under the same model is therefore

This different expression can be useful, but it answers a different question. In parameter space, recovering a coefficient from a small singular value requires division by that singular value, which can amplify noise. In fitted-data space, applying \(A\) multiplies that coefficient by the singular value again.

For a general noise covariance matrix \(\Sigma\), the coordinates \(\langle\varepsilon,u_j\rangle\) need not have equal variances or be uncorrelated. The isotropic-noise sum above then does not apply unchanged: the covariance must account for \(\langle u_j,\Sigma u_k\rangle\), including cross-direction terms. The simple sum formula is justified here by \(\operatorname{Cov}(\varepsilon)=\sigma^2I_m\).

Common Pitfall

• Wrong reasoning: “The trace variance is \(\sigma^2\sum_j q_\lambda(s_j^2)^2\), because \(q_\lambda\) is the filter response.” That expression is the variance of the fitted observations, not the parameter estimate.
• Correction: For parameter variance, the coefficient multiplying the observed noise is \(s_jg_\lambda(s_j^2)=q_\lambda(s_j^2)/s_j\). Squaring it gives the contribution \(\sigma^2q_\lambda(s_j^2)^2/s_j^2\). In particular, TSVD contributes \(\sigma^2/s_j^2\) for each retained parameter direction.