Spectral Filters Under Correlated Noise

Why the Noise Basis Matters

Spectral regularization is often analyzed by expressing the data in the left singular-vector basis of the forward operator. With uncorrelated noise in that basis, each coefficient can be studied separately. But correlation in the original observation coordinates need not disappear under this transformation: the transformed covariance can have off-diagonal entries. The filter still acts mode by mode, while the noise can couple the errors across modes.

This distinction matters for risk calculations and for choosing a useful coordinate system. We first derive the estimator’s bias and covariance in the ordinary singular-vector basis. We then show how whitening, followed by a singular-value decomposition, gives coordinates with identity noise covariance. A generalized eigenbasis expresses the same construction directly in terms of the forward operator and the noise covariance.

Model and Spectral Filter

Work in finite-dimensional real spaces. Let $A$ be an $m$-by-$p$ matrix with full column rank, and suppose the data satisfy $y=Ax+\varepsilon$, where $x\in\mathbb{R}^p$ is fixed. Assume $E[\varepsilon]=0$ and $\operatorname{Cov}(\varepsilon)=\Sigma$, with $\Sigma$ positive semidefinite. No Gaussian assumption is needed for the mean and covariance calculations below.

Write a compact singular-value decomposition $A=U S V^\mathsf{T}$, where $U^\mathsf{T}U=I_p$, $V^\mathsf{T}V=I_p$, and $S=\operatorname{diag}(s_1,\ldots,s_p)$ with $s_i>0$. A spectral filter specified by a function $g$ estimates $x$ as $g(A^\mathsf{T}A)A^\mathsf{T}y$. Define $h_i=s_i^2g(s_i^2)$, the filter factor for mode $i$. Equivalently, the estimate is

\hat{x}=V\operatorname{diag}\!\left(\frac{h_i}{s_i}\right)U^\mathsf{T}y.

The filter factors determine how much of each signal mode is retained. For example, ridge regularization with parameter $\alpha>0$ has $g(\lambda)=1/(\lambda+\alpha)$, so $h_i=s_i^2/(s_i^2+\alpha)$. Let $\theta=V^\mathsf{T}x$ be the signal coordinates and $\eta=U^\mathsf{T}\varepsilon$ the noise coordinates. Their noise covariance is

C=\operatorname{Cov}(\eta)=U^\mathsf{T}\Sigma U.

Even if $\Sigma$ is diagonal in the observation coordinates, $C$ need not be diagonal. It is $C$, rather than $\Sigma$ alone, that describes the noise dependence among the singular modes used by this estimator.

Definition: For the fixed-signal model above, the bias is $E[\hat{x}]-x$, the covariance of the estimator is $\operatorname{Cov}(\hat{x})$, and its mean-squared error under squared Euclidean loss is $E[\|\hat{x}-x\|^2]$. The mean-squared error is the sum of squared bias and the trace of the estimator covariance.

Worked Example: Ridge Risk with Correlated Singular Coordinates

Suppose $A=\operatorname{diag}(2,1)$, $x=(1,-2)^\mathsf{T}$, and the noise covariance in the left singular-vector basis is $C=\begin{pmatrix}1&0.6\\0.6&4\end{pmatrix}$. Use ridge regularization with $\alpha=1$ to calculate the bias, variance, and mean-squared error.

Here $s_1=2$ and $s_2=1$, so the ridge filter factors are $h_1=4/(4+1)=0.8$ and $h_2=1/(1+1)=0.5$. The error in signal coordinates is $(h_i-1)\theta_i+(h_i/s_i)\eta_i$. Thus the bias coordinates are $(-0.2)(1)=-0.2$ and $(-0.5)(-2)=1$. The squared bias is $(-0.2)^2+1^2=1.04$.

The noise multipliers are $h_1/s_1=0.4$ and $h_2/s_2=0.5$. The estimator covariance in these coordinates is therefore

\begin{pmatrix}0.4&0\\0&0.5\end{pmatrix} \begin{pmatrix}1&0.6\\0.6&4\end{pmatrix} \begin{pmatrix}0.4&0\\0&0.5\end{pmatrix} = \begin{pmatrix}0.16&0.12\\0.12&1\end{pmatrix}.

Its trace, the variance under squared Euclidean loss, is $0.16+1=1.16$. The mean-squared error is $1.04+1.16=2.20$. The covariance between the two estimation errors is $0.12$, but it does not enter this particular scalar risk: the squared Euclidean norm sums the two squared coordinate errors, without a cross-product between them.

Risk Decomposition in the Singular-Vector Basis

Let $H=\operatorname{diag}(h_1,\ldots,h_p)$ and $D=\operatorname{diag}(h_1/s_1,\ldots,h_p/s_p)$. The coefficient estimate is $V^\mathsf{T}\hat{x}=H\theta+D\eta$. Since $V$ is orthogonal, transforming the error into these coefficient coordinates preserves its Euclidean norm.

Theorem: Under the stated model, $E[\hat{x}]-x=V(H-I)\theta$, and $\operatorname{Cov}(\hat{x})=VDCDV^\mathsf{T}$. Consequently, $$ E[\|\hat{x}-x\|^2] = \|(H-I)\theta\|^2+\operatorname{tr}(DCD) = \sum_{i=1}^p(1-h_i)^2\theta_i^2 +\sum_{i=1}^p\frac{h_i^2}{s_i^2}C_{ii}. $$

Proof. Substituting $y=Ax+\varepsilon$ into the estimator gives $\hat{x}=V(H\theta+D\eta)$, because $U^\mathsf{T}Ax=SV^\mathsf{T}x=S\theta$. Taking expectations and using $E[\eta]=0$ gives the stated bias. Since $\operatorname{Cov}(\eta)=C$, the covariance of $D\eta$ is $DCD$; transforming back by $V$ gives $VDCDV^\mathsf{T}$.

For the risk, write the coefficient error as $(H-I)\theta+D\eta$. Its expectation is $(H-I)\theta$, and the random part has mean zero. Expanding the squared norm, the cross term has expectation zero. Therefore the risk is the squared norm of the mean plus $E[\|D\eta\|^2]$. For any random vector $Z$, $E[\|Z\|^2]=\operatorname{tr}(\operatorname{Cov}(Z))+\|E[Z]\|^2$, as follows by expanding each squared coordinate. Hence the variance term is $\operatorname{tr}(DCD)$. Its $i$th diagonal entry is $(h_i/s_i)^2C_{ii}$, which gives the displayed sum. $\square$

This result separates two consequences of correlated noise. Under squared Euclidean loss, only the marginal variances $C_{ii}$ contribute to the scalar variance term. The off-diagonal elements do, however, determine covariances between estimated modes. They matter for other quadratic losses: if the error is measured by $W$, a symmetric positive semidefinite matrix, the variance contribution is $\operatorname{tr}(WVDCDV^\mathsf{T})$, which can depend on off-diagonal entries of $C$.

Worked Example: When Off-Diagonal Covariance Enters the Loss

Use the estimator covariance $K=\begin{pmatrix}0.16&0.12\\0.12&1\end{pmatrix}$ from the preceding example, but measure error using the quadratic loss $e^\mathsf{T}We$, where $W=\begin{pmatrix}1&1\\1&1\end{pmatrix}$. This matrix is positive semidefinite because $e^\mathsf{T}We=(e_1+e_2)^2$.

The variance contribution to this loss is $\operatorname{tr}(WK)$. Multiplication gives $WK=\begin{pmatrix}0.28&1.12\\0.28&1.12\end{pmatrix}$, so $\operatorname{tr}(WK)=0.28+1.12=1.40$. Equivalently, the variance of the sum of the two errors is $0.16+1+2(0.12)=1.40$. If the off-diagonal covariance were incorrectly discarded, the answer would be $0.16+1=1.16$, which is not the variance under this loss.

Whitening and the Generalized Eigenbasis

When $\Sigma$ is positive definite, define the whitened observations $z=\Sigma^{-1/2}y$ and the whitened operator $B=\Sigma^{-1/2}A$, where $\Sigma^{-1/2}$ is the symmetric inverse square root. The whitened noise $\xi=\Sigma^{-1/2}\varepsilon$ has mean zero and covariance $I_m$. If $B=Q T R^\mathsf{T}$ is a compact singular-value decomposition, with positive singular values $t_i$, then $Q^\mathsf{T}\xi$ has covariance $I_p$. A filter applied to $B$ therefore has uncorrelated noise coordinates in this basis.

Theorem: Suppose $\Sigma$ is positive definite and $A$ has full column rank. The estimator $$ \hat{x}_{w} = R\operatorname{diag}\!\left(\frac{k_i}{t_i}\right)Q^\mathsf{T}\Sigma^{-1/2}y $$ has coefficient error $R^\mathsf{T}(\hat{x}_{w}-x)=(K-I)R^\mathsf{T}x+T_k\zeta$, where $K=\operatorname{diag}(k_i)$, $T_k=\operatorname{diag}(k_i/t_i)$, and $\zeta=Q^\mathsf{T}\Sigma^{-1/2}\varepsilon$ has covariance $I_p$. Thus its squared Euclidean risk is $$ \sum_{i=1}^p(1-k_i)^2(R^\mathsf{T}x)_i^2+\sum_{i=1}^p\frac{k_i^2}{t_i^2}. $$

Proof. Since $B R=QT$, we have $Q^\mathsf{T}Bx=TR^\mathsf{T}x$. Also, $z=Bx+\xi$, so $Q^\mathsf{T}z=TR^\mathsf{T}x+\zeta$. Multiplication by $\operatorname{diag}(k_i/t_i)$ gives the stated coefficient error. Finally, $\operatorname{Cov}(\zeta)=Q^\mathsf{T}\Sigma^{-1/2}\Sigma\Sigma^{-1/2}Q=I_p$. The bias coordinates are $(k_i-1)(R^\mathsf{T}x)_i$, and the noise variances are $k_i^2/t_i^2$. Summing their squared-bias and variance contributions proves the risk expression. $\square$

The construction also has a generalized eigenvector interpretation. Let $q_i$ be column $i$ of $Q$, and set $u_i=\Sigma^{-1/2}q_i$. Then $u_i^\mathsf{T}\Sigma u_j=q_i^\mathsf{T}q_j$, so these vectors are orthonormal in the inner product defined by $\Sigma$. Because $AA^\mathsf{T}u_i=t_i^2\Sigma u_i$, they solve the generalized eigenproblem for the pair $(AA^\mathsf{T},\Sigma)$. Moreover, $u_i^\mathsf{T}\varepsilon$ are uncorrelated with variance one, and $u_i^\mathsf{T}A r_j=t_i$ when $r_j$ is column $j$ of $R$ and $i=j$, and zero otherwise. Thus the generalized basis diagonalizes the signal modes while standardizing their noise coordinates.

Whitening changes the geometry of the data fit: discrepancies are measured after multiplication by $\Sigma^{-1/2}$, rather than in the original unweighted observation norm. It is appropriate when the noise covariance is known and positive definite, or when a justified estimate of it is available. If $\Sigma$ is singular, whitening on its range requires separate treatment of zero-variance directions; the positive-definite result cannot simply be applied with an ordinary inverse.

Worked Example: Whitening a Diagonal Noise Model

Let $A=\operatorname{diag}(4,1)$, $\Sigma=\operatorname{diag}(4,1)$, and use ridge filtering with $\alpha=1$ after whitening. The whitened operator is $B=\Sigma^{-1/2}A=\operatorname{diag}(2,1)$, and its noise covariance is the identity.

The whitened singular values are $t_1=2$ and $t_2=1$. Therefore the ridge factors are $k_1=4/(4+1)=0.8$ and $k_2=1/(1+1)=0.5$. The estimator’s noise multipliers in the whitened singular coordinates are $k_1/t_1=0.4$ and $k_2/t_2=0.5$. Since each noise coordinate has variance one, the two contributions to variance are $0.4^2=0.16$ and $0.5^2=0.25$. The total variance is $0.41$. Any fixed signal $x$ adds squared bias $0.2^2x_1^2+0.5^2x_2^2$, using the right singular vectors of $B$, which here are the coordinate axes.

Common Pitfall and Practical Implications

A frequent error is to assume that a diagonal spectral filter makes the entire estimation problem coordinatewise independent. The filter is diagonal in its chosen basis, but the transformed noise covariance may not be. The correction is to calculate $U^\mathsf{T}\Sigma U$ before making claims about mode independence. The trace-risk formula happens to involve only its diagonal entries under squared Euclidean loss; that does not make the off-diagonal covariances vanish.

Whitening provides a principled route to coordinatewise bias-variance analysis when the covariance is positive definite: transform both the observations and the forward operator, then decompose the transformed operator. The resulting singular values and filter factors need not match those from the unwhitened problem. Consequently, risk comparisons must specify both the estimator and the loss being used. Whitening restores uncorrelated noise coordinates, not a universal equivalence between differently weighted estimation procedures.

Key takeaway: In the ordinary singular basis, compute the transformed covariance $C=U^\mathsf{T}\Sigma U$. For Euclidean risk, the filter-mode variances use $C_{ii}$, while off-diagonal entries describe cross-mode dependence and can affect other losses. Whitening and the associated generalized eigenbasis yield identity noise covariance when $\Sigma$ is positive definite.

Check Your Understanding

Use the model and notation developed above to answer these questions.

Why can a diagonal covariance matrix in observation coordinates become non-diagonal in the left singular-vector basis?
For squared Euclidean loss, which entries of $C$ contribute to the variance term, and why?
How does the variance contribution change for a quadratic loss $e^\mathsf{T}We$ that is not diagonal in the singular-vector basis?
State the covariance of the whitened noise and the additional decomposition used to obtain uncorrelated spectral coordinates.
What generalized eigenproblem do the whitened left singular vectors solve after transforming back to the original observation coordinates?