Spectral Cut-Off and Truncated SVD

Why Truncate Singular Values?

In an inverse problem, the data may contain information about a target in directions that the forward operator measures only weakly. Dividing by a small singular value attempts to recover those directions, but it also magnifies measurement noise. Spectral cut-off addresses this trade-off by inverting only singular values at or above a chosen threshold and setting the other estimated coordinates to zero.

The rule is a hard decision: each singular direction is either retained or discarded. Its bias and variance can therefore be calculated one coordinate at a time. Those calculations also make clear why a threshold that is too low can produce a noisy estimate, while one that is too high can remove substantial signal.

Model and Definition

Consider a finite-dimensional linear inverse problem $y=Ax+\varepsilon$, where $A:\mathbb{R}^d\to\mathbb{R}^m$ is fixed, $x\in\mathbb{R}^d$ is the target, and $y\in\mathbb{R}^m$ is observed. Assume the noise has mean zero and isotropic covariance:

E[\varepsilon]=0,\qquad \operatorname{Cov}(\varepsilon)=\sigma^2 I_m.

Write a singular value decomposition of $A$ using its positive singular values $s_1\geq s_2\geq\cdots\geq s_r>0$, with corresponding orthonormal left and right singular vectors $u_j$ and $v_j$. Thus $Av_j=s_j u_j$. Complete the right singular vectors to an orthonormal basis of $\mathbb{R}^d$; vectors $v_{r+1},\ldots,v_d$, if any, span the null space of $A$. Define the target and noise coordinates by $\theta_j=\langle v_j,x\rangle$ and $\varepsilon_j=\langle u_j,\varepsilon\rangle$ for $j\leq r$. Then $\langle u_j,y\rangle=s_j\theta_j+\varepsilon_j$.

Definition: For a threshold $\tau>0$, let $J_\tau=\{j\in\{1,\ldots,r\}:s_j\geq\tau\}$. The spectral cut-off estimator, also called the truncated SVD estimator, is $$ \hat x_\tau=\sum_{j\in J_\tau}\frac{\langle u_j,y\rangle}{s_j}v_j. $$ It estimates every discarded or unidentifiable right-singular-vector coordinate by zero. Equivalently, its spectral filter is $g_\tau(s)=s^{-1}$ for $s\geq\tau$ and $g_\tau(s)=0$ for $0<s<\tau$.

The convention $s_j\geq\tau$ means a singular value exactly equal to the threshold is retained. The estimator is the ordinary least-squares inverse on the span of the retained right singular vectors, with no estimate in the other directions. When $A$ has a nontrivial null space, no data-based method can identify the component of $x$ in that space without additional assumptions; this estimator sets that component to zero.

Worked Example: Constructing the Truncated Estimate

Suppose $A$ has three singular values $4,2,0.5$, and the observed coordinates $\langle u_j,y\rangle$ are $4.2,-3.7,1.8$. With threshold $\tau=1$, the first two singular values are retained and the third is discarded.

The retained coordinates are $4.2/4=1.05$ and $-3.7/2=-1.85$. Hence $\hat x_1=1.05v_1-1.85v_2+0v_3$. The third data coordinate, $1.8$, does not enter the estimate: dividing it by $0.5$ would amplify it, and every other contribution in that direction, by a factor of two.

Coordinatewise Bias, Variance, and Risk

In a retained direction, dividing the data coordinate by $s_j$ gives $\theta_j+\varepsilon_j/s_j$. In a discarded direction the estimate is zero. Since the noise is mean zero and isotropic, $E[\varepsilon_j]=0$ and $\operatorname{Var}(\varepsilon_j)=\sigma^2$. These facts give the coordinatewise properties directly.

Theorem: Under the stated model and noise assumptions, for $1\leq j\leq r$, $$ E[\langle v_j,\hat x_\tau\rangle]-\theta_j = \begin{cases} 0,&s_j\geq\tau,\\ -\theta_j,&s_j<\tau, \end{cases} \qquad \operatorname{Var}(\langle v_j,\hat x_\tau\rangle) = \begin{cases} \sigma^2/s_j^2,&s_j\geq\tau,\\ 0,&s_j<\tau. \end{cases} $$ For $j>r$, the estimator’s coordinate is zero, so its bias is $-\theta_j$ and its variance is zero. Consequently, its expected squared Euclidean error is $$ E[\|\hat x_\tau-x\|^2] = \sum_{\substack{1\leq j\leq r\\s_j<\tau}}\theta_j^2 +\sum_{j=r+1}^d\theta_j^2 +\sigma^2\sum_{j\in J_\tau}\frac{1}{s_j^2}. $$

Proof. If $s_j\geq\tau$, then $\langle v_j,\hat x_\tau\rangle=\langle u_j,y\rangle/s_j=\theta_j+\varepsilon_j/s_j$. Its expectation is $\theta_j$, and its variance is $\sigma^2/s_j^2$. If $s_j<\tau$, the estimator’s coordinate is identically zero, giving bias $-\theta_j$ and variance zero. The same zero-coordinate argument applies in the null space.

For each coordinate, the expected squared error equals its variance plus the square of its bias: $E[(Z-a)^2]=\operatorname{Var}(Z)+(E[Z]-a)^2$. The right singular vectors form an orthonormal basis, so the squared Euclidean error is the sum of the squared coordinate errors. Summing their expectations gives the displayed risk formula. No independence between noise coordinates is required for this calculation. $\square$

This result separates the two costs of thresholding. Discarding an identifiable coordinate contributes its squared signal $\theta_j^2$ as squared bias. Retaining it contributes noise variance $\sigma^2/s_j^2$. A small $s_j$ is costly to invert because that variance grows as $s_j$ shrinks.

Worked Example: Calculating Bias and Variance

Let the singular values be $4,2,0.5$, the target coordinates be $(1,-2,3)$, and the noise covariance be $(1/4)I$. For $\tau=1$, the first two directions are retained. The bias coordinates are $0,0,-3$, so the squared bias is $9$.

The variances in the retained coordinates are $\frac{1}{4}/4^2=1/64$ and $\frac{1}{4}/2^2=1/16=4/64$. The discarded coordinate has variance zero. Thus the expected squared error is $9+1/64+4/64=581/64=9.078125$. The large risk here is chiefly the cost of discarding the true third coordinate, whose squared size is $3^2=9$.

What Changing the Threshold Does

As $\tau$ increases, fewer singular values satisfy $s_j\geq\tau$. The variance term in the risk can only decrease, because retained coordinates are removed from its sum. The squared-bias term can only increase, because each newly discarded coordinate adds $\theta_j^2$. Thus increasing the threshold lowers or preserves variance while raising or preserving squared bias. The total risk need not move in only one direction.

The exact risk change from a single coordinate is especially informative. Suppose a direction with singular value $s_j>0$ is switched from discarded to retained, with all other decisions unchanged. Its contribution changes from $\theta_j^2$ to $\sigma^2/s_j^2$. Therefore its inclusion reduces expected squared error precisely when $\theta_j^2>\sigma^2/s_j^2$; exclusion reduces risk when the inequality is reversed. This comparison describes an oracle decision because it depends on the unknown target coordinate.

Result: For a fixed coordinate $j$, retaining it rather than discarding it changes expected squared error by $$ \frac{\sigma^2}{s_j^2}-\theta_j^2. $$ Thus the coordinatewise risk-minimizing decision retains the coordinate when $\theta_j^2>\sigma^2/s_j^2$, discards it when $\theta_j^2<\sigma^2/s_j^2$, and is indifferent at equality. A single threshold imposes a common ordering by singular value; it cannot in general make every coordinate’s oracle decision.

To verify the result, the risk contribution when discarded is $\theta_j^2$, from squared bias and zero variance. When retained it is $\sigma^2/s_j^2$, from zero bias and coordinate variance. Subtracting the discarded risk from the retained risk gives $\sigma^2/s_j^2-\theta_j^2$. Its sign gives the stated comparison. $\square$

Worked Example: Comparing Threshold Choices

Take singular values $(3,1,0.25)$, target coordinates $(1,1,0.1)$, and noise covariance $(1/4)I$. With threshold $\tau=0.5$, the first two directions are kept. The squared bias is $0.1^2=0.01$, and the variance is $(1/4)(1/3^2+1/1^2)=(1/4)(10/9)=5/18$. The total risk is $1/100+5/18=259/900$, approximately $0.288$.

Lower the threshold to $0.25$, retaining all three directions. The squared bias becomes zero, but the variance rises to $(1/4)(1/9+1+1/0.25^2)=(1/4)(154/9)=77/18$, approximately $4.278$. The weak third direction has only $0.01$ squared signal but adds variance $4$.

Instead, raising the threshold to $1.5$ retains only the singular value $3$. The squared bias is $1^2+0.1^2=1.01$, and the variance is $(1/4)/9=1/36$, giving risk $467/450$, approximately $1.038$. In this example, $\tau=0.5$ has lower risk than either of the other two choices. This does not make it a universally optimal threshold: the comparison depends on the target coordinates as well as the noise level and singular values.

Threshold Boundaries and Unidentifiable Directions

The hard cut-off changes discontinuously when the threshold crosses a singular value. If $s_j=\tau$, the convention used here retains coordinate $j$; a threshold even slightly above $s_j$ discards it. The estimator and its risk can therefore jump at threshold values equal to singular values. This is a property of the hard decision, not a numerical approximation to a smooth change.

A zero singular value is different from a small positive one. The corresponding right singular vector lies in the null space of $A$, so changing its coefficient does not change $Ax$ at all. It cannot be recovered by dividing a data coordinate by a singular value; there is no such data coordinate to invert. The truncated SVD sets that coefficient to zero, and its squared error includes the unobserved coefficient’s square.

Worked Example: A Singular Value Exactly at the Cut-Off

Let $A$ have singular values $2$ and $0.5$, target coordinates $(0.5,1)$, and noise covariance $(0.04)I$. Set $\tau=0.5$, so the second coordinate is retained by the $s_j\geq\tau$ convention. If the observed singular-vector data coordinates are $1.1$ and $0.4$, the estimate is $(1.1/2,0.4/0.5)=(0.55,0.8)$. Its coordinate variances are $0.04/4=0.01$ and $0.04/0.25=0.16$, with zero bias, so its total risk is $0.17$.

If the threshold is raised just above $0.5$, the second coordinate is discarded. The estimate becomes $(0.55,0)$; its second-coordinate squared bias is $1$, and its variance there is zero. The first-coordinate variance remains $0.01$, giving total risk $1.01$. The example shows both the boundary convention and how a small threshold change can trade variance for substantial bias.

Common Mistake

A tempting but incorrect argument is: “A larger singular value always means its coordinate should be retained, so choosing a threshold is only about selecting the smallest singular value.” A larger singular value does reduce the variance cost of inversion, but that alone does not determine whether retaining a coordinate improves risk. The coordinate’s signal size matters too: the comparison is between $\theta_j^2$ and $\sigma^2/s_j^2$. Moreover, a threshold chosen from the singular values alone cannot generally implement the coordinatewise risk-minimizing decisions because it does not use the unknown $\theta_j$.

The practical interpretation is more measured: the threshold controls which directions are inverted and how much noise is admitted into the estimate. Its choice balances signal discarded against noise amplified. Spectral cut-off is useful precisely because this balance is explicit, even though the best threshold depends on information that may not be available.

Check Your Understanding

Use the model and threshold convention from this tutorial.

For a retained coordinate with singular value $s_j$, derive its variance under isotropic noise with covariance $\sigma^2 I_m$.
What squared-error risk contribution does a discarded, identifiable coordinate make?
If the threshold is increased, which risk component can only decrease, and which can only increase?
Why does the truncated SVD set a null-space coordinate to zero rather than estimate it by inversion?
For a coordinate with $\theta_j^2=\sigma^2/s_j^2$, what is the difference in risk between retaining and discarding it?