Reading a Filter Profile
A spectral regularization method does not treat every direction in an inverse problem equally. Its action depends on the corresponding singular value: directions with large singular values may be retained, while directions with small singular values may be attenuated or removed. Comparing methods therefore means comparing their multipliers at the same singular values, not just comparing their parameter names.
Let \(T\) have singular system \(T v_j=\sigma_j u_j\), with \(\sigma_j>0\). Write \(x_j=\langle x^\dagger,v_j\rangle\) for the true signal coordinate and suppose the observed data are \(y=Tx^\dagger+\varepsilon\). The data coordinate along \(u_j\) is \(y_j=\langle y,u_j\rangle=\sigma_jx_j+\varepsilon_j\), where \(\varepsilon_j=\langle\varepsilon,u_j\rangle\). A spectral filter with function \(g\) estimates the \(v_j\)-coordinate as \(\hat{x}_j=g(\sigma_j)y_j\).
This distinction is central. The signal enters the data after multiplication by \(\sigma_j\), whereas data noise is already present in the data coordinate. Consequently, a method can retain some signal in a direction while applying a different-sized multiplier to noise in that direction.
The Three Profiles
Use a cutoff \(\tau>0\) for truncated singular value decomposition (TSVD), a penalty parameter \(\alpha>0\) for Tikhonov regularization, and \(t\) iterations with step size \(\eta\) for Landweber iteration. In the formulas below, the TSVD convention is to retain a singular value exactly equal to the cutoff.
For Landweber, we will take \(0<\eta\leq 1/\sigma_1^2\), where \(\sigma_1\) is the largest singular value. This ensures \(0\leq 1-\eta\sigma_j^2<1\) for every positive singular value in the spectrum. The formulas show three different shapes: TSVD switches abruptly from zero to one signal retention, Tikhonov changes smoothly with \(\sigma^2\), and Landweber retention depends on both \(\sigma\) and the iteration count.
Worked Example: A Three-Value Finite Spectrum
Let the singular values be \(1,\ 0.5,\ 0.1\). Compare TSVD with \(\tau=0.25\), Tikhonov with \(\alpha=0.25\), and two Landweber iterations with \(\eta=1\). The step size is valid because \(1\leq 1/\sigma_1^2=1\).
For TSVD, the first two singular values meet the cutoff and the third does not. The signal multipliers are therefore \((1,1,0)\), and the data-noise multipliers are \((1,2,0)\), since the retained noise multipliers are \(1/1\) and \(1/0.5\).
For Tikhonov, calculate \(h=\sigma^2/(\sigma^2+0.25)\). At the three singular values this gives \(1/1.25=0.8\), \(0.25/0.5=0.5\), and \(0.01/0.26=1/26\approx0.0385\). Dividing each signal multiplier by its singular value gives the noise multipliers \(0.8,\ 1,\ 5/13\approx0.3846\).
For Landweber, \(h=1-(1-\sigma^2)^2\). The three values are \(1,\ 1-0.75^2=7/16=0.4375\), and \(1-0.99^2=0.0199\). Dividing by \(\sigma\), the noise multipliers are \(1,\ 0.875,\ 0.199\).
The smallest singular value is treated differently by each method: TSVD discards it, while Tikhonov and Landweber retain a small fraction of its signal. Their noise multipliers are also different, so “retains some signal” does not by itself describe how the method treats the observed data.
How the Profiles Behave on a Decaying Spectrum
A finite table makes each coordinate visible; a decaying sequence reveals the pattern across increasingly weak directions. Consider \(\sigma_j=2^{-j}\) for \(j=1,2,\ldots\). Compare the same three methods using \(\tau=1/4\), \(\alpha=1/16\), and Landweber with \(\eta=1,\ t=4\). Here \(\sigma_1=1/2\), so the Landweber step size satisfies \(1\leq1/\sigma_1^2=4\).
Worked Example: Four Coordinates of a Geometrically Decaying Spectrum
For TSVD, \(\sigma_j\geq1/4\) holds exactly for \(j=1,2\). Thus \(h=(1,1,0,0)\) and \(g=(2,4,0,0)\) on the first four coordinates.
For Tikhonov, \(\alpha/\sigma_j^2=4^{j-2}\), so \(h_j=1/(1+4^{j-2})\). This gives signal multipliers \((0.8,\ 0.5,\ 0.2,\ 1/17)\). Since \(g_j=h_j/\sigma_j\), the noise multipliers are \((1.6,\ 2,\ 1.6,\ 16/17)\).
For Landweber, calculate \(h_j=1-(1-4^{-j})^4\), then divide by \(2^{-j}\) to obtain \(g_j\). The results, rounded to six decimal places, are:
| Coordinate \(j\) | \(\sigma_j\) | Landweber \(h_j\) | Landweber \(g_j\) |
|---|---|---|---|
| 1 | 0.5 | 0.683594 | 1.367188 |
| 2 | 0.25 | 0.227524 | 0.910095 |
| 3 | 0.125 | 0.061050 | 0.488403 |
| 4 | 0.0625 | 0.015534 | 0.248539 |
For instance, at \(j=2\), \(h_2=1-(15/16)^4=14911/65536\approx0.227524\), so \(g_2=h_2/(1/4)\approx0.910095\). The profiles show that the TSVD cutoff makes a hard boundary. Tikhonov and four Landweber steps instead attenuate the weaker directions gradually, but at different rates.
Signal and Noise Through the Same Filter
The multiplier formulas yield a coordinate-by-coordinate decomposition without any assumption about a noise distribution. Substituting \(y_j=\sigma_jx_j+\varepsilon_j\) into \(\hat{x}_j=g(\sigma_j)y_j\) gives the signal contribution \(h(\sigma_j)x_j\) and noise contribution \(g(\sigma_j)\varepsilon_j\). The difference from the true coordinate is therefore \((h(\sigma_j)-1)x_j+g(\sigma_j)\varepsilon_j\). These are separate contributions; for particular data they can reinforce or cancel one another.
Proof. By definition, \(y_j=\sigma_jx_j+\varepsilon_j\) and \(\hat{x}_j=g(\sigma_j)y_j\). Distributing \(g(\sigma_j)\) gives \(\hat{x}_j=g(\sigma_j)\sigma_jx_j+g(\sigma_j)\varepsilon_j=h(\sigma_j)x_j+g(\sigma_j)\varepsilon_j\). Subtract \(x_j\) from both sides to obtain the stated error. \(\square\)
This decomposition explains why comparing only signal multipliers is incomplete. For example, a retained TSVD coordinate has signal multiplier one, but its noise multiplier is \(1/\sigma_j\). Conversely, a small signal multiplier does not imply a zero noise contribution unless the noise multiplier is also zero or that data coordinate has no noise.
Landweber at a Fixed Iteration Count
A useful way to interpret a fixed number of Landweber steps is to examine its behavior as \(\sigma\) becomes small. Put \(z=\eta\sigma^2\), so \(0<z\leq1\). The geometric-series identity gives
Every term in the sum is between zero and one, and its first term is one. Consequently, \(z\leq h_{\mathrm{LW},t}(\sigma)\leq tz\). Dividing by \(\sigma>0\) gives the corresponding bounds for the data-noise multiplier.
Proof. With \(z=\eta\sigma^2\), the finite geometric-series formula yields \(1-(1-z)^t=z\sum_{k=0}^{t-1}(1-z)^k\). Since \(0\leq1-z<1\), each summand is at most one, while the first is one. Thus \(z\leq h\leq tz\). Dividing by \(\sigma\) proves the two bounds for \(g=h/\sigma\). For fixed \(t\), the upper bounds \(t\eta\sigma^2\) and \(t\eta\sigma\) both tend to zero as \(\sigma\) tends to zero. For fixed positive \(\sigma\), \(0\leq1-\eta\sigma^2<1\), so \((1-\eta\sigma^2)^t\) tends to zero; the formulas for \(h\) and \(g=h/\sigma\) then give the stated limits. \(\square\)
The limits clarify the role of iteration count. At a fixed number of steps, sufficiently small singular values receive little signal retention and have small data-noise multipliers. Increasing the number of steps changes that profile: for each fixed positive singular value, the signal multiplier approaches one and the noise multiplier approaches the unregularized inverse multiplier \(1/\sigma\). This is a coordinate-wise statement, not a guarantee that a single iteration count is best for every spectrum and noise realization.
Worked Example: Separating Signal and Noise Contributions
Take singular values \((1,\ 0.2,\ 0.04)\), true signal coordinates \(x=(1,1,1)\), and data-noise coordinates \(\varepsilon=(0.1,-0.1,0.1)\). Then the observed coordinates are \(y_j=\sigma_jx_j+\varepsilon_j\), so \(y=(1.1,\ 0.1,\ 0.14)\). Compare the same parameter choices as before: TSVD with \(\tau=0.1\), Tikhonov with \(\alpha=0.04\), and five Landweber steps with \(\eta=1\). The step size is valid since \(\sigma_1=1\).
TSVD keeps the first two coordinates, giving \(h=(1,1,0)\) and \(g=(1,5,0)\). Its signal contribution is \((1,1,0)\); its noise contribution is \((0.1,-0.5,0)\). Adding them gives \(\hat{x}=(1.1,0.5,0)\).
For Tikhonov, the signal multipliers are \(1/1.04=25/26\), \(0.04/0.08=1/2\), and \(0.0016/0.0416=1/26\). The noise multipliers are \(25/26,\ 0.2/0.08=5/2,\ 0.04/0.0416=25/26\). Thus its signal contribution is \((25/26,1/2,1/26)\), and its noise contribution is \((5/52,-1/4,5/52)\). Adding coordinatewise gives \(\hat{x}=(55/52,1/4,7/52)\), approximately \((1.0577,0.25,0.1346)\).
For five Landweber steps, the signal multipliers are \(1,\ 1-0.96^5=0.1846273024,\) and \(1-0.9984^5\approx0.007974441\). Dividing by the singular values gives noise multipliers \(1,\ 0.923136512,\) and approximately \(0.199361023\). The signal contribution is approximately \((1,0.184627302,0.007974441)\); the noise contribution is approximately \((0.1,-0.092313651,0.019936102)\). Their sum is \(\hat{x}\approx(1.1,0.092313651,0.027910543)\).
The second-coordinate noise is negative, so it partly cancels the signal contribution rather than simply increasing the estimate. Keeping the two contributions separate makes that cancellation visible and prevents the multiplier comparison from being mistaken for a prediction of the realized error alone.
Common Pitfall and Practical Comparison
Wrong reasoning: “Tikhonov has a smaller signal multiplier at a small singular value, so it must multiply the noise in that coordinate by the same small amount.” This confuses the multiplier on the true signal with the multiplier on the data coordinate. The signal multiplier is \(h=\sigma g\), while the noise multiplier is \(g=h/\sigma\). At \(\sigma=0.1\), for example, a signal multiplier of \(0.2\) corresponds to a noise multiplier of \(2\), not \(0.2\).
Correction: For each singular value, calculate \(h\) and \(g\) separately. Then use \(hx_j\) for the signal contribution and \(g\varepsilon_j\) for the noise contribution. A useful comparison table is:
| Method | Signal profile | Data-noise profile |
|---|---|---|
| TSVD | Retains or removes each direction abruptly | Inverts retained directions; sets discarded directions to zero |
| Tikhonov | Smooth attenuation depending on \(\sigma^2/(\sigma^2+\alpha)\) | Applies \(\sigma/(\sigma^2+\alpha)\) |
| Landweber | Gradually increases with iteration count | Applies the signal multiplier divided by \(\sigma\) |
Worked spectra are useful because they turn qualitative descriptions such as “smooth shrinkage” or “hard cutoff” into coordinate-specific calculations. For any new spectrum, record the singular values, parameters, signal multipliers, and noise multipliers in separate columns. That simple discipline keeps the method’s effect on the target signal distinct from its effect on observed noise.
Check Your Understanding
Use the multiplier formulas and show the intermediate calculations.
- For \(\sigma=0.2\), calculate the Tikhonov signal and data-noise multipliers when \(\alpha=0.04\).
- With TSVD cutoff \(\tau=0.1\), what are the signal and data-noise multipliers at \(\sigma=0.1\)? How would they change if the cutoff convention used a strict inequality?
- For a Landweber iteration with \(t=3\), \(\eta=1\), and \(\sigma=0.5\), calculate \(h\) and \(g\).
- Explain why a signal multiplier of \(0.2\) at singular value \(0.1\) corresponds to a data-noise multiplier of \(2\).
- For fixed \(t\), what happens to both Landweber multipliers as \(\sigma\) tends to zero? Contrast this with the limit as \(t\) tends to infinity for a fixed positive \(\sigma\).