Choosing Regularization from a Noise Bound
Balancing bias against variance describes a trade-off, but it does not by itself tell us which regularization parameter to use for a particular data set. In some inverse problems, we instead know a deterministic bound on the data error. Morozov’s discrepancy principle uses that information directly: it selects a parameter for which the residual is commensurate with the known noise level.
The principle is most transparent for a linear inverse problem. Let \(A:X\to Y\) be a compact linear operator between Hilbert spaces, let \(x^\dagger\in X\) be the unknown solution, and observe \(y^\delta=Ax^\dagger+e\), where the noise satisfies \(\|e\|_Y\leq\delta\) for a known \(\delta>0\). For Tikhonov regularization with parameter \(\alpha>0\), define \(x_\alpha^\delta\) as the minimizer of \(\|Ax-y^\delta\|_Y^2+\alpha\|x\|_X^2\). The residual is \(r(\alpha)=\|Ax_\alpha^\delta-y^\delta\|_Y\).
The Residual in Spectral Coordinates
Write the positive singular values of \(A\) as \(s_j\), with corresponding left singular vectors \(u_j\), and set \(\lambda_j=s_j^2\). Decompose the data as \(y^\delta=\sum_j y_j u_j+y_\perp\), where \(y_j=\langle y^\delta,u_j\rangle_Y\) and \(y_\perp\) is orthogonal to the closure of the range of \(A\). The Tikhonov filter leaves a fraction \(\alpha/(\lambda_j+\alpha)\) of each data coordinate in the residual; the orthogonal component cannot be fitted by any \(Ax\).
The formula makes the two limiting behaviors visible. As \(\alpha\) decreases toward zero, the fitted-range part of the residual vanishes, but \(y_\perp\) remains. As \(\alpha\) increases, the filter factors increase toward one and the residual approaches the norm of the data. Thus, increasing regularization generally increases the data residual.
Proof. Since \(Ax^\dagger\) belongs to the range of \(A\), its projection onto the orthogonal complement of the closure of that range is zero. Therefore \(y_\perp\) is the corresponding projection of \(e\), and \(\|y_\perp\|_Y\leq\|e\|_Y\leq\delta\). In particular, \(r(\alpha)\) tends to a value no larger than \(\delta\) as \(\alpha\) decreases to zero, which is strictly below \(\tau\delta\).
For every \(\alpha>0\), each factor \(\alpha/(\lambda_j+\alpha)\) is continuous and strictly increasing in \(\alpha\). The series defining \(r(\alpha)^2\) is continuous: on any convergent sequence of positive parameter values, its summands converge coordinatewise and are bounded by \(|y_j|^2\), whose sum is finite. Dominated convergence for series therefore permits passage of the limit through the sum. The same coordinate formula shows that \(r(\alpha)^2\) is strictly increasing if some \(y_j\ne0\). Its limits at zero and infinity are \(\|y_\perp\|_Y\) and \(\|y^\delta\|_Y\), respectively; the limit at infinity also follows by dominated convergence. Since the target \(\tau\delta\) lies strictly between these limits, continuity gives a solution and strict increase gives uniqueness. \(\square\)
The assumptions matter. If \(\|y^\delta\|_Y\leq\tau\delta\), the residual may never reach the target at a finite parameter. If all \(y_j\) vanish, the residual is constant, so uniqueness also fails. The noise bound alone does not rule out either case.
Worked Examples
Worked Example: A Scalar Discrepancy Equation
Consider \(A:\mathbb{R}\to\mathbb{R}\) given by multiplication by \(\sqrt{1.178}\), and data \(y^\delta=1+1.178/0.440786\approx3.672499\). Use \(\delta=0.5\) and \(\tau=2\), so the target residual is \(\tau\delta=1\). Find the parameter and check the nearby value \(\alpha=0.4406\).
Here \(\lambda=1.178\), so the residual is \(r(\alpha)=\alpha y^\delta/(1.178+\alpha)\). Setting it equal to one gives \(\alpha y^\delta=1.178+\alpha\), hence \(\alpha=1.178/(y^\delta-1)=0.440786\) by the definition of \(y^\delta\). Thus the solution rounds to \(\alpha\approx0.4408\).
At \(\alpha=0.4406\), compare the residual with its value at the root: \(r(0.4406)^2=[0.4406(1.178+0.440786)/(0.440786(1.178+0.4406))]^2\approx0.999386\). It is below the target squared, which is \(1\). The root is therefore above \(0.4406\); the displayed equation gives the more precise value \(0.440786\), not \(0.4406\).
Worked Example: A Two-Coordinate Spectrum
Let \(A\) be the diagonal matrix with entries \(2\) and \(1\), and let \(y^\delta=(1,1)\). Then \(\lambda_1=4\), \(\lambda_2=1\), and \(r(\alpha)^2=\alpha^2/(4+\alpha)^2+\alpha^2/(1+\alpha)^2\). Choose \(\tau=2\) and \(\delta=\sqrt{0.29}/2\), so the target residual is \(\sqrt{0.29}\).
At \(\alpha=1\), the two residual coordinates are \(1/(4+1)=1/5\) and \(1/(1+1)=1/2\). Therefore \(r(1)^2=(1/5)^2+(1/2)^2=1/25+1/4=0.04+0.25=0.29\), and \(r(1)=\sqrt{0.29}=\tau\delta\). The discrepancy parameter is exactly \(\alpha=1\). Both data coordinates are nonzero in positive singular directions, so the residual is strictly increasing and this solution is unique.
This example also shows why the spectrum matters: the two coordinates have different filter factors, so their contributions to the residual do not grow at the same rate.
Worked Example: A Residual Floor from Unfittable Data
Take \(A=\operatorname{diag}(1,0)\), true solution \(x^\dagger=(3,0)\), and noise \(e=(0,2)\). Then \(y^\delta=(3,2)\) and \(\|e\|=2\), so use \(\delta=2\) and \(\tau=1.2\). The target residual is \(2.4\).
The first coordinate can be fitted with Tikhonov residual \(3\alpha/(1+\alpha)\); the second is orthogonal to the range of \(A\) and remains equal to \(2\). Consequently \(r(\alpha)^2=9\alpha^2/(1+\alpha)^2+4\). Setting this equal to \(2.4^2=5.76\) gives \(9\alpha^2/(1+\alpha)^2=1.76\), or \(3\alpha/(1+\alpha)=\sqrt{1.76}\). Solving yields \(\alpha=\sqrt{1.76}/(3-\sqrt{1.76})\approx0.793\).
As \(\alpha\) decreases to zero, the residual tends to \(2\), exactly the orthogonal noise component, while the selected residual is \(2.4\). The residual cannot fall below that floor, but the discrepancy target lies above it and below \(\|y^\delta\|=\sqrt{13}\), so the parameter exists.
Existence, Interpretation, and a Common Pitfall
In practice, the theorem gives a useful checklist before solving the discrepancy equation. The noise bound controls the residual floor: because the exact signal lies in the closure of the range, the part of the observed data orthogonal to that closure can come only from noise, and its norm is at most \(\delta\). The safety factor ensures \(\tau\delta\) exceeds that floor. A further check, \(\|y^\delta\|_Y>\tau\delta\), ensures the target is below the limiting residual at very large \(\alpha\). When both conditions hold and the data have a nonzero fitted component, a root search for \(r(\alpha)-\tau\delta=0\) has a unique answer.
Common mistake: “The noise is bounded by \(\delta\), so choose \(\alpha\) until the residual equals \(\delta\).” This ignores that the residual of a regularized fit includes both noise and regularization effects; it need not equal the actual noise norm. The standard discrepancy equation uses \(\tau\delta\) with \(\tau>1\), and its solvability must be checked. Another mistake is to assume that a large parameter always reaches any target: \(r(\alpha)\) approaches \(\|y^\delta\|_Y\), so a target at or above that limit is not reached at a finite \(\alpha\).
Morozov’s principle is a deterministic choice rule, not a universal risk minimizer. Its appeal is that it uses a stated error bound rather than requiring an estimate of the unknown solution’s bias or a variance model. Its validity depends on the noise bound being credible and on the forward model being appropriate. If the noise level is poorly specified, the selected parameter inherits that error; if the residual target is infeasible, the equation itself signals that the stated conditions and data do not support the requested choice.
Check Your Understanding
Use the spectral residual formula and the assumptions stated above.
- Why is the discrepancy target multiplied by a factor \(\tau>1\), rather than set equal to the noise bound itself?
- What is the limit of the Tikhonov residual as \(\alpha\) decreases to zero, and why is it no larger than \(\delta\) under the noise-bound assumption?
- Which conditions in the theorem ensure that the discrepancy equation has a unique finite solution?
- For \(A=\operatorname{diag}(1,0)\), explain why a nonzero second data coordinate contributes a residual floor for every \(\alpha>0\).
- What can be concluded if \(\|y^\delta\|_Y\leq\tau\delta\)?