From Source Smoothness to Bias
A spectral filter’s residual describes which components of a target the regularized solution fails to recover. A source condition describes how much of the target lies at each spectral scale. Their combination yields a quantitative bound on the bias. The central calculation is scalar: multiply the residual magnitude at a spectral value by the source factor at that value, then take a supremum.
Let \(T\) be a compact, positive, self-adjoint operator on a Hilbert space, with positive eigenvalues \(\mu_j\) and corresponding orthonormal eigenvectors \(e_j\). Write \(L=\|T\|\). A filter \(g_\alpha\), indexed by a regularization parameter \(\alpha>0\), defines the regularized solution \(f_\alpha=g_\alpha(T)T f_*\) for a target \(f_*\). Its residual function is \(r_\alpha(\lambda)=1-\lambda g_\alpha(\lambda)\). Thus the bias is the difference \(f_\alpha-f_*\), not the residual itself applied to \(f_*\).
The source condition says that \(f_*\) has no component in the kernel of \(T\): \(T^\beta\) annihilates that component. It does not require \(w\) itself to have zero kernel component. That distinction matters when bounding the sum of squared spectral coefficients of \(w\).
The Spectral Supremum Behind the Bound
For a particular operator, the relevant quantity is the supremum over its positive spectral values, rather than necessarily over every number between zero and \(L\). Define
Here \(\sigma(T)\) denotes the spectrum of \(T\). Since \(T\) is compact, its nonzero spectral values are positive eigenvalues. A bound on the full interval \(0<\lambda\le L\), as used in the definition of qualification, is sufficient for bounding \(M_\beta(\alpha;T)\). But the full-interval supremum can be larger than the one for a specific operator.
Proof. Choose an orthonormal eigenbasis for the closure of the range of \(T\), and write \(w_j=\langle w,e_j\rangle\). The component of \(w\) in the kernel is omitted from these coordinates, so Parseval’s identity gives \(\sum_j|w_j|^2\le\|w\|^2\), not necessarily equality. The source condition gives \(f_*=\sum_j\mu_j^\beta w_j e_j\). Since \(g_\alpha(T)T=I-r_\alpha(T)\) on the positive-eigenvalue subspace, the bias has coordinates $$ f_\alpha-f_*=-r_\alpha(T)f_*=-\sum_j r_\alpha(\mu_j)\mu_j^\beta w_j e_j. $$ Consequently, $$ \|f_\alpha-f_*\|^2 =\sum_j |r_\alpha(\mu_j)|^2\mu_j^{2\beta}|w_j|^2 \le M_\beta(\alpha;T)^2\sum_j|w_j|^2 \le M_\beta(\alpha;T)^2\|w\|^2. $$ Taking square roots proves the first inequality. The qualification bound implies \(M_\beta(\alpha;T)\le C_\beta\alpha^\beta\), proving the second. \(\square\)
The theorem separates the operator calculation from the filter calculation. The source condition contributes \(\lambda^\beta\); the filter contributes \(|r_\alpha(\lambda)|\). Qualification supplies a uniform bound on their product. No eigenvalue-decay assumption is needed for this bias inequality.
Worked Filter Calculations
Worked Example: Tikhonov Qualification and Its Bias Bound
For Tikhonov regularization, \(g_\alpha(\lambda)=1/(\lambda+\alpha)\), so $$ r_\alpha(\lambda)=1-\frac{\lambda}{\lambda+\alpha} =\frac{\alpha}{\lambda+\alpha}. $$ For \(0<\beta\le1\), set \(t=\lambda/\alpha\). Then $$ \lambda^\beta |r_\alpha(\lambda)| =\alpha^\beta\frac{t^\beta}{1+t}. $$
If \(0<\beta<1\), differentiating \(h(t)=t^\beta/(1+t)\) gives $$ h'(t)=\frac{t^{\beta-1}\{\beta-(1-\beta)t\}}{(1+t)^2}. $$ The derivative changes from positive to negative at \(t=\beta/(1-\beta)\), so this point is the maximum over \(t>0\). Substitution yields $$ \sup_{t>0}\frac{t^\beta}{1+t} =\beta^\beta(1-\beta)^{1-\beta}. $$ For \(\beta=1\), \(t/(1+t)\le1\), with supremum 1 as \(t\) tends to infinity. Thus Tikhonov has qualification at least 1, with \(C_\beta=\beta^\beta(1-\beta)^{1-\beta}\) for \(0<\beta<1\) and \(C_1=1\). The theorem gives \(\|f_\alpha-f_*\|\le C_\beta\alpha^\beta\|w\|\) for a source of order \(\beta\le1\).
To see why this qualification does not extend beyond 1, fix a positive \(\lambda\). For \(\beta>1\), the ratio \(\lambda^\beta|r_\alpha(\lambda)|/\alpha^\beta\) equals \(\lambda^\beta/[\alpha^{\beta-1}(\lambda+\alpha)]\), which diverges as \(\alpha\) decreases to zero. No constant independent of \(\alpha\) can give the qualification bound of order \(\beta>1\).
Worked Example: Iterated Tikhonov and a Higher-Order Bound
The \(m\)-fold iterated Tikhonov residual is \(r_\alpha(\lambda)=(\alpha/(\lambda+\alpha))^m\), where \(m\) is a positive integer. With \(t=\lambda/\alpha\), $$ \lambda^\beta|r_\alpha(\lambda)| =\alpha^\beta\frac{t^\beta}{(1+t)^m}. $$ For \(0<\beta<m\), differentiation gives a unique maximum at \(t=\beta/(m-\beta)\). Indeed, the logarithmic derivative is \(\beta/t-m/(1+t)\), which is zero precisely there and changes from positive to negative. Evaluating at that point gives $$ \sup_{t>0}\frac{t^\beta}{(1+t)^m} =\frac{\beta^\beta(m-\beta)^{m-\beta}}{m^m}. $$ At \(\beta=m\), the expression is \(t^m/(1+t)^m\), bounded above by 1, with supremum 1. Hence the bias is at most the displayed constant times \(\alpha^\beta\|w\|\) for every \(0<\beta\le m\).
For \(\beta>m\), \(t^\beta/(1+t)^m\) grows without bound as \(t\) tends to infinity. More concretely, at a fixed positive \(\lambda\), the ratio of \(\lambda^\beta|r_\alpha(\lambda)|\) to \(\alpha^\beta\) grows like a positive constant times \(\alpha^{m-\beta}\) as \(\alpha\) decreases to zero. Thus the uniform qualification order is \(m\), not larger.
Worked Example: Using the Actual Spectrum and Checking the Bias Sign
Let \(T\) be diagonal with eigenvalues \(1,1/4,1\), let \(\alpha=1/4\), and let \(f_*=(1,1,1)\). This is a source of order 1, since \(f_*=Tw\) with \(w=(1,4,1)\) and \(\|w\|=\sqrt{18}\). For Tikhonov regularization, the residual values at the three coordinates are $$ r_\alpha(1)=\frac{1}{5},\qquad r_\alpha(1/4)=\frac{1}{2},\qquad r_\alpha(1)=\frac{1}{5}. $$ Thus \(r_\alpha(T)f_*=(1/5,1/2,1/5)\). The bias is the negative of this vector: $$ f_\alpha-f_*=-r_\alpha(T)f_* =(-1/5,-1/2,-1/5). $$ Its norm is \(\sqrt{1/25+1/4+1/25}=\sqrt{33/100}=\sqrt{33}/10\).
The actual spectral supremum for \(\beta=1\) is \(M_1(\alpha;T)=\max\{1/5,(1/4)(1/2)\}=1/5\). The theorem therefore gives \(\|f_\alpha-f_*\|\le\sqrt{18}/5\), which is valid because \(33/100\le18/25\). The full-interval qualification bound uses \(C_1\alpha=1/4\), yielding the looser bound \(\sqrt{18}/4\). This example shows both why the spectrum-specific supremum can sharpen the result and why the residual-applied vector must not be confused with the bias.
A Further Derivation: Landweber Iteration
For Landweber iteration with step size \(\eta>0\) and \(k\) iterations, assume \(0<\eta L\le1\). Its residual is \(r_k(\lambda)=(1-\eta\lambda)^k\). Set \(\alpha=1/(\eta k)\). For any \(\beta>0\), the elementary inequality \(1-y\le e^{-y}\) for \(0\le y\le1\) gives $$ \lambda^\beta(1-\eta\lambda)^k \le \lambda^\beta e^{-\eta k\lambda}. $$ Put \(x=\eta k\lambda\), so the right-hand side is \(\alpha^\beta x^\beta e^{-x}\). Differentiating \(x^\beta e^{-x}\) gives \(x^{\beta-1}e^{-x}(\beta-x)\); its maximum is at \(x=\beta\), with value \(\beta^\beta e^{-\beta}\). Therefore $$ \sup_{0<\lambda\le L}\lambda^\beta|r_k(\lambda)| \le \left(\frac{\beta}{e}\right)^\beta\alpha^\beta. $$ Applying the theorem gives \(\|f_k-f_*\|\le(\beta/e)^\beta\alpha^\beta\|w\|\). The calculation makes the iteration dependence explicit: the bias bound scales as \((\eta k)^{-\beta}\) for a source of order \(\beta\).
Interpreting and Using the Bound
A bias bound is a deterministic statement about how well a filter recovers a target under a source condition. It does not by itself bound the variability of an estimate constructed from noisy data. It also does not say that the bound is attained: the supremum may occur at a spectral value that is absent from the operator, or the source element \(w\) may place little weight on the corresponding eigenspace.
A common mistake is to replace the spectrum-specific supremum by a claimed maximum at a particular eigenvalue without checking the spectrum, or to assert equality between \(\sum_j|w_j|^2\) and \(\|w\|^2\) while omitting kernel coordinates. The correction is to first bound the scalar multiplier over the positive spectral values and then use \(\sum_j|w_j|^2\le\|w\|^2\). Another sign error is to call \(r_\alpha(T)f_*\) the bias: with the convention \(f_\alpha-f_*=-r_\alpha(T)f_*\), it is the negative of the bias. A norm is unchanged by this sign, but the vector identity is not.
Check Your Understanding
Use the residual convention and spectral supremum carefully.
- For a source condition \(f_*=T^\beta w\), write the bias as a function of \(r_\alpha(T)\) and \(f_*\).
- Why is \(\sum_j|w_j|^2\le\|w\|^2\) the generally valid inequality when the eigenvectors \(e_j\) span only the positive-eigenvalue subspace?
- For Tikhonov regularization, what is the residual function, and for which source orders does its standard qualification bound apply?
- In the diagonal example, what vector is \(r_\alpha(T)f_*\), and what vector is \(f_\alpha-f_*\)?
- Why can the supremum over the actual positive spectrum yield a sharper bias bound than the qualification supremum over the full interval?