From Penalized Least Squares to a Spectral Filter
Kernel ridge regression (KRR) is often introduced as a minimization problem: fit the observed responses while penalizing the size of the fitted function in a reproducing kernel Hilbert space. The same solution can be read spectrally. Along an eigen-direction with eigenvalue \( \mu \), the fitted signal is multiplied by \( \mu/(\mu+\lambda) \). Deriving that factor makes clear what the regularization parameter does and why its numerical value depends on how the loss is scaled.
We use the averaged squared-error convention: the residual sum of squares is divided by the sample size \(n\), and the penalty is \( \lambda\|f\|_{\mathcal H}^2 \). Here \( \lambda>0 \). With this convention, the eigenvalues relevant to the filter are those of a normalized operator; using an unnormalized sum of squares changes the parameter appearing beside the eigenvalues.
The Operator Form of the Objective
For each input \(x_i\), write \(K_{x_i}=K(x_i,\cdot)\in\mathcal H\), so the reproducing property gives \(f(x_i)=\langle f,K_{x_i}\rangle_{\mathcal H}\). Define the sampling operator \(A:\mathcal H\to\mathbb R^n\) by \((Af)_i=f(x_i)/\sqrt n\), and write \(z=y/\sqrt n\), where \(y=(y_1,\ldots,y_n)^\mathsf T\). The objective becomes \( \|Af-z\|_2^2+\lambda\|f\|_{\mathcal H}^2 \). The adjoint satisfies \(A^*v=n^{-1/2}\sum_i v_i K_{x_i}\).
The operator \(A^*A\) acts on the RKHS. In particular, $$ A^*Af=\frac{1}{n}\sum_{i=1}^n f(x_i)K_{x_i}. $$ It is positive and self-adjoint, and has finite rank. Its positive eigenvalues are therefore a finite list (counting multiplicity), though its null space may be infinite-dimensional. The normal equation for the objective is $$ (A^*A+\lambda I)f=A^*z. $$ The penalty adds \( \lambda \) to every eigenvalue of \(A^*A\), making the equation uniquely solvable even in directions that are not identified by the sampled inputs.
Proof. Put \(T=A^*A\). For every \(h\in\mathcal H\), $$ \langle (T+\lambda I)h,h\rangle_{\mathcal H} =\|Ah\|_2^2+\lambda\|h\|_{\mathcal H}^2 \geq \lambda\|h\|_{\mathcal H}^2. $$ Thus \(T+\lambda I\) is bounded, self-adjoint, and bounded below by \( \lambda I \), so it has a bounded inverse. Set \(f_\lambda=(T+\lambda I)^{-1}A^*z\). Expanding the objective at \(f_\lambda+h\), and using \((T+\lambda I)f_\lambda=A^*z\), gives $$ \|A(f_\lambda+h)-z\|_2^2+\lambda\|f_\lambda+h\|_{\mathcal H}^2 =\|Af_\lambda-z\|_2^2+\lambda\|f_\lambda\|_{\mathcal H}^2+\|Ah\|_2^2+\lambda\|h\|_{\mathcal H}^2. $$ The last two terms are nonnegative and their sum is zero only when \(h=0\), proving existence and uniqueness of the minimizer. For a positive-eigenvalue direction \(u_j\), the normal equation gives $$ \langle f_\lambda,u_j\rangle_{\mathcal H} =\frac{\langle A^*z,u_j\rangle_{\mathcal H}}{\mu_j+\lambda}. $$ If \(z=A f^\dagger\), then \(A^*z=T f^\dagger\), and the coefficient is \( \mu_j\langle f^\dagger,u_j\rangle_{\mathcal H}/(\mu_j+\lambda) \). This is the claimed multiplier on the signal. \(\square\)
Why the Multiplier Is \( \mu/(\mu+\lambda) \)
The distinction in the theorem is important. The inverse normal equation applies \(1/(\mu+\lambda)\) to the right-hand side \(A^*z\). But when the data are generated by a signal \(z=A f^\dagger\), that right-hand side already contains a factor \( \mu \) in direction \(u_j\), because \(A^*A f^\dagger=T f^\dagger\). Combining the two factors produces \( \mu/(\mu+\lambda) \). Equivalently, the regularized estimate of the signal is the result of applying \(T(T+\lambda I)^{-1}\) to that signal.
For positive \( \mu \), the multiplier lies strictly between zero and one. It is small when \( \mu\ll\lambda \), so that direction is strongly shrunk; it is close to one when \( \mu\gg\lambda \), so that direction is largely retained. On a zero-eigenvalue direction, the signal multiplier is zero. Such a direction is invisible to \(A\), and the penalty selects zero rather than attempting to recover an unobserved component.
Worked Example: Shrinking Two Eigen-Directions
Suppose \(A^*A\) has orthonormal eigenvectors \(u_1,u_2\) with eigenvalues \(4\) and \(1\), and the signal is \(f^\dagger=3u_1-2u_2\). Let \(z=A f^\dagger\) and choose \( \lambda=2 \). The spectral multipliers are $$ r_2(4)=\frac{4}{4+2}=\frac{2}{3},\qquad r_2(1)=\frac{1}{1+2}=\frac{1}{3}. $$ Therefore $$ f_\lambda=\frac{2}{3}(3)u_1+\frac{1}{3}(-2)u_2=2u_1-\frac{2}{3}u_2. $$ The eigenvalue-4 component is retained more strongly than the eigenvalue-1 component. Directly from the normal equation, the right-hand-side coefficients are \(4(3)=12\) and \(1(-2)=-2\); dividing by \(4+2\) and \(1+2\) gives \(2\) and \(-2/3\), respectively, confirming the calculation.
Connection to the Kernel Gram Matrix
The same fit can be expressed using only an \(n\)-dimensional matrix. The matrix \(AA^*\) has entries \(K(x_i,x_j)/n\), so it is the normalized Gram matrix \(G=K_n/n\), where \(K_n\) denotes the usual Gram matrix. The positive eigenvalues of \(AA^*\) and \(A^*A\) agree: if \(A^*A u=\mu u\) with \(\mu>0\), then \(Au\ne0\) and \(AA^*(Au)=\mu Au\); the reverse follows by applying \(A^*\) to a positive-eigenvalue eigenvector of \(AA^*\).
To verify this representation, use \(f_\lambda=A^*(AA^*+\lambda I)^{-1}z\), which follows by multiplying out the normal equation. Applying \(A\) gives \(Af_\lambda=AA^*(AA^*+\lambda I)^{-1}z=G(G+\lambda I)^{-1}z\). Since \(G=K_n/n\) and \(z=y/\sqrt n\), converting back from scaled responses gives the stated fitted response. Thus an eigenvalue \( \kappa \) of \(K_n\) has fitted-response multiplier $$ \frac{\kappa}{\kappa+n\lambda}. $$ This is the same as \( \mu/(\mu+\lambda) \) with \( \mu=\kappa/n \), the corresponding eigenvalue of the normalized Gram matrix.
Worked Example: Checking the Normalization
Suppose \(n=4\), and the Gram matrix \(K_n\) has an eigenvalue \( \kappa=8 \). With the averaged-loss objective and \( \lambda=0.5 \), the eigenvalue of \(G=K_n/n\) is \( \mu=8/4=2 \). The multiplier is $$ \frac{\mu}{\mu+\lambda}=\frac{2}{2+0.5}=0.8. $$ Using the unnormalized Gram eigenvalue gives the same answer: $$ \frac{\kappa}{\kappa+n\lambda}=\frac{8}{8+4(0.5)}=\frac{8}{10}=0.8. $$ Using \(8/(8+0.5)\) would be incorrect for this objective: it mixes an unnormalized eigenvalue with a parameter from an averaged-loss convention.
How the Regularization Convention Changes
If the objective is written with an unnormalized sum of squared errors, $$ \sum_{i=1}^n(y_i-f(x_i))^2+\rho\|f\|_{\mathcal H}^2, $$ then it is \(n\) times the averaged-loss objective with \( \lambda=\rho/n \). Multiplying an entire objective by a positive constant does not change its minimizer, so the two conventions agree when \( \rho=n\lambda \). In terms of the unnormalized Gram eigenvalue \( \kappa \), the filter is then \( \kappa/(\kappa+\rho) \).
A further convention writes the penalty as \( \alpha^2\|f\|_{\mathcal H}^2 \). In that notation the filter is \( \mu/(\mu+\alpha^2) \); the parameter added to the eigenvalue is \( \alpha^2 \), not \( \alpha \). These conventions are not contradictory. The denominator must use the same scale and parameter as the stated objective.
Worked Example: The Complement Is the Residual Multiplier
Consider one eigen-direction with \( \mu=0.25 \), signal coefficient \( \theta=4 \), and \( \lambda=0.25 \). The fitted coefficient is $$ \frac{\mu}{\mu+\lambda}\theta =\frac{0.25}{0.25+0.25}(4)=2. $$ The signal coefficient not retained by the fit is \(4-2=2\). Its multiplier is $$ 1-\frac{\mu}{\mu+\lambda}=\frac{\lambda}{\mu+\lambda} =\frac{0.25}{0.5}=\frac{1}{2}. $$ Thus \( \lambda/(\mu+\lambda) \) describes the fraction left out of the signal, while \( \mu/(\mu+\lambda) \) describes the fraction retained. If the squared-penalty convention uses \( \alpha^2=\lambda=0.25 \), then \( \alpha=0.5 \) gives the same fit.
Common Mistakes and Why the Filter Matters
A frequent error is to call \( \lambda/(\mu+\lambda) \) the fitted-signal multiplier. It is instead the complement of that multiplier, and in the noiseless signal calculation it gives the fraction of the signal coefficient left as residual. Another error is to state the correct-looking ratio while using eigenvalues and a regularization parameter from incompatible normalizations. Checking the objective itself prevents both mistakes.
The spectral view makes KRR’s action interpretable direction by direction: regularization does not shrink every component by one common constant. Its effect depends on how the eigenvalue compares with the penalty parameter. This is useful when reading operator or Gram-matrix formulas, comparing implementations that scale their loss differently, and interpreting which components of a fitted signal are strongly attenuated.
Check Your Understanding
Work through these questions by identifying the objective and the relevant eigenvalue scale.
- For \( \mu=3 \) and \( \lambda=1 \), what fraction of a signal coefficient is retained, and what fraction is left out?
- Why does the normal-equation coefficient \(1/(\mu+\lambda)\) not by itself equal the fitted-signal multiplier?
- If the loss is an unnormalized sum and the penalty coefficient is \( \rho \), what is the Gram-eigenvalue multiplier?
- A Gram matrix has eigenvalue \(12\), the sample size is \(6\), and the averaged-loss parameter is \( \lambda=1 \). Calculate the multiplier using both normalizations.
- What does KRR return in a direction in the null space of the sampling operator, and why?