Kernel Rescaling and Regularization Calibration

Why Kernel Scale and Regularization Must Be Calibrated Together

A kernel has an amplitude as well as a shape. Multiplying every kernel value by the same positive constant preserves which inputs are similar relative to one another, but it changes the size of the associated covariance spectrum and the scale of the RKHS penalty. Consequently, a regularization value that worked with one kernel scale does not generally carry over unchanged to another.

This is important when kernels are normalized, when a kernel is multiplied by a constant during model construction, or when a tuning value is transferred between software implementations. The key principle is precise: under the usual kernel-ridge objective with average squared error, multiplying the kernel by \(c>0\) requires multiplying the regularization parameter by \(c\) to preserve the fitted function.

Kernel Rescaling and the Covariance Spectrum

Let \(K\) be a measurable, symmetric, positive semidefinite kernel on an input space with distribution \(P_X\). Assume \(E[K(X,X)]<\infty\), so its covariance (integral) operator on \(L_2(P_X)\) is positive and trace class. It is defined by

Write the nonzero eigenvalues of \(T_K\), counted with multiplicity, as \(\mu_1,\mu_2,\ldots\). For a positive constant \(c\), define \(K_c=cK\).

The covariance operator is linear in the kernel. Thus the spectrum scales by \(c\), while the effective dimension is preserved only when its regularization argument scales by the same factor.

To verify the effective-dimension identity, use the eigenvalues of \(T_K\) to write \(N_{K_c}(\lambda)=\sum_j c\mu_j/(c\mu_j+\lambda)\). Dividing numerator and denominator of each summand by \(c\) gives \(\sum_j\mu_j/(\mu_j+\lambda/c)=N_K(\lambda/c)\). The sums are well-defined because the operators are positive and trace class, and each summand is bounded above by \(c\mu_j/\lambda\), whose sum is finite.

What Rescaling Does to the RKHS Penalty

The spectrum is only one part of the calibration. Kernel ridge regression penalizes the RKHS norm, and rescaling the kernel changes that norm too. Let \(\mathcal H_K\) denote the reproducing kernel Hilbert space of \(K\). For \(K_c=cK\), the two RKHSs contain the same functions, but their inner products differ.

Here is a proof. Give the common set of functions the inner product \(\langle f,g\rangle_{\mathcal H_{K_c}}=c^{-1}\langle f,g\rangle_{\mathcal H_K}\). This is an inner product and makes the space complete, since multiplying a norm by a positive constant does not change completeness. For each input \(x\), let \(K_x=K(\,\cdot\,,x)\). The section of \(K_c\) at \(x\) is \(cK_x\). By the reproducing property in \(\mathcal H_K\),

Thus the proposed inner product has reproducing kernel \(K_c\), so it gives its RKHS. The norm identity follows directly from the definition of that inner product. \(\square\)

Consider kernel ridge regression with training inputs \(x_1,\ldots,x_n\), responses collected in \(y\), and the average-loss objective

If the kernel is replaced by \(K_c\), its objective with parameter \(\lambda_c\) uses the penalty \(\lambda_c\|f\|_{\mathcal H_{K_c}}^2=(\lambda_c/c)\|f\|_{\mathcal H_K}^2\). It matches the original objective exactly when \(\lambda_c/c=\lambda\).

This equality of objectives proves equality of their minimizers whenever the minimizer is unique. For \(\lambda>0\), the squared RKHS norm makes the objective strictly convex in \(f\), so its minimizer is unique.

Gram-Matrix Verification

The same calibration can be checked directly for the fitted predictor. Let \(G\) be the \(n\times n\) Gram matrix with entries \(G_{ij}=K(x_i,x_j)\). Kernel ridge regression with the average-loss convention has coefficient vector \(\alpha=(G+n\lambda I_n)^{-1}y\), and prediction \(\hat f(x)=\sum_{i=1}^n\alpha_iK(x_i,x)\).

With \(K_c=cK\), the Gram matrix becomes \(cG\). Choosing \(\lambda_c=c\lambda\) gives

The new prediction is therefore \(\sum_i(\alpha_i/c)cK(x_i,x)=\sum_i\alpha_iK(x_i,x)\), identical to the original prediction for every \(x\). This derivation also shows where the factor \(n\) appears: it comes from using average squared error. If the objective instead uses the sum of squared errors, the Gram formula has \(\lambda I_n\) rather than \(n\lambda I_n\), but the kernel-scale conversion remains \(\lambda_c=c\lambda\).

Worked Spectrum and Effective Dimension

Worked Example: A Small Kernel Scale

Common Mistakes and Practical Interpretation

• Keeping \(\lambda\) fixed after changing kernel amplitude. The same numerical value does not represent the same regularization. If the kernel is multiplied by \(c\), the matching value is \(c\lambda\).
• Scaling \(\lambda\) in the opposite direction. The RKHS norm squared becomes \(1/c\) as large for the same function, so the coefficient on that norm must become \(c\lambda\), not \(\lambda/c\).
• Comparing eigenvalues without comparing the regularization scale. The relevant effective-dimension terms are \(c\mu_j/(c\mu_j+\lambda_c)\). Matching them requires matching \(\lambda_c/c\), not merely noting that the eigenvalues have changed.
• Assuming shape determines a tuning value. Two kernels can have the same relative pattern of similarities and different overall amplitudes. Tuning values are meaningful only relative to the kernel scale and the objective convention.

In practice, a reported tuning value is not portable without its kernel normalization and loss convention. A data-dependent normalization can even make the scale vary across analyses, so the calibration must use the actual multiplicative factor applied to the kernel. This scaling identity does not say that rescaling improves prediction; it says that a pure change of units leaves the fit unchanged only when regularization is converted consistently.