Choosing Lambda by Bias–Variance Balance

Why Choose Regularization by Balancing?

In kernel ridge regression, the regularization parameter $\lambda$ controls a tradeoff. Larger values shrink the fitted function more strongly, typically increasing approximation bias; smaller values admit more complexity, which can increase variance. The useful scale is therefore not determined by bias or variance alone, but by how their bounds change together as $\lambda$ varies.

The purpose here is to optimize a risk bound, not to claim that its minimizer is the exact minimizer of the unknown prediction risk. The ingredients are the source-condition bias bound and the effective-dimension variance bound. Their balance gives a principled regularization scale once the smoothness and effective-dimension growth are specified.

Risk Bound and Source-Condition Bias

Let $T$ be a bounded, positive, self-adjoint covariance operator on a Hilbert space, and let $\|T\|$ denote its operator norm. For the effective-dimension expression below, assume also that $T$ is trace class. Define the prediction norm of $h$ by $\|T^{1/2}h\|$. A source condition writes the target as $f_*=T^r g$, where $g$ has bounded norm and $r$ quantifies smoothness relative to $T$.

Definition: For $0<r\leq 1$, the source condition with radius $R$ is $f_*=T^r g$ for some $g$ satisfying $\|g\|\leq R$. The effective dimension at regularization level $\lambda>0$ is $$ \mathcal{N}(\lambda)=\operatorname{Tr}\!\left[T(T+\lambda I)^{-1}\right]. $$

In the population ridge problem, the regularized target is $f_\lambda=(T+\lambda I)^{-1}Tf_*$. Its bias is $f_\lambda-f_*=-\lambda(T+\lambda I)^{-1}f_*$. The following bound makes the covariance scale explicit; omitting it can give an incorrect prediction-norm bound when $\|T\|$ is not at most one.

Result: Under the source condition above, $$ \|T^{1/2}(f_\lambda-f_*)\|^2\leq \|T\|R^2\lambda^{2r}. $$ In particular, if $\|T\|\leq 1$, the right-hand side is at most $R^2\lambda^{2r}$.

To verify the result, use the spectral theorem for $T$. For each spectral value $t\geq 0$, the multiplier in the Hilbert-space bias is $\lambda t^r/(t+\lambda)$. When $0<r\leq1$, this is at most $\lambda^r$: writing $u=t/\lambda$, it is $\lambda^r u^r/(1+u)$, and $u^r/(1+u)\leq1$ for both $u\leq1$ and $u\geq1$. Thus $\|f_\lambda-f_*\|\leq R\lambda^r$. Since $\|T^{1/2}h\|^2\leq\|T\|\|h\|^2$, the stated prediction-norm bound follows.

Suppose the available prediction-risk bound has the form $$ \mathcal{R}_n(\lambda)\leq A\lambda^{2r}+\frac{V}{n}\mathcal{N}(\lambda), $$ where $A>0$ is a bias constant, $V>0$ is a variance constant, and $n$ is the sample size. For the source bound just derived, one may take $A=\|T\|R^2$. The variance term’s dependence on $\mathcal{N}(\lambda)$ is the effective-dimension bound; the optimization below treats the displayed right-hand side as a proxy to minimize.

Power-Law Effective Dimension: The Balancing Rule

Suppose that for the regularization range of interest, an upper bound on effective dimension is $\mathcal{N}(\lambda)\leq C\lambda^{-q}$, with $C,q>0$. Setting $K=VC/n$, it is enough to minimize $G_n(\lambda)=A\lambda^{2r}+K\lambda^{-q}$. The first term increases with $\lambda$, while the second decreases.

Theorem: Let $A,K,r,q>0$. On the full domain $(0,\infty)$, the function $$ G_n(\lambda)=A\lambda^{2r}+K\lambda^{-q} $$ has a unique global minimizer $$ \lambda_*=\left(\frac{qK}{2rA}\right)^{1/(2r+q)}. $$ At this minimizer, $K\lambda_*^{-q}=(2r/q)A\lambda_*^{2r}$.

Proof. Differentiating on $(0,\infty)$ gives $$ G_n'(\lambda)=2rA\lambda^{2r-1}-qK\lambda^{-q-1}. $$ Since $\lambda^{q+1}>0$, the sign of this derivative is the sign of $2rA\lambda^{2r+q}-qK$. The latter expression is strictly increasing in $\lambda$, is negative as $\lambda$ approaches zero, and is positive for sufficiently large $\lambda$. It has exactly one zero, at the displayed $\lambda_*$. Therefore $G_n$ decreases up to $\lambda_*$ and increases after it. Also, $G_n$ tends to infinity at both domain endpoints, so this stationary point is the unique global minimizer. The equation defining it gives $2rA\lambda_*^{2r+q}=qK$; dividing by $q\lambda_*^q$ yields the stated balance. $\square$

Substituting $K=VC/n$, the scale is proportional to $n^{-1/(2r+q)}$ when the other constants are fixed. The minimized proxy is proportional to $n^{-2r/(2r+q)}$: indeed, at the optimum the variance term is $2r/q$ times the bias term, and $\lambda_*^{2r}$ is proportional to $n^{-2r/(2r+q)}$. These statements describe the optimized bound under the assumed effective-dimension inequality, not an unconditional guarantee that the actual risk has exactly that rate.

A domain restriction matters. If the allowed values are $0<\lambda<1$, the formula is a minimizer on that open interval only when $0<\lambda_*<1$. If $\lambda_*\geq1$, the function decreases throughout the interval, so its infimum is approached at the excluded endpoint $1$ and there is no minimizer on $(0,1)$. On the closed interval $0<\lambda\leq1$, the minimizer is $\min\{\lambda_*,1\}$. The same check must be made whenever the effective-dimension bound is valid only over a restricted range.

Worked Example: A Polynomial Effective-Dimension Bound

Suppose $r=1$, $A=2$, and $\mathcal{N}(\lambda)\leq 3\lambda^{-1/2}$. Let $V=4$ and $n=1000$. Find the minimizer of the resulting proxy and check the bias–variance balance.

Here $q=1/2$ and $K=VC/n=4(3)/1000=0.012$. The unconstrained minimizer is $$ \lambda_*=\left(\frac{(1/2)(0.012)}{2(1)(2)}\right)^{1/(2+1/2)} =(0.0015)^{0.4}\approx 0.0742. $$

This is inside $(0,1)$, so it is admissible if that is the chosen domain. The bias term is $2(0.0742)^2\approx0.0110$. The variance term is $0.012(0.0742)^{-1/2}\approx0.0441$. The balance predicts a variance-to-bias ratio of $2r/q=4$; the calculated terms have ratio approximately $0.0441/0.0110\approx4.01$, with the small discrepancy due to rounding. The proxy is approximately $0.0551$.

When the Effective Dimension Does Not Have a Power Law

The derivative method also applies when a different effective-dimension envelope is more appropriate. In particular, a logarithmic envelope leads to a different balance equation, while finite-rank effective dimension can remain bounded as $\lambda$ decreases. In those settings, applying the power-law formula without checking its assumptions can give a misleading scale.

Worked Example: Exact Optimization for a Rank-One Operator

Let $T$ have one eigenvalue equal to $1$ and all other eigenvalues equal to zero. Then $\mathcal{N}(\lambda)=1/(1+\lambda)$. For $A=4$, $r=1$, and $V/n=9$, minimize $$ G(\lambda)=4\lambda^2+\frac{9}{1+\lambda},\qquad 0<\lambda\leq1. $$

The derivative is $G'(\lambda)=8\lambda-9/(1+\lambda)^2$, and the second derivative is $G''(\lambda)=8+18/(1+\lambda)^3>0$. Thus the derivative is strictly increasing. At $\lambda=1/2$, $G'(1/2)=4-9/(3/2)^2=4-4=0$, so strict convexity makes $1/2$ the unique minimizer. Its proxy value is $4(1/2)^2+9/(3/2)=1+6=7$.

Here the variance contribution approaches $9$, rather than diverging, as $\lambda$ decreases to zero. The power-law divergence assumption used in the previous theorem does not describe this example; direct optimization gives the relevant balance.

Worked Example: A Logarithmic Effective-Dimension Envelope

Suppose $\mathcal{N}(\lambda)\leq 3\log(e/\lambda)$ for $0<\lambda\leq1$, with $A=2$, $r=1/2$, $V=10$, and $n=1000$. The proxy is $$ G(\lambda)=2\lambda+0.03\log(e/\lambda), $$ because $VC/n=10(3)/1000=0.03$.

Its derivative is $G'(\lambda)=2-0.03/\lambda$. This derivative is negative for $0<\lambda<0.015$, zero at $0.015$, and positive for $\lambda>0.015$. Therefore the unique minimizer on $(0,1]$ is $\lambda_*=0.015$. Equivalently, balancing the derivative gives $\lambda_*=K/(2rA)=0.03/2=0.015$.

The bias term there is $2(0.015)=0.03$. The variance-envelope term is $0.03\log(e/0.015)\approx0.03(5.1997)\approx0.1560$, so the proxy is approximately $0.1860$. Unlike the power-law case, the two terms themselves are not equal at this optimizer: the derivative of the logarithmic term, rather than equality of the terms, determines the balance.

Common Mistakes and Practical Interpretation

Dropping the covariance scale in the prediction bias. A bound in the ordinary Hilbert-space norm is not automatically the same bound in prediction norm. The inequality $\|T^{1/2}h\|^2\leq\|T\|\|h\|^2$ contributes the factor $\|T\|$. Only when $\|T\|\leq1$ can it be replaced by $1$ in this argument.
Using a stationary-point formula without checking the domain. Differentiation finds a candidate, not necessarily a minimizer on the allowed set. Check both that it lies in the range where the bound applies and whether the endpoints are included.
Calling the optimized proxy the exact risk. The effective-dimension and bias expressions are bounds. Their optimizer selects a scale for that bound; it need not coincide with the unknown risk minimizer.
Using a power-law balance for a non-power-law variance term. The formula depends on $\mathcal{N}(\lambda)\leq C\lambda^{-q}$. For logarithmic or finite-rank behavior, optimize the corresponding proxy instead.

The practical conclusion is conditional but useful: once a valid bias bound, a valid effective-dimension envelope, and a regularization domain are specified, minimizing their sum produces a transparent data-size and smoothness dependent choice. The constants matter for the numerical value, while the exponents determine how the scale changes with sample size. If the smoothness or envelope is uncertain, selecting $\lambda$ from this calculation alone does not resolve that uncertainty.

Check Your Understanding

Use the assumptions and distinctions made in this tutorial when answering.

Why does the prediction-norm bias bound include $\|T\|$, and under what condition can that factor be replaced by $1$?
For an envelope $\mathcal{N}(\lambda)\leq C\lambda^{-q}$, write the minimizer of the power-law proxy in terms of $A,V,C,n,r,$ and $q$.
At the power-law optimum, what is the ratio of the variance term to the bias term?
If the unconstrained minimizer is greater than $1$, what happens on the open domain $(0,1)$, and what happens on the closed domain $(0,1]$?
Why should the power-law optimizer not be applied directly to a logarithmic effective-dimension bound?