Effective Dimension and the Variance Term

Why Count Directions by Their Contribution?

A kernel covariance operator can have infinitely many eigen-directions. Yet kernel ridge regression does not use all of them equally: regularization suppresses directions whose covariance eigenvalues are small relative to the regularization parameter. Counting every nonzero eigenvalue equally would therefore miss the central feature of the estimator. Effective dimension replaces that crude count with a sum of fractional contributions, one for each direction.

This quantity is especially useful when interpreting the variance side of kernel ridge risk. It describes how many directions remain statistically active at a chosen regularization scale. It is not, by itself, an exact variance formula in every sampling model. In particular, an exact variance calculation can involve squared weights, while effective dimension uses the weights themselves. Keeping those quantities distinct is important.

Definition from the Covariance Operator

Let $\mathcal H$ be a separable reproducing kernel Hilbert space with kernel $k$, and let $X$ have the input distribution. Write $k_X=k(X,\cdot)\in\mathcal H$. Assume $E\|k_X\|_{\mathcal H}^2<\infty$. The population covariance operator $T:\mathcal H\to\mathcal H$ is

T=E[k_X\otimes k_X], \qquad (k_X\otimes k_X)g=\langle g,k_X\rangle_{\mathcal H}k_X.

Under the stated moment condition, $T$ is positive, self-adjoint, and trace class. In particular, it is compact and its positive eigenvalues, repeated according to multiplicity, can be listed as $\mu_1\geq\mu_2\geq\cdots>0$, with $\sum_j\mu_j<\infty$. The list may be finite. For a regularization parameter $\lambda>0$, the operator $T+\lambda I$ is invertible and its regularized inverse is bounded.

Definition: The effective dimension at regularization level $\lambda>0$ is $$ \mathcal N(\lambda)=\operatorname{Tr}\!\left(T(T+\lambda I)^{-1}\right). $$ It is the trace of the regularized covariance operator. Equivalently, it is the sum of the regularized covariance eigenvalues after each is scaled by its eigenvalue plus $\lambda$.

The trace is well defined and finite: $T(T+\lambda I)^{-1}$ is positive trace class, since its eigenvalues are bounded above by $\mu_j/\lambda$, and $\sum_j\mu_j/\lambda<\infty$. The notation $\mathcal N(\lambda)$ emphasizes that this is a scale-dependent dimension, not necessarily the algebraic rank of $T$.

Spectral Representation and Interpretation

Theorem: If $T$ is positive and trace class with positive eigenvalues $(\mu_j)$, then for every $\lambda>0$, $$ \mathcal N(\lambda)=\sum_j\frac{\mu_j}{\mu_j+\lambda}. $$ Each eigen-direction contributes a number between zero and one.

Proof. By the spectral theorem for compact self-adjoint operators, choose an orthonormal eigenbasis for the closure of the range of $T$, with $Te_j=\mu_j e_j$. On $e_j$, the operator $T+\lambda I$ has eigenvalue $\mu_j+\lambda$; its inverse has eigenvalue $1/(\mu_j+\lambda)$. Therefore $T(T+\lambda I)^{-1}$ has eigenvalue $\mu_j/(\mu_j+\lambda)$ on $e_j$. On the null space of $T$, it has eigenvalue zero. Taking its trace gives the stated sum. The sum converges because \[ 0\leq \frac{\mu_j}{\mu_j+\lambda}\leq\frac{\mu_j}{\lambda} \] and $T$ is trace class. $\square$

The ratio $\mu_j/(\mu_j+\lambda)$ is close to one when $\mu_j$ is much larger than $\lambda$, and close to zero when $\mu_j$ is much smaller. Thus directions with substantial covariance relative to the regularization scale count almost fully; weak directions count only fractionally. This is the sense in which effective dimension measures the number of directions contributing.

Worked Example: A Three-Direction Covariance

Suppose $T$ has eigenvalues $4,1,\frac14$, and take $\lambda=1$. Compute its effective dimension.

The three directional contributions are \[ \frac{4}{4+1}=\frac45,\qquad \frac{1}{1+1}=\frac12,\qquad \frac{1/4}{1/4+1}=\frac15. \] Therefore \[ \mathcal N(1)=\frac45+\frac12+\frac15 =\frac{8}{10}+\frac{5}{10}+\frac{2}{10} =\frac{15}{10}=\frac32. \] The operator has rank three, but its effective dimension at this scale is $1.5$: the first direction contributes $0.8$, the second $0.5$, and the third $0.2$. This value is a weighted count, not a claim that the space literally has dimension $1.5$.

How Regularization Changes Effective Dimension

Theorem: For positive trace-class $T$, $\mathcal N(\lambda)$ is nonincreasing in $\lambda$. If $T$ has finite rank $r$, then $$ \lim_{\lambda\downarrow0}\mathcal N(\lambda)=r, \qquad \lim_{\lambda\to\infty}\mathcal N(\lambda)=0. $$ For any trace-class $T$, $0\leq \mathcal N(\lambda)\leq \operatorname{Tr}(T)/\lambda$.

Proof. For each positive eigenvalue $\mu_j$, the function $w_j(\lambda)=\mu_j/(\mu_j+\lambda)$ decreases as $\lambda$ increases, since \[ w_j'(\lambda)=-\frac{\mu_j}{(\mu_j+\lambda)^2}<0. \] Summing these nonincreasing terms shows that $\mathcal N(\lambda)$ is nonincreasing. The trace bound follows term by term from \[ \frac{\mu_j}{\mu_j+\lambda}\leq\frac{\mu_j}{\lambda}, \] and summing gives $\mathcal N(\lambda)\leq \operatorname{Tr}(T)/\lambda$.

If $T$ has finite rank $r$, there are exactly $r$ positive eigenvalues. For each, $\mu_j/(\mu_j+\lambda)\to1$ as $\lambda\downarrow0$, so the finite sum tends to $r$. As $\lambda\to\infty$, each term tends to zero, and the finite sum tends to zero. $\square$

For infinite-rank $T$, the same finite-rank limit must not be applied without care. In fact, $\mathcal N(\lambda)$ tends to infinity as $\lambda\downarrow0$ when infinitely many eigenvalues are positive: for any chosen number $m$, the first $m$ terms each tend to one, so the limit inferior is at least $m$, and $m$ is arbitrary. At every fixed positive $\lambda$, however, the trace-class condition still ensures that the effective dimension is finite.

Worked Example: The Same Spectrum at Two Scales

Let the positive eigenvalues be $3,1,\frac13$. Compare effective dimension at $\lambda=1$ and $\lambda=3$.

At $\lambda=1$, \[ \mathcal N(1)=\frac{3}{4}+\frac{1}{2}+\frac{1/3}{1+1/3} =\frac34+\frac12+\frac14=\frac32. \] At $\lambda=3$, \[ \mathcal N(3)=\frac{3}{6}+\frac{1}{4}+\frac{1/3}{3+1/3} =\frac12+\frac14+\frac{1}{10} =\frac{17}{20}=0.85. \] The larger regularization level reduces every directional weight, so the total decreases from $1.5$ to $0.85$. The small-eigenvalue direction is most strongly suppressed relative to its contribution at the smaller regularization level.

Connection to the Kernel Ridge Variance Term

The definition explains the geometry of regularization; a simple population score-noise calculation makes the variance connection more explicit. Let $A=(T+\lambda I)^{-1}$, and consider a centered random perturbation $\xi\in\mathcal H$ with covariance operator $\operatorname{Cov}(\xi)=(\sigma^2/n)T$. Such a covariance has the same eigen-directions as $T$, with noise variance proportional to $\mu_j$ in direction $e_j$. The induced perturbation in the fitted element is $\delta=A\xi$. Measure prediction error in the $T$-seminorm, $\|\delta\|_T^2=\langle\delta,T\delta\rangle_{\mathcal H}$.

Derived identity: Under these covariance assumptions, $$ E\|\delta\|_T^2 =\frac{\sigma^2}{n}\sum_j\left(\frac{\mu_j}{\mu_j+\lambda}\right)^2. $$ This squared-weight factor is bounded above by $(\sigma^2/n)\mathcal N(\lambda)$, but it is generally not equal to it.

To verify the identity, use $Ae_j=(\mu_j+\lambda)^{-1}e_j$. The coordinate $\langle \xi,e_j\rangle$ has mean zero and variance $(\sigma^2/n)\mu_j$. Hence the $e_j$ coordinate of $\delta=A\xi$ has variance $(\sigma^2/n)\mu_j/(\mu_j+\lambda)^2$. Prediction error weights its squared coordinate by $\mu_j$, so the expected contribution from this direction is \[ \mu_j\frac{\sigma^2}{n}\frac{\mu_j}{(\mu_j+\lambda)^2} =\frac{\sigma^2}{n}\left(\frac{\mu_j}{\mu_j+\lambda}\right)^2. \] Summing the nonnegative coordinate contributions gives the identity. The sum is finite because each squared weight is at most $\mu_j/\lambda$. Finally, $0\leq\mu_j/(\mu_j+\lambda)\leq1$, so its square is no larger than itself; summing proves the stated comparison. This calculation isolates the role of effective dimension without asserting that every finite-sample kernel ridge variance equals this population expression.

Worked Example: Effective Dimension Versus Squared Weights

Take three eigenvalues whose ratios $\mu_j/\lambda$ are $32,8,2$. Their effective-dimension weights are $32/33,8/9,2/3$, so \[ \mathcal N(\lambda)=\frac{32}{33}+\frac89+\frac23 =\frac{96}{99}+\frac{88}{99}+\frac{66}{99} =\frac{250}{99}\approx2.525253. \] For the population score-noise calculation, the squared weights sum to \[ \left(\frac{32}{33}\right)^2+ \left(\frac89\right)^2+ \left(\frac23\right)^2 =\frac{1024}{1089}+\frac{64}{81}+\frac49. \] Using common denominator $9801$, note that $1089\cdot9=9801$ and $81\cdot121=9801$. Thus \[ \frac{1024}{1089}=\frac{9216}{9801},\qquad \frac{64}{81}=\frac{7744}{9801},\qquad \frac49=\frac{4356}{9801}, \] and the squared-weight sum is \[ \frac{9216+7744+4356}{9801} =\frac{21316}{9801}\approx2.174880. \] It is indeed smaller than the effective dimension $250/99\approx2.525253$, as the term-by-term inequality predicts. The first quantity is the variance factor in this specified calculation; the second is the effective dimension.

What the Quantity Does—and Does Not—Say

Effective dimension is useful because it compresses an entire spectrum into a single number at the scale set by $\lambda$. When the covariance has many eigenvalues much larger than $\lambda$, many directions count nearly fully. When most eigenvalues are much smaller, the effective dimension can be far below the rank. This helps explain why regularization can limit variance even when the underlying RKHS is infinite-dimensional.

A common mistake is to say that $\mathcal N(\lambda)$ is the exact number of fitted directions, or to substitute it automatically for the variance. Neither statement is generally correct. It is a trace-weighted dimension, and a variance calculation depends on the covariance of the noise and the way the estimator uses the data. In the score-noise calculation above the factors are squared; effective dimension bounds their sum because each weight lies in $[0,1]$. Finite-sample bounds require additional control of the empirical covariance operator and noise, so they do not follow from the trace identity alone.

A second pitfall is to confuse $\operatorname{Tr}(T)$ with effective dimension. The trace sums raw eigenvalues, whereas effective dimension sums $\mu_j/(\mu_j+\lambda)$. Rescaling the kernel changes the eigenvalues and therefore changes the effective dimension at a fixed $\lambda$; the regularization scale must be interpreted relative to the covariance scale.

Key takeaway: $\mathcal N(\lambda)=\operatorname{Tr}(T(T+\lambda I)^{-1})=\sum_j\mu_j/(\mu_j+\lambda)$ is a regularization-dependent, fractional count of covariance directions. It describes how many directions remain active and naturally controls related variance expressions, but it is not automatically identical to the variance itself.

Check Your Understanding

Use the spectral weights and assumptions stated above to answer these questions.

If $T$ has eigenvalues $2$ and $1/2$, what is $\mathcal N(1)$? Show each directional contribution.
Why is $\mathcal N(\lambda)$ finite for every $\lambda>0$ when $T$ is positive and trace class?
For a finite-rank operator, what does $\mathcal N(\lambda)$ approach as $\lambda$ decreases to zero, and why?
In the population score-noise calculation, why is the sum of squared weights no larger than effective dimension?
Explain why effective dimension is not generally the algebraic rank of $T$ at a positive regularization level.