Mercer-Series Risk Calculations with Eigenvalue Multiplicities

Why Multiplicities Matter

A Mercer spectrum is often summarized by how quickly its eigenvalues decrease. But when several eigenfunctions share an eigenvalue, each one contributes to the prediction risk. The relevant sums therefore count eigenvalues with their multiplicities, not merely the number of distinct eigenvalues. This distinction can change the order of an effective dimension or variance calculation.

There is a second point to keep explicit: positive-eigenvalue eigenfunctions need not span the whole prediction space. Kernel ridge shrinkage cannot recover a target component in the null space of the kernel operator. Its squared norm remains part of the prediction bias, no matter how the positive eigenvalues are grouped.

Blockwise Mercer Coordinates

Let $T$ be a positive, compact, trace-class kernel integral operator on $L^2(P)$. Write its distinct positive eigenvalues as $\mu_b$, with multiplicity $m_b$, and choose orthonormal eigenfunctions $\phi_{br}$, $1\le r\le m_b$, so that $T\phi_{br}=\mu_b\phi_{br}$. Assume these eigenfunctions, together with an orthonormal basis for $\ker(T)$, give a complete orthonormal basis of $L^2(P)$. Trace class means $\sum_b m_b\mu_b<\infty$.

For a target $f\in L^2(P)$, let $f_{br}=\langle f,\phi_{br}\rangle$, and let $f_0=\Pi_{\ker(T)}f$ be its projection onto the null space. In population coordinates, ridge regularization at $\lambda>0$ multiplies each positive-eigenvalue coefficient by $\mu_b/(\mu_b+\lambda)$; its null-space component is zero. To calculate a noise contribution, use the following diagonal sequence model:

Definition: In the diagonal noise model, the estimated coefficient is $\hat f_{br}=\frac{\mu_b}{\mu_b+\lambda}f_{br}+\frac{\xi_{br}}{\mu_b+\lambda}$, where $E[\xi_{br}]=0$, $\operatorname{Var}(\xi_{br})=\sigma^2\mu_b/n$, and the estimator has zero null-space component. These conditions specify the coefficientwise risk calculation below; they do not assert that an empirical KRR fit has this exact finite-sample distribution.

Definition: The squared $L^2(P)$ bias and effective dimension are $$ B^2(\lambda;f)=\|f_0\|_{L^2(P)}^2+ \sum_b\sum_{r=1}^{m_b} \left(\frac{\lambda}{\mu_b+\lambda}\right)^2 f_{br}^2, \qquad \mathcal{N}(\lambda)=\sum_b m_b\frac{\mu_b}{\mu_b+\lambda}. $$

The effective dimension counts each eigenfunction separately: the contribution of a block is $m_b\mu_b/(\mu_b+\lambda)$. It is finite because $\mu_b/(\mu_b+\lambda)\leq\mu_b/\lambda$ and $\sum_bm_b\mu_b<\infty$.

Risk Formula and Its Derivation

Theorem: Under the diagonal noise model, the expected squared prediction error is $$ E\!\left[\|\hat f-f\|_{L^2(P)}^2\right] = \|f_0\|_{L^2(P)}^2+ \sum_b\sum_{r=1}^{m_b} \left(\frac{\lambda}{\mu_b+\lambda}\right)^2 f_{br}^2 +\frac{\sigma^2}{n}\sum_bm_b\frac{\mu_b}{(\mu_b+\lambda)^2}. $$

For a positive-eigenvalue coordinate, subtracting $f_{br}$ from the estimated coefficient gives $$ \hat f_{br}-f_{br} =-\frac{\lambda}{\mu_b+\lambda}f_{br} +\frac{\xi_{br}}{\mu_b+\lambda}. $$ The first term is deterministic and the second has mean zero. Consequently, $$ E[(\hat f_{br}-f_{br})^2] = \left(\frac{\lambda}{\mu_b+\lambda}\right)^2 f_{br}^2 +\frac{\operatorname{Var}(\xi_{br})}{(\mu_b+\lambda)^2} = \left(\frac{\lambda}{\mu_b+\lambda}\right)^2 f_{br}^2 +\frac{\sigma^2}{n}\frac{\mu_b}{(\mu_b+\lambda)^2}. $$

On the null space, the estimator is zero, so its error is $-f_0$ and its squared error is $\|f_0\|_{L^2(P)}^2$. Parseval’s identity adds the squared errors over the complete orthonormal basis. Taking expectations of the nonnegative sum is justified by Tonelli’s theorem. The bias sum is bounded by $\|f\|_{L^2(P)}^2$; the variance sum is finite because $\mu_b/(\mu_b+\lambda)^2\leq\mu_b/\lambda^2$ and the operator is trace class. This proves the formula. $\square$

The variance sum also has the useful blockwise form $\frac{\sigma^2}{n}\sum_bm_b\mu_b/(\mu_b+\lambda)^2$. It is not the effective dimension itself: each block contributes a different function of its eigenvalue. The inequality $\mu/(\mu+\lambda)^2\leq \lambda^{-1}\mu/(\mu+\lambda)$ gives the general comparison $$ \frac{\sigma^2}{n}\sum_bm_b\frac{\mu_b}{(\mu_b+\lambda)^2} \leq \frac{\sigma^2}{n\lambda}\mathcal{N}(\lambda). $$

Worked Example: Two Eigenvalue Blocks and a Null Component

Suppose $T$ has eigenvalue $4$ with multiplicity $2$, eigenvalue $1$ with multiplicity $3$, and a nontrivial null space. Set $\lambda=1$. Let the two coefficients in the first block be $1,2$, the three coefficients in the second block be $1,-1,2$, and suppose $\|f_0\|_{L^2(P)}^2=1/4$.

For eigenvalue $4$, the shrinkage factor is $4/(4+1)=4/5$, so the squared-bias factor is $(1/5)^2$. The squared coefficients sum to $1^2+2^2=5$, giving a positive-block bias contribution $5/25=1/5$. For eigenvalue $1$, the squared-bias factor is $(1/2)^2$, and the squared coefficients sum to $1+1+4=6$, giving $6/4=3/2$. Including the null space, the total squared bias is $1/4+1/5+3/2=39/20$.

The effective dimension is $2(4/5)+3(1/2)=8/5+3/2=31/10$. The variance factor before multiplying by $\sigma^2/n$ is $2(4/25)+3(1/4)=8/25+3/4=107/100$. Thus the risk is $39/20+(107/100)\sigma^2/n$. Omitting the null component would incorrectly reduce the squared bias by $1/4$.

How Multiplicities Change the Sums

When eigenvalues are listed with repetition as $\tilde\mu_1,\tilde\mu_2,\ldots$, the effective dimension can equivalently be written $\sum_j\tilde\mu_j/(\tilde\mu_j+\lambda)$. Block notation groups equal terms, replacing each repeated term by its multiplicity. The grouping is only bookkeeping; it must preserve every eigenfunction’s contribution.

Worked Example: Effective Dimension for Repeated Eigenvalues

Consider two blocks: eigenvalue $1$ with multiplicity $4$, and eigenvalue $1/4$ with multiplicity $2$. At $\lambda=1/2$, the first block contributes $4(1/(1+1/2))=4(2/3)=8/3$. The second contributes $2((1/4)/(1/4+1/2))=2(1/3)=2/3$. Hence $\mathcal{N}(1/2)=10/3$.

The variance sum, before its factor $\sigma^2/n$, is $4(1/(3/2)^2)+2((1/4)/(3/4)^2)=16/9+8/9=8/3$. Counting just the two distinct eigenvalues would instead give an effective dimension of $2/3+1/3=1$, which is not the dimension of the regularized feature space: it ignores the repeated directions.

Multiplicity can itself grow as the eigenvalues decrease. The following calculation shows how this changes the regularization dependence of both the effective dimension and the variance.

Worked Example: Polynomial Eigenvalues With Growing Multiplicities

Let $\mu_b=b^{-4}$ and $m_b=b$, for $b=1,2,\ldots$. This is trace class because $\sum_bm_b\mu_b=\sum_bb^{-3}<\infty$. For small $\lambda$, put $B=\lfloor\lambda^{-1/4}\rfloor$; the indices up to $B$ have $\mu_b$ at least approximately $\lambda$.

For $b\leq B$, $\mu_b\geq\lambda$, so $\mu_b/(\mu_b+\lambda)$ lies between $1/2$ and $1$. The effective-dimension contribution of these indices is therefore between $\frac12\sum_{b\leq B}b$ and $\sum_{b\leq B}b$, both of order $B^2$. For $b>B$, use $\mu_b/(\mu_b+\lambda)\leq\mu_b/\lambda$; the remaining contribution is at most $\lambda^{-1}\sum_{b>B}b^{-3}$, which is of order $\lambda^{-1}B^{-2}$, also of order $B^2$. Since $B$ is of order $\lambda^{-1/4}$, this proves $\mathcal{N}(\lambda)$ is of order $\lambda^{-1/2}$.

For the variance sum, when $b\leq B$, $\mu_b\geq\lambda$ implies $\frac{1}{4\mu_b}\leq\frac{\mu_b}{(\mu_b+\lambda)^2}\leq\frac{1}{\mu_b}$. After multiplying by $m_b=b$, summing these bounds gives order $\sum_{b\leq B}b^5$, or $B^6$. For $b>B$, the summand satisfies $\mu_b/(\mu_b+\lambda)^2\leq\mu_b/\lambda^2$, so the tail is at most $\lambda^{-2}\sum_{b>B}b^{-3}$, again of order $B^6$. A lower bound of order $B^6$ already comes from the first $B$ indices. Thus the variance sum is of order $B^6=\lambda^{-3/2}$, and the risk variance is of order $(\sigma^2/n)\lambda^{-3/2}$. Both orders reflect the growing multiplicities.

Common Pitfalls and Interpretation

Dropping multiplicities. Summing once per distinct eigenvalue undercounts both effective dimension and variance. Use $m_b$ in every block contribution.
Assuming positive eigenfunctions are complete. If $\ker(T)$ is nontrivial, its target projection contributes $\|f_0\|^2$ to the squared bias. It is not represented by the positive-eigenvalue sum.
Calling a sequence-model identity an empirical-sample identity. The risk theorem above follows from the stated coefficientwise noise model. Empirical KRR can have additional dependence and random-design effects; the formula should not be applied to it without suitable assumptions and a separate argument.

These calculations matter because effective dimension summarizes how many directions are appreciably retained at a given regularization level, while the variance sum records how strongly noise in those directions affects prediction. When eigenvalues repeat or multiplicities grow, the block structure is part of the spectrum’s statistical behavior, not a cosmetic detail.

Check Your Understanding

Work through each question by counting eigenfunctions, not just distinct eigenvalues.

Why can the positive-eigenvalue bias sum fail to equal the full squared bias when $T$ has a nontrivial null space?
A block has eigenvalue $\mu$ and multiplicity $m$. Write its contributions to effective dimension and to the variance sum.
For eigenvalue $2$ with multiplicity $3$, calculate the block’s effective-dimension contribution at $\lambda=1$.
In the polynomial-spectrum example, what role does the cutoff $B$ play in separating the two parts of each sum?
Which assumption in the diagonal noise model determines the coefficientwise variance contribution, and why does it not by itself establish an empirical KRR risk formula?