Ridge Eigenvalue Regularization and Its Bias

Why Regularize Small Eigenvalues?

A sample covariance matrix can have very small eigenvalues, or be singular when the dimension is large relative to the sample size. This can make its inverse unstable or undefined. Two direct repairs are to raise every eigenvalue by the same amount, called ridge regularization, or to replace only eigenvalues below a chosen threshold, called eigenvalue flooring. Both can improve conditioning, but they alter the covariance estimate and therefore have a cost.

To compare that cost carefully, distinguish three quantities: the change from the observed sample covariance, the estimator's statistical bias relative to the population covariance, and the error in one particular sample. They are not interchangeable.

Definitions and Basic Quantities

Let \(X_1,\ldots,X_n\) be independent and identically distributed \(p\)-dimensional random vectors with mean \(\mu\) and covariance matrix \(\Sigma\). Assume \(n\geq 2\) and \(E\|X_i\|^2<\infty\). The unbiased sample covariance is

and \(E[S]=\Sigma\). Since \(S\) is symmetric and positive semidefinite, write its eigenvalues in descending order as \(\lambda_1\geq\cdots\geq\lambda_p\geq 0\), with orthonormal eigenvectors \(u_1,\ldots,u_p\). Thus \(S=\sum_{i=1}^p\lambda_i u_i u_i^\top\).

Both estimators are positive definite. Flooring changes only eigenvalues below \(\tau\); ridge increases every eigenvalue, including the largest. For a positive-definite matrix \(A\), its spectral condition number is \(\kappa(A)=\lambda_{\max}(A)/\lambda_{\min}(A)\). The Frobenius norm is \(\|A\|_F=(\sum_{i,j}A_{ij}^2)^{1/2}\).

Conditioning and Frobenius Change

Proof. Both regularized matrices have the same eigenvectors as \(S\), with eigenvalues given by their definitions. The largest and smallest floored eigenvalues are \(\max(\lambda_1,\tau)\) and \(\max(\lambda_p,\tau)\); the ridge eigenvalues are ordered as before, so their extremes are \(\lambda_1+\rho\) and \(\lambda_p+\rho\). Dividing each largest eigenvalue by its smallest gives the stated condition numbers.

In the eigenbasis of \(S\), the flooring difference has eigenvalues \((\tau-\lambda_i)_+\), while the ridge difference is \(\rho I_p\). For a symmetric matrix, the squared Frobenius norm is the sum of its squared eigenvalues. Therefore the squared norms of the two differences are respectively \(\sum_i(\tau-\lambda_i)_+^2\) and \(p\rho^2\). \(\square\)

The ridge condition number is strictly smaller than the original condition number when \(\lambda_1>\lambda_p\): adding \(\rho\) increases the denominator proportionally more than the numerator. Flooring leaves the largest eigenvalue unchanged when \(\lambda_1\geq\tau\), while lifting the minimum to at least \(\tau\). These formulas also show why a matching value of \(\rho\) and \(\tau\) does not mean matching modifications.

Statistical Bias: What Can Be Said Exactly?

The statistical bias of an estimator \(\widehat\Sigma\) is the matrix \(E[\widehat\Sigma]-\Sigma\), when that expectation exists. For ridge, the bias is particularly simple because the added matrix is deterministic. For flooring, the adjustment depends on the random eigenvalues and eigenvectors of \(S\); flooring the population eigenvalues is generally not the same operation as taking the expectation of a floored sample covariance.

Proof. Linearity of expectation gives \(E[S+\rho I_p]-\Sigma=E[S]+\rho I_p-\Sigma=\rho I_p\). Its squared Frobenius norm is \(p\rho^2\).

For flooring, \(S_{\mathrm{floor}}(\tau)-S=\sum_i(\tau-\lambda_i)_+u_i u_i^\top\) is positive semidefinite for every observed \(S\), since each coefficient is nonnegative. For any fixed vector \(v\), it follows that \(v^\top(S_{\mathrm{floor}}(\tau)-S)v\geq0\). Taking expectations preserves this inequality. Also, using \(E[S]=\Sigma\), the bias is \(E[S_{\mathrm{floor}}(\tau)-S]\), as claimed. \(\square\)

If fourth moments are finite, the squared Frobenius risks in the following identity are finite. For any unbiased \(S\) and any estimator \(\widehat\Sigma\) with finite squared Frobenius error, expanding the square around its expectation gives

The cross term vanishes because \(E[\widehat\Sigma-E[\widehat\Sigma]]=0\). For ridge, the variance term is unchanged from that of \(S\), since subtracting an expectation removes the fixed shift \(\rho I_p\). Consequently,

Thus ridge has a precise Frobenius bias, and under squared Frobenius loss it increases the risk of the unbiased sample covariance by exactly \(p\rho^2\). Its benefit is improved conditioning, not a reduction in this particular loss relative to \(S\). Flooring has a different bias-variance tradeoff; the formulas above do not determine its total risk without further information about the distribution of \(S\).

A Nearly Singular Gaussian Comparison

Consider a Gaussian population with covariance \(\Sigma=\operatorname{diag}(4,1,0.01)\). Its condition number is \(4/0.01=400\), so the third direction has very little population variance. Suppose a realized sample covariance is \(S=\operatorname{diag}(3.6,0.8,0.002)\). This is an illustration of one observed matrix, not a claim about the expected error of either estimator. Use \(\tau=\rho=0.1\).

Choosing and Interpreting a Regularization

The floor \(\tau\) has the units of a covariance eigenvalue: it sets the minimum variance allowed in any direction. The ridge parameter \(\rho\) has the same units, but it adds that amount in every direction. Both choices therefore depend on the scale of the variables. Rescaling the variables changes the covariance eigenvalues and should prompt corresponding reconsideration of the regularization amount.

A floor can be attractive when the goal is to avoid a small denominator without unnecessarily changing large eigenvalues. Ridge is simple and shifts every direction uniformly in the sample eigenbasis. The appropriate choice depends on how the regularized matrix will be used and on the loss of interest. Conditioning alone does not determine statistical accuracy.

Common Pitfall