Effective Degrees of Freedom for Ridge

Why Ridge Has Effective, Rather Than Ordinary, Degrees of Freedom

In ordinary least squares, the fitted values are a projection of the observed response onto the space spanned by the model columns. The trace of that projection matrix equals the dimension of the space, so it counts the fitted model’s degrees of freedom. Ridge regression changes the fitted values by shrinking contributions from predictor directions. Its smoother matrix is generally not a projection, and its trace need not be an integer. That trace provides a useful measure of how much flexibility the ridge fit uses.

The key is to define the model and penalty precisely. Here, the predictors are centered, the intercept is not penalized, and the ridge penalty is \(\lambda\) times the sum of squared slope coefficients. For \(\lambda>0\), the ridge fit is a linear function of the response. Its smoother matrix makes it possible to quantify the fit’s complexity, while a singular-value decomposition reveals which predictor directions contribute most.

The Ridge Model and Its Smoother Matrix

Let \(y\) be the \(n\)-vector of observed responses and let \(X\) be the \(n\times p\) design matrix of centered predictor columns. With an intercept, write the ridge objective as

Here \(\beta_0\) is the intercept, \(\boldsymbol{\beta}\) is the \(p\)-vector of slopes, and \(\lambda\) is the penalty parameter for this stated objective. Since the columns of \(X\) are centered, the intercept estimate is \(\bar{y}\); equivalently, the slope fit can be obtained by fitting the centered response \(y_c=y-\bar{y}\mathbf{1}\). This separates the intercept from the predictor contribution.

For \(\lambda>0\), the matrix \(X^\mathsf{T}X+\lambda I_p\) is invertible, even if \(X\) is rank deficient. The centered ridge fitted values are \(X(X^\mathsf{T}X+\lambda I_p)^{-1}X^\mathsf{T}y_c\). Adding back the intercept gives the full fitted-value smoother matrix:

The first term maps \(y\) to its mean in every observation, and the second maps the centered response to the ridge predictor fit. Because \(X\) is centered, its columns are orthogonal to \(\mathbf{1}\), so these contributions are separated. The intercept term has trace one. At \(\lambda=0\), do not use the displayed inverse formula unless \(X^\mathsf{T}X\) is known to be invertible. For any rank, the zero-penalty predictor fit is instead the least-squares projection onto \(\operatorname{col}(X)\).

Deriving the Singular-Value Formula

Let \(r=\operatorname{rank}(X)\), and write a reduced singular-value decomposition using only the positive singular values:

The columns of \(U_r\) are orthonormal vectors in observation space, and those of \(V_r\) are orthonormal vectors in predictor space. Substituting this decomposition into the predictor part of the smoother and using \(V_r^\mathsf{T}V_r=I_r\) gives

Taking the trace uses the fact that for compatible matrices, \(\operatorname{tr}(AB)=\operatorname{tr}(BA)\). The trace of the intercept term is one, and each diagonal entry in the singular-direction representation contributes its shrinkage weight. Therefore:

The zero-penalty result also follows by taking the limit in the singular-value formula: each positive singular value contributes \(d_j^2/(d_j^2+0)=1\). Singular directions with zero singular value are not included in the sum, since they do not belong to the column space of \(X\). Thus the zero-penalty statement remains valid when \(X\) is rank deficient.

Worked Calculations

How the Penalty Changes Effective Degrees of Freedom

Each predictor direction contributes the fraction \(d_j^2/(d_j^2+\lambda)\). For \(\lambda\geq 0\), this contribution is one at zero penalty, lies strictly between zero and one for positive penalty, and approaches zero as the penalty grows without bound. The intercept remains unpenalized under the model defined here. For \(\lambda>0\), differentiating the sum makes the direction of change explicit:

Thus increasing \(\lambda\) strictly decreases effective degrees of freedom when the predictor design has positive rank. At zero penalty, the value is \(1+r\); as \(\lambda\) tends to infinity, it approaches one, the intercept contribution. If there are no predictor directions, \(r=0\), it is one for every penalty. Without an intercept, the corresponding predictor-only formula omits the leading one.

Effective degrees of freedom can be fractional because ridge shrinks directions rather than simply retaining or discarding them. The trace summarizes the total influence of the response on its fitted values. It is useful for describing model flexibility, but it is not the number of nonzero slope coefficients and should not be interpreted as such. For a fixed design and penalty, it depends on the design’s singular values; changing predictor scaling or the penalty convention changes the numerical setup and must be accounted for when comparing values.

Common Mistake

A common error is to substitute \(\lambda=0\) into \((X^\mathsf{T}X+\lambda I_p)^{-1}\) even when the design is rank deficient. In that case \(X^\mathsf{T}X\) is singular and the inverse does not exist. The correction is to use the projection onto \(\operatorname{col}(X)\), equivalently \(XX^+\), at zero penalty; its trace is \(r\). Another related error is to count every predictor column as a separate direction. When columns are dependent, only the rank \(r\) counts at zero penalty, and only the positive singular values appear in the ridge formula.