Effective Degrees of Freedom Along a Penalty Path

Why Measure Complexity Along a Penalty Path?

A penalty path displays fitted models over a range of penalty strengths. As the penalty increases, ridge coefficients are generally shrunk toward zero, while lasso coefficients can become exactly zero. The coefficient values describe how estimates change, but they do not directly summarize how much flexibility remains in the fitted model. Effective degrees of freedom provide a complexity measure that accounts for shrinkage as well as, in the lasso, selection of a smaller active set.

For an ordinary least-squares fit, the rank of the design matrix is a natural degrees-of-freedom count. For a penalized fit, counting coefficients alone can be misleading: a ridge coefficient may be nonzero but heavily shrunk. The trace of the fit’s response-to-fit map captures that shrinkage in a linear smoother. GLM fits are nonlinear in the data, so the corresponding ridge quantity is usually computed from a local Fisher-scoring approximation. Lasso GLM active-set counts are likewise useful approximations, not universal exact counts.

Definitions: Smoother Degrees of Freedom and the GLM Approximation

Definition: Suppose a fitted vector $\hat y$ is a linear function of a response vector $y$, so that $\hat y=Sy$ for a fixed matrix $S$. Its effective degrees of freedom are $\operatorname{tr}(S)$. More generally, for a differentiable fit $\hat y(y)$, the local degrees of freedom at $y$ are $\operatorname{tr}\{\partial\hat y/\partial y^\top\}$, when this derivative exists. The latter is a local sensitivity measure; its expectation is one standard definition of degrees of freedom for a data-dependent fitting rule, under conditions ensuring the expectation exists.

For a GLM, let $X$ be the design matrix, $o$ a known offset vector, $\eta=o+X\beta$ the linear predictor, and $\mu_i=E[Y_i]$. Consider a ridge penalty $\lambda\beta^\top D\beta/2$, where $\lambda\geq0$ and $D$ is positive semidefinite. Its zero diagonal entries can designate unpenalized coefficients, such as an intercept. At a fitted value, the Fisher-scoring weight for observation $i$ is $w_i=(d\mu_i/d\eta_i)^2/\{\phi V(\mu_i)\}$, with any prior observation weight included in $w_i$. Write $W=\operatorname{diag}(w_1,\ldots,w_n)$.

Definition: The local Fisher effective degrees of freedom for a ridge GLM fit are $$ \operatorname{df}_{\mathrm{ridge}}(\lambda) =\operatorname{tr}\left\{(X^\top W X+\lambda D)^{-1}X^\top W X\right\}, $$ when the inverse exists. The weights are evaluated at the fitted GLM means. This is the trace of the local weighted least-squares smoother; for a non-Gaussian GLM it is generally a local approximation to the sensitivity of the full nonlinear fit.

This definition follows from the working response $z_i=\eta_i+(y_i-\mu_i)d\eta_i/d\mu_i$. With a known offset, the ridge IRLS subproblem is

\frac{1}{2}\left\|W^{1/2}(z-o-X\beta)\right\|^2 +\frac{\lambda}{2}\beta^\top D\beta.

Thus the working least-squares target for the coefficients is $W^{1/2}(z-o)$, not $W^{1/2}z$. With the offset held fixed, the local fitted linear predictor is $o+X\hat\beta$; its derivative with respect to $z$ is $X(X^\top W X+\lambda D)^{-1}X^\top W$. Taking its trace gives the stated formula by cyclic invariance of trace.

Worked Example: A Ridge Trace From Information Eigenvalues

Consider a centered Gaussian regression with two penalized predictors. Suppose the eigenvalues of $X^\top X$ are $9$ and $4$, and use ridge penalty $\lambda\|\beta\|^2/2$. Find the effective degrees of freedom at $\lambda=3$, and compare it with the unpenalized value.

The ridge smoother is $S=X(X^\top X+\lambda I)^{-1}X^\top$. The two eigen-directions of $X^\top X$ contribute $d/(d+\lambda)$ to its trace. Therefore $$ \operatorname{df}_{\mathrm{ridge}}(3) =\frac{9}{9+3}+\frac{4}{4+3} =\frac{3}{4}+\frac{4}{7} =\frac{21+16}{28} =\frac{37}{28}\approx1.321. $$

At zero penalty, the ordinary least-squares fitted-value matrix is the projection onto the two-dimensional column space, so its trace is $2$. As the penalty rises from zero to $3$, the effective degrees of freedom fall from $2$ to approximately $1.321$, although both coefficients may remain nonzero.

Ridge Trace Formula and Its Limits

The eigenvalue calculation above is a general result for a fixed-design ridge smoother. It makes explicit why a nonzero ridge coefficient need not contribute a full degree of freedom: a direction with a small information eigenvalue is shrunk more strongly at the same penalty.

Theorem: Let $A=X^\top X$ be positive semidefinite with positive eigenvalues $d_1,\ldots,d_r$, where $r=\operatorname{rank}(X)$. For $\lambda>0$, the ridge smoother $S_\lambda=X(A+\lambda I)^{-1}X^\top$ has $$ \operatorname{tr}(S_\lambda)=\sum_{j=1}^{r}\frac{d_j}{d_j+\lambda}. $$ At $\lambda=0$, the ordinary least-squares fitted-value projection has trace $r$, including when $A$ is singular.

Proof. For $\lambda>0$, $A+\lambda I$ is positive definite, so its inverse exists. Cyclic invariance of trace gives $$ \operatorname{tr}(S_\lambda) =\operatorname{tr}\{(A+\lambda I)^{-1}X^\top X\} =\operatorname{tr}\{(A+\lambda I)^{-1}A\}. $$ Diagonalize $A=Q\operatorname{diag}(d_1,\ldots,d_r,0,\ldots,0)Q^\top$, with $Q$ orthogonal. In this basis, $(A+\lambda I)^{-1}A$ has diagonal entries $d_j/(d_j+\lambda)$ for $1\leq j\leq r$ and zero entries for the zero eigenvalues. Summing the diagonal entries proves the formula for positive $\lambda$. At zero penalty, the fitted values are the orthogonal projection of $y$ onto the column space of $X$. An orthogonal projection has eigenvalue $1$ on that space and $0$ on its orthogonal complement, hence trace $r$. This separate argument does not require $A$ to be invertible. $\square$

Each term $d_j/(d_j+\lambda)$ decreases as $\lambda$ increases, since its derivative is $-d_j/(d_j+\lambda)^2<0$. This monotonicity applies to the fixed-design ridge smoother. In a GLM, the weights $W$ themselves depend on the fitted means and can change along the path. The trace at each fit remains a useful local complexity summary, but the fixed-weight eigenvalue argument alone does not prove that the fully refitted GLM trace must decrease monotonically at every step.

Worked Example: Ridge GLM With an Unpenalized Intercept

At one fitted logistic-regression iteration, suppose the Fisher information $X^\top W X$, in intercept-then-slope order, is $\operatorname{diag}(4,9,1)$. The intercept is unpenalized and both slopes receive a unit ridge penalty, so $D=\operatorname{diag}(0,1,1)$. Calculate the local effective degrees of freedom for $\lambda=3$.

The penalized information matrix is $$ X^\top W X+3D=\operatorname{diag}(4,12,4). $$ Multiplying its inverse by the unpenalized information gives diagonal entries $1$, $9/12$, and $1/4$. Thus $$ \operatorname{df}_{\mathrm{ridge}}(3) =1+\frac{9}{12}+\frac{1}{4} =1+\frac{3}{4}+\frac{1}{4} =2. $$

The intercept contributes one because it is not penalized. The two slope directions contribute $0.75$ and $0.25$, respectively. The result is a local Fisher-scoring complexity measure at these weights, not an assertion that the nonlinear logistic fit is globally a linear smoother.

Lasso GLMs: Active-Set Approximations

For lasso, the objective includes $\lambda\sum_{j\in P}|\beta_j|$, where $P$ is the set of penalized coefficient indices; an intercept, if included in $\beta$, is excluded from $P$. Define the active set $A=\{j\in P:\hat\beta_j\ne0\}$. Within a region where the active set and the signs of its coefficients do not change, the absolute-value penalty is linear in the active coefficients. It therefore adds no curvature to the local quadratic problem. This motivates an active-set degrees-of-freedom approximation.

Definition: For a lasso GLM, a common local active-set approximation is the rank of the weighted design matrix containing the unpenalized coefficients and the active predictors. If that matrix has full column rank, the approximation is the number of those columns. More generally, under a fixed local Fisher approximation and a stable active set, the active-set smoother has trace equal to that rank.

To see the basis of the approximation, restrict the local weighted least-squares problem to the active coefficients and the unpenalized coefficients. With the active signs fixed, the lasso penalty is linear, so its derivative shifts the local solution but does not change the quadratic curvature $X_A^\top W X_A$. If this matrix is invertible, the derivative of the fitted working values with respect to the working response is $$ S_A=W^{1/2}X_A(X_A^\top W X_A)^{-1}X_A^\top W^{1/2}. $$ This is a projection onto the column space of $W^{1/2}X_A$. Its trace is the rank of that matrix. If columns are dependent, the rank—not the raw number of active coefficients—is the relevant local count. In a full GLM fit, changing weights, coefficient signs, or active set can make this only an approximation to data sensitivity.

Worked Example: An Orthogonal Lasso Working Subproblem

Consider a local IRLS quadratic subproblem with an unpenalized intercept and three orthogonal slope directions. Let their information values be $5,2,1$, and let the unpenalized least-squares targets for the slopes be $2,1.5,0.4$. The coordinatewise lasso solution for a positive target $t_j$ is $\max(t_j-\lambda/d_j,0)$, where $d_j$ is its information value. Find the active-set approximation at $\lambda=1$ and $\lambda=3$.

At $\lambda=1$, the three slope estimates are $$ \max(2-1/5,0)=1.8,\qquad \max(1.5-1/2,0)=1,\qquad \max(0.4-1,0)=0. $$ The first two slopes are active. Including the unpenalized intercept gives three independent fitted directions, so the approximation is $3$.

At $\lambda=3$, the slope estimates are $$ \max(2-3/5,0)=1.4,\qquad \max(1.5-3/2,0)=0,\qquad \max(0.4-3,0)=0. $$ Only the first slope remains active. Including the intercept gives an active-set approximation of $2$. In this orthogonal quadratic example, increasing the penalty removes a direction. In correlated or nonlinear GLM fits, active sets need not evolve so simply.

Reading a Penalty Path and Avoiding Pitfalls

Effective degrees of freedom complement coefficient plots, cross-validation results, and predictive-risk summaries. A ridge trace reports how much each information direction contributes after shrinkage. A lasso active-set approximation reports the dimension of the currently selected local fit. Comparing these summaries at candidate penalty values helps distinguish a model with many weakly contributing ridge directions from one with fewer active lasso predictors.

Identify the quantity being summarized.
For ridge, use the local Fisher trace at the fitted weights. For lasso, identify the active predictors and the rank of their weighted design, including unpenalized terms.

Compare fits along the path.
Record the penalty, the relevant degrees-of-freedom summary, and the predictive criterion at each fit. A smaller count indicates less local flexibility, not automatically better prediction.

Check whether the approximation is stable.
Near an active-set change, or when weights shift substantially, a lasso count or ridge local trace may change abruptly or imperfectly reflect the sensitivity of the full refitted model.

Common mistake: “There are four nonzero ridge coefficients, so the fit has four degrees of freedom.” This ignores shrinkage: nonzero ridge directions can contribute much less than one each. Use the ridge trace instead. A related error is to treat the number of nonzero lasso coefficients as an exact GLM degrees-of-freedom formula in every dataset. The active-set rank is a useful local approximation, but dependence among predictors, changing GLM weights, and changes in the active set matter. Finally, the unpenalized intercept should not be included in the lasso penalty merely because it appears in the coefficient vector; nor should a known offset be treated as a coefficient to penalize.

Key takeaway: Ridge effective degrees of freedom account for continuous shrinkage through a local trace; lasso GLM degrees of freedom are commonly approximated by the rank of the active weighted design. Both describe model complexity along a penalty path, but GLM versions must be interpreted with their local-approximation assumptions in view.

Check Your Understanding

Use the definitions and derivations above to answer the following.

Why can a ridge fit with several nonzero coefficients have effective degrees of freedom substantially below the number of coefficients?
For eigenvalues $6$ and $2$, calculate the fixed-design ridge degrees of freedom at $\lambda=2$.
In the local GLM weighted least-squares problem, what response vector must be used when the linear predictor includes a known offset $o$?
Why is the rank of the active weighted design more appropriate than the raw number of nonzero lasso coefficients when predictors are linearly dependent?
What changes in interpreting a ridge trace when the GLM weights are refitted at each penalty value?