Effective Degrees of Freedom in Smoothing Splines

Why a Smoothing Spline Has Effective Degrees of Freedom

A smoothing spline can be flexible without assigning a separate, unrestricted parameter to every visible bend in the fitted curve. Its curvature penalty controls how strongly the fit responds to the observed responses. To describe that flexibility, it is useful to ask how much each response value can influence the fitted values overall. The answer is summarized by the spline’s effective degrees of freedom.

The word “effective” matters: this quantity need not be an integer. It measures the total influence of the data on the fit, rather than counting a fixed number of ordinary regression coefficients. We will express that influence using a matrix, then use the eigenvalues of the curvature penalty to show precisely how it changes with the penalty strength.

The Smoothing Spline as a Linear Smoother

Suppose the predictor values $x_1,\ldots,x_n$ are fixed and distinct, and the observed responses are $y_1,\ldots,y_n$. A smoothing spline chooses a smooth function by balancing fit to the responses against a penalty on curvature. In the usual formulation, the objective is the sum of squared residuals plus $\lambda$ times the integrated squared second derivative. Here $\lambda\geq 0$ is the smoothing parameter: a larger value puts more weight on avoiding curvature.

For the degrees-of-freedom calculation, we need only the fitted values at the observed predictor values. Write $\hat{\mathbf y}=(\hat y_1,\ldots,\hat y_n)^T$ and $\mathbf y=(y_1,\ldots,y_n)^T$. The curvature penalty can be represented, for these fitted values, by a symmetric positive semidefinite matrix $K$. Thus the fitted-value problem has the form

\hat{\mathbf y} = \underset{\mathbf f\in\mathbb R^n}{\operatorname{argmin}} \left\{ (\mathbf y-\mathbf f)^T(\mathbf y-\mathbf f) +\lambda\,\mathbf f^T K\mathbf f \right\}.

This finite-dimensional representation is the one needed below. The matrix $K$ is determined by the predictor values and the curvature penalty; it is not an additional matrix of observed responses.

Definition: A linear smoother is a fitting rule for which the vector of fitted values can be written $\hat{\mathbf y}=S_\lambda\mathbf y$, where $S_\lambda$ is a matrix that depends on the predictor values and smoothing parameter, but not on the observed response vector. The effective degrees of freedom of this smoother are $\operatorname{tr}(S_\lambda)$, the sum of the diagonal entries of its smoother matrix.

Differentiating the objective with respect to $\mathbf f$ and setting the derivative to zero gives

-2(\mathbf y-\hat{\mathbf y})+2\lambda K\hat{\mathbf y}=0, \qquad (I+\lambda K)\hat{\mathbf y}=\mathbf y.

Because $K$ is positive semidefinite, $I+\lambda K$ is invertible for $\lambda\geq 0$. Therefore

\hat{\mathbf y}=S_\lambda\mathbf y, \qquad S_\lambda=(I+\lambda K)^{-1}.

This equation establishes the linear-smoother property: for fixed predictor values and $\lambda$, changing the responses changes the fitted values linearly. The formula describes fitted values at the observed predictor values, not a claim that the fitted curve itself is a straight line.

Eigenvalues Explain the Effective Degrees of Freedom

For a cubic smoothing spline with at least two distinct predictor values, constant and linear fitted-value patterns have zero curvature penalty. They form a two-dimensional null space of $K$. All remaining directions, if there are any, have positive penalty eigenvalues. Let $\mu_1,\ldots,\mu_n$ be the eigenvalues of $K$, including multiplicities. Since $K$ is symmetric, it has an orthonormal eigenbasis, so the smoother has eigenvalues $1/(1+\lambda\mu_i)$ in the same directions.

Theorem: For a cubic smoothing spline with $m\geq 2$ distinct predictor values and $n=m$ observations at those distinct values, the effective degrees of freedom are $$ \operatorname{df}_{\mathrm{eff}}(\lambda) = \operatorname{tr}(S_\lambda) = \sum_{i=1}^{n}\frac{1}{1+\lambda\mu_i}. $$ Two eigenvalues are zero, corresponding to constant and linear trends. If $n>2$, the other $n-2$ eigenvalues are positive. Consequently, $\operatorname{df}_{\mathrm{eff}}(0)=n$, and its limit as $\lambda$ tends to infinity is 2. If $n=2$, both eigenvalues are zero, so the effective degrees of freedom are 2 for every $\lambda$.

To justify the formula, diagonalize $K=Q\operatorname{diag}(\mu_1,\ldots,\mu_n)Q^T$, where $Q$ is orthogonal. Substitution into $S_\lambda=(I+\lambda K)^{-1}$ gives

S_\lambda = Q\operatorname{diag}\left( \frac{1}{1+\lambda\mu_1},\ldots, \frac{1}{1+\lambda\mu_n} \right)Q^T.

The trace is unchanged by an orthogonal change of basis, so summing these diagonal eigenvalues yields the stated result. Each zero-penalty direction contributes $1$ for every $\lambda$. Each positive-penalty direction contributes a value that starts at $1$ when $\lambda=0$ and decreases toward $0$ as $\lambda$ grows.

For $n>2$, at least one positive eigenvalue is present. Differentiating the sum shows that the degrees of freedom decrease strictly with $\lambda$:

\frac{d}{d\lambda}\operatorname{df}_{\mathrm{eff}}(\lambda) = -\sum_{i:\,\mu_i>0} \frac{\mu_i}{(1+\lambda\mu_i)^2} <0.

For $n=2$, there are no positive eigenvalues and the derivative is zero. Thus the general statement is that effective degrees of freedom are nonincreasing as the penalty grows, and strictly decreasing when there are more than two distinct predictor values. In the large-penalty limit with at least two distinct predictor values, the fit is restricted to the unpenalized space of linear trends in $x$; the corresponding fitted values are those from least squares on an intercept and $x$.

Worked Example: Calculating Effective Degrees of Freedom From Eigenvalues

Consider a hypothetical smoothing-spline calculation with four distinct predictor values. Suppose its curvature-penalty matrix has eigenvalues $0,0,1,4$. For $\lambda=0.5$, calculate the effective degrees of freedom.

The two zero eigenvalues contribute $1$ each. The two positive eigenvalues contribute $1/(1+0.5)$ and $1/(1+4(0.5))$. Therefore,

\operatorname{df}_{\mathrm{eff}}(0.5) = 1+1+\frac{1}{1.5}+\frac{1}{3} = 2+0.6667+0.3333 = 3.000.

The smoother has 3.000 effective degrees of freedom. This is not a claim that the curve has exactly three ordinary coefficients. It says that, in total, the data influence the fitted values as much as three fully retained directions would. The result lies between the interpolation limit of 4 and the large-penalty limit of 2.

Effective Degrees of Freedom and Data Influence

The trace definition has a direct interpretation through the diagonal entries of $S_\lambda$. The entry $S_{\lambda,ii}$ measures how the fitted value at $x_i$ responds to a change in the observed response $y_i$, holding the other responses fixed. Summing those own-observation influences gives the effective degrees of freedom. Off-diagonal entries also matter to how fitted values borrow information across observations, even though they do not appear individually in the trace.

A related interpretation follows under the response model $\mathbf y=\boldsymbol\mu+\boldsymbol\varepsilon$, with fixed predictors and errors having mean zero and covariance matrix $\sigma^2 I$. Then

\operatorname{Cov}(\hat{\mathbf y}) = \operatorname{Cov}(S_\lambda\mathbf y) = \sigma^2 S_\lambda S_\lambda^T.

This covariance formula is valid under the stated error assumptions. The trace of $S_\lambda$, however, is defined from the fitting rule and does not require normally distributed errors. It is a summary of the smoother’s responsiveness, not an estimate of the number of observations or a count of knots.

Worked Example: Seeing Shrinkage in Eigen-coordinates

Use the same hypothetical eigenvalues $0,0,1,4$. Suppose the observed response vector, expressed in the orthonormal eigenbasis of $K$, has coordinates $(10,8,3,-2)$. Find the fitted coordinates and effective degrees of freedom when $\lambda=1$.

In an eigenvector direction with eigenvalue $\mu_i$, the fitted coordinate is the observed coordinate multiplied by $1/(1+\lambda\mu_i)$. Thus the first two coordinates, which have zero penalty, are unchanged. The third is multiplied by $1/2$, and the fourth by $1/5$:

\hat{\mathbf y}_{\mathrm{eig}} = \left( 10,\ 8,\ \frac{3}{2},\ -\frac{2}{5} \right) = (10,\ 8,\ 1.500,\ -0.400).

The effective degrees of freedom are the sum of the same four multipliers:

\operatorname{df}_{\mathrm{eff}}(1) = 1+1+\frac{1}{2}+\frac{1}{5} = 2.700.

The fit retains the constant and linear directions fully, while shrinking the two curved directions. The coordinate values are in an eigenbasis for explanation; transforming them back with the eigenvectors gives the fitted values in the original observation order.

Important Special Case: One Distinct Predictor Value

The two-dimensional limiting linear space requires at least two distinct predictor values. With only one distinct predictor value, there is only one distinct fitted-value location. A constant fitted value can represent its fit with zero curvature penalty, and the fitted-value space has dimension one. In this case $K$ is the $1\times1$ zero matrix, so the effective degrees of freedom are 1 for every $\lambda$. This case must not be assigned the two-degree-of-freedom limit that applies when there are at least two distinct predictor values.

Worked Example: Only One Distinct Predictor Value

Suppose three hypothetical measurements are taken at the same predictor value, $x=5$, with responses $2,4,6$. There is just one distinct predictor location. The fit at that location is the mean response, since this minimizes the sum of squared errors:

\hat f(5)=\bar y=\frac{2+4+6}{3}=4.

There is one fitted value shared by the three observations. For the effective-degrees-of-freedom calculation on this one-location fitted-value space, $K=(0)$ and $S_\lambda=(1)$, regardless of the value of $\lambda$. Hence

\operatorname{df}_{\mathrm{eff}}(\lambda)=\operatorname{tr}(S_\lambda)=1.

The three responses supply information about one fitted-value coordinate; they do not create three distinct predictor locations or two independent trend directions.

Common Mistake and Why the Quantity Matters

A common mistake is to say, “A smoothing spline has $n$ degrees of freedom because it uses $n$ observed responses,” or, at the other extreme, “It always has two degrees of freedom because a heavily smoothed fit is linear.” Both statements ignore the penalty strength and the number of distinct predictor values. At $\lambda=0$, the fitted values can retain all $n$ response directions. As $\lambda$ increases, the curved directions are progressively shrunk. Only in the large-penalty limit with at least two distinct predictor values does the effective degrees of freedom approach 2. With one distinct predictor value it is 1 throughout.

Effective degrees of freedom are useful because they summarize the flexibility of a fit on a scale comparable across different penalty values. A lower value indicates that fewer response directions are being retained, but it does not by itself establish that the resulting curve is appropriate for a particular application. The number is a property of the smoother and the predictor locations, not a guarantee about prediction quality or a causal interpretation.

Key takeaway: For a smoothing spline, $S_\lambda=(I+\lambda K)^{-1}$, and effective degrees of freedom are $\operatorname{tr}(S_\lambda)=\sum_i(1+\lambda\mu_i)^{-1}$. With at least two distinct predictor values, the two unpenalized trend directions remain while penalized curvature directions shrink as $\lambda$ grows. With one distinct predictor value, the effective degrees of freedom are 1 for every $\lambda$.

Check Your Understanding

Answer each question using the definitions and results in this tutorial.

For a smoother matrix $S_\lambda$, what is the definition of effective degrees of freedom?
A penalty matrix has eigenvalues $0,0,2$. Calculate the effective degrees of freedom at $\lambda=1$.
Why does each zero eigenvalue contribute 1 to effective degrees of freedom for every nonnegative $\lambda$?
For three distinct predictor values, what are the effective degrees of freedom at $\lambda=0$ and their limit as $\lambda$ tends to infinity?
Why is it incorrect to claim that the large-penalty limit is always 2 when there is only one distinct predictor value?