Selecting Smoothing Strength for a Smoothing Spline

Why the Smoothing Parameter Must Be Chosen

A smoothing spline can adapt to curved patterns without requiring a fixed list of knot locations, but its shape depends on how strongly curvature is penalized. A weak penalty can let the fitted curve follow random variation in the observed responses. A strong penalty can suppress genuine structure. Choosing the smoothing parameter is therefore a predictive and interpretive decision: the aim is to retain useful features without treating every fluctuation as signal.

Let the observed data be $(x_i,y_i)$, for $i=1,\ldots,n$, where each $y_i$ is a quantitative response and each $x_i$ is a quantitative predictor. A smoothing spline estimates a function $f$ by balancing squared errors at the observed predictor values against a penalty on curvature.

Definition: For a smoothing parameter $\lambda\geq 0$, a smoothing spline minimizes $$ \sum_{i=1}^{n}\{y_i-f(x_i)\}^2+\lambda\int \{f''(t)\}^2\,dt. $$ The first term measures lack of fit at the observations. The second penalizes curvature. A larger $\lambda$ places more weight on smoothness relative to fitting the observed responses; a smaller $\lambda$ allows more flexibility. Under the usual smoothing-spline setup, the fitted function is a natural cubic smoothing spline.

For a given data set, the fitted values at the observed predictors can be written as $\hat{\mathbf y}=S_\lambda\mathbf y$, where $S_\lambda$ is the smoothing matrix and $\mathbf y$ is the vector of observed responses. Its trace, $\operatorname{tr}(S_\lambda)$, is the effective degrees of freedom. As $\lambda$ increases, the fitted curve generally becomes less flexible and its effective degrees of freedom decrease. The limiting fit is a straight-line least-squares regression; with very little penalty, the fit can approach interpolation when the predictor values are distinct.

A useful selection rule asks how well each candidate value of $\lambda$ predicts observations not used to fit that candidate. Cross-validation makes this question operational. It does not prove that a selected curve is the true relationship, and it does not replace checking whether the resulting shape is reasonable in context.

Leave-One-Out Cross-Validation

In leave-one-out cross-validation (LOOCV), each observation is set aside in turn. The spline is fitted to the other $n-1$ observations, and the fitted function predicts the response at the omitted predictor. The squared prediction errors are averaged. The selected value of $\lambda$ is the one with the smallest average.

Definition: Write $\hat f_{-i,\lambda}(x_i)$ for the prediction at $x_i$ from the smoothing spline fitted with observation $i$ omitted. The LOOCV score is $$ \operatorname{CV}(\lambda)=\frac{1}{n}\sum_{i=1}^{n}\{y_i-\hat f_{-i,\lambda}(x_i)\}^2. $$ The score is in squared response units. Its square root is in the response's original units.

Smaller LOOCV scores indicate better average prediction on the omitted observations among the candidate smoothing parameters considered. This is a comparison, not a guarantee of low prediction error for every future case. The result can depend on the observed sample, the candidate values examined, and the loss being assessed (here, squared error).

A Shortcut for a Linear Smoother

Refitting the spline $n$ times for every candidate $\lambda$ can be computationally costly. For a fixed $\lambda$, a smoothing spline is a linear smoother: its fitted response vector is $S_\lambda\mathbf y$. Let $\hat y_i$ be the fitted value using all observations, $e_i=y_i-\hat y_i$ the ordinary residual, and $S_{\lambda,ii}$ the $i$th diagonal element of the smoothing matrix. For a smoothing spline fitted by the stated penalized least-squares criterion, the exact leave-one-out residual can be obtained from the full-data fit when $1-S_{\lambda,ii}\ne 0$:

y_i-\hat f_{-i,\lambda}(x_i)=\frac{e_i}{1-S_{\lambda,ii}}.

The denominator accounts for the fact that observation $i$ helped determine its own fitted value. When its diagonal leverage $S_{\lambda,ii}$ is large, omitting it can change its prediction substantially. This shortcut is useful when calculating LOOCV scores, but it does not remove the need to compare candidate values of $\lambda$.

Justification: For fixed $\lambda$, deleting observation $i$ removes only its squared-error term from the smoothing-spline penalized least-squares objective. The rank-one update for this quadratic minimization gives $$ \hat y_i=a_i+S_{\lambda,ii}(y_i-a_i), $$ where $a_i=\hat f_{-i,\lambda}(x_i)$. Thus $e_i=(1-S_{\lambda,ii})(y_i-a_i)$, which gives the displayed quotient when $1-S_{\lambda,ii}\ne 0$. If $1-S_{\lambda,ii}=0$, the quotient is undefined and the deleted-data fit must be evaluated directly.

Worked Example: Calculating a LOOCV Score

Suppose four hypothetical observations are used to compare three smoothing parameters. For each candidate, the full-data residuals $e_i$ and diagonal values $S_{\lambda,ii}$ are shown below. The diagonal values are equal within each candidate here to keep the arithmetic transparent.

$\lambda$	Full-data residuals $e_i$	$S_{\lambda,ii}$
0.2	0.25, −0.50, 0.25, 0	0.75
1	0.60, −0.40, 0.20, −0.40	0.60
10	1.50, −1.00, 0.50, −1.00	0.50

For $\lambda=0.2$, the leave-one-out residuals are the full-data residuals divided by $1-0.75=0.25$: $1,-2,1,0$. Thus $$ \operatorname{CV}(0.2)=\frac{1^2+(-2)^2+1^2+0^2}{4}=\frac{6}{4}=1.50. $$

For $\lambda=1$, divide each residual by $1-0.60=0.40$, giving $1.5,-1,0.5,-1$. Hence $$ \operatorname{CV}(1)=\frac{1.5^2+(-1)^2+0.5^2+(-1)^2}{4} =\frac{4.50}{4}=1.125. $$

For $\lambda=10$, divide by $1-0.50=0.50$, giving $3,-2,1,-2$. Therefore $$ \operatorname{CV}(10)=\frac{3^2+(-2)^2+1^2+(-2)^2}{4} =\frac{18}{4}=4.50. $$

Among these candidates, $\lambda=1$ has the smallest estimated mean squared prediction error, $1.125$ squared response units. This comparison selects $\lambda=1$ from the three values shown; it does not establish that no value between them would perform better.

Generalized Cross-Validation

Generalized cross-validation (GCV) is a convenient approximation to LOOCV. Rather than use a separate leverage adjustment for every observation, it uses the average influence represented by the effective degrees of freedom. For a smoothing matrix $S_\lambda$, the GCV score is

Definition: For candidates with $\operatorname{tr}(S_\lambda)<n$, the generalized cross-validation score, with residual sum of squares $\operatorname{RSS}(\lambda)=\sum_{i=1}^{n}(y_i-\hat y_i)^2$, is $$ \operatorname{GCV}(\lambda)= \frac{\operatorname{RSS}(\lambda)/n} {\{1-\operatorname{tr}(S_\lambda)/n\}^2}. $$ Candidates with $\operatorname{tr}(S_\lambda)=n$ are excluded because this formula is undefined. Choose the candidate $\lambda$ with the smallest GCV score among those with $\operatorname{tr}(S_\lambda)<n$, provided the candidate set and fitting procedure are otherwise held fixed.

The numerator is the average squared error in the fit to the observed data. The denominator increases the score to account for model flexibility: a fit using more effective degrees of freedom is penalized. GCV often gives a useful, computationally efficient approximation to LOOCV, but it is not identical to LOOCV in every data set. In particular, replacing the individual diagonal values by their average can matter when the observations have very unequal leverage.

Worked Example: Comparing GCV Scores

For a hypothetical data set with $n=8$, three candidate smoothing parameters give the following residual sums of squares and effective degrees of freedom.

$\lambda$	$\operatorname{RSS}(\lambda)$	$\operatorname{tr}(S_\lambda)$
0.1	10	6
1	14	4
5	20	2.5

For $\lambda=0.1$, $$ \operatorname{GCV}(0.1)=\frac{10/8}{(1-6/8)^2} =\frac{1.25}{0.25^2}=20.0. $$ For $\lambda=1$, $$ \operatorname{GCV}(1)=\frac{14/8}{(1-4/8)^2} =\frac{1.75}{0.50^2}=7.00. $$ For $\lambda=5$, $$ \operatorname{GCV}(5)=\frac{20/8}{(1-2.5/8)^2} =\frac{2.50}{0.6875^2}\approx 5.289. $$

The smallest GCV score is for $\lambda=5$. Although its in-sample RSS is the largest of these three, its lower effective degrees of freedom reduce the flexibility adjustment enough to give the smallest GCV score. The score is on the squared response scale, but its main use here is to rank the candidates fitted to the same data.

What the Selected Curve Should Look Like

Cross-validation selects a smoothing strength by a numerical prediction criterion. After selection, inspect the fitted curve together with the observations and, when helpful, with less- and more-smoothed candidates. This is not a substitute for the criterion; it checks whether the selected curve is compatible with the data's broad pattern and the subject matter.

A curve may be undersmoothed when it is insufficiently smoothed: it follows small, erratic fluctuations that are plausibly noise rather than stable features of the relationship. A curve may be oversmoothed when it is too smooth: it suppresses genuine structure, such as a broad bend supported by the data. These labels describe the curve's relationship to the signal of interest, which is not directly observed. A wiggly curve is not automatically undersmoothed if the underlying relationship genuinely changes rapidly, and a nearly straight curve is not automatically oversmoothed if a straight trend adequately describes the relationship.

Worked Example: Interpreting a Selected Curve

Imagine a hypothetical data set relating a measured concentration $x$, in milligrams per litre, to a response $y$, in response units. Across the observed range, the plotted points suggest a broad rise followed by a shallow bend. Suppose cross-validation selects $\lambda=2$ from the candidate grid, and the fitted curve follows that broad pattern without turning sharply at isolated observations.

To interpret the choice, compare the selected fit with two nearby candidates. If a smaller $\lambda$ produces several short, abrupt turns that are not supported by a sustained pattern in the observations, that fit may be undersmoothed. If a larger $\lambda$ produces an almost straight line that misses the broad bend apparent across multiple observations, that fit may be oversmoothed. The selected curve is a reasonable compromise if it retains the broad rise and bend without following every local fluctuation. This visual check describes the fit; it does not prove that the bend is a causal effect of concentration.

The selection criterion still matters: if $\operatorname{CV}(2)=3.2$ response-units squared and the nearest candidates have scores $3.8$ and $3.6$ response-units squared, $\lambda=2$ has the best estimated prediction performance among those candidates. The differences are modest, so it would be unwise to portray the selected value as uniquely certain. The final fitted spline is then refitted using all observations at $\lambda=2$, since cross-validation used omitted observations only to evaluate candidate performance.

Conditions, Choice of Criterion, and Common Pitfalls

The usual interpretation of these criteria assumes that the observations are appropriate for the intended prediction task. In a standard regression setting, the predictor values are treated as fixed or conditioned upon, and responses are independent with a common underlying mean function plus random errors. Equal error variance is a customary assumption for interpreting squared-error prediction comparisons in the usual way. Normal errors are not required to calculate CV or GCV. If observations are dependent, such as repeated measurements on the same person, leaving out one row at a time may not represent prediction for a new independent case; the validation scheme must respect how the data were collected.

The penalty and validation criterion also answer different parts of the problem. The penalty defines the family of candidate curves and how $\lambda$ trades fit against curvature. CV or GCV compares prediction performance within that family. A selected $\lambda$ cannot correct a poor predictor, an inappropriate response scale, biased data collection, or a mismatch between the candidate model and the scientific question.

Choosing the smallest training RSS. A more flexible curve will generally fit the observed responses at least as closely, so training RSS alone does not assess prediction on omitted data. Use a validation criterion when the goal is predictive performance.
Calling a small $\lambda$ oversmoothing. A small penalty allows more curvature and can lead to undersmoothing. Oversmoothing means the curve is too smooth and misses meaningful structure.
Reading a CV minimum as certainty. The minimum identifies the best candidate under the chosen criterion and candidate set. It does not establish that the selected curve is the true relationship or that its exact smoothing parameter is known without uncertainty.
Ignoring the scale of the criterion. CV and GCV here average squared response errors, so their units are squared response units. A score of 4 is not an average prediction error of 4 response units; its square root is on the response scale.

Key takeaway: Select $\lambda$ by comparing cross-validation or generalized cross-validation scores, then inspect the fitted curve for a sensible balance. Undersmoothing follows too much small-scale variation; oversmoothing suppresses too much structure.

Check Your Understanding

Answer these questions using the definitions and calculations above.

As $\lambda$ increases, what happens to the relative weight on curvature and to the usual flexibility of the fitted smoothing spline?
A candidate has $e_i=0.6$ and $S_{\lambda,ii}=0.4$. Calculate its leave-one-out residual using the linear-smoother identity.
Why can a candidate with a larger training RSS have a smaller GCV score than a more flexible candidate?
A curve follows isolated bumps closely, while a more strongly smoothed curve removes a broad bend supported by many observations. Which fit might be undersmoothed, and which might be oversmoothed? Explain.
What does the smallest LOOCV score establish, and what does it not establish?

\(\lambda\)	\(\operatorname{RSS}(\lambda)\)	\(\operatorname{tr}(S_\lambda)\)
0.1	10	6
1	14	4
5	20	2.5