Regression Splines Versus Smoothing Splines

Two Ways to Control a Flexible Curve

A spline can follow a curved relationship without requiring one polynomial to describe the whole range of $x$. But a flexible curve needs a rule that controls how much it can bend. A regression spline usually controls flexibility by choosing a spline basis with a specified set of knots. A smoothing spline controls it by penalizing curvature, with a smoothing parameter that determines the strength of the penalty.

Both methods can be fitted to the same observations and can produce similar-looking curves. They differ, however, in what the analyst specifies: a regression spline uses a finite set of knot locations, while a smoothing spline uses a penalty to regulate curvature. The distinction matters when explaining why a curve is flexible, comparing fits, and deciding what has been held fixed.

Definition: A fixed-knot regression spline is a spline represented by a finite basis whose knot locations and degree are specified for the fit. Its coefficients are estimated by least squares. A smoothing spline is a function chosen to balance fit to the observations against a penalty on curvature. For the usual cubic smoothing spline, the criterion is $$ \sum_{i=1}^{n}\{y_i-f(x_i)\}^2+\lambda\int \{f''(t)\}^2\,dt, $$ where $\lambda\geq 0$ is the smoothing parameter.

For comparison, suppose the observations follow $Y_i=f(x_i)+\varepsilon_i$, with independent errors having mean zero and common variance $\sigma^2$. These assumptions describe a standard regression setting; the curve-fitting distinction itself does not require normally distributed errors. Normality is needed for some familiar small-sample inference procedures, not for defining either fit.

Where Flexibility Enters

A fixed-knot regression spline has a chosen degree and knot configuration. For example, a continuous piecewise-linear spline with one knot at $c$ can be written $f(x)=\beta_0+\beta_1x+\beta_2(x-c)_+$, where $(x-c)_+=\max(0,x-c)$. The knot marks where the linear part of the curve is allowed to change. A cubic regression spline uses cubic pieces joined under specified smoothness constraints. In either case, the fitted coefficients are selected to minimize the residual sum of squares for that basis.

The knot locations are part of the model specification. If they are set in advance, fitting the coefficients does not move them. A different knot placement or number of knots changes the candidate curves available to the regression. More or better-placed knots can allow more local flexibility, but the count alone does not tell the whole story: degree, constraints, and the locations themselves also matter.

A cubic smoothing spline instead uses the curvature integral in its criterion. At $\lambda=0$, there is no penalty, and with distinct $x_i$ it can interpolate the observed responses. As $\lambda$ grows, curves with substantial curvature become more costly. In the limit of very strong penalization, the fitted curve approaches a straight least-squares line, because a straight line has $f''(t)=0$. For intermediate values, the result trades some data fit for less curvature.

In the usual cubic smoothing spline, the fitted function has knots at the distinct observed $x$-values. These knots are not chosen as a small set of locations to specify by hand; the penalty regulates the effective flexibility of the fit. The effective degrees of freedom summarize that flexibility, but they are not simply the number of observed $x$-values or a count of manually selected knots.

A Fixed-Knot Fit Worked Out

Worked Example: Least Squares With One Prespecified Knot

Consider hypothetical observations $(x_i,y_i)=(0,0),(1,1),(2,0),(3,1),(4,0)$. Fit a continuous piecewise-linear regression spline with the knot fixed at $c=2$. Use the basis $1,x,(x-2)_+$.

The fitted values have the form $\hat y_i=\hat\beta_0+\hat\beta_1x_i+\hat\beta_2(x_i-2)_+$. The design rows, in the order of the observations, are $(1,0,0),(1,1,0),(1,2,0),(1,3,1),(1,4,2)$. Least squares gives the normal equations $$ \begin{aligned} 5\hat\beta_0+10\hat\beta_1+3\hat\beta_2&=2,\\ 10\hat\beta_0+30\hat\beta_1+11\hat\beta_2&=4,\\ 3\hat\beta_0+11\hat\beta_1+5\hat\beta_2&=1. \end{aligned} $$

Subtracting twice the first equation from the second gives $10\hat\beta_1+5\hat\beta_2=0$, so $\hat\beta_1=-\hat\beta_2/2$. Subtracting $0.6$ times the first equation from the third gives $5\hat\beta_1+3.2\hat\beta_2=-0.2$. Substitution yields $0.7\hat\beta_2=-0.2$, hence $\hat\beta_2=-2/7$, $\hat\beta_1=1/7$, and $\hat\beta_0=2/7$.

The fitted values at $x=0,1,2,3,4$ are therefore $2/7,3/7,4/7,3/7,2/7$. The residuals are $-2/7,4/7,-4/7,4/7,-2/7$, so $$ \operatorname{SSE}=\sum_{i=1}^{5}(y_i-\hat y_i)^2 =\frac{4+16+16+16+4}{49} =\frac{8}{7}\approx 1.143. $$

This fit’s knot was fixed at $x=2$; the observations determined the coefficients but did not relocate the knot. The calculation assumes the stated piecewise-linear basis and ordinary least squares. It does not claim that this basis is the best choice for the data.

The Penalty Makes a Different Trade-Off

For a smoothing spline, the data-fit term is the residual sum of squares, and the second term charges for curvature. A larger $\lambda$ does not add or remove a manually selected knot; it changes the relative importance of fitting the observations and keeping the curve smooth. Multiplying the entire criterion by a positive constant would not change its minimizer, but the scale of $\lambda$ depends on the scale of the penalty and data.

Key result: If a smoothing spline is represented in a finite basis with coefficient vector $\boldsymbol{\beta}$, its criterion can be written $$ (\mathbf y-\mathbf B\boldsymbol{\beta})^\mathsf{T}(\mathbf y-\mathbf B\boldsymbol{\beta}) +\lambda\boldsymbol{\beta}^\mathsf{T}\mathbf P\boldsymbol{\beta}, $$ where $\mathbf B$ is the basis matrix and $\mathbf P$ represents the curvature penalty. When the penalized system has a unique solution, minimizing this quadratic criterion gives $$ \hat{\boldsymbol{\beta}} =(\mathbf B^\mathsf{T}\mathbf B+\lambda\mathbf P)^{-1} \mathbf B^\mathsf{T}\mathbf y. $$

To justify the result, expand the first term as $\mathbf y^\mathsf{T}\mathbf y-2\boldsymbol{\beta}^\mathsf{T}\mathbf B^\mathsf{T}\mathbf y+\boldsymbol{\beta}^\mathsf{T}\mathbf B^\mathsf{T}\mathbf B\boldsymbol{\beta}$. Differentiating the criterion with respect to $\boldsymbol{\beta}$ and setting the derivative to zero gives $-2\mathbf B^\mathsf{T}\mathbf y+2\mathbf B^\mathsf{T}\mathbf B\boldsymbol{\beta}+2\lambda\mathbf P\boldsymbol{\beta}=0$. Rearranging gives the stated solution when the matrix is invertible. This shows directly how the penalty changes the coefficient estimates; it does not turn the smoothing spline into an ordinary unpenalized fixed-knot regression spline.

Worked Example: Comparing the Two Limits on the Same Data

Use the five observations from the previous example. First, the least-squares straight line is $\hat y=\hat\beta_0+\hat\beta_1x$. Since $\bar{x}=2$, $\bar{y}=2/5=0.4$, and $\sum (x_i-\bar{x})(y_i-\bar{y})=0$, the slope is $0$ and the intercept is $0.4$. Its fitted values are all $0.4$, and its residual sum of squares is $$ \operatorname{SSE}_{\text{line}} =3(0-0.4)^2+2(1-0.4)^2 =3(0.16)+2(0.36)=1.20. $$

At the other extreme, the cubic smoothing spline with $\lambda=0$ interpolates these distinct $x$-values, so its fitted values are $0,1,0,1,0$ and its residual sum of squares is $0$. Its curvature penalty is positive. For this equally spaced data, the natural cubic interpolant has second derivatives at the five data locations of $0,-30/7,36/7,-30/7,0$. Integrating the squared second derivative across the four unit-width intervals gives $$ \int \{f''(t)\}^2\,dt =\frac{300+372+372+300}{49} =\frac{192}{7}\approx 27.43. $$

The interpolation fit therefore has criterion $0+\lambda(192/7)$, while the straight line has criterion $1.20+\lambda(0)$. For example, at $\lambda=0.02$, these two criterion values are approximately $0.549$ and $1.20$, respectively. At $\lambda=0.05$, they are approximately $1.371$ and $1.20$. This comparison shows how the penalty changes the relative cost of these two endpoint candidates. It does not establish that the smoothing spline at either positive value of $\lambda$ is exactly one of these two curves; an intermediate spline may have a smaller criterion than both.

Comparing the Methods in Practice

Feature	Fixed-knot regression spline	Smoothing spline
Main flexibility control	Degree, knot count and knot locations, with smoothness constraints	Strength of the curvature penalty, set by $\lambda$
Role of knots	A specified, usually small set defines the pieces of the curve	Usually has knots at distinct observed $x$-values; the penalty controls the fit
Fitting criterion	Least squares for the chosen basis	Residual sum of squares plus a curvature penalty
What changing the control does	Changing knots or basis changes the available curve family	Changing $\lambda$ changes the fit-penalty balance within the smoothing-spline method

Worked Example: Choosing What to Compare

Suppose an analyst fits both methods to the same hypothetical response data and uses the same held-out observations for prediction assessment. A fixed-knot spline has a held-out mean squared prediction error of $2.4$ response-units squared, while a smoothing spline selected by a prespecified validation procedure has a value of $2.0$ response-units squared. The difference is $2.4-2.0=0.4$ response-units squared in favor of the smoothing spline on that validation set.

This result compares predictive performance under those particular fitting and validation choices; it does not show that smoothing splines must outperform regression splines. A fair comparison uses the same outcome, observations, prediction targets, and loss measure. It also reports how the fixed knots were chosen and how $\lambda$ was selected, rather than attributing the difference to the method name alone.

A fixed-knot spline can be useful when knot positions have a clear scientific or design-based justification, or when a compact basis is wanted. A smoothing spline is useful when one wants to regulate overall curvature through a tuning parameter rather than nominate a small knot set. Neither choice removes the need to consider the observed $x$-range, the purpose of the fit, and how flexibility was selected.

Common Mistakes and Why the Distinction Matters

“A smoothing spline has no knots.” In the usual cubic smoothing spline, knots occur at the distinct observed $x$-values. The important distinction is that curvature penalization, not a manually chosen small knot set, controls flexibility.
“More knots always mean a better fit.” More knots can expand the family of curves available to an unpenalized regression spline, but they do not guarantee better generalization. The knot placement, smoothness constraints, and evaluation goal also matter.
“A larger $\lambda$ moves the knots.” The smoothing parameter changes the penalty strength. It does not serve as a rule for relocating a fixed knot in a regression spline.
“The smoothing spline with $\lambda=0$ is automatically the best fit.” It interpolates the observations and has zero residual sum of squares, but that fact alone says nothing about how well it predicts new observations. Its curvature can be large.
“The two methods are interchangeable because both use splines.” They can represent related kinds of curves, but their fitting criteria and flexibility controls differ. A penalized regression spline is a possible hybrid approach, not the same thing as ordinary least squares with fixed knots.

Key takeaway: A fixed-knot regression spline controls flexibility through its specified basis and knot configuration. A smoothing spline controls flexibility through a curvature penalty and $\lambda$. Compare the fitted curves only after making clear which choices were fixed, which were selected, and how the fits were evaluated.

Check Your Understanding

Answer each question by distinguishing the curve representation from the rule used to fit it.

In a fixed-knot regression spline, what is estimated from the data, and what remains fixed when the knots are prespecified?
What does increasing $\lambda$ do in a cubic smoothing spline’s objective?
For the five-point example, why does the straight-line fit have zero curvature penalty but a positive residual sum of squares?
Why is a smoothing spline with $\lambda=0$ not necessarily a good choice for predicting new observations?
When comparing held-out prediction errors from two spline methods, what must be kept consistent for the comparison to be meaningful?