Why Diagnose a Spline Fit?
A spline can represent changing curvature without requiring one global polynomial shape. That flexibility is useful, but it also means that a fitted curve can look convincing while leaving systematic structure in the data—or can follow local fluctuations that are not stable features of the relationship. Residuals and knot sensitivity provide two complementary checks: residual plots ask what remains after fitting, while alternative knot placements ask how much the result depends on a modeling choice.
The goal is not to find a plot that proves a spline is correct. A diagnostic plot is evidence to consider alongside the data, the model's purpose, and the assumptions behind the fit. A small sample can make patterns difficult to distinguish from chance, and no residual pattern by itself establishes a cause.
Residuals for a Spline Model
Suppose observations are \((x_i,y_i)\), for \(i=1,\ldots,n\), and a spline model specifies a mean response \(f(x_i)\). The fitted value \(\hat{y}_i\) estimates that mean at \(x_i\). The residual \(e_i\) is the observed response minus the fitted value:
A positive residual means the observation lies above the fitted curve; a negative residual means it lies below. Residuals have the same units as the response. For ordinary least squares, the spline fit is a linear combination of basis functions, including a constant basis function. The usual working assumptions are that the errors have mean zero, are independent, and have a common variance. Normal errors are additionally used for some exact small-sample inference, but normality is not required to draw a residual plot.
In a residual-versus-fitted plot, a smooth sequence of positive residuals followed by negative ones can indicate that the fitted mean is systematically too low in one range and too high in another. In a residual-versus-predictor plot, the horizontal axis preserves the observations' order along the predictor, making it easier to see whether departures are localized near a knot or at a boundary. The two plots can tell different stories: fitted values need not increase steadily with \(x\), and observations with similar fitted values can occur at different predictor values.
Look for sustained patterns, not isolated signs. A few positive and negative residuals are expected even when a model is useful. A broad run of residuals on one side of zero, or a repeated bend across the predictor range, deserves attention. These are prompts to inspect the fit, not automatic instructions to add knots.
Worked Example: Reading Residuals From a Spline Fit
Consider six hypothetical observations. A least-squares linear spline with one interior knot at \(x=3\) has the following fitted values. The listed fit is also a straight line, which is included in this spline model.
| \(x_i\) | \(y_i\) | \(\hat{y}_i\) | \(e_i=y_i-\hat{y}_i\) |
|---|---|---|---|
| 1 | 12 | 11 | 1 |
| 2 | 10 | 12 | -2 |
| 3 | 14 | 13 | 1 |
| 4 | 15 | 14 | 1 |
| 5 | 13 | 15 | -2 |
| 6 | 17 | 16 | 1 |
For example, at \(x=2\), \(e_i=10-12=-2\), so the observed response is 2 units below its fitted value. The sum of squared errors is
The residuals are not all on one side of zero, and this small set does not show a sustained run of residuals. But six observations provide limited evidence: the alternating signs alone neither establish a problem nor certify an adequate fit. The plot should be read for a broad, repeatable pattern, not judged by whether every point lies close to zero.
Why Least-Squares Residuals Sum to Zero
For a least-squares spline fit with an intercept, the residuals sum to zero, apart from rounding. This fact is a useful arithmetic check, but it is not evidence that the model has captured the relationship: positive and negative residuals can cancel while retaining a clear pattern.
To see why the sum is zero, let the fitted values be generated by a design matrix \(X\) of spline basis columns, with a column of ones for the intercept. The residual sum of squares is \(\sum_i(y_i-\hat{y}_i)^2\). At the least-squares minimum, changing the intercept by a small amount \(c\) changes every fitted value by \(c\). Differentiating the sum of squares with respect to \(c\) and setting the derivative to zero gives:
so \(\sum_i e_i=0\). More generally, least squares makes the residuals orthogonal to every fitted basis column: \(X^\mathsf{T}e=0\). A nonzero residual sum in an ordinary least-squares fit with an intercept can therefore flag a calculation, rounding, or model-description mismatch. A penalized fit or a fit without an intercept need not satisfy this zero-sum result.
Comparing Alternative Knot Placements
Knot sensitivity is assessed by refitting the same type of spline with different plausible knot locations and comparing the resulting diagnostics. The comparison is most informative when it keeps the observations, response scale, spline degree, number of knots, and fitting method fixed, changing only the locations being examined. If the number of basis functions also changes, a difference in fit may reflect both complexity and knot placement.
For each fit, inspect residual-versus-fitted and residual-versus-predictor plots as well as the fitted curve. Ask whether the residual pattern changes meaningfully and whether the curve's important features remain in roughly the same locations and have similar shapes. Training SSE is useful to report, but it cannot settle the comparison: least-squares fitting minimizes SSE for each candidate, and a lower value does not by itself show that the change is meaningful or dependable.
Worked Example: Checking Sensitivity Across Knot Locations
Suppose a hypothetical data set contains 12 observations. Three ordinary least-squares cubic spline fits each use the same number of interior knots and the same basis construction. Only the knot locations differ. The following are hypothetical, unrounded fit summaries; each fit includes an intercept.
| Knot placement | SSE | Sum of residuals |
|---|---|---|
| Earlier placement | 38.6 | 0 |
| Middle placement | 29.4 | 0 |
| Later placement | 31.1 | 0 |
The middle placement has the smallest SSE. Relative to the earlier placement, its reduction is \(38.6-29.4=9.2\) response-units squared, or
That reduction identifies a difference in training fit, not a verdict that the middle placement is best. The next check is to compare the plots and fitted curves. If the residual pattern changes substantially and a feature of the fitted curve shifts when the knots move, the interpretation may be knot-sensitive. If the curves and residual patterns are similar, the numerical SSE difference may matter less in context. A residual sum of zero in all three fits is expected from the intercept; it does not distinguish them.
When Flexibility Hides a Warning
A highly flexible spline can fit observed responses extremely closely. That may leave little visible residual structure, but an almost perfect in-sample fit does not show that the curve captures a stable relationship. In particular, if modest knot changes produce visibly different curves between observations, apparent detail in the fitted curve deserves caution. Knot sensitivity is about dependence on the chosen locations; it is not a formal proof that a curve is overfit.
Worked Example: A Near-Perfect Fit That Still Warrants Caution
Imagine six hypothetical observations at \(x=0,1,2,3,4,5\), with responses \(10,13,11,16,12,18\), respectively. A sufficiently rich spline basis with enough independent parameters to interpolate these six points can have fitted values \(10,13,11,16,12,18\). Each residual is \(y_i-\hat{y}_i=0\), so the in-sample SSE is
The observed responses rise by 3, fall by 2, rise by 5, fall by 4, and rise by 6 across successive predictor values. The residual plot contains only zeros, so it cannot reveal how the interpolating curve behaves between those observed values. Suppose two alternative knot placements produce interpolated predictions of 12.4 and 14.1 response units at \(x=2.5\). The difference is \(14.1-12.4=1.7\) response units, despite both fits matching all six observations exactly. This hypothetical comparison shows why zero residuals are not enough: the curve between data points can be sensitive to knot placement. The example raises a diagnostic concern; it does not, by itself, identify which curve should be used.
A Practical Diagnostic Sequence
A disciplined comparison helps keep visual impressions separate from what the calculations establish.
Confirm \(e_i=y_i-\hat{y}_i\), identify the response units, and check that the residual sum is approximately zero when ordinary least squares includes an intercept.
Look for sustained runs or bends in residual-versus-fitted and residual-versus-predictor plots. Note where any pattern occurs; do not treat a single unusual point as a repeated trend.
Hold the data and spline complexity fixed where possible. Compare residual plots, SSE, and the fitted curves over the observed predictor range.
State what changes across fits and what remains stable. Do not claim that a residual plot proves the assumptions or that the smallest SSE proves a knot placement is best.
Common Mistakes
- Reading a zero residual sum as a clean diagnostic. The sum is zero by the least-squares intercept condition; residuals can still show a systematic pattern.
- Choosing knots solely because they lower training SSE. SSE summarizes fit to the observations used to estimate the model. It does not alone establish that a knot placement is more informative or stable.
- Calling any wiggle evidence of overfitting. Small samples naturally produce irregular residuals. Look for meaningful sensitivity across reasonable placements and report the limited evidence.
- Confusing residual plots with causal evidence. A residual pattern can point to a mismatch between the fitted mean and the observed data, but it does not establish why the mismatch occurs.
Check Your Understanding
Use the definitions and diagnostic principles above to answer these questions.
- An ordinary least-squares spline includes an intercept. What condition does this impose on the sum of its residuals, and why does that condition not guarantee an adequate fit?
- An observation has \(y_i=24\) response units and fitted value \(\hat{y}_i=27\) response units. Find its residual and interpret its sign in context.
- What distinct information can a residual-versus-predictor plot provide compared with a residual-versus-fitted plot?
- Two spline fits use the same observations and number of knots but different knot locations. One has lower SSE. What additional evidence should be inspected before interpreting the difference?
- A flexible spline has zero residuals at every observed predictor value, but alternative knot placements give different predictions between observations. What diagnostic concern does this raise, and what does it not prove?