Boundary Behavior and Spline Overfitting

Why a Flexible Trajectory Needs Careful Checking

A spline can represent gradual changes in a longitudinal marker without imposing one rigid curve over the full follow-up period. But flexibility is not automatically evidence that every bend in the fitted trajectory is real. Near the beginning or end of observed follow-up, data are often sparse and the fitted curve can be sensitive to a few observations. Within the observed range, an overly flexible spline can also turn random variation into rapid local ups and downs.

These concerns matter in joint models because the longitudinal trajectory is part of a model used alongside the event process. A poorly supported bend can affect how the marker is summarized and, in turn, how its relationship with event risk is represented. The aim here is not to remove all curvature. It is to check whether the fitted shape is stable, sufficiently precise, and useful for predicting data not used to fit it.

Model and Assumptions for Shape Diagnostics

Let \(Y_{ij}\) be the longitudinal response for subject \(i\) at time \(t_{ij}\), and let \(b_i\) denote that subject’s random effects. For a generalized longitudinal submodel, write

Here \(g\) is the response link, \(x_i\) contains fixed covariates, \(B(t)\) is a vector of spline basis functions, \(\gamma\) is its coefficient vector, and \(z_{ij}^\top b_i\) represents subject-specific effects. For example, a binary marker may use the logit link, while a count marker may use the log link. The fitted trajectory on the response scale is obtained by applying \(g^{-1}\) to the linear predictor; the shape on the response scale need not look like the shape on the link scale.

Shape diagnostics are meaningful only relative to the model and the observed time range. They assume that the response distribution and link are reasonable for the marker, that the spline basis and its boundary rules are known, and that uncertainty accounts for the fitted model’s coefficient covariance. In a joint model, interpretation also depends on how random effects and the event process enter the likelihood. A plot of the fixed spline component alone is not the same as a subject-specific fitted trajectory.

Begin by plotting the fitted trajectory over the times actually observed, marking the data density or measurement times. Inspect the link-scale curve as well as the response-scale curve, and identify whether a plotted line is population-level or conditional on particular random effects. Avoid interpreting a curve far outside the observed time range as data-supported. Even a spline with controlled tails cannot create empirical information where follow-up is absent.

Curvature and Its Uncertainty

For a fixed covariate profile and reference random effects, let the time-varying part of the linear predictor be \(\eta(t)=B(t)^\top\gamma\). Its first and second derivatives are

The second derivative describes curvature on the link scale. If \(\widehat{\gamma}\) has estimated covariance matrix \(\widehat{\Sigma}_{\gamma}\), then the fitted linear predictor at a fixed time has estimated variance

This follows from the variance rule for a linear combination: for a coefficient vector with covariance \(\widehat{\Sigma}_{\gamma}\), the variance of \(B(t)^\top\widehat{\gamma}\) is \(B(t)^\top\widehat{\Sigma}_{\gamma}B(t)\). Thus, the pointwise standard error is the square root of this expression. A pointwise 95% Wald interval on the link scale is \(\hat{\eta}(t)\pm1.96\,SE\{\hat{\eta}(t)\}\); transforming both endpoints by \(g^{-1}\) gives a corresponding interval on the response scale.

A pointwise band describes uncertainty at each selected time separately; it is not a 95% simultaneous band for the entire curve. Moreover, coefficient intervals for curvature or visual uncertainty bands do not test whether the whole shape is correct. They are evidence to consider alongside the data support, model fit, and validation.

Detecting Local Oscillation

Local oscillation is often easiest to notice by inspecting changes over nearby times on the link scale. A sequence of alternating increases and decreases over short intervals may reflect noise being followed too closely, especially when there are few observations in those intervals. But a genuine transient response can also change direction. The diagnostic question is whether the pattern is well supported and reproducible, not whether the curve is perfectly smooth.

Check the observation times, fitted values, and uncertainty together. A bend where there are no nearby observations is less convincing than one supported by repeated measurements across subjects. Compare fitted shapes under a modest reduction in spline flexibility, keeping the response distribution, link, covariates, and other modeling choices fixed where possible. If a small change in basis or knot placement substantially changes a local wiggle, that sensitivity is itself a warning. It is not an automatic instruction to select the smoother curve.

Held-Out Predictions and Model Comparison

A direct way to assess whether additional spline flexibility is useful is to compare predictions for data excluded from fitting. Split by subject, not by individual measurement: placing some measurements from one subject in the training set and others in the test set can make validation optimistic because the fitted random effects have already used information from that subject. Fit each candidate model using the training subjects, then predict the held-out measurements without using their longitudinal outcomes to estimate their random effects. The resulting comparison assesses prediction for new subjects under the specified prediction target; it is not a dynamic prediction that updates from a held-out patient’s history.

Use a score suitable for the response. For binary outcomes, the Brier score is the average squared difference between the observed value \(y\in\{0,1\}\) and its predicted probability \(\hat p\): \((y-\hat p)^2\). For counts or ordinal responses, use a score appropriate to the fitted predictive distribution, such as a log score when the probabilities are available. Compare scores on the same held-out observations. Lower average loss is better for these scores, but one split can be noisy; repeated subject-level splits or cross-validation provide a more stable comparison when feasible.

Choosing a Response to a Diagnostic Warning

The main alternatives to an unstable flexible spline are not interchangeable fixes. A lower-dimensional parametric trajectory imposes a stronger shape assumption and can be more stable, but may miss real changes. A natural cubic spline constrains the tails to be linear beyond its boundary knots, reducing tail curvature by construction; that constraint is not evidence that the data support a linear tail. A penalized spline discourages excessive roughness while allowing the data to determine some of the smoothness, but its result still depends on the penalty and its estimation. A simpler basis, alternative boundary rule, or penalized fit can therefore be useful sensitivity analyses.

For a warning near the boundary, first check how much data support the end of follow-up and whether the plotted region extends beyond it. Then compare the boundary shape and uncertainty under a reasonable alternative specification. For apparent interior oscillation, inspect local data support and sensitivity to flexibility. In either case, consider held-out performance and retain the more flexible curve only when its added structure is credible and useful.

Common failure modes include treating every wiggle as false, treating a smooth-looking line as proof of reliability, and choosing a model solely because it has the smallest training error. Another mistake is to let a boundary constraint conceal extrapolation: a linear tail imposed by a natural spline remains an assumption outside the data, not a discovery. Finally, held-out scores assess predictive performance for the chosen target and split; they do not prove that a fitted curve is the true biological trajectory.