Spline Complexity for Generalized Responses

Why Spline Complexity Needs Careful Comparison

A flexible trajectory can represent changes in a binary, ordinal, count or bounded biomarker that a straight line cannot. But adding spline basis functions also adds opportunities to fit sampling noise. In a joint model, this choice matters twice: it shapes the longitudinal mean and can change the estimated association between the latent marker trajectory and event risk. The goal is therefore not to maximize flexibility, but to choose a trajectory that fits adequately, gives defensible uncertainty, and remains stable under reasonable perturbations of the data or specification.

The response family and link should be chosen for the marker’s type. Spline complexity is a separate decision: it controls the shape of the predictor over time, not the response distribution. For instance, a binary marker may have a spline predictor on the logit scale; its predicted probability is obtained by applying the inverse link. Candidate models should use the same response family, link, covariates, and event-model structure when the purpose is to compare trajectory complexity.

Candidate Trajectories in a Joint Model

For subject \(i\) at visit \(j\), a generalized longitudinal submodel can be written as

Here \(g\) is the link, \(\mathbf{B}(t)\) is the spline basis, \(\boldsymbol{\theta}\) contains its coefficients, and \(\mathbf{b}_i\) denotes subject-specific random effects. For example, with a binary marker, \(g\) may be the logit link and \(\mu_i(t)\) is a conditional probability. The event component may use the latent marker value in a hazard such as

where \(m_i(t)\) is the specified latent marker feature. To isolate the effect of spline complexity, keep the event model and the definition of \(m_i(t)\) unchanged across candidates. A different trajectory can nevertheless alter \(\hat{\alpha}\), because it changes the estimated latent marker history.

A piecewise polynomial spline of degree \(d\) has polynomial pieces joined at knots; with the usual B-spline construction, the basis dimension depends on both \(d\) and the knot sequence. Greater degree or more knots can make a richer shape possible, but neither guarantees a better model. A smoothing penalty offers another route: retain a sufficiently rich basis and discourage excessive curvature. For a coefficient vector \(\boldsymbol{\theta}\), a common penalized objective is

where \(\ell\) is the relevant log likelihood, \(\mathbf{K}\) is a penalty matrix, and \(\lambda\geq 0\) controls penalty strength. The penalty is often constructed to penalize curvature while leaving selected low-order components, such as a constant or linear trend, unpenalized. A larger \(\lambda\) then means more shrinkage in the penalized directions.

What the Penalty Does: A Short Derivation

The effect of a quadratic penalty is easiest to see in a local weighted least-squares approximation, such as one used during iteratively reweighted fitting of a generalized model. Let \(\mathbf{z}\) be the working response, \(\mathbf{B}\) the evaluated basis matrix, and \(\mathbf{W}\) the working weight matrix. The penalized estimate minimizes

Differentiating with respect to \(\boldsymbol{\theta}\), setting the derivative to zero, and solving gives

Thus the fitted working values are \(\mathbf{S}_{\lambda}\mathbf{z}\), where \(\mathbf{S}_{\lambda}=\mathbf{B}(\mathbf{B}^{\top}\mathbf{W}\mathbf{B}+\lambda\mathbf{K})^{-1}\mathbf{B}^{\top}\mathbf{W}\). The effective degrees of freedom are commonly approximated by \(\mathrm{tr}(\mathbf{S}_{\lambda})\). They generally decrease as the penalty suppresses more flexible directions, though the exact behavior depends on the design and penalty matrix. In a generalized joint model this is a local approximation, not an exact universal degrees-of-freedom formula. It explains why counting all basis coefficients as freely estimated parameters can misstate the complexity of a penalized fit.

Comparing Fit Without Confusing Complexity Measures

For unpenalized candidate joint models fitted to the same observations, the maximized marginal log likelihood can be compared using information criteria. With \(k\) estimated parameters and \(n\) independent subjects, the familiar forms are

The likelihood here should reflect the joint model, including the longitudinal and event data and integration over subject-specific random effects. A lower criterion is preferred among the candidates being compared. For penalized models, a criterion must account appropriately for effective complexity; simply inserting the number of basis coefficients into an unpenalized AIC formula is not generally justified. The likelihood used for scoring should also be the likelihood, not merely the penalized objective value.

Tuning a Smoothing Penalty With Validation

When candidates share a rich basis and differ in \(\lambda\), subject-level cross-validation can assess predictive performance without relying only on an in-sample fit. In each fold, fit the joint model on training subjects and evaluate a prespecified predictive log score for held-out subjects. A joint marginal predictive score integrates over the random effects for a new subject; it should not use that subject’s held-out marker values to estimate their random effects before scoring those same values. Splitting visits from the same patient across training and validation sets can leak subject-specific information and give an overly optimistic assessment.

Choose the validation target to match the scientific use. A joint predictive score can include marker and event information. A marker-only score answers a narrower question about marker prediction and should not be described as assessing the full joint model. Repeated folds or repeated partitions help quantify variation in the score. A one-standard-error rule is a practical simplicity preference: choose the most strongly penalized candidate whose mean score is within one estimated standard error of the best mean score. It is a selection convention, not a test that models have equal predictive performance.

Uncertainty and Stability Are Part of the Choice

Fit scores alone do not show whether a complicated curve is well determined. Examine uncertainty in the estimated trajectory on the scale relevant to interpretation, and check the estimated event-association parameter under the candidate specifications. For a binary marker, for example, uncertainty may be shown for predicted probabilities as well as for the link-scale predictor. These quantities are related by the inverse link but are not interchangeable. Uncertainty calculations should reflect the fitted joint model and its dependence structure; treating repeated observations as independent can understate uncertainty.

Stability asks whether conclusions change substantially under reasonable perturbations, such as resampling subjects, modestly changing knot locations, or changing the number of knots by one. A subject-level bootstrap can summarize the spread of an estimated probability at a clinically relevant time or of the event-association estimate. Instability is a warning to inspect the data support, convergence and model specification; it is not automatic proof that the more flexible model is wrong. Conversely, a narrow interval is not a reason by itself to prefer a model if its shape is implausibly constrained or its predictive fit is poor.

Diagnostics, Alternatives, and Common Pitfalls

Before interpreting a complexity comparison, verify that every candidate uses the same subjects, response and event definitions, and likelihood integration settings. Check convergence messages, gradients, Hessian behavior, and whether different starting values lead to materially different solutions. A higher-dimensional spline can make likelihood optimization and integration more difficult; an apparent fit improvement may reflect a poorly converged comparison. Plot estimated trajectories and uncertainty on the link and response scales, as appropriate, and inspect whether changes in the estimated event association are consequential.

The main alternatives answer different questions. A small set of prespecified polynomial or low-knot models offers transparent, discrete comparisons, often using AIC or BIC when likelihood assumptions and parameter counting are appropriate. A penalized spline searches a continuum of smoothness levels and can be tuned by validation or a suitable information criterion using effective complexity. Cross-validation emphasizes predictive performance, while information criteria trade fit against complexity using an approximation based on likelihood and parameter count. None substitutes for checking the response distribution, link, event model or adequacy of the observed time coverage.

• Mistake: “The largest knot count has the best fit, so it is the best model.” The unpenalized likelihood cannot decrease merely because a nested model is made more flexible, so its increase alone does not balance added complexity. Compare an appropriate penalized criterion or validation score, and assess uncertainty and stability.
• Mistake: treating \(\lambda\) as another knot count. Knot count changes the available basis; \(\lambda\) shrinks selected combinations within a basis. State both the basis and penalty when describing a fitted trajectory.
• Mistake: using visits as independent validation units. Measurements from one subject share latent effects. Split by subject for new-subject prediction, and integrate over random effects in the held-out predictive score.
• Mistake: selecting the model with the narrowest intervals. Overly restrictive curves can produce narrow uncertainty while missing genuine temporal structure. Consider fit and calibration as well as uncertainty.

This comparison does not eliminate the need for scientific judgment. Sparse observations in parts of follow-up can make different curves hard to distinguish, and a selected model is still conditional on the chosen response family, link, random-effects structure and event association. Detailed boundary behavior and spline overfitting require particular care where data are limited near the ends of follow-up. Report the basis, degree, knot locations or selection rule, penalty and tuning procedure, criterion values, and important sensitivity findings so that the complexity choice can be assessed and reproduced.