Cross-Validation for Joint-Model Prediction

Why Cross-Validate at the Subject Level?

A joint model uses longitudinal measurements and time-to-event outcomes together. Its predictions depend on estimated model parameters, and often on subject-specific latent effects. Evaluating predictions on the same subjects used to estimate those quantities can make performance look better than it will be for new subjects. Cross-validation addresses this optimism by repeatedly fitting the model on one set of subjects and evaluating predictions on a separate set.

The unit of splitting matters. A subject may contribute many longitudinal records but only one event or censoring time. If some of that subject’s records enter the training data while other records are treated as held out, the held-out predictions are not independent of the training process. A model may already have learned information about that person. The solution is to assign each subject, with all of their records and survival outcome, to exactly one fold.

Write subject \(i\)’s longitudinal measurements as \(Y_i(t)\), its observed event or censoring time as \(T_i\), and its event indicator as \(\Delta_i\), where \(\Delta_i=1\) indicates an observed event and \(\Delta_i=0\) indicates censoring. Let \(I_k\) be the set of subjects in fold \(k\). The training set for that fold is all subjects not in \(I_k\). The model is refitted from the beginning for each training set.

Define the Prediction Task Before Splitting

Cross-validation evaluates a particular prediction task, not an abstract model in isolation. Specify the prediction time, the survival quantity to predict, and what information about a held-out subject is available at that time. For example, a task may predict survival through a fixed horizon \(u\) using only information available at baseline, or predict survival beyond a landmark time \(t_0\) using measurements recorded on or before \(t_0\).

For a landmark prediction, write the held-out subject’s available history as \(H_i(t_0)\). A prediction might be the conditional survival probability \(\hat S_i(u\mid t_0,H_i(t_0))\), for \(u>t_0\). The history may be used to infer a subject-specific latent effect if that matches the intended prediction procedure. But measurements recorded after \(t_0\), or the subject’s future event outcome, must not be used to construct the prediction. At baseline, no follow-up history is available, so a prediction must instead use baseline information and the model’s population distribution for latent effects.

This distinction prevents a subtle form of leakage: withholding a subject from parameter estimation does not by itself make a prediction valid if future information from that subject is then used to personalize it. The allowed information must match the real use of the prediction.

What Must Be Refit Within Each Fold?

Every data-dependent part of the modeling pipeline must be learned using the training subjects only. This includes the joint-model parameters, baseline hazard estimates, longitudinal and survival submodel parameters, and any estimated covariance or random-effects distribution. If the pipeline includes imputation, scaling, variable screening, or other preprocessing, its learned quantities must also come from the training fold. Apply those training-derived quantities to the held-out fold without re-estimating them there.

For a fixed, prespecified model, the essential procedure is:

The cross-validated estimate for a subject-level loss \(L_i\) is the average of the losses from the predictions made while each subject was held out:

For a loss such as the Brier score, smaller values indicate better predictions. For a score defined so that larger values are better, retain that score’s direction when interpreting the result. If folds have unequal sizes, the pooled subject-level average above weights each fold according to its number of subjects; simply averaging fold means would give equal weight to folds of different sizes.

Why the Cross-Validation Estimate Has This Form

The justification is that each subject’s loss is evaluated using a fit that excludes that subject. Under independent, exchangeable sampling of subjects from the target population, a randomly assigned held-out fold behaves like a sample of new subjects relative to the model fitted on the remaining subjects. The average held-out loss therefore estimates predictive risk for the fitting procedure trained on approximately \(n(K-1)/K\) subjects.

More precisely, if a fixed fitting procedure applied to a training sample \(D\) produces predictions \(\hat f_D\), its expected risk is the expected loss on a new subject drawn from the same target population. For each fold, the held-out subjects approximate such new subjects relative to that fold’s training data. Averaging their losses estimates the procedure’s risk. This argument requires that the held-out outcomes are used for scoring, not for choosing or altering that fold’s fitted model.

Cross-validation does not exactly estimate the risk of a model fitted on all \(n\) subjects: each fold’s model uses fewer than \(n\) subjects. With fewer training subjects, performance may be slightly worse than that of the final fit using all available data. The estimate is also conditional on the modeling procedure and prediction task that were evaluated.

Assumptions, Diagnostics, and Common Failure Modes

The interpretation of ordinary subject-level cross-validation relies on several conditions. Subjects should be independent sampling units for the intended target population, or the folds should otherwise respect the dependence structure. The folds should represent the prediction setting of interest. The model must be fitted successfully in each training set, and censoring must be handled appropriately by any metric that requires it. For landmark predictions, the recorded history used to personalize predictions must stop at the landmark.

Inspect the fold assignments and model fits rather than treating the final average as sufficient evidence. Check that every subject appears in exactly one held-out fold, that no subject contributes records to both training and held-out sets, and that each fold has enough events and follow-up for stable fitting. Record convergence warnings, boundary estimates, and failed fits. A metric calculated only from folds where fitting succeeded can be misleading if failure is related to the data or outcome; report failures and address them transparently.

Other warning signs include implausibly strong performance relative to the apparent estimate, predictions that use measurements after the stated prediction time, or preprocessing that was estimated using all subjects. A large difference between folds may reflect instability, uneven event counts, or genuine differences in their held-out subjects. The folds are not independent replications: their training sets overlap, so their metric values should not be treated as independent observations for a routine standard-error calculation.

• Splitting measurement rows instead of subjects. This puts information about one person in both sets. Assign the whole subject to one fold.
• Fitting once on all subjects. Predictions from that fit are apparent predictions, not cross-validated predictions. Refit the joint model separately for every training fold.
• Using held-out outcomes to revise the fit. Held-out outcomes are for scoring. Do not use them to alter model parameters or the prediction pipeline for that fold.
• Using future history for a landmark prediction. This leaks information unavailable at the prediction time. Restrict subject-specific updating to the permitted history.

Cross-validation differs from apparent performance because the latter evaluates predictions on the fitting data. It differs from a single holdout split because each subject is evaluated once rather than only the subjects assigned to one test set, though a single split may be useful when a sufficiently large independent test sample is available. Bootstrap optimism correction is another internal-validation approach; it estimates and corrects optimism using bootstrap resamples rather than fold-specific fits. Each approach has its own sampling variability and assumptions.

Use cross-validation to assess a prespecified model-building procedure. If the same cross-validation results are also used to choose among many models or tuning options, the reported best performance can be optimistic. The selection problem requires a separate validation design; it should not be resolved by interpreting the selected model’s same cross-validation score as an unbiased final estimate.