Why Evaluate Covariate Models by Prediction?
A covariate model can fit the observed data well yet predict new subjects poorly. This can happen when a strategy captures chance patterns in the development sample, uses too many covariate terms for the available information, or is evaluated using records that indirectly reveal the outcomes being predicted. Cross-validation addresses a focused question: how well does a specified modeling strategy predict data that were not used to fit it?
In nonlinear mixed-effects models, the unit of prediction matters. Measurements from the same subject are usually dependent through that subject’s random effects. Therefore, randomly assigning individual records to training and validation sets can answer an easier, different question than predicting a new subject. The validation design must match the intended use of the model.
Set the Prediction Target Before Making Folds
Write \(y_i\) for the complete vector of observations from subject \(i\), \(x_i\) for that subject’s covariates, and \(b_i\) for the subject-specific random effects. A candidate model \(M\) specifies a population parameter vector \(\theta_M\), a random-effects distribution, and the conditional model for the observations. In a typical NLME formulation, \(b_i\) follows a population distribution such as \(N(0,\Omega_M)\), and the observations are modeled conditional on \(b_i\), the covariates, and the model parameters.
For prediction in a new subject, that subject’s outcomes have not informed an estimate of their random effects. A population-level predictive distribution therefore averages over the random-effects distribution:
Here \(\widehat{\theta}_M\) is estimated using only the training subjects. For a squared-error loss, the prediction is the mean of this predictive distribution. Other choices include a negative log predictive density, which scores the full predictive distribution rather than just its mean. The same target and loss should be used to compare candidate covariate strategies.
If the intended task is instead to predict later measurements for a subject whose earlier measurements are already available, the target is conditional on that permitted history. The prediction may use information about that subject’s random effects learned from the earlier measurements. That is a valid but different prediction problem: validation must hold out later records while allowing only the earlier records that would truly be available at prediction time.
What the Cross-Validation Score Estimates
Let \(L_i(M)\) be the chosen loss for held-out subject \(i\), calculated from that subject’s covariates, outcomes, and predictions made by a model fitted without that subject. If each subject is weighted equally, a cross-validation estimate of mean subject-level loss is
For example, if \(L_i(M)\) is that subject’s mean squared prediction error, lower values are better and the units are the square of the measurement units. Taking the square root of the average squared error gives an RMSE in the original measurement units. Equal subject weighting gives each person equal influence; weighting all observations equally answers a different question when subjects contribute different numbers of records.
The basic justification is that each held-out subject is scored by a model that did not use that subject’s outcomes. Under independent sampling of subjects from the target population, the held-out loss therefore provides information about prediction error for subjects not used to fit that fold’s model. Averaging across folds uses each subject as held out once. This does not make the score exactly equal to the risk of a final model fitted to all \(n\) subjects: each fold model is fitted to fewer subjects. Nor does it remove uncertainty due to which subjects happened to be sampled.
A fair comparison uses the same folds for all candidate strategies, refits each model on each training set, and applies the same prediction target and loss. Any data-dependent step—including covariate screening, transformations chosen from the data, or imputation—must be performed using training data only. Otherwise, information from the held-out outcomes can influence the model being evaluated.
Decide whether validation concerns entirely new subjects or future observations from subjects with an available history.
For new-subject prediction, assign subject identifiers to folds, then place every record for each subject in that subject’s fold.
For each fold and candidate strategy, use training data alone to estimate NLME parameters and perform any data-driven screening or tuning.
Predict the held-out units according to the target, calculate the prespecified loss, and aggregate comparable losses across folds.
Worked Example: Comparing Two Covariate Strategies
Suppose eight hypothetical subjects are divided into four subject-level folds, with two subjects per fold. The outcome is a concentration measured in mg/L. For each subject, use the mean squared error across that subject’s held-out records as the subject-level loss, measured in \((\mathrm{mg/L})^2\). A baseline strategy \(M_0\) and a covariate strategy \(M_1\) are both fitted separately on each fold’s training subjects.
The hypothetical subject-level losses, in fold order, are \(M_0: 4,1,9,4,1,4,9,0\) and \(M_1: 1,4,4,1,1,1,4,0\). For \(M_0\), the mean loss is
For \(M_1\), the mean loss is
The corresponding subject-weighted RMSE values are \(\sqrt{4.00}=2.00\) mg/L for \(M_0\) and \(\sqrt{2.00}=1.414\) mg/L for \(M_1\). Under this particular validation design and loss, \(M_1\) has lower estimated prediction error. This comparison alone does not establish that its covariate effect is nonzero in the population or that the model will perform better in every future population.
Why Subject-Level Folds Matter
If one subject’s records appear in both training and validation sets, the fitted model may benefit from information about that subject that would not be available for a genuinely new subject. In an NLME model, repeated measurements can reveal a subject’s deviation from the population through their random effects. Using some records to estimate that deviation and then evaluating predictions on the same subject’s remaining records can make validation error look artificially small for a new-subject prediction target.
The problem is not that record-level splitting is always invalid. It is inappropriate when the intended target is a new subject. For forecasting later observations from known subjects, splitting within subjects can be appropriate if the split respects time order and the model uses only the history available at the forecast time. The split design defines what “held out” means; it cannot be chosen independently of the prediction question.
Worked Example: Recognizing Subject Leakage
Imagine a hypothetical data set with six subjects, each measured four times. The aim is to predict concentrations for new subjects. A random record split puts some records from every subject in training and the rest in validation. Its reported pooled validation mean squared error is \(0.64\;(\mathrm{mg/L})^2\). The resulting RMSE is
Now regroup the same subjects so that all four records from each subject are held out together, and evaluate the new-subject prediction target. Suppose the pooled held-out mean squared error is \(4.41\;(\mathrm{mg/L})^2\). Then
The record-split result is not evidence that new-subject prediction has an RMSE of 0.800 mg/L: training included other records from the validation subjects. The grouped result is the relevant estimate for the stated target. The difference in these hypothetical results illustrates how leakage can produce an overly favorable score; it does not imply that grouped errors must always be larger in every data set.
Selection, Diagnostics, and Alternatives
Cross-validation can compare a prespecified set of candidate strategies, but selecting the strategy with the smallest cross-validation score and then reporting that same minimum as an unbiased estimate of its performance can be optimistic. The minimum was chosen partly because of favorable validation noise. If the goal is to estimate the performance of the entire selection procedure, use nested cross-validation: an inner process selects the covariate strategy using only the outer training subjects, and an outer held-out fold evaluates that selected strategy. The outer scores then assess the selection procedure, not just one model chosen in advance.
Worked Example: Nested Cross-Validation for a Selection Procedure
Consider a hypothetical three-fold outer cross-validation. Within each outer training set, an inner subject-level cross-validation selects one of two covariate strategies. That selected strategy is refitted on the full outer training set and evaluated on the outer held-out subjects. Suppose the three outer-fold mean squared errors are \(1.20\), \(1.80\), and \(1.50\;(\mathrm{mg/L})^2\), with equal numbers of held-out subjects per fold. The estimated mean loss for this selection procedure is
The corresponding RMSE is \(\sqrt{1.50}=1.225\) mg/L. This estimates performance of the procedure that selects by inner cross-validation and then refits, under this outer-fold design. It is not a direct estimate of either candidate’s performance selected once using all available subjects. If outer folds contain different numbers of subjects, aggregate subject losses directly or use an appropriately weighted fold average rather than treating unequal fold means as equally precise.
Useful checks include confirming that subject identifiers never cross folds, verifying that all preprocessing is repeated within training data, and examining fold-specific losses rather than only their average. Large differences among folds can signal sensitivity to a few subjects, sparse covariate groups, or an unstable fitting process. NLME fits may also fail to converge in some training folds; failures should be recorded and investigated, not silently dropped, because excluding difficult folds can distort the comparison. A model that repeatedly fails to fit is not a dependable prediction strategy merely because its successful fits score well.
Cross-validation is one tool among several, and the tools answer different questions. Information criteria summarize fit with a complexity adjustment under their modeling conditions; likelihood-based tests and Wald statistics address evidence about parameters under their assumptions. Bootstrap stability examines how selected effects vary under resampling. Cross-validation focuses on held-out prediction. Agreement among these approaches can be informative, but a prediction score is not a substitute for inference about a population covariate effect, and a statistically supported effect is not by itself proof of improved prediction.
| Approach | Primary question | Important distinction |
|---|---|---|
| Subject-level cross-validation | How well does the fitting strategy predict held-out subjects under a chosen loss? | Requires folds and scoring that match the prediction target. |
| Likelihood-based comparison or information criterion | How does model fit compare after accounting for model complexity under specified assumptions? | Does not directly measure held-out prediction error. |
| Bootstrap stability | How often or how consistently are effects selected across resampled data sets? | Selection stability is not the same quantity as predictive accuracy. |
Cross-validation also has limitations. With few independent subjects, fold-level results can be highly variable, and each fold fit uses only a fraction of the available subjects. Predictions are specific to the covariate distribution, measurement schedule, and population represented by the sampled subjects. If subjects are clustered within sites or families and the goal is prediction at a new site or within a new family, folds may need to hold out those higher-level units instead. No resampling design can make an unrepresentative development sample representative of a different target population.
Common Mistake: Treating a Lower Score as Evidence of an Effect
A common error is to conclude, “The covariate strategy has the lower cross-validation error, so the covariate has a real population effect.” The first clause may describe a comparison under a particular prediction target and loss; the second is an inferential claim that does not follow from that comparison. A covariate can improve prediction without a precisely estimated effect, and a population effect can exist without materially improving prediction in the validation setting. Report the predictive comparison as such, and use appropriate inferential analyses when the question concerns population effects.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For prediction in new subjects, should repeated records from one subject be assigned to different folds? Why?
- A model has a subject-weighted cross-validation mean squared error of \(9.00\;(\mathrm{mg/L})^2\). What is its RMSE, and what does that value summarize?
- Why can selecting the lowest-scoring covariate strategy and reporting that same minimum as its performance be optimistic?
- What prediction target can justify splitting records within a subject?
- Does a lower cross-validation loss establish that a covariate has a nonzero population effect?
Solutions. (1) No. All records from a subject should be in one fold, because training on some records can reveal information about that subject and make new-subject performance appear better than it is. (2) The RMSE is \(\sqrt{9.00}=3.00\) mg/L. It summarizes the square root of the average subject-level squared prediction error for held-out subjects under that validation design; it is not a guarantee about an individual prediction. (3) The winning minimum may partly reflect favorable validation noise, because the strategies were compared and the smallest score was selected. Nested cross-validation can assess the selection procedure. (4) Forecasting future observations for subjects whose earlier records are available can justify within-subject splitting, provided training uses only history available before the forecast. (5) No. Lower predictive loss concerns prediction under the selected target and loss; it does not by itself establish a population effect.