Why the Development Sample Can Make a Model Look Better
A joint model uses longitudinal measurements and time-to-event information to estimate quantities such as a patient’s event risk by a specified horizon. Its predictions may depend on estimated population parameters, the association between the longitudinal and event processes, and the patient’s observed measurement history. When the model is fitted and evaluated on the same patients, the evaluation reflects both predictive information and the model’s adaptation to those patients. This can make performance appear better than it will be for new patients.
The distinction matters whether performance is summarized by a discrimination measure, a calibration measure, or a prediction score. Harrell’s concordance, horizon-specific calibration, and the Brier score summarize different properties, but none automatically turns a development-sample calculation into an estimate of performance for future patients. A metric can be calculated correctly and still answer the wrong validation question.
The target must be stated clearly. One target is the performance of the particular fitted model \(\hat f_D\) for new patients from the target population. Another is the average performance of the entire development procedure, including model fitting and any data-driven selection, across hypothetical repetitions of development samples. These targets differ: the second includes variation in which model would be fitted from one sample to another.
What “Optimism” Means
For a metric where larger values indicate better performance, define the apparent advantage, or optimism, as the difference between apparent and out-of-sample performance. For a loss where smaller values are better, such as the Brier score, it is convenient to define optimism as out-of-sample loss minus apparent loss. With these conventions, positive optimism means the development-sample result looks better than the out-of-sample target.
In a repeated-sampling formulation, the generalization gap compares the metric evaluated on the data used to fit a model with the metric evaluated on independent data. For a larger-is-better metric \(M\), one expression is
where \(D^\ast\) is an independent sample from the target population, and the inner expectation holds the fitted model fixed. The equation describes a target contrast; it does not say that the gap is always positive for every metric, data set, or fitting procedure. Positive optimism is common when a flexible model has learned sample-specific features, but noise, finite samples, metric definitions, or changes in the target population can produce a small, zero, or even negative observed difference.
Why Reusing the Development Data Creates Optimism
A fitted model is chosen in part because its parameter estimates represent patterns in the development data. Some of those patterns are genuine and will recur in new patients; some reflect sampling variation. Reusing the same data for evaluation rewards the model for fitting both. An independent evaluation does not give the model that same advantage.
In a joint model, estimation can involve longitudinal fixed effects, random-effects distributions, event-model parameters, and parameters linking the longitudinal and event processes. If investigators compare candidate association structures, choose predictors, or tune model complexity using the same data, those decisions add adaptation. The apparent performance of the selected model can then reflect both parameter fitting and selection for favorable results.
The following derivation gives a simple illustration of the same mechanism. It is not a formula for the optimism of a survival metric. Suppose a Gaussian linear model is correctly specified, has a fixed design matrix \(X\) of full column rank \(p\), and has independent errors with mean zero and variance \(\sigma^2\). The training outcome is \(Y=X\beta+\varepsilon\), with \(\varepsilon\sim N(0,\sigma^2 I)\). The fitted values are \(HY\), where \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\) is a projection matrix with trace \(p\). Therefore the training residual sum of squares is \(\varepsilon^\mathsf{T}(I-H)\varepsilon\). Taking expectations gives \(\sigma^2\operatorname{tr}(I-H)=\sigma^2(n-p)\). For an independent replicate \(Y^\ast=X\beta+\varepsilon^\ast\) at the same design points, the prediction errors are \(\varepsilon^\ast-X(\hat\beta-\beta)\). The independent noise contributes \(n\sigma^2\), and the expected squared fitted-mean error is \(\sigma^2\operatorname{tr}(H)=p\sigma^2\). Thus expected test error is \(n\sigma^2+p\sigma^2\), rather than the smaller training residual total.
The assumptions here are deliberately restrictive: a correctly specified Gaussian model, fixed design, and an independent replicate at those same design points. The result illustrates how fitting parameters can reduce training error without reducing future prediction error. It should not be transferred numerically to a joint model’s concordance or censored-data prediction score.
Worked Example: Training and Test Error in a Simple Analogue
Suppose the Gaussian illustration has \(n=40\) observations, \(p=8\) fitted coefficients, and error variance \(\sigma^2=4\) squared units. Assume the design is full rank and the model assumptions just stated hold. The expected training MSE is
The expected test MSE at an independent replicate of the same design is
The expected difference is \(4.8-3.2=1.6\) squared units. The lower training error is not evidence that the model predicts new outcomes more accurately; it partly reflects using the training outcomes to estimate eight coefficients. This calculation illustrates the mechanism, not an optimism correction for a joint-model metric.
Interpreting Apparent Metrics in Context
The direction of “better” depends on the metric. For concordance, a larger value generally indicates better discrimination under the metric’s definition. For the Brier score, a smaller value indicates smaller squared prediction error. Calibration is different again: closeness to ideal calibration, rather than simply a larger or smaller summary, is the goal. Optimism comparisons must respect the metric’s scale and interpretation.
For censored event times, a metric may use weighting or other methods to account for incomplete outcome observation. Its interpretation depends on the relevant assumptions, including appropriate handling of censoring and adequate support for the weights where inverse-probability weighting is used. Estimating a metric correctly under those assumptions does not, by itself, address reuse of the development data. Conversely, an apparent-versus-out-of-sample comparison is not meaningful if the metric’s censoring assumptions or target horizon are inappropriate.
Worked Example: Apparent and Independent Brier Scores
A hypothetical joint model predicts event risk by 2 years. Suppose event status by 2 years is fully observed for each patient in both samples, so this illustration needs no censoring weights. For the five development patients, predicted risks and event indicators produce squared errors \(0.04, 0.09, 0.01, 0.16,\) and \(0.00\). The apparent Brier score is
For five hypothetical new patients, evaluated with the fitted model held fixed, suppose the squared errors are \(0.09, 0.16, 0.04, 0.25,\) and \(0.04\). The independent-sample score is
Here the apparent advantage on the lower-is-better loss scale is \(0.116-0.060=0.056\). The development patients’ average squared prediction error is \(0.056\) lower in this small illustration. This difference is not proof that the model has exactly \(0.056\) optimism in its target population: five patients per sample give a noisy comparison, and the samples may not represent the target population well. The example also cannot be used to claim that the difference is statistically significant.
Worked Example: Concordance Is Not a Prediction-Error Loss
Suppose a hypothetical fitted joint model has apparent Harrell concordance \(0.74\) on its development patients. A separate, suitable evaluation sample gives concordance \(0.66\), using the same prediction horizon and concordance definition. Since higher concordance indicates better ranking under that definition, the observed apparent advantage is
The development-sample concordance is 0.08 higher on this scale. It is not appropriate to describe the difference as an 8% increase in the probability that a particular patient will experience an event, or as an 8-percentage-point increase in calibration. Concordance summarizes ordering of comparable pairs; it is not an absolute-risk error measure. The comparison assumes the samples and censoring treatment make the same target estimand meaningful. The difference alone does not establish its uncertainty or identify its cause.
Evaluation Must Reproduce the Intended Prediction Task
For a joint model, an evaluation can look optimistic for a reason distinct from parameter overfitting: information unavailable at the intended prediction time may have entered the prediction. For example, a prediction made at a landmark time should use only longitudinal history available by that time. Using later measurements to estimate an individual’s latent trajectory and then treating the result as a prediction made at the earlier landmark leaks future information into the evaluation.
This distinction is important. Optimism concerns the advantage from fitting or selecting a model on the evaluation patients. Leakage concerns whether the prediction uses information that would not exist when the real prediction is made. An evaluation can suffer from either problem, both, or neither. A well-defined target specifies the population, prediction time, horizon, available patient history, event outcome, and censoring treatment.
Worked Example: Detecting Information Leakage
A hypothetical model is intended to predict 1-year event risk at month 6, using longitudinal measurements collected through month 6. An analyst calculates each development patient’s predicted risk using that patient’s full longitudinal record through month 12, then reports the apparent 1-year score from month 6. Even if the model parameters were otherwise appropriate, this does not evaluate the intended prediction task: months 7–12 measurements were not available at the month-6 prediction time.
The correction is to construct each prediction using only measurements observed by month 6, and to define outcomes and follow-up consistently with the 1-year horizon. If the model was also fitted on these patients, the resulting metric remains apparent performance and may still be optimistic. Correct timing prevents leakage; it does not make development-sample evaluation out of sample.
Diagnostics, Alternatives, and Limitations
A large apparent score by itself is not a diagnostic of successful prediction. Useful warning signs include a complex model relative to the available information, data-driven variable or association-structure selection, unusually strong apparent calibration or discrimination, and a marked performance decline on an independent evaluation. These are clues, not proof: differences can also reflect sampling variation, differences in patient mix, outcome ascertainment, or censoring.
The main evaluation approaches answer related but different questions. The table summarizes their roles without prescribing a particular correction procedure.
| Evaluation result | What it summarizes | Main caution |
|---|---|---|
| Apparent performance | Metric for the model on its own development patients | Can be favorable because fitting and evaluation reuse the same data |
| Optimism-adjusted internal estimate | An attempt to account for development-sample adaptation | Depends on the adjustment and its implementation; it is not external validation |
| Independent validation | Performance for patients not used to fit the evaluated model | May be imprecise or poorly transportable if the validation population differs |
Uncertainty intervals and optimism address different issues. An interval quantifies uncertainty under the sampling and analysis procedure used to construct it. An interval around an apparent metric does not change the fact that the metric was evaluated on the fitting data. A narrow interval can therefore precisely describe an apparent quantity while leaving its gap from out-of-sample performance unresolved.
Likewise, an apparent result is not useless. It can describe fit to the development sample, identify gross problems, and support preliminary comparisons made with appropriate caution. Its limitation is interpretive: without an approach that addresses reuse of data, it should not be presented as an unbiased estimate of performance for future patients.
Common Mistake
Correction: A low apparent Brier score describes errors in the development sample, and its interval describes uncertainty according to the interval procedure. Neither alone removes the advantage that may arise from fitting on those same patients. State the target population and prediction task, and distinguish the apparent estimate from an estimate intended to represent new patients.
Check Your Understanding
Answer each question, then compare with the solutions.
- A joint model has apparent concordance \(0.70\) and suitable out-of-sample concordance \(0.64\). What is the apparent advantage, and what does it mean?
- A model has apparent Brier score \(0.12\) and out-of-sample Brier score \(0.17\). Calculate optimism using the lower-is-better convention.
- Does a positive optimism value have to occur for every metric and every fitted model? Explain.
- Why does using measurements collected after a prediction landmark create a problem different from fitting and evaluating on the same patients?
- What does a narrow uncertainty interval around an apparent metric establish, and what does it not establish?
Solutions. 1. The advantage is \(0.70-0.64=0.06\); on this concordance scale, the development-sample value is 0.06 higher, not 6% more accurate in every sense. 2. The loss optimism is \(0.17-0.12=0.05\), so the apparent loss is 0.05 lower in this comparison. 3. No. Positive optimism is common when fitting adapts to sample-specific patterns, but the observed gap can be zero or negative because of sampling variation, metric behavior, or population differences. 4. Later measurements are unavailable at the prediction time, so using them leaks future information; same-sample evaluation instead concerns reuse of patients for fitting and evaluation. 5. It quantifies uncertainty for the apparent quantity under the interval method’s assumptions. It does not turn the apparent estimate into an out-of-sample estimate or remove optimism.