Why Validate a Frozen Model Elsewhere or Later?
A prediction model can perform well in the data used to develop it and still perform differently when applied to new people. The new cohort may have a different distribution of risk factors, a different range of outcomes, or different procedures for measuring the predictors and outcome. Validation in a separately recruited cohort or at a later time tests how the already specified model performs under those conditions.
For a genetic prediction model, “already specified” must mean more than retaining the same published genetic weights. The score calculation, allele coding, variant handling, any non-genetic predictors, transformations, and rule for converting a score into a predicted outcome must all be specified and applied consistently. If parameters are re-estimated using the validation outcomes, the resulting analysis is no longer an evaluation of the original frozen model.
A later cohort from the same health system can be temporally independent while still sharing important features with the development setting. A separately recruited cohort may differ in recruitment, clinical practice, or measurement. These labels describe how the cohort was obtained; they do not by themselves guarantee that the validation is unbiased or representative of every population in which the model might eventually be used.
Define the Validation Target and Freeze the Procedure
Before computing metrics, state the intended prediction target and the population to which the validation result is meant to apply. Specify the outcome definition, prediction horizon, eligibility criteria, and setting. For example, predicting a diagnosis within five years among adults recruited at a particular set of clinics is a more precise target than predicting “disease risk” in general.
The model must then be applied as it would be in the intended use. A genetic score requires consistent variant definitions, effect alleles, and weight alignment. If the score is combined with age or clinical variables, their units and transformations must also match the frozen specification. Outcome measurement needs equivalent care: a change in diagnostic criteria, follow-up length, or ascertainment can change what counts as an event. Such differences should be documented rather than silently treated as model failure or success.
The analysis should use metrics suited to the prediction target. For a binary outcome, discrimination, calibration, and an overall score such as the Brier score describe different aspects of performance. For a quantitative outcome, prediction error can be summarized in outcome units. Earlier material on validation metrics defines these measures; here the key task is to compare their estimates across settings without confusing a difference in estimates with a proven difference in underlying performance.
Worked Example: Checking Event Calibration in a New Cohort
A frozen model predicts the probability of a disease event within five years for 100 people recruited at a new clinic. The sum of the 100 predicted probabilities is 18. In the observed follow-up, 20 people have the event. Calculate the observed-to-expected ratio and interpret it.
The expected event count is the sum of the individual predicted probabilities, \(E=18\), and the observed count is \(O=20\). Thus
In this cohort, the observed event count is about 1.11 times the expected count. Equivalently, there were two more observed events than the model’s summed predictions. This is a summary of aggregate agreement, not proof that every risk prediction is too low by the same amount; the ratio does not describe how well the model ranks individuals or how predictions behave across the risk range.
The interpretation also depends on the outcome and follow-up being measured as specified. If the clinic used a different event definition or the observed follow-up did not correspond to five years, the comparison would not evaluate the same prediction target. A finite cohort can show an observed-to-expected ratio other than one even when predictions are calibrated in the relevant population.
Comparing Development and Validation Performance
A development-sample estimate and an external or temporal validation estimate answer different questions. Apparent performance in the data used to fit or select a model can be optimistic, particularly when the same observations informed model construction. Independent holdout validation or cross-validation estimates are designed to address that concern, but they too have sampling uncertainty and may target a different population from the new validation cohort.
Report the metric, its uncertainty, the cohort size, and the outcome frequency or distribution alongside each estimate. A numerical difference between two point estimates does not alone establish that performance changed. For a formal comparison, uncertainty must account for how the estimates were obtained: metrics from two independent cohorts have different sampling variation from metrics computed on the same individuals. The interval or test should match that design. Even a statistically clear difference may be small in practical terms, so interpretation should remain tied to the intended use.
One useful conditional justification concerns the observed-to-expected ratio. Treat the validation predictions \(\hat p_1,\ldots,\hat p_n\) as fixed, let \(Y_i\) be the binary event indicator, \(O=\sum_iY_i\), and \(E=\sum_i\hat p_i\). If \(E>0\) and the stronger conditional calibration statement \(E(Y_i\mid\hat p_1,\ldots,\hat p_n)=\hat p_i\) holds for every individual, then linearity of conditional expectation gives
This result is conditional on the stated predictions and calibration assumption. It does not say that a realized cohort’s ratio must equal one. Nor does the weaker statement that observed and expected event counts have equal unconditional expectations, by itself, imply that the expected ratio is one.
Worked Example: Comparing AUC Across Recruitment Periods
Suppose a development analysis reported an AUC of 0.82. A later-time cohort was scored with the frozen model, and its AUC was 0.75. To see how the latter estimate can arise, consider four cases and four non-cases with the following predicted scores. Assume there are no tied scores.
Cases have scores 0.9, 0.7, 0.6, and 0.4. Non-cases have scores 0.8, 0.5, 0.3, and 0.2. The AUC is the fraction of case–non-case pairs in which the case has the higher score. There are \(4\times4=16\) such pairs. The case with score 0.9 exceeds all four non-case scores; the case with score 0.7 exceeds three; the case with score 0.6 exceeds three; and the case with score 0.4 exceeds two. Therefore,
In this small illustration, a randomly selected case has a higher score than a randomly selected non-case in 75% of pairs. The validation point estimate is 0.07 lower than the development estimate of 0.82. This calculation alone does not show that the model’s underlying discrimination has declined: the validation estimate has sampling uncertainty, and the two cohorts may differ in the range and mix of cases and non-cases. A narrower or broader case spectrum can change AUC even when the scoring rule is unchanged. Report the uncertainty and cohort characteristics before drawing a conclusion about a temporal change.
Case Mix and Measurement Can Change Performance
Case mix refers to the distribution of characteristics in the validation population that affect prediction or the observed outcome. A later cohort may have a different age distribution, baseline risk, referral pattern, disease severity, or prevalence of the outcome. The model’s predictions can therefore have a different distribution, and metrics may change even though the model’s coefficients have not.
The type of metric matters. AUC describes ranking of cases relative to non-cases and can depend on how distinct or heterogeneous those groups are. Measures that depend on absolute risks are also affected by event frequency and baseline risk. For quantitative outcomes, error depends on the range and scale of the observed values. These are reasons to report cohort composition and outcome summaries, not reasons to dismiss an inconvenient validation result.
Measurement changes can be equally important. Genotype platforms, imputation procedures, variant quality rules, allele coding, or score standardization can alter the predictor values. Changes in phenotype definitions, laboratory assays, follow-up, or event ascertainment can alter the outcome. The validation report should distinguish the intended model application from departures in the data-generation process. If the implementation differs from the frozen specification, describe the deviation and do not present the result as an unqualified evaluation of that specification.
Worked Example: Comparing Quantitative-Trait Prediction Error
A frozen model predicts a quantitative trait in the same units in two cohorts. Its development-sample RMSE was 0.80 g/L. In a later cohort, the observed and predicted values for four hypothetical participants are as follows.
| Participant | Observed (g/L) | Predicted (g/L) | Prediction error (g/L) |
|---|---|---|---|
| A | 10 | 11 | −1 |
| B | 12 | 11 | 1 |
| C | 14 | 13 | 1 |
| D | 16 | 15 | 1 |
Using error \(Y_i-\hat Y_i\), the validation RMSE is the square root of the mean squared error:
The validation RMSE is 1.00 g/L, compared with 0.80 g/L in development, a point-estimate increase of 0.20 g/L. The errors here are all within 1 g/L, but four observations are far too few to establish a stable population-level difference. In a real comparison, the uncertainty and the outcome’s distribution in each cohort matter. If the later cohort used an assay with a shifted scale, the change might partly reflect measurement rather than a change in the model’s predictive relationship.
How to Report a Validation Result
A clear report lets readers judge what was evaluated and where the result applies. Describe the cohort’s source and recruitment period, the target population and outcome definition, and the prediction horizon. State that the score and prediction procedure were frozen, and document any differences in predictor or outcome measurement. Give the relevant performance estimates with uncertainty, along with enough information about outcome frequency or distribution and case mix to interpret them.
If the model is subsequently updated or recalibrated using validation data, distinguish that analysis from the primary evaluation of the frozen model. The updated procedure may be useful, but its performance in those same data is not an independent validation of the updated model. Any claim about performance in another population should be limited to what the validation design and cohort support.
Common Mistakes
- Calling a model external validation after refitting it. If outcome data from the validation cohort are used to change its weights or prediction rule, the result is not performance of the original frozen model.
- Reading a point-estimate difference as proof of deterioration. A difference between metrics must be interpreted with their uncertainty and the cohorts’ designs.
- Treating case mix as irrelevant. Cohort composition can alter measured performance, particularly for metrics sensitive to the distribution of cases, non-cases, or outcomes.
- Ignoring changed measurement. Differences in variant processing, outcome definitions, assays, or follow-up may mean the model was not evaluated on equivalent inputs or targets.
- Overinterpreting an observed-to-expected ratio. A finite-sample ratio need not equal one under calibration; the conditional expectation result requires predictions treated as fixed and the stated conditional calibration assumption.
Check Your Understanding
Answer each question using the model, cohort, and measurement context in the prompt.
- What distinguishes temporal validation from validation in a separately recruited cohort, and what must be fixed before either evaluation?
- A validation cohort has an observed-to-expected ratio of 1.15. What does this ratio summarize, and what can it not establish on its own?
- Why can the AUC differ between a development cohort and a later cohort even when the model’s scoring rule is unchanged?
- A genetic score uses different allele coding in the validation data than in development. Why is this a threat to interpreting the result as validation of the frozen model?
- Why should a report include uncertainty and information about outcome measurement and case mix when comparing performance estimates?