Two Evaluations, Two Targets
A genomic prediction model can perform well in cross-validation and less well in a separate cohort—or the reverse. That difference is not automatically a contradiction. Cross-validation and external validation use different data arrangements, and generally estimate performance for different targets. Interpreting them requires stating what model was evaluated, what population the observations represent, and whether the external data remained independent of model development.
In internal cross-validation, a development data set is repeatedly divided into training and validation folds. A full analysis pipeline is fitted using each training portion, then assessed on observations held out from that fitting process. The resulting estimate concerns the model-building procedure applied to data from the development setting, usually with fewer observations per training fit than the full development sample. For high-dimensional genomic analyses, feature screening, imputation, dimension reduction, tuning, and threshold selection must be handled without using the held-out fold; otherwise the estimate can be optimistic.
External validation instead evaluates a specified fitted model in a separate cohort that was not used to choose its features, preprocessing, hyperparameters, calibration, or decision threshold. It asks how that locked model performs on the external observations. Independence is a data-use condition, not simply a different file name or institution: overlap, indirect reuse, or repeated adaptation to the external results weakens the claim of independent evaluation.
For a loss measure, let \(A\) denote the full development procedure and \(f_D\) the fitted model obtained from development data \(D\). If \(P\) is the development-data distribution, an idealized cross-validation target is the expected loss of a model trained on a subset of size \(m\) drawn from \(P\):
Here \(L\) is a specified loss, such as squared prediction error or misclassification loss. In contrast, for an external cohort drawn from distribution \(Q\), validation of the locked model targets:
These expressions clarify why the estimates need not agree. Cross-validation averages performance across fitted models and held-out folds; external validation assesses one fitted model on another cohort. The training size \(m\) in cross-validation may be smaller than the size of \(D\), and \(Q\) may represent a different sampling frame from \(P\). A sampling frame is the process and eligibility conditions determining which individuals could enter a cohort. The two estimates are directly comparable only with care about these differences and the performance measure being reported.
What Must Be Fixed and What Must Be Independent
A credible external evaluation starts by specifying the prediction task and evaluation plan before inspecting external outcomes. This includes the outcome definition and follow-up window, the predictor measurements available at prediction time, the performance measures, and any classification threshold. The fitted coefficients and all learned processing steps must also be fixed. For example, if development data determine imputation values, gene filtering, scaling constants, principal components, or a probability calibration map, those same fitted quantities are applied to external observations; they are not re-estimated using the external cohort.
The external cohort must be independent in the relevant sense. Participants should not have contributed to model fitting or tuning, and external outcomes should not have informed repeated revisions that are then reported as if the final evaluation were untouched. A cohort can be geographically separate yet not independent if samples overlap. Conversely, a cohort collected at the same institution may provide independent evaluation if its participants and outcomes were genuinely held out and the model was locked in advance.
The evaluation plan also needs enough data for the intended metrics. A small external cohort can yield a noisy estimate, especially for rare outcomes, subgroup measures, or calibration summaries. An estimate from an independent cohort is not automatically precise or representative. Report the denominator and uncertainty alongside a metric, and describe the cohort's recruitment and eligibility rather than treating “external” as a guarantee of broad generalizability.
Worked Example: Comparing Cross-Validated and External Accuracy
Suppose a hypothetical genomic classifier is evaluated by pooled out-of-fold predictions for 100 people from a development cohort. It classifies 82 correctly. The locked classifier is then evaluated in an independent cohort of 80 people, where it classifies 58 correctly. Assume each person contributes one independent observation, the same outcome definition and threshold are used, and no external results were used to alter the classifier.
The development-cohort cross-validated accuracy is:
The external-cohort accuracy is:
The observed difference, external minus cross-validated accuracy, is \(0.725-0.820=-0.095\), or 9.5 percentage points lower in the external cohort. This is a descriptive difference, not proof that the model has deteriorated or that the cohorts differ for any particular reason. The estimates have sampling variability, the cross-validation models were fitted to subsets rather than the entire development cohort, and the cohorts may have different sampling frames. A defensible report gives both values, their denominators, and uncertainty, then describes relevant design differences without claiming that the difference is itself causal evidence.
Sampling Frame and the Meaning of a Metric
A sampling-frame difference can change the interpretation of a metric even when some aspects of the prediction mechanism are similar. This is especially important when cohorts have different proportions of cases and controls. Accuracy and predictive values depend on outcome prevalence. Sensitivity and specificity are conditional on outcome status, but they too can differ across cohorts if the mix of disease severity, measurement conditions, or other relevant characteristics within outcome groups differs.
The following example holds sensitivity and specificity fixed solely to show the arithmetic effect of different outcome prevalence. It does not assert that those rates will remain fixed in real cohorts. That assumption would need substantive justification and empirical assessment.
Worked Example: Outcome Prevalence Changes Predictive Value
Imagine a hypothetical classifier with sensitivity \(0.80\) and specificity \(0.90\). In a development sampling frame, suppose 10% of 1,000 people have the outcome. There are 100 people with the outcome, of whom \(0.80(100)=80\) test positive. Of 900 people without the outcome, \(0.90(900)=810\) test negative, so \(900-810=90\) are false positives. The positive predictive value is therefore:
Now consider an external sampling frame with 30% outcome prevalence among 1,000 people, while retaining the same assumed sensitivity and specificity. There are 300 cases, with \(0.80(300)=240\) true positives. There are 700 non-cases, with \(0.10(700)=70\) false positives. Thus:
Under these stipulated conditions, positive predictive value is about 47.1% in the first frame and 77.4% in the second. The classifier's assumed sensitivity and specificity did not change, but the probability that a positive prediction is a true positive did. This is why external performance should be interpreted in the context of the cohort's outcome frequency and recruitment. It would be incorrect to transfer the first PPV to the second frame without checking whether the relevant prevalence and conditional performance are comparable.
Different metrics answer different questions. Accuracy summarizes the fraction correctly classified at a particular threshold; sensitivity and specificity describe correct classification within outcome groups; AUC summarizes ranking across thresholds; and Brier score summarizes squared error of predicted probabilities. Calibration concerns agreement between predicted probabilities and observed outcome frequencies. A sampling-frame difference can affect these measures in different ways, and a single metric cannot explain every discrepancy. The earlier material on calibration assessment provides the meaning of calibration measures; here the central point is to report the same prespecified measures in each evaluation and interpret them against each cohort's design.
Worked Example: Computing and Comparing Brier Scores
For binary outcomes \(y_i\in\{0,1\}\) and predicted probabilities \(\hat{p}_i\), the Brier score is the mean squared probability error, \(\frac{1}{n}\sum_{i=1}^n(\hat{p}_i-y_i)^2\); smaller values indicate better probability predictions on that measure. Consider four hypothetical out-of-fold predictions with outcomes \(1,0,1,0\) and predicted probabilities \(0.8,0.2,0.6,0.4\). Their squared errors are:
The cross-validated Brier score for these four observations is \((0.04+0.04+0.16+0.16)/4=0.100\). In a hypothetical independent cohort, suppose outcomes are \(1,0,0,0\) and the locked model predicts \(0.7,0.3,0.4,0.1\). The squared errors are \(0.09, 0.09, 0.16,\) and \(0.01\), giving an external Brier score of \((0.09+0.09+0.16+0.01)/4=0.0875\). Numerically, the external score is lower. With only four observations in each illustration, this comparison is far too unstable to support a claim that the model is better in the external cohort. The two samples may also represent different sampling frames; the calculation illustrates the metric, not a general conclusion about model transport.
Justifying the Difference in Targets
The distinction follows directly from what is averaged. In cross-validation, each held-out observation is scored using a model that was fitted without that fold. If the analysis is leakage-safe and observations are suitably exchangeable within the development sampling scheme, the held-out losses provide information about the procedure's performance on new observations from that scheme. The folds are not independent experiments, and cross-validation is not an evaluation of one model fitted on all development observations.
In external validation, the model \(f_D\) is fixed before external outcomes are used. If external observations are independent draws from \(Q\), the sample mean of their losses estimates the expected loss of that fixed model in \(Q\). For \(n_E\) external observations, writing \(\ell_i=L(Y_i,f_D(X_i))\), the empirical mean is:
Under independent sampling from \(Q\), with a finite loss variance, this mean has expectation \(R_Q(f_D)\) and standard error \(\sigma_{\ell,Q}/\sqrt{n_E}\), where \(\sigma_{\ell,Q}\) is the standard deviation of individual losses in \(Q\). This justification depends on the sampling and independence assumptions; clustered or repeated observations require an analysis that respects their dependence. The analogous uncertainty for cross-validation is more involved because training sets and folds overlap, so treating fold scores as independent observations is generally not justified.
State the outcome, prediction time, intended population or sampling frame, and prespecified performance measures.
Record the fitted pipeline, predictors, processing rules, and threshold. Do not use external outcomes to revise them while still presenting the result as an untouched evaluation.
Check for participant overlap and describe differences in eligibility, recruitment, outcome frequency, and measurement. These details determine how directly the results can be compared.
Give each metric with its denominator and uncertainty, and interpret differences as evidence about the evaluated model and cohorts—not as an automatic explanation of why performance changed.
Choosing Between Evaluation Approaches
| Approach | Primary target | Main strength | Main limitation |
|---|---|---|---|
| Internal cross-validation | Performance of a development procedure under the development sampling conditions | Uses development data efficiently while holding observations out of each fit | Can be optimistic if the pipeline leaks information; fold fits use less data than the final fit |
| External validation | Performance of one locked model in an independent cohort | Directly tests the fitted model beyond the data used to develop it | May be imprecise or narrowly representative; performance is specific to that cohort's sampling conditions |
| Single internal holdout | Performance of a fitted model on a reserved portion of the development sampling frame | Simple separation between fitting and assessment | Can waste scarce genomic samples and gives a result dependent on one split; it is not external validation merely because it was held out |
These approaches are complementary rather than interchangeable. Cross-validation is useful during development and for estimating how a procedure may perform within its source setting. A genuinely independent cohort adds evidence about the final model in another sample. Neither alone guarantees performance in every intended clinical setting. A marked external difference should prompt checks of data independence, outcome definitions, predictor availability, thresholds, metric denominators, uncertainty, and sampling-frame characteristics. The next topic treats cohort and population shift more directly; the present distinction is that a difference between estimates is a finding to interpret, not by itself a diagnosis of its cause.
Common Mistakes and Limitations
- Calling any held-out data “external.” A random split from the same development cohort is an internal holdout. External validation requires an independent cohort, not just an unused subset.
- Re-fitting before external assessment. Re-estimating preprocessing or coefficients with external observations changes the object being evaluated. If outcomes guide the change, the external set has also become part of development.
- Comparing metrics without checking their definitions. A changed outcome window, threshold, denominator, or handling of indeterminate predictions can make two nominally identical metrics incomparable.
- Explaining every difference as model failure. Sampling variation, training-set size, cohort composition, and measurement differences can contribute. The metric difference alone does not identify a cause.
- Assuming external means universally generalizable. An independent cohort supports inference about its sampled setting, subject to its size and design. It does not establish performance in all populations.
External validation also has practical limitations. Suitable cohorts may be small, outcomes may be defined differently, or genomic measurements may not be available in a compatible form. If an external model is recalibrated or otherwise updated after assessment, the updated model can be evaluated, but its results should be distinguished from the original locked-model validation. Clear separation between evaluation and updating preserves the meaning of both results.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A team randomly holds out 20% of one cohort, fits its model on the other 80%, and reports performance on the held-out portion. Is this external validation? Explain.
- A locked model has cross-validated accuracy of 0.84 and external accuracy of 0.76. Name two reasons the estimates might differ, and state one detail needed before interpreting the difference.
- Why must an external cohort's outcomes not be used to select a threshold if the team wants to report an untouched external evaluation?
- Suppose an external cohort has a different outcome prevalence. Can its positive predictive value be assumed equal to the development cohort's positive predictive value if sensitivity and specificity are unchanged? Explain.
- What does a cross-validation estimate target that is not exactly the same as the target of external validation?
Solutions. (1) No. It is an internal holdout from the same cohort, not an independent external cohort. (2) Possible reasons include sampling variability, smaller training sets in cross-validation, and differences in sampling frame or measurement. The cohort sizes and outcome definitions are among the details needed. (3) Choosing a threshold using external outcomes adapts the model to the evaluation data, so the resulting performance no longer evaluates the originally locked decision rule independently. (4) No. Predictive value depends on prevalence as well as sensitivity and specificity; the arithmetic in the prevalence example shows why. (5) Cross-validation estimates performance of a development procedure trained on subsets of the development data, whereas external validation estimates performance of one fixed fitted model in the external cohort.