External Validation Across Clinical Sites

Why Validate Across Sites?

A model can perform well on the data used to develop it and still produce unreliable predictions in a different hospital. A new site may treat a different patient population, collect measurements using different instruments or schedules, or encode clinical events differently. Even when the model itself is unchanged, these differences can affect the inputs it receives and the outcomes against which its predictions are evaluated.

External validation asks how a fixed model performs on eligible data from a site that did not contribute to its development or tuning. The goal is not to make the model fit the new site better. It is to obtain a credible assessment of its performance there, and to make clear what population, measurements, workflow, and data definitions that assessment represents. The earlier material on analytical validation of model predictions provides the general context for evaluating prediction performance; here, the central concern is whether the evaluation remains valid across clinical sites.

Set Up a Credible Site Evaluation

Before calculating a performance measure, define the prediction task precisely. Specify the intended population, prediction time, inputs available at that time, outcome and measurement window, and unit of analysis. For a dosing model, one unit might be a patient-level dosing decision with a predicted concentration and a later measured concentration. If multiple decisions or measurements come from one patient, record that structure rather than treating all rows as independent.

The model must be locked for the evaluation: its parameters, preprocessing, feature definitions, and prediction rules should be fixed in advance. Any exclusions, handling of missing inputs, outcome definitions, and primary performance measures should also be specified before examining site results. If an input is unavailable or defined differently at the new site, document how the prespecified model handled it. Changing the model, selecting a new threshold, or revising preprocessing after seeing the validation outcomes turns the evaluation into model adaptation; performance on those same data is no longer an independent assessment of the adapted procedure.

A useful site description covers four connected areas:

• Population: eligibility, sample size, clinical characteristics relevant to the model, and how patients entered the evaluated cohort.
• Measurements: assay or instrument, units, sampling time, measurement procedures, and whether the reference outcome matches the model’s intended target.
• Workflow: how and when inputs were recorded, whether model-relevant values were available at prediction time, and any differences in usual care that affect the evaluated task.
• Data definitions: coding systems, variable mappings, missing-value rules, time stamps, and operational definitions for doses, outcomes, and clinical events.

These details are not background decoration. For example, a concentration described as a predose trough is not a comparable reference outcome if the local record includes postdose samples under the same label. A performance statistic can be computed precisely and still answer the wrong question if the prediction and reference measurement do not align.

Performance Measures and Their Assumptions

For patient-level prediction occasion \(i\), let \(\hat{y}_i\) be the model prediction and \(y_i\) the corresponding reference outcome on the same scale and at the prespecified time. Define the prediction error as \(e_i=\hat{y}_i-y_i\). Positive errors indicate overprediction under this convention; negative errors indicate underprediction. Common summaries include mean error (bias), mean absolute error (MAE), and root mean squared error (RMSE):

Bias summarizes average signed error and can conceal offsetting individual errors. MAE summarizes average error magnitude in the outcome’s units. RMSE also uses the outcome’s units, but gives greater weight to larger errors because errors are squared before averaging. None is automatically the right primary measure: choose according to the prediction task and clinical consequences. If decisions depend on whether an error exceeds a clinically justified tolerance, report that proportion as well, with the tolerance defined in advance.

These summaries require correctly paired predictions and reference outcomes that represent the intended target. The simple formulas describe the observed evaluation sample; they do not by themselves provide uncertainty intervals or establish performance in every future patient at the site. If observations are independent, uncertainty for mean error can be estimated from its sample standard deviation \(s_e\) using \(SE(\bar{e})=s_e/\sqrt{n}\). A \(t\)-interval additionally relies on an appropriate sampling model and, for small samples, an approximately normal distribution of errors. With repeated observations per patient, an analysis that treats every row as independent will generally understate uncertainty; resampling or inference should preserve patient-level clusters.

For a model that supplies prediction intervals, evaluate their empirical coverage as well as their width. If a nominal 90% prediction interval is \([L_i,U_i]\), occasion \(i\) is covered when \(L_i\leq y_i\leq U_i\). The observed coverage is the number covered divided by the number evaluated. Its interpretation depends on valid reference outcomes and the prespecified prediction task; nominal coverage is a repeated-sampling target, not a guarantee that a particular patient’s outcome lies in the interval.

Separate Performance from Site Composition

An overall site metric depends both on how predictions perform within patient groups and on how common those groups are at that site. Therefore, report relevant population characteristics and, where sample size permits, performance by prespecified clinically relevant strata. This can reveal whether a site-wide summary reflects a different mix of patients rather than a uniform change in performance. Small subgroup estimates are uncertain, so they should not be presented as definitive rankings.

Diagnostics, Alternatives, and Limits

Before interpreting performance, check whether the evaluation data support the intended comparison. Verify that predictions were generated by the locked model, that each prediction is paired with the correct patient and outcome time, and that units and transformations agree. Examine the counts and reasons for excluded records, missing inputs, and missing outcomes. Compare local measurement schedules and definitions with the protocol. A strikingly large error can reflect a genuine model limitation, but it can also signal a time-alignment, unit-conversion, or coding problem; investigate data provenance without silently changing the prespecified analysis.

External validation has a different role from the main alternatives. Internal validation, such as resampling within development data, estimates performance under reuse of the development source and can help assess optimism, but it does not independently test a new site’s data practices. A held-out site evaluation directly assesses performance in that site’s evaluated setting, but a single site cannot establish performance in all hospitals. Local model updating may be considered if performance is inadequate, but the data used to update should not then be described as an independent test of the updated model.

A site evaluation also has limits that a clean metric cannot remove. A small cohort yields imprecise estimates; selective availability of measurements can make the evaluated cases unrepresentative; and a site’s workflow may change which patients receive measurements or when they are taken. Differences in population, measurement, workflow, and definitions should be reported rather than collapsed into a single claim that the model “works” or “fails” at the site. The result supports a bounded conclusion about the evaluated data and task.

A Practical Reporting Sequence