What an External Validation Report Must Establish
An external validation report is useful only when a reader can tell exactly what was predicted, which model made the predictions, and how predictions were compared with observations in a new data source. A performance number without those details is difficult to interpret: a small prediction error for one target, sampling design or patient group does not automatically describe performance for another.
The analysis should be specified before the validation results are examined. In particular, state the target quantity and prediction time, the eligible population, the prediction mode, the primary performance measures and the uncertainty procedure. For a locked-model evaluation, do not use validation concentrations to re-estimate model parameters or select a better-performing analysis. This follows the distinction between validation and model updating established in the earlier material on Prediction Without Re-Estimation and Distinguishing Validation From Model Updating.
A clear report separates three things. The prediction task defines the intended target and setting. The analysis describes how each prediction and performance estimate was obtained. The result gives estimates, uncertainty and context. That separation makes it possible to distinguish a model's performance from differences caused by changes in data definitions, measurement procedures or analysis choices.
Specify the Task and Lock the Analysis
Begin with the clinical and quantitative question in operational terms. For example: “For adults receiving the specified oral regimen, predict each patient's plasma concentration at 8 hours after the first dose using baseline covariates and dosing information, without using any concentration from that patient.” State whether the target is a concentration, exposure summary or other quantity; give its time point, units, treatment regimen and eligible population.
Also say whether the predictions are population predictions or conditional predictions. Population predictions use the prespecified model and available non-concentration information, without conditioning on that patient's measured concentrations. Conditional predictions use a defined set of that patient's observations to inform the prediction. These answer different questions, so the report must identify which was evaluated and must not describe one as the other.
“Locked” should mean more than naming a model. Report the model version or specification, parameter values, covariate relationships, random-effects assumptions, residual model, and any software or implementation details needed to reproduce predictions. State how the input data were mapped to model inputs, how missing or invalid values were handled, and what happened when a patient fell outside the model's supported input range. Any departure from the prespecified procedure should be documented rather than silently incorporated.
Worked Example: Writing a Prespecified Prediction Task
Suppose a hypothetical external study enrolled adults given a fixed oral dose and recorded plasma concentrations at 8 hours. The intended analysis uses baseline age, body weight, dose and dosing time to produce one population prediction per eligible patient. No 8-hour or other validation concentration is used to estimate that patient's prediction.
A concise task statement is: “We evaluated the locked population model's prediction of plasma concentration in mg/L at 8 hours after the first oral dose in adults receiving the specified regimen. Predictions used the recorded baseline covariates and dosing history; no validation concentrations were used to condition individual predictions.”
This statement identifies the target, units, timing, population, prediction mode and permitted inputs. It does not claim the results apply to other doses, ages, sampling times or conditional predictions. Those claims would require evidence from relevant validation tasks.
Describe the External Data and Harmonization
The report should identify the external data source and the rules by which patients and records entered the analysis. Give the numbers screened, excluded and analyzed, with reasons for exclusions; describe the sampling design and relevant study characteristics. Report the number of independent patients and, when applicable, the number and arrangement of observations per patient. These details determine both the scope of the validation and the appropriate unit for uncertainty calculations.
Describe how variables were harmonized before predictions were generated. State how dose, dose time, sample time, concentration units, covariates, assay values below quantification limits and missing values were represented. Explain how discrepancies in definitions or measurement procedures were resolved. The earlier material on Harmonizing External PK Datasets and Assay and Protocol Differences in External Validation addresses these issues in depth; a case-study report should give the actual decisions made, not merely say that the data were “cleaned.”
Include a compact table of the study population and prediction inputs. Report summaries that help readers judge the intended scope, such as dose, age, body weight, sampling time and concentration range, as appropriate. If the external cohort differs from the development data or has limited covariate support, describe that difference. State which patients or records could not be evaluated and why. A summary table cannot replace the data definitions and processing rules required for reproduction.
Calculate and Present Performance
Choose measures that correspond to the prespecified task and state their formulas and units. For paired observed and predicted concentrations \(C_i^{\mathrm{obs}}\) and \(C_i^{\mathrm{pred}}\), define the prediction error as \(e_i=C_i^{\mathrm{pred}}-C_i^{\mathrm{obs}}\). Mean error describes average signed error, mean absolute error describes average error magnitude, and root mean squared error gives greater weight to larger errors. The earlier material on Prediction Error Metrics for External PK Validation covers the broader choice of metrics.
Give the sample size and units alongside each estimate. Do not rely on a single summary if it could conceal important differences across the prediction range or relevant subgroups. Report calibration and prediction-interval coverage when those were prespecified and are relevant to the intended use; state how they were calculated and give their uncertainty. The earlier material on Calibration of External PK Predictions, Coverage of External Prediction Intervals, Time- and Concentration-Dependent External Performance, and Subgroup Performance in External Validation provides the corresponding measures and considerations.
Worked Example: Calculating Prediction Errors
The following five paired values are hypothetical concentrations in mg/L. The prespecified task is population prediction at one target time. For these illustrative data, the predictions are 1.2, 1.8, 5.0, 6.0 and 12.0 mg/L; the corresponding observations are 1.0, 2.0, 4.0, 8.0 and 10.0 mg/L.
The errors, calculated as prediction minus observation, are \(0.2,-0.2,1.0,-2.0,2.0\) mg/L. Therefore the mean error is
The mean absolute error is
The root mean squared error is
The small positive mean error indicates a slight average overprediction in this five-patient illustration, while MAE and RMSE describe error magnitude. The five observations are not enough to establish stable performance in a target population; the arithmetic demonstrates the reported measures, not a general clinical conclusion.
Quantify Uncertainty and Check Interpretation
A performance estimate from an external sample is uncertain because another sample from the same target setting would generally give a different estimate. State the uncertainty method, the resampling unit, the number of replicates, and how interval limits were obtained. For a patient-level bootstrap, resample patients with replacement and recompute the complete prespecified performance statistic in each replicate. If observations are clustered within patients, resample patients rather than treating records as independent. If the external sampling design involves strata or clusters, the resampling scheme should respect that design.
A bootstrap interval describes sampling uncertainty for the chosen performance estimand under the assumptions represented by the resampling scheme. It does not measure uncertainty about every possible population, eliminate bias from a nonrepresentative validation sample, or guarantee the same performance in a different setting. Report the interval with the point estimate, units and analysis population. If some bootstrap replicates cannot produce a statistic, state how often this occurred and how it was handled.
Worked Example: Reporting Bootstrap Uncertainty
Consider a separate, hypothetical external cohort of 60 adults. Suppose the prespecified analysis produced a mean error of \(0.18\) mg/L and an RMSE of \(1.42\) mg/L. A patient-level bootstrap with 2,000 replicates produced a 95% percentile interval of \(-0.11\) to \(0.47\) mg/L for mean error and \(1.12\) to \(1.79\) mg/L for RMSE. These are illustrative outputs, not results from a real study.
A reproducible report would say that, among the 60 evaluated patients, mean prediction error was \(0.18\) mg/L, with a 95% patient-level bootstrap interval from \(-0.11\) to \(0.47\) mg/L; RMSE was \(1.42\) mg/L, with an interval from \(1.12\) to \(1.79\) mg/L. The mean-error interval includes zero, so these data do not establish a nonzero average signed error at the corresponding two-sided 5% level. That does not establish absence of bias. The interval quantifies uncertainty in the mean-error estimate, not the range of errors expected for an individual patient.
To reproduce this procedure, the report would also name the patient as the resampling unit, give the random-number seed and software implementation where applicable, and specify whether the full statistic—including any prespecified data-processing steps—was recalculated in every replicate. The interval should not be presented without those implementation details.
Report Coverage, Limitations and Failure Modes
For prediction intervals, report the intended nominal level, how the intervals were generated, and the observed fraction of measurements they contained. For instance, an observed coverage of 49 out of 60 is \(49/60=0.817\), or 81.7%. If the nominal interval level was 90%, this estimate is below the nominal level, but sampling uncertainty matters before interpreting the difference. State whether coverage was calculated per patient or per record and use the same unit when quantifying uncertainty.
Worked Example: Interpreting Prediction-Interval Coverage
In the five-patient hypothetical example above, suppose the respective prediction intervals were \([0.5,2.0]\), \([0.8,3.5]\), \([2.0,8.0]\), \([3.0,7.0]\) and \([8.0,15.0]\) mg/L. The observed concentrations were 1.0, 2.0, 4.0, 8.0 and 10.0 mg/L. Four observations fall within their intervals; the fourth, 8.0 mg/L, exceeds its upper limit of 7.0 mg/L.
Empirical coverage is \(4/5=0.800\), or 80.0%. If these were nominal 90% prediction intervals, the point estimate is below nominal coverage. With only five patients, however, it is highly imprecise; it should not be reported as persuasive evidence of a systematic coverage deficit. A real report would present an appropriate uncertainty interval and explain the prediction unit and interval construction.
A useful report includes limitations close to the findings, rather than leaving readers to infer them. Limitations may include a small or selected cohort, sparse sampling, restricted covariate support, assay or protocol differences, missing input data, or limited representation of particular patient groups. These constrain transportability: successful prediction in the observed validation sample is not proof of performance in every clinical setting. Describe the target population to which the evidence most directly applies.
Several reporting errors can make a technically correct calculation misleading:
- Changing the analysis after seeing the results. Selecting a prediction mode, subgroup or primary metric because it looks favorable makes the reported estimate harder to interpret as a prespecified validation result. Label exploratory analyses clearly.
- Leaving the prediction mode unstated. A conditional prediction informed by patient concentrations cannot be presented as a population prediction made without those concentrations.
- Ignoring dependence. Resampling concentration records independently when several belong to the same patient can understate uncertainty. Match the resampling unit to the independent sampling unit.
- Overgeneralizing a summary. A cohort-average metric does not establish good prediction for every individual or subgroup. Report the evaluated population and relevant limitations.
- Calling validation an update. If validation data are used to change parameter estimates or model structure, the resulting analysis is no longer an evaluation of the untouched locked model. Report model updating as a separate analysis.
A common mistake is to conclude, “The model is externally validated, so it is suitable for all patients.” That conclusion exceeds the evidence. External validation estimates performance in a specified external setting, using a specified task and dataset; it supports claims only to the extent that the data and design represent the intended use. A more defensible conclusion names the cohort, prediction task, results and remaining uncertainty.
The main alternatives answer different questions. Internal validation evaluates performance using data related to model development, often with resampling; external validation assesses performance in an independent data source. Model updating uses new data to revise a model and must not be conflated with testing the locked version. A report may present these analyses together, but should label each analysis, its data and its purpose separately.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A report predicts concentrations using one measured concentration from each external patient. What must it say about prediction mode, and why?
- In 40 independent patients, 32 observations fall within their prespecified prediction intervals. Calculate empirical coverage and state what else should be reported.
- A study has three concentration records per patient. What should be resampled for a patient-level bootstrap, and why?
- A team changes a fixed-effect parameter after inspecting external validation results, then reports performance for the changed model. Is that a locked-model validation result?
Solutions. (1) It must identify the predictions as conditional if the measured concentrations inform the individual predictions; these are not population predictions made without patient concentrations. It should state which observations are used and how. (2) Coverage is \(32/40=0.800\), or 80.0%. Report the nominal interval level, interval construction, patient-level uncertainty and the population and prediction unit. (3) Resample patients, carrying each selected patient's records together. The patient is the independent sampling unit, so treating the 120 records as independent would ignore within-patient dependence. (4) No. Changing the parameter using validation results is model updating. Performance of the changed model should be labeled as an updated-model analysis and distinguished from performance of the untouched locked model.