Validation Starts With the Decision
A model can predict an outcome well on average and still be unreliable for the decision it is meant to support. For example, small errors in predicted average exposure may be acceptable for describing a population but consequential if a dosing decision depends on whether an individual’s exposure crosses a safety threshold. Validation must therefore test the predictions that matter for the stated context of use, not merely demonstrate that the model can reproduce its development data.
The earlier material on defining a context of use and linking model outputs to decision rules provides the starting point: state who or what the model applies to, when it will be used, what it predicts, and how that prediction informs an action. Designing validation means translating those statements into a plan for data, comparisons, and performance criteria. The criteria should be specified before examining validation results; otherwise, it is easy to select a favorable summary after seeing the data.
Specify What Must Be Validated
Begin by writing the prediction target precisely. Identify the outcome, prediction time, prediction horizon, and unit of analysis. For a concentration prediction, for example, distinguish a concentration at a specified sampling time from an individual average concentration over a dosing interval. For a treatment-response prediction, state whether the output is a probability of response by a specified time or a predicted change on a continuous scale. A validation comparison is only meaningful when the model prediction and the observed reference outcome correspond to the same target.
Next, describe the intended population and use conditions. Relevant features may include disease severity, age, organ function, concomitant medications, clinical setting, dose regimen, and the time at which the prediction would be made. These details guide the validation data selection. A dataset is not fit for validation merely because it contains the same outcome variable: it must also provide appropriate observations under conditions relevant to the intended use.
The model must be fixed before it is evaluated on validation data. This includes its structure, parameter estimates, covariate relationships, and any rules for handling input data or producing predictions. If validation outcomes are used to revise the model and performance is then reported on those same outcomes, the result is no longer an independent evaluation. It is development evidence, and a fresh evaluation is needed to estimate performance without that reuse.
Name the outcome, prediction time, prediction horizon, and unit of analysis that match the context of use.
Specify the population, setting, treatments, and data collection conditions needed to represent the intended use.
Choose measures and thresholds that reflect how prediction errors could affect the decision.
Report results, uncertainty, departures from the plan, and the limits on the populations or conditions supported by the evidence.
Choose Data That Can Answer the Validation Question
Validation data should contain the inputs that would be available at the intended prediction time and a suitable reference outcome. If the model predicts a future event, information collected after that event cannot be used as a predictor in the validation exercise. Doing so creates information leakage: the prediction appears to use information that would not actually have been available when the decision was made.
The unit used to separate development and validation data also matters. Repeated records from one patient are not independent patients. Randomly putting some records from each patient into each dataset can make validation look stronger because the model-development process has already seen information from those same patients. If the intended use is prediction for new patients, separate patients. If the intended use is prediction at new sites, a site-level separation may be more informative. A temporal validation, in which later data are kept aside, may be relevant when practice or data collection changes over time.
An internal holdout set, a temporal evaluation, and an evaluation at new sites answer related but different questions. A holdout set can provide an independent assessment when it is kept untouched throughout model development. Temporal or new-site data can assess transport to a later period or different setting. Resampling methods, such as cross-validation, reuse the available development dataset in structured splits and can help estimate development optimism, but they do not substitute for an independent evaluation in a distinct population or setting. The choice depends on the intended use and the evidence available; no single design establishes performance in every possible setting.
Worked Example: Selecting Data for New-Patient Prediction
A hypothetical model predicts a patient’s drug concentration at 24 hours after a dose. It will be used to support dose selection for new patients at hospitals similar to those in the intended setting. The development dataset contains repeated concentration measurements for 300 patients from three hospitals.
A random row-level split is unsuitable for the main validation: records from the same patient could appear in both development and validation data. Instead, the team holds out all records from 60 patients before fitting the model, ensuring no patient contributes to both sets. The remaining 240 patients are used for development. The held-out patients have concentration measurements at 24 hours, and the model inputs used to predict those concentrations are available at the intended prediction time.
This design addresses prediction for patients not used in fitting, provided the held-out patients and measurement process are relevant to the intended use. It does not by itself establish performance at hospitals with different practice patterns, because all three hospitals may contribute patients to both sets. A separate hospital-level or later-time evaluation would address a different transport question. The plan should state this limit rather than describe the patient-level holdout as evidence for every hospital.
Set Performance Criteria That Reflect Consequences
A performance measure should summarize an aspect of prediction that matters to the decision. For a continuous outcome, possible measures include mean error, which shows systematic over- or underprediction, and mean absolute error, which summarizes typical error magnitude in the outcome’s units. For a threshold-based action, sensitivity and specificity can describe the proportions of relevant events correctly flagged and non-events correctly not flagged. No measure is sufficient for every use: a low average error can conceal important errors near a decision threshold, while a high sensitivity can come with many false alarms.
Acceptance criteria should be selected before the validation results are examined. They may be informed by the decision threshold, the consequences of false positive and false negative classifications, and the precision needed for the decision. A numerical cutoff is not universally acceptable: a mean absolute error of 2 units may be modest for one outcome and unacceptable for another. The criterion needs context, units, and a rationale. The sample size also matters, because a performance estimate from few relevant outcomes can be imprecise even when its observed value meets a target.
For a simple decision with two types of error, suppose a false negative has cost \(C_{FN}\) and a false positive has cost \(C_{FP}\), with costs expressed in the same decision-relevant units. If the respective error probabilities in the intended population are \(P(FN)\) and \(P(FP)\), the expected error cost per decision is:
This follows by dividing the population into the mutually exclusive error events: the cost is \(C_{FN}\) when a false negative occurs, \(C_{FP}\) when a false positive occurs, and zero for a correctly classified case under this simplified accounting. Taking the probability-weighted average gives the expression above. It illustrates why validation criteria should reflect the decision: when missing an event is especially costly, sensitivity may deserve greater emphasis than overall accuracy. The calculation is only as useful as its assumptions; real decisions may involve more outcomes, different actions, or costs not captured by a simple two-error summary.
Worked Example: Evaluating a Threshold-Based Flag
A hypothetical model flags patients predicted to be at high risk of a safety event, prompting closer monitoring. In a validation dataset of 100 patients, 20 experience the event. The model flags 18 of those 20 and also flags 12 of the 80 patients who do not experience it. For this example, the team has pre-specified point-estimate targets of sensitivity at least 0.85 and specificity at least 0.80.
There are 18 true positives and 2 false negatives among the 20 patients with events. Thus sensitivity is:
Among the 80 patients without events, 68 are not flagged and 12 are flagged. Thus specificity is:
Both observed point estimates exceed their respective targets. However, only 20 patients experienced the event, so the sensitivity estimate is based on a relatively small denominator. Meeting a point-estimate target does not show that the underlying sensitivity is known precisely or that it will remain above 0.85 in the intended population. The validation report should include uncertainty intervals and the pre-specified rule for interpreting them; the team should not present these two point estimates alone as conclusive evidence.
To illustrate the role of decision consequences, suppose the hypothetical cost assigned to a false negative is 10 units and the cost of a false positive is 1 unit. The observed error cost for these 100 patients is \(2(10)+12(1)=32\) units, or \(32/100=0.32\) units per patient. Under the same simplified costs, flagging nobody would produce \(20(10)=200\) units, or 2.00 per patient; flagging everybody would produce \(80(1)=80\) units, or 0.80 per patient. On these assumptions, the model’s observed error cost is lower than either alternative. These illustrative calculations do not establish that the assigned costs are clinically appropriate or that the observed frequencies will apply elsewhere.
Worked Example: Aligning a Summary With the Target Population
A hypothetical model predicts an outcome for two patient groups. The intended population is 70% patients with typical renal function and 30% patients with impaired renal function. In a validation sample, the mean absolute prediction error is 1.0 mg/L for the typical-function group and 2.4 mg/L for the impaired-function group. Suppose the acceptance criterion is a target-population mean absolute error no greater than 1.5 mg/L.
If the validation sample contains equal numbers from the two groups, its unweighted overall mean absolute error is \(0.5(1.0)+0.5(2.4)=1.70\) mg/L. This describes the validation sample’s group mix, not the specified target population. To calculate the corresponding target-population summary, weight each group’s error by its target-population proportion:
The weighted estimate is below the pre-specified 1.5 mg/L criterion, whereas the unweighted sample summary is above it. The conclusion depends on the target population stated in the validation plan. The group-specific results should also be reported: the weighted summary does not erase the larger error in patients with impaired renal function. This estimate is only credible if the relevant groups are adequately represented and the group proportions used for weighting are appropriate for the intended population.
Interpret Results Without Overstating Them
A validation conclusion should connect results to the pre-specified criteria and intended use. Report which predictions were evaluated, the data source and separation design, the number of independent patients or other relevant units, the observed performance measures with units, and uncertainty where appropriate. Describe important departures from the intended population or data conditions. If a criterion is not met, the result is evidence that the model has not demonstrated the specified performance for that use; it is not proof that every prediction is wrong.
Common warning signs include unexpectedly low errors after extensive model revision using the validation outcomes, missing representation of patients central to the intended use, predictors that would not be available at decision time, or a summary dominated by common cases while rare but consequential cases are poorly assessed. Also check whether the reference outcome was collected comparably across groups and whether missing outcomes could change the conclusion. These are design and interpretation checks; detailed examination of patterns between predictions and observed outcomes is a separate part of evaluating model performance.
A frequent mistake is to conclude, “The model is validated because the overall error is small.” That reasoning skips the intended prediction, population, decision, and acceptance criteria. The correction is to ask whether the relevant prediction was tested in suitable data, whether the pre-specified measures meet the criteria, and whether uncertainty and limitations permit the intended decision. Validation supports a bounded claim tied to the evaluated use, not a blanket label of validity.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A model predicts an outcome for new patients, but repeated records from each patient are randomly divided between development and validation sets. What is the concern?
- A model has a low mean absolute error but often misses patients whose outcomes exceed a safety threshold. Why might the summary be insufficient?
- In a hypothetical validation set, 24 of 30 patients with an event are flagged. Calculate sensitivity and state what the result does not establish.
- The intended population is 80% group A and 20% group B. Their mean absolute errors are 1.2 and 3.0 units, respectively. Calculate the target-population mean absolute error.
Solutions. 1. Records from the same patient appear in both sets, so the evaluation may benefit from patient-specific information already represented during development. For new-patient prediction, separate patients rather than records. 2. The average error may hide consequential errors near the decision threshold; assess threshold-related performance using pre-specified measures tied to the costs of missed events and false alarms. 3. Sensitivity is \(24/30=0.80\), or 80%. This point estimate alone does not establish its precision or guarantee the same sensitivity in the intended population. 4. The weighted mean absolute error is \(0.80(1.2)+0.20(3.0)=0.96+0.60=1.56\) units.