Why Lock the Model Before External Outcomes Are Examined?
An external validation asks how a model developed elsewhere performs on new data. To make that assessment meaningful, the model being assessed must be defined before its predictions or performance are compared with the external outcomes. Otherwise, decisions made after seeing those outcomes can make the model appear better than it would have appeared under a genuinely independent evaluation.
“Locking” is not just saving a model file. It means specifying the model, its fitted values, its data-processing rules, and the computational procedure sufficiently precisely that the validation analysis cannot quietly change the candidate being evaluated. External study design information may be available while planning the analysis; the critical safeguard is that external outcomes do not determine the model choices being assessed.
Locking protects against a particular threat: outcome-informed selection. It does not guarantee that a model is correct, transportable, or unbiased for every performance measure. Differences in population, sampling, assay, or clinical practice can still affect performance. The purpose is narrower and essential: ensure that the external outcomes are used to assess the prespecified model, rather than to help construct the model that is then claimed to have been assessed.
What Must Be Fixed?
A useful lock makes the model choices explicit enough that two analysts implementing it would not have to make consequential decisions by looking at the external concentrations. At minimum, the record should cover the following.
- Structural model: the disposition and absorption structure, such as the number of compartments, elimination pathway, absorption model, lag-time treatment, and any transit or other processes. State the parameterization and the equations connecting parameters to concentration predictions.
- Covariate model: which covariates affect which parameters; the functional form, transformations, reference values, and centering; and any prespecified interactions. State what happens if a covariate is missing or outside the development range. Do not decide after looking at validation outcomes that a covariate should be added, removed, or transformed differently.
- Random-effects model: which parameters have interindividual or other random effects, their distributions, and whether their covariance or correlation is estimated or fixed. For example, state whether random effects on clearance and volume are correlated, independent, or absent.
- Residual-error model: specify the error form and its parameter values. A proportional model, an additive model, and a combined model imply different variability across concentration levels; they are not interchangeable implementation details.
- Fitted values and execution: identify the parameter estimates, their units, constraints, software and relevant version, and computational settings needed to reproduce the model. Record any transformations or rules required before model input. A model description without the fitted values and executable details may not uniquely define the candidate.
The lock should also identify the data and analysis versions, the date of approval, and who had access to external outcomes before approval. The earlier material on Designing an External Validation Dataset addresses dataset planning; here the emphasis is on fixing the candidate model and analysis-relevant model choices before those outcomes can influence them.
Worked Examples of Locking Decisions
Worked Example: Fixing a Covariate Relationship
Suppose a hypothetical development analysis specifies typical clearance \(CL_{\mathrm{typ}}\) at a body weight of 70 kg as 5.20 L/h, with a power relationship \(CL_{\mathrm{typ},i}=5.20(WT_i/70)^{0.75}\) L/h. Before external outcomes are examined, the power form, exponent, reference weight, fitted value, and units are recorded. What typical clearance does the locked formula assign to a person weighing 80 kg, before any individual random effect is applied?
Substitute \(WT_i=80\) kg and evaluate the formula:
The multiplier is approximately 1.105 to four significant figures, and the clearance computed from the unrounded multiplier is approximately 5.75 L/h. This calculation is a consequence of the already locked formula; it is not evidence that the weight relationship is appropriate in the external population. If external outcomes later look poorly predicted, changing the exponent because of that observation would be model updating, not an unchanged validation of this specification.
Worked Example: Locking the Residual-Error Form
Consider a hypothetical locked combined residual model in which the residual standard deviation at prediction \(C\) is \(\sqrt{\sigma_{\mathrm{add}}^2+(\sigma_{\mathrm{prop}}C)^2}\). Suppose the prespecified values are \(\sigma_{\mathrm{add}}=0.50\) mg/L and \(\sigma_{\mathrm{prop}}=0.20\), and consider a predicted concentration of 10 mg/L. Calculate the residual standard deviation.
The additive component is 0.50 mg/L. The proportional component is \(0.20(10)=2.00\) mg/L. Thus:
The units remain mg/L because the expression under the square root is in \((\mathrm{mg/L})^2\). Replacing this model after seeing external concentrations with an additive-only error of 0.50 mg/L would produce a much narrower residual distribution at 10 mg/L. That change could alter conclusions about predictive adequacy, so the error form and fitted values belong in the lock.
Worked Example: Deciding Whether a Change Is Part of Validation
In a hypothetical evaluation, the locked development model has one-compartment disposition, first-order absorption, a weight effect on clearance, independent random effects on clearance and volume, and a combined residual model. Before external outcomes are unblinded, the external study team reports that body weight is recorded in pounds rather than kilograms. Is converting weight to kilograms a model change?
A prespecified, unit-correct conversion is an input-processing operation, not a new covariate relationship. For a recorded weight of 176 lb, the conversion is \(176/2.20462\approx79.83\) kg. The locked weight function can then be applied using kilograms and its 70 kg reference value. This is appropriate only if the conversion rule is documented, applied consistently, and does not depend on observed concentrations.
By contrast, fitting a new weight exponent because the external concentration residuals appear systematically high would use validation outcomes to alter the model. That is model updating. The resulting analysis may be useful, but its performance is not an independent assessment of the original locked model.
Why Outcome-Informed Selection Can Look Too Good
The basic selection problem can be shown without assuming a particular PK model. Suppose two candidate models have the same true expected validation loss \(R\), but their observed loss estimates are \(R+\epsilon_1\) and \(R+\epsilon_2\). Assume for illustration that the errors have mean zero and are not identical. If the analyst reports the smaller observed loss after examining the external outcomes, the reported quantity is:
Taking expectations and using \(E(\epsilon_1)=E(\epsilon_2)=0\) gives:
The inequality is strict when the difference between the two estimated errors is nonzero with positive probability. Thus, even when neither candidate is truly better, selecting the one that looks better on the observed validation outcomes tends to report an overly favorable loss. The same reasoning applies when many alternatives are tried, such as covariate forms, random-effects structures, or residual models. It does not imply that every post hoc change must improve apparent performance; it shows why choosing based on the observed outcomes invalidates the usual interpretation of a prespecified external assessment.
Locking Workflow and Process Checks
State the structural, covariate, random-effects, and residual-error choices, along with the fitted parameter values and intended model outputs. Resolve ambiguity while using development information, not external outcome performance.
Record equations, units, reference values, input transformations, missing-data rules, software, and computational settings. Specify how any known external design features are handled without using outcome patterns to choose a model.
Check that the code runs, units and transformations are consistent, and its outputs agree with the documented specification using development or artificial test inputs. This is an implementation check, not an opportunity to revise the model using validation performance.
Version the specification, code, and parameter file; record approval and access dates; and retain an audit trail. Where feasible, separate the people preparing the model from those who can inspect external outcomes before the lock is approved.
Warning signs include an undocumented change to an exponent, a parameter value that differs from the approved file, a new residual model added after examining concentration plots, or repeated model runs followed by reporting only the most favorable one. These are process failures even if the final model seems scientifically plausible. A useful safeguard is a change log that states what changed, when, why, and whether external outcomes were accessible at the time.
Alternatives, Limits, and Common Pitfalls
| Approach | What it answers | Consequence for interpretation |
|---|---|---|
| Locked, outcome-blind model assessment | How does the prespecified candidate perform on new data? | Preserves an independent assessment against outcome-informed model selection, subject to the study and metric assumptions. |
| Outcome-informed model selection or parameter adjustment | Can the model be adapted to the observed external data? | May be useful for development, but apparent performance on those same outcomes is not an untouched validation result. |
| Development on one dataset followed by assessment on a separate, untouched dataset | How does the updated candidate perform on data not used to update it? | Can provide a new independent assessment if the new assessment data remain unused during updating. |
A common mistake is to treat “the model structure was locked” as sufficient while allowing the covariate relationship, random-effects specification, or residual-error model to change after inspecting external outcomes. Correction: these are all parts of the candidate model. Freeze them, along with fitted values and material implementation choices, or clearly identify subsequent analyses as updating rather than validation of the original candidate.
Another mistake is to claim that locking makes validation unbiased in every sense. It prevents one source of optimism—selection based on the validation outcomes—but cannot ensure that the external sample represents every intended use population, remove sampling uncertainty, or correct measurement and protocol differences. A locked model can perform poorly, and that result is still informative. If outcome-informed changes are scientifically warranted, report them as model updating and assess the updated candidate with data not used to make those changes.
Check Your Understanding
Answer each question, then compare with the worked solutions below.
- Name four distinct model components, beyond the structural equations alone, that should be covered by a model lock.
- A weight exponent is changed after analysts observe a concentration-dependent pattern in external residuals. Is the result a validation of the original locked model? Explain.
- For \(CL_{\mathrm{typ},i}=5.20(WT_i/70)^{0.75}\) L/h, calculate typical clearance at 70 kg and state why the reference value matters.
- Why can choosing the lower of two observed validation losses produce an optimistic reported loss, even if both candidates have equal true loss?
- Does locking guarantee unbiased performance in every external population? Give one limitation.
Solutions. (1) Examples include the covariate model, random-effects structure, residual-error model, fitted parameter values, and implementation rules. (2) No. The exponent was selected using validation outcomes, so the analysis is outcome-informed updating rather than an untouched assessment of the original model. (3) At 70 kg, \(WT_i/70=1\), so \(CL_{\mathrm{typ},i}=5.20(1)^{0.75}=5.20\) L/h. The reference defines the weight at which the typical clearance equals the stated baseline value. (4) The smaller observed loss is selected partly because of random estimation error; algebraically, the expected minimum is \(R-E|\epsilon_1-\epsilon_2|/2\), which is no greater than the common true loss \(R\). (5) No. Locking limits outcome-informed selection, but does not ensure the external sample represents all intended populations or remove sampling uncertainty and data-collection differences.