Why Freeze the Model?
An external validation asks how a model performs on data not used to develop it. That question has a clear answer only if the model and the rules that turn input data into predictions are settled before anyone uses the external outcomes to make choices. Otherwise, the evaluation data have quietly become development data: they have helped shape the model being evaluated.
Freezing is more than saving a final equation. An NLME model includes a structural description of the response, covariate relationships, distributions for between-subject variability, and a residual-error model. Its fitted parameter estimates and the processing steps that supply its inputs matter too. A change in any of these can change predictions, even if the model's name remains the same.
The earlier material on defining the external prediction target and designing temporal and geographic validation establishes what population and setting are to be evaluated. Here the practical question is narrower: what exactly must be held fixed so that the planned evaluation remains a genuine external test?
What Must Be Frozen?
A useful way to describe an NLME model is \(y_{ij}=f(t_{ij},D_{ij},x_{ij};\theta,b_i)+\epsilon_{ij}\), where \(y_{ij}\) is the response for subject \(i\) at observation \(j\), \(f\) is the structural model, \(t_{ij}\) is time, \(D_{ij}\) represents dosing information, \(x_{ij}\) represents covariates, \(\theta\) contains population parameters, and \(b_i\) represents subject-level random effects. A typical specification assigns \(b_i\sim N(0,\Omega)\) and defines a residual-error distribution, often through a variance model \(R_i\). This notation is illustrative; particular models can use other distributions and error structures.
The frozen object includes both the model's form and the estimated values that define its behavior. For example, it is not enough to freeze the choice of a one-compartment structure if the clearance estimate is then changed after examining external concentrations. The complete specification should also cover any alternative models or averaging rules that are part of the planned evaluation.
| Component | Choices or values to lock | Why it matters |
|---|---|---|
| Structural model | Compartments, absorption or elimination form, parameterization, and structural parameter estimates | Changing the function or its estimates changes the concentration-time predictions. |
| Covariate model | Included covariates, functional forms, interactions, reference values, and estimated effects | Adding, removing, or re-estimating an effect changes how individual inputs map to parameters. |
| Variability and residual error | Random-effects structure and covariance estimates \(\Omega\); residual-error form and estimated parameters | These affect the modeled distribution of responses and the uncertainty represented by predictions. |
| Data and prediction pipeline | Record inclusion, transformations, units, missing-data rules, and allowed prediction inputs | Different processing or information can yield different inputs and predictions under the same equation. |
Parameter estimates should be recorded with their units and parameterization. A numerical value for clearance is not interchangeable across unit systems, and a covariate effect per kilogram is not the same as an effect per standardized unit unless the transformation is defined. Likewise, freeze rules governing which observations or subjects are eligible and how repeated records are handled. These are model-evaluation decisions, not harmless clerical details.
Freeze prediction rules as well. State what information is available at the prediction time, and what operation is used to obtain a prediction from the fitted model. A validation intended to assess predictions available before observing a subject's external responses cannot use those responses to estimate that subject's random effects. If a different prediction target permits additional information, that permission must be set in advance and applied consistently; it cannot be introduced after seeing which choice improves performance.
Worked Examples of Freezing and Leakage
Worked Example: Locking a Structural Model and Its Estimates
Hypothetical setup. A development analysis selected a one-compartment model for an intravenous bolus and estimated clearance \(CL=5.00\) L/h and volume \(V=50.0\) L. Before external outcomes are opened, the validation plan specifies a dose of 500 mg and a prediction time of 4.00 h for a worked calculation. For this simplified calculation, assume the dose is available, the subject is represented by the population parameters, and there is no additional dose or covariate complication.
For this model the concentration is \(C(t)=(D/V)\exp[-(CL/V)t]\). Substitution gives
The structural form, estimates, and units are part of the frozen model. Suppose someone looks at external outcomes and lowers the clearance estimate to 4.50 L/h. The resulting calculation is \(10.0e^{-0.360}\approx6.977\) mg/L. The difference is not a neutral correction: external outcomes have changed the model being evaluated. Any performance result from this revised model is no longer an untouched external evaluation.
Worked Example: Outcome-Driven Updating Creates Optimism
Hypothetical setup. In a small external evaluation, two observed concentrations are 12 and 14 mg/L. The frozen model predicts 10 and 10 mg/L. Use mean absolute error (MAE), the average absolute difference between prediction and observation, just to illustrate the effect of using outcomes to alter the model.
With the frozen predictions, the absolute errors are \(|12-10|=2\) mg/L and \(|14-10|=4\) mg/L. Therefore
After seeing these outcomes, an analyst adds 3 mg/L to both predictions, producing 13 and 13 mg/L. The resulting MAE is \((|12-13|+|14-13|)/2=(1+1)/2=1.00\) mg/L. This improvement is not evidence that the original frozen model performed with an MAE of 1.00 mg/L. The added offset was chosen using the same outcomes against which performance was then assessed.
If the offset is to be estimated as a calibration step, that is a model-update analysis, not a frozen external validation. The updated procedure needs evaluation on data not used to choose the offset.
Worked Example: Freezing a Covariate Transformation
Hypothetical setup. In development, body weight has mean 70 kg and standard deviation 10 kg. The covariate is defined as \(z=(\text{weight}-70\ \mathrm{kg})/(10\ \mathrm{kg})\), and the estimated effect on log clearance is 0.20 per one-unit increase in \(z\). Assume the external weights are 70 and 90 kg, and these values are available before outcomes are examined.
Using the frozen development transformation, the two standardized values are \(z_1=(70-70)/10=0\) and \(z_2=(90-70)/10=2\). Their covariate contributions to log clearance are \(0.20(0)=0\) and \(0.20(2)=0.400\), respectively. The second subject's contribution is therefore 0.400 log-clearance units relative to the stated reference, under this model.
If the analyst instead recenters and rescales using the external weights, their mean is 80 kg and their sample standard deviation is \(\sqrt{[(70-80)^2+(90-80)^2]/(2-1)}\approx14.14\) kg. The resulting standardized values are approximately \(-0.7071\) and \(0.7071\), so the same numerical coefficient gives contributions of approximately \(-0.1414\) and \(0.1414\). Those are not the frozen covariate calculations. Re-estimating the transformation while retaining the original coefficient changes the model's predictions.
Using external covariates without outcomes is not automatically leakage in every evaluation design; it can be allowed if the target and protocol explicitly permit it. But the transformation rule must be specified in advance, and an external transformation must not be chosen because it makes predictions agree better with external outcomes. A change in the mean and scale can also be mathematically compensated by changing the coefficient and reference; that compensation is itself a model reparameterization and must be defined, not improvised after evaluation.
Why Outcome-Informed Choices Bias Evaluation
Suppose an analyst considers several candidate models and chooses the one with the smallest observed external loss. Even if each candidate's performance estimate is noisy, selecting the smallest observed loss preferentially selects a candidate that benefited from favorable noise. The selected loss is therefore not an independent assessment of a prechosen model.
The basic justification follows from an inequality. Let \(L_1,\ldots,L_K\) be the observed losses for \(K\) candidate models on the same external data. For every set of values,
Taking expectations preserves the inequality, so \(E[\min_k L_k]\leq E[L_1]\). This does not by itself quantify the bias for a particular set of correlated models, but it shows why reporting the best loss after searching the evaluation data is optimistic relative to evaluating a model fixed in advance. The same selection logic applies when outcomes are used to choose a covariate, an outlier threshold, a residual model, or a preprocessing option.
This argument assumes the candidates were compared using the same evaluation data and the selected result is reported as though the candidate had been fixed beforehand. A prespecified comparison of several models can answer a different, explicitly defined question, but it must report the comparison as planned rather than silently presenting the winner as an untouched single-model test.
Practical Audit, Failure Modes, and Limits
Before external outcomes are accessed, preserve a dated protocol or analysis plan, the final model specification, parameter estimates, covariate definitions, data-processing code, and software settings needed to reproduce the analysis. Record which external fields may be used and when they become available. A versioned model file or checksum can help establish that the evaluated artifact was not replaced during analysis; it cannot, by itself, establish that the surrounding data choices were unbiased.
Common warning signs include selecting a structural model after comparing external fit, changing a covariate effect after inspecting residual patterns, defining exclusions after seeing unusual external outcomes, or choosing among missing-data rules based on which produces the most favorable result. A code review should also check for scripts that read outcome columns before predictions or prespecified analyses are complete. If a decision was influenced by external outcomes, document it, classify the analysis as updated or exploratory as appropriate, and do not describe its apparent performance as a pristine validation of the original frozen model.
Freezing does not guarantee transportability, correct measurement, or accurate predictions. It does not eliminate legitimate differences between development and external populations, nor does it settle whether a model is clinically useful. It makes the evaluation question interpretable by preventing the answer from influencing the object under evaluation. Nor does freezing prohibit all future improvements: a changed model may be appropriate for later use, but its performance needs an evaluation design that does not reuse outcomes used to make that change.
State the prediction target, permitted information, and time at which predictions are meant to be available.
Record structural and covariate forms, parameter estimates, variability and residual components, data rules, and prediction procedures.
Generate the planned predictions without using external outcomes to choose or revise any of the locked elements.
If an outcome-informed change is made, report it as an updated analysis and seek new untouched data for an independent evaluation.
Check Your Understanding
Answer each question, then compare with the concise solutions below.
- Name four distinct NLME model components that should be frozen before external outcomes are examined.
- An analyst changes an estimated covariate effect after inspecting external residuals but keeps the same structural equation. Is the model still frozen? Why?
- When might using external covariate values without external outcomes be allowed, and what must be specified in advance?
- A model is modified after its external evaluation outcomes are reviewed. What is needed to assess the modified model independently?
Solutions. (1) Examples include the structural form and parameter estimates, covariate effects and transformations, random-effects covariance structure and estimates, and residual-error model; data-processing and prediction rules must also be fixed. (2) No. The external residuals informed the covariate-effect change, so the evaluated model was adapted using the evaluation outcomes. (3) It may be allowed when the evaluation target and protocol permit access to those predictors; the allowed fields, transformation rule, and timing must be prespecified and must not be chosen using outcomes. (4) Evaluate it on new data whose outcomes were not used to make or select the modification.