Population and Conditional Prediction Performance

Two Prediction Targets, Two Questions

An external PK model can predict concentrations for a new patient in more than one way. A population prediction asks how well the locked model predicts without using that patient’s measured concentrations. An individual conditional prediction asks how well it predicts after using specified observations from that same patient to learn about their individual random effects. These are different prediction tasks, not competing calculations of one identical target.

That distinction matters when interpreting a performance summary. A prediction informed by a patient’s earlier concentrations can be useful for forecasting their later concentrations, but its performance does not describe what the model can predict for a patient with no concentration data. Conversely, a population prediction assesses the model’s predictions before individual concentration information becomes available.

Terminology can vary across software and reports. In particular, a typical-individual prediction \(f(x,\theta,0)\) is not generally the same as the population-average prediction \(E_{\eta}[f(x,\theta,\eta)]\). The latter averages model predictions over the random-effects distribution. For nonlinear models, taking the expectation over random effects can change the prediction. A validation report should state which quantity it calls a population prediction and how it was calculated.

What Data Inform Each Prediction?

For a population prediction, the individual’s measured concentrations must not be used to tune that individual’s random effects. The prediction can use the locked population model, the individual’s permitted covariates, and the dosing and sampling information available at the prediction time. If those covariates or dosing records are unavailable or inconsistent in the external data, the prediction is not based on the same information set as the intended use.

For an individual conditional prediction, the validation concentrations used to estimate the individual effect are part of the prediction information. It is therefore essential to identify which measurements were used and which measurement or measurements are being predicted. A legitimate prospective question might be: “After observing this patient’s first two concentrations, how well does the model predict their next concentration?” Using those first two observations to estimate the patient effect is appropriate. Using the target concentration itself to estimate that effect, and then reporting the resulting prediction as a forecast of that same concentration, gives the prediction access to its answer.

Conditioning on individual observations need not mean re-estimating the population model. With the model locked, individual data may update the estimate of \(\eta_i\) while \(\theta\) remains fixed. If the external data are used to change population parameter estimates, that is model updating rather than a prediction based only on the locked model.

A Simple Conditional Prediction Derivation

A Gaussian random-intercept example shows precisely how individual data change a prediction. Let \(z_{ij}\) be log concentration, and suppose, for the purpose of this derivation, that the model is

Assume that the random effects and residual errors are mutually independent, that residual errors are independent across observations, and that \(\mu\), \(\omega^2\), and \(\sigma^2\) are known and locked. For one individual, let \(r_j=z_{ij}-\mu\) be the residual from the population prediction for each of \(m\) conditioning observations. The likelihood for \(\eta_i\) is proportional to \(\exp[-\sum_j(r_j-\eta_i)^2/(2\sigma^2)]\), and its prior density is proportional to \(\exp[-\eta_i^2/(2\omega^2)]\). Multiplying these densities and completing the square gives a normal conditional distribution with mean and variance

where \(\bar r=m^{-1}\sum_{j=1}^{m}r_j\). The conditional mean is a shrunken version of the mean residual: it lies between zero and \(\bar r\) when the variances are positive. If between-individual variability is small relative to residual variability, the estimate is pulled more strongly toward zero. If multiple informative observations are available, the data can provide more information about the individual effect. This derivation describes a simple Gaussian model; in nonlinear PK models, conditional estimates may require numerical methods and the relationship need not have this closed form.

Typical-Individual and Marginal Predictions

The phrase “population prediction” can obscure a second distinction: prediction for a typical individual versus the average prediction across the population. In the log-normal random-intercept model, the typical-individual prediction on the log scale is \(\mu\). If \(C\) is concentration and the model implies \(\log C=\mu+\eta\) with \(\eta\sim N(0,\omega^2)\), then the arithmetic mean concentration over individuals is

The final equality follows from the moment-generating property of a normal random variable. In contrast, \(\exp(\mu)\) is the median of the log-normal distribution, and its geometric mean. A reported average prediction on the concentration scale should therefore specify whether it is a typical-individual value or an average over the random-effects distribution.

Constructing a Fair Performance Comparison

A performance summary is meaningful only when its prediction target, information set, evaluation observations, and scale are clear. To compare population and conditional predictions directly, apply the same chosen summary to predictions of the same held-out observations. Keep the population parameters fixed in both analyses. For the conditional analysis, specify the conditioning measurements in advance; for the population analysis, do not use those measurements to estimate individual effects. This paired comparison helps ensure that a difference reflects the additional individual information rather than a different set of patients or outcomes.

The two summaries still answer different applied questions. Population-prediction performance addresses settings such as initial dose selection when no patient concentrations are available. Conditional-prediction performance addresses settings where some concentrations have already been measured and are used to personalize predictions. A better conditional summary does not show that the model can make equally good predictions before therapeutic drug monitoring data are collected. Nor does a weaker population summary invalidate the model’s potential usefulness for later, data-informed forecasting.

A fair evaluation of conditional predictions should reserve evaluation outcomes that were not used for conditioning. Common designs include using earlier observations to condition and later observations to evaluate, or separating observations into conditioning and evaluation sets according to a prespecified rule. The chosen design should reflect how predictions would be used in practice. If the goal is to forecast the next sample, for example, condition only on information available before that sample.

Assumptions, Diagnostics, and Limitations

Population and conditional prediction comparisons rely on several practical conditions. The locked model must define the same structural, random-effects, and residual-error assumptions for the external population. Doses, times, units, covariates, and relevant data handling must be aligned with the intended prediction task. Conditional predictions additionally depend on the quality and timing of the individual observations used to estimate random effects. If these measurements are sparse, noisy, or uninformative for the relevant PK parameters, conditional predictions may remain close to population predictions or be unstable.

Useful checks include confirming that no evaluation outcome entered the individual-effect estimation, verifying that all predictions used the intended dosing and sampling records, and documenting the number and timing of conditioning observations. Inspect whether conditional effects are extreme or poorly determined, and whether the conclusions change substantially when the conditioning set changes. A conditional prediction that looks unusually successful when evaluated on the same observations used to estimate its individual effect is a warning sign of information leakage, not evidence of strong forecasting performance.

These approaches are alternatives only in the sense that they represent different information settings. Population prediction is appropriate when no individual concentration data are available for prediction. Conditional prediction is appropriate when the target is explicitly data-informed prediction after specified observations. A single summary should not be used to answer both questions without labeling the prediction type and information set. Both approaches are also limited by the adequacy of the locked model and by whether the external validation population and data-collection process support the intended use.