Why Define the Target Before Evaluating Predictions?
An external prediction result is meaningful only relative to a stated question. “How well does the model predict concentration?” leaves important choices unresolved: concentration in which patients, at what future time, using what information available for each patient, and on what scale? Two analyses can use the same fitted nonlinear mixed-effects (NLME) model and the same external records yet answer different questions because they make different choices on these points.
The external prediction target is the specific prediction quantity and population to which an evaluation refers. It is not simply the external dataset: the dataset is observed, whereas the target describes the patients, prediction settings, information, and outcomes of interest. Defining that target also separates a prediction question from model selection. Selection criteria summarize aspects of fit or penalized fit; they do not, by themselves, specify which future patients and outcomes matter.
For an NLME model, let \(Y_i(t)\) denote patient \(i\)'s outcome at time \(t\), such as concentration, and let \(\eta_i\) denote patient-specific random effects. A model may describe the outcome through a structural function of time, dose, covariates, and random effects, together with a residual-error model. The target definition does not change those model components. It determines the setting in which a model-based prediction is to be judged.
Write \(t_0\) for the prediction time and \(h\) for the prediction horizon. The outcome of interest is then \(Y_i(t_0+h)\). Let \(H_i(t_0)\) denote the patient information allowed at \(t_0\), such as dosing and covariate information available by then and, if the target permits it, previously observed outcomes. Future dosing or covariate information must be described as part of the planned prediction setting; it is not past patient history. Finally, let \(Q^*\) denote the distribution of eligible patients and prediction settings in the target population.
Five Elements of a Precise Target
State the patient characteristics, eligibility conditions, and prediction settings that define whom the result concerns. Where relevant, specify how subgroups are represented in the target distribution \(Q^*\).
State when the prediction is issued and how far ahead the outcome is measured. A prediction at 24 hours for an outcome at 36 hours has a 12-hour horizon, not a 36-hour horizon.
Name the outcome, its units, the measurement time or window, and whether the prediction concerns the original scale or a specified transformation.
List what is available when the prediction is issued. Do not define a target that uses information collected after that time unless the question explicitly concerns retrospective reconstruction rather than prospective prediction.
Specify whether interest is in a point summary, such as a mean or median, or in a predictive distribution. For a population-level objective, also state how patients or settings are weighted.
These elements distinguish two related objects. A patient-level prediction target concerns the predictive quantity for a patient with the permitted information \(H_i(t_0)\) and specified prediction setting. A population-level performance target averages a chosen prediction loss over the target population. In general form, if \(\widehat{\mathcal P}_M\) is the predictive distribution from model \(M\), and \(L\) is a prespecified loss, the latter can be written as
This notation makes the dependencies visible: the target population \(Q^*\), horizon \(h\), permitted history, outcome definition, predictive quantity, and loss all help determine the question. The formula is a definition of the target performance quantity, not a claim that any particular evaluation procedure estimates it without bias.
Worked Examples
Worked Example: A Population Mix Changes the Target Average
Suppose a hypothetical external prediction question concerns patients receiving a specified regimen in two settings: 60% are outpatients and 40% are inpatients. At a fixed prediction time and horizon, suppose the model's mean predicted concentrations within those settings are 8.0 mg/L and 12.0 mg/L, respectively. Find the target-population average prediction for this specified mix.
The target mixture assigns weights \(0.60\) and \(0.40\), which sum to 1. The average prediction is therefore
The result is 9.6 mg/L for the stated target mix. It is not automatically the right average for a population with different setting proportions. For example, equal weights would produce \(0.50(8.0)+0.50(12.0)=10.0\) mg/L. The patient mix is part of the target, not a harmless descriptive detail.
Worked Example: Separating Prediction Time From Horizon
In a hypothetical dosing setting, a model is to issue a prediction at hour 24 for the concentration measured at hour 36. The intended information rule permits dose records, covariates, and concentration measurements recorded no later than hour 24. The future dose schedule from hour 24 to hour 36 is supplied as a planned regimen. State the target's time and information components.
The prediction time is \(t_0=24\) hours and the outcome time is 36 hours. Thus
Permitted patient history is restricted to information available at or before hour 24. The planned future regimen is a prediction-setting input, not an observed patient history available at hour 24. A concentration measurement recorded at hour 30 cannot be used as history for this target: it occurs \(30-24=6\) hours after prediction time. Using it would answer a different, later-updated prediction question. This definition does not specify how to generate or score the prediction; it fixes the question those later operations must address.
Worked Example: Original-Scale Mean Versus Log-Scale Prediction
Suppose, hypothetically, that the predictive distribution for a future concentration \(Y\) is specified by \(\log(Y/(1\ \text{mg/L}))\sim N(\log(10),0.25)\). Here the variance on the log scale is \(0.25\), so the log-scale standard deviation is \(0.5\). Compare the median concentration with the arithmetic mean concentration.
The median of a lognormal distribution is obtained by exponentiating its log-scale mean:
For a normal random variable \(Z\sim N(\mu,\sigma^2)\), the lognormal mean is \(\exp(\mu+\sigma^2/2)\) after accounting for the stated concentration unit. Therefore,
The median is 10 mg/L, while the arithmetic mean is about 11.33 mg/L. These are different predictive summaries on the original concentration scale. A target that asks for a log-scale prediction or a median is not interchangeable with one that asks for the arithmetic mean concentration.
Why the Scale and Information Rule Matter
A transformation can change the predictive quantity, not merely its notation. For a positive outcome, predicting \(\log Y\) and then exponentiating a predicted log-scale mean generally gives a median under a lognormal model, not an arithmetic mean. Thus the target should state whether an error or summary is defined in concentration units or on a transformed scale. It should also name the units and measurement definition, such as a concentration at a specified sampling time rather than an unspecified “concentration.”
The permitted-history rule has a similar effect. A prediction based only on baseline covariates answers a different question from one that may use concentrations observed during treatment. The latter may be useful for updating a patient's prediction, but it is not a baseline prediction. If the intended use is to predict before the first dose, allowing later concentrations changes the target and can make performance appear stronger than it would be at the intended decision point.
Population weighting matters when the target includes subgroups with different frequencies or prediction settings. An unweighted average over the external records targets the mix represented by those records. It does not necessarily target a separately specified clinical population. The target should say whether the aim is performance in the observed external mix, a defined mixture, or a particular subgroup.
Justifying a Population Average
A population average prediction follows directly from conditioning on patient characteristics. Let \(X_i\) denote characteristics that define relevant subgroups, and suppose the target population assigns probability \(Q^*(X_i=x_k)=w_k\) to subgroup \(k\). For subgroup-specific mean predictions \(m_k\), the law of total expectation gives the target mean as the weighted sum of subgroup means:
The weights must represent the target distribution and sum to 1. This identity explains the calculation in the first example. It also shows why a single overall mean can conceal a target mismatch: changing \(Q^*\) changes the weights and can change the target mean even when every subgroup mean remains fixed. For continuous characteristics, the corresponding expression is an integral over their target distribution.
Alternatives, Diagnostics, and Limits
| Choice | Question answered | What changes if the alternative is chosen? |
|---|---|---|
| Population prediction | What prediction summarizes a defined population or subgroup? | A patient-specific target instead conditions on that patient's permitted history. |
| Single horizon | How well is the outcome predicted at one stated future time? | A different horizon or a trajectory target concerns different outcomes. |
| Original outcome scale | What is predicted in the outcome's natural units? | A transformed scale changes summaries and the interpretation of deviations. |
| Point summary | Which single summary, such as a mean or median, is the target? | A predictive-distribution target concerns uncertainty as well as a central value. |
Several checks can reveal an incompletely specified or internally inconsistent target:
- Can a reviewer identify exactly who belongs to the target population and how settings or subgroups are weighted?
- Is the outcome time strictly after the stated prediction time, with the horizon calculated correctly?
- Could every listed input actually be known when the prediction is supposed to be issued?
- Are the outcome, transformation, summary, and units mutually consistent?
- Would a different target population, history rule, or horizon answer a substantively different question? If so, those choices should not be left implicit.
Defining a target does not ensure that the external data represent that target, that the outcome is measured comparably, or that the NLME model is adequate. Those are separate concerns for validation design and interpretation. The definition makes such concerns visible; it cannot resolve them by itself.
Common Mistake
Check Your Understanding
Answer each question using the target-definition principles above. Short solutions follow the questions.
- A prediction is issued at hour 8 for an outcome at hour 20. What is the horizon?
- A target population contains 25% of subgroup A and 75% of subgroup B. Their mean predictions are 4 mg/L and 8 mg/L. What is the target-population average prediction?
- Why is a prediction using an outcome observed after prediction time not valid under a rule permitting only information available at prediction time?
- Under the lognormal distribution in the third worked example, which is larger: the arithmetic mean or median, and why?
- Name two target elements that must be reconsidered if the intended use changes from a baseline prediction to a prediction updated after treatment begins.
Solutions. (1) \(20-8=12\) hours. (2) \(0.25(4)+0.75(8)=1+6=7\) mg/L. (3) The outcome occurs in the future relative to the prediction time, so it is unavailable under the stated information rule; using it changes the prediction question. (4) The arithmetic mean is larger: it is approximately 11.33 mg/L, compared with the 10 mg/L median, because the lognormal distribution is right-skewed. (5) At minimum, revise the permitted-history rule and specify the updated prediction time and horizon; the target population or outcome definition may also need to be reconsidered.