Why a Predictive Check Needs a Target
A predictive check is only interpretable when it is clear what feature of the data is being checked. “Compare the data with simulations” is not yet a complete instruction: the comparison could concern a population median, a subgroup-specific proportion, or one patient’s next concentration. Those targets answer different questions, and they can give different impressions of model adequacy.
A useful starting point is to state three things before looking at a check: the level (population, subgroup, or individual), the observed-data quantity to summarize, and the corresponding simulated quantity. The comparison is meaningful only when the observed and simulated quantities use the same definition, observations, conditioning information, and scale.
In an NLME model, a typical response can be represented as \(Y_{ij}=f(t_{ij},x_i;\theta,\eta_i)+\epsilon_{ij}\), where \(\theta\) describes population parameters, \(x_i\) contains subject covariates, \(\eta_i\) represents subject-specific random effects, and \(\epsilon_{ij}\) represents residual variation. The distribution of \(\eta_i\), the residual model, and the structural model together determine what outcomes the model can predict. The target specifies which aspect of those outcomes is being examined; it does not change the model itself.
A predictive check generally treats the observed design as the frame for the comparison: the simulated quantity is calculated at the relevant times, doses, covariates, and records. Whether the simulations are conditional on estimated model parameters, include additional uncertainty, or use other conditioning choices is a separate specification. In every case, the target must say what is held fixed and what is allowed to vary. The procedures for generating replicate data are considered separately; here the priority is defining the question those replicates must answer.
Three Levels of Prediction
A population-level target summarizes responses across the population represented by the data. Examples include the median concentration at a specified time across all eligible subjects or the overall proportion of observations above a threshold. Such a target usually reflects variation among subjects as well as residual variation, according to the predictive distribution being used. It does not describe every subject equally well: a good population summary can coexist with poor predictions for particular individuals.
A subgroup-level target applies a specified summary within a defined subset, such as subjects in a dose group or a covariate category. The subgroup must be defined clearly, and the same definition must be used for the observed and simulated data. For example, “the median in the lower-weight group” requires a stated rule for membership in that group. A subgroup comparison can reveal systematic discrepancies that an overall population summary hides.
An individual-level target concerns a particular subject’s response or trajectory. There are two importantly different prediction questions here. A prediction for a new measurement on a subject may condition on information available before that measurement, including suitable earlier observations that inform the subject-specific random effects. A population prediction for a particular subject’s covariates, by contrast, may integrate over the random-effects distribution rather than condition on that subject’s records. State which question is intended. If the measurement being checked was also used to estimate that subject’s random effect, apparent agreement can be artificially optimistic.
| Target level | Example quantity | Interpretation |
|---|---|---|
| Population | Median response at a specified time across eligible subjects | Whether the model reproduces an overall distributional feature |
| Subgroup | Proportion above a threshold among subjects in a defined dose group | Whether the model reproduces that feature within the group |
| Individual | One subject’s held-out response, or a trajectory summary | Whether the model predicts that subject under stated conditioning information |
Define Both Sides of the Comparison
For any chosen target, write down the rule that turns a data set into the quantity being checked. If the target is a median, specify which records enter the median and the time window or time point. If the target is a proportion, state the event, the denominator, and the subgroup. If the target is an individual prediction, state the subject, outcome, time, and conditioning information. Then apply that same rule to the observed data and to each replicate data set.
This requirement can be justified directly. Suppose the target is the median concentration at time \(t^\ast\) among a specified set of subjects. Then both quantities must be formed by selecting that set and time point, and taking the median. If the observed quantity is a median but the simulated quantity is a mean, the difference combines model behavior with a change in summary definition. It cannot be attributed cleanly to either one.
Here \(b\) indexes a replicate data set. The formula emphasizes that the same function \(T\) and the same relevant design \(D\) are used on both sides. Under a fully specified predictive model, if the observed data and replicate data were generated under the same conditions, their target quantities would be comparable draws from the same predictive mechanism. In practice, model parameters are often estimated from the observed data, so this comparison is generally a model check rather than an exact test with a guaranteed reference distribution.
Worked Example: A Population Median at One Time
Consider hypothetical concentrations, in mg/L, from five different subjects at 4 hours after dose: 8, 10, 12, 14, and 16. The target is the population median at 4 hours for this set of subjects.
Observed quantity. Sorting the five concentrations gives the same ordered list, so the middle value is \(T_{\mathrm{obs}}=12\) mg/L.
Suppose three hypothetical replicate data sets, evaluated at the same 4-hour time and across the same five subject designs, give concentrations whose medians are 10, 12, and 13 mg/L. Applying the same median rule gives replicate target values of 10, 12, and 13 mg/L. The observed median of 12 mg/L is one of these illustrative replicate medians. This small example demonstrates alignment of the summaries; three replicates are not enough to support a stable assessment of model adequacy.
If instead the simulated values were summarized by their mean, the resulting comparison would not answer the stated median question. Likewise, comparing the observed 4-hour median with simulated medians at a different time would change the target.
Worked Example: A Subgroup Proportion Above a Threshold
Suppose a hypothetical study has two prespecified dose groups and the target is the proportion of concentrations above 5 mg/L within each group. In the lower-dose group, 3 of 8 observations exceed 5 mg/L. In the higher-dose group, 6 of 10 exceed it.
The observed proportions are \(3/8=0.375\), or 37.5%, for the lower-dose group and \(6/10=0.600\), or 60.0%, for the higher-dose group. The denominator is the number of eligible observations in each group, and the event is strictly “above 5 mg/L.”
For illustration, suppose three hypothetical replicate data sets, evaluated on those same group memberships and denominators, produce lower-dose counts of 2, 4, and 3 above threshold, and higher-dose counts of 5, 7, and 6. The corresponding simulated proportions are \(2/8=0.250\), \(4/8=0.500\), and \(3/8=0.375\) in the lower-dose group; they are \(5/10=0.500\), \(7/10=0.700\), and \(6/10=0.600\) in the higher-dose group.
The comparison is made separately by group: the observed 37.5% is compared with the lower-dose replicate proportions, and the observed 60.0% with the higher-dose replicate proportions. Pooling all subjects would define an overall proportion instead and could conceal a subgroup-specific mismatch. These illustrative replicates show how to form the target; they are not a formal test.
Worked Example: One Subject’s Held-Out Measurement
Suppose a hypothetical patient has earlier observations that are available for prediction, and the check targets a later concentration at 8 hours that was not used to estimate the patient-specific prediction. Given those earlier observations, assume the model’s predictive mean for the held-out concentration is 6.0 mg/L and its predictive standard deviation is 1.0 mg/L. The observed held-out concentration is 7.2 mg/L.
Define the individual target as the standardized prediction discrepancy \(T=(Y-\mu_{\mathrm{pred}})/\sigma_{\mathrm{pred}}\), where \(\mu_{\mathrm{pred}}\) and \(\sigma_{\mathrm{pred}}\) are the conditional predictive mean and standard deviation for this subject and time. The observed target is \(T_{\mathrm{obs}}=(7.2-6.0)/1.0=1.2\).
Suppose three hypothetical replicated held-out concentrations are 5.4, 6.3, and 7.5 mg/L, using the same conditional mean and standard deviation. Their standardized targets are \((5.4-6.0)/1.0=-0.6\), \((6.3-6.0)/1.0=0.3\), and \((7.5-6.0)/1.0=1.5\). The observed value, 1.2, is compared with these replicate values on the same standardized scale. It is above two of the three and below one; that is a description of this tiny illustrative set, not a calibrated probability statement.
The conditioning information matters: if the 7.2 mg/L observation had also been used to estimate the patient-specific prediction, the check would no longer assess prediction of a held-out measurement in this way. The target and interpretation would need to be stated differently.
Justification: Why the Target Must Match
The basic logic is parallel evaluation. Imagine applying the same deterministic rule \(T\) to the observed data and to data generated under the model. If the model and conditioning scheme are appropriate for the question, differences in \(T_{\mathrm{obs}}\) and \(T_{\mathrm{rep}}\) can indicate that the model does not reproduce that feature. This logic fails if the rule differs across the two sides.
For instance, let \(T_{\mathrm{obs}}\) be a median and let \(U_{\mathrm{rep}}\) be a mean. The comparison \(T_{\mathrm{obs}}-U_{\mathrm{rep}}\) is not a discrepancy in one defined target: it compares two different functions of data. Similarly, comparing an individual’s observed response with a simulated population median mixes levels of aggregation. Consistent summaries do not guarantee a good model, but inconsistent summaries prevent a clear interpretation even when the model is adequate.
State whether the question concerns the population, a defined subgroup, or an individual. For an individual target, state what subject-specific information is conditioned on.
Specify the response, time or records, summary or event, inclusion rule, and units. Define the denominator when the target is a proportion.
Use the same target definition and relevant design. Check that groups, time points, records, and transformations match.
A check of one summary does not establish that every part of the model is adequate. Describe what the comparison does and does not assess.
Diagnostics, Alternatives, and Limitations
Several warning signs suggest that a predictive check is not addressing its stated target. The observed and simulated summaries may use different time points, units, transformations, thresholds, or group definitions. The denominator may differ because missing or excluded records were handled inconsistently. A population summary may be presented as if it established individual predictive performance, or one subject’s conditional prediction may be interpreted as an unconditional population prediction. Each problem changes the question or weakens the comparison.
The earlier material on sparse sampling and individual fit displays, unequal sampling and subject-weighted summaries, and eta and epsilon shrinkage is relevant when choosing what to summarize. For example, pooling all records can give subjects with more samples greater weight, whereas calculating a subject-level summary first and then summarizing across subjects gives subjects more equal weight. Neither choice is universally correct: the target must specify which weighting reflects the scientific question.
Predictive checks complement, rather than replace, residual diagnostics and parameter-based evaluation. Residual plots can show patterns across time, predictions, or covariates that a single numerical target misses. Conversely, a precisely defined predictive summary can directly assess whether the model reproduces a feature of the data that matters scientifically. A good result for one target does not prove adequacy for other targets, and a poor result does not by itself identify the source of misspecification.
A particularly important limitation concerns reuse of information. If a subject’s outcome contributes to estimating that subject’s random effect and is then checked against a prediction conditioned on that estimate, the resulting agreement may be optimistic. For a prospective individual prediction question, distinguish the information available before the target measurement from the measurement being evaluated. Also state whether a subgroup was defined before examining outcomes; defining it after seeing the outcomes can make the resulting check difficult to interpret.
Common Mistakes
- Using different statistics on the two sides. Comparing an observed median with a simulated mean is not a check of one common target. Apply the same summary rule to both.
- Leaving the level implicit. An overall result does not necessarily describe covariate subgroups or individual predictions. Name the population, subgroup, or subject explicitly.
- Calling a conditional prediction a population prediction. State whether individual random-effect information is used or predictions integrate over population heterogeneity.
- Using the tested outcome to construct its own prediction. This can make an individual-level check look better than a prediction of an unseen outcome. Identify the conditioning data and separate them from the outcome being checked.
- Overgeneralizing a successful check. Agreement for one time point, threshold, or summary supports only that feature; it does not establish that the full model is adequate.
Check Your Understanding
Answer each question by identifying the target and the corresponding comparison. Short solutions follow the questions.
- A model check compares the observed median concentration at 12 hours with simulated means at 12 hours. What is the problem?
- A team wants to check the proportion above 2 mg/L among subjects receiving a specified regimen. Name two details that must be fixed in the target definition.
- A patient’s observed concentration is used to estimate that patient’s random effect and is then compared with a prediction using that estimate. Why might the comparison be optimistic?
- What does a population-level median check establish about individual predictions?
- Why can pooling all records produce a different target from first calculating one summary per subject?
Solutions. (1) The observed and simulated quantities are different statistics. Calculate medians on both sides, or explicitly redefine the target to be a mean on both sides. (2) Specify the regimen-based subgroup and the threshold event, and also state which records enter the denominator, including the relevant time or sampling rule. (3) The checked outcome has contributed to the individual-specific estimate used to predict it, so the prediction is not independent of that outcome and agreement can be overly favorable. A held-out prediction requires conditioning only on information available before the measurement. (4) It assesses whether the model reproduces that population median under the stated design and predictive scheme; it does not establish that each individual is well predicted. (5) Subjects with more records contribute more observations to a pooled summary. A subject-weighted summary gives each subject a defined contribution first, so it answers a different question.