Why Check Predictions at Two Levels?
A candidate PD model can describe the average response well while missing important differences among subjects. The reverse can also occur: predictions may track some individuals closely while the model systematically misses the population’s central tendency. Comparing observations with predictions at both levels makes these failures easier to distinguish during model building.
The comparison is meaningful only when the prediction is clearly defined. In a nonlinear mixed-effects model, a prediction for a typical subject is not always the same as the average prediction across the population. Individual predictions, in turn, use subject-specific information and therefore answer a different question from population predictions.
Model and Prediction Definitions
Let \(Y_{ij}\) be the observed PD response for subject \(i\) at observation \(j\), with the observation’s time, concentration, and covariates denoted collectively by \(d_{ij}\). Write a candidate model as
Here, \(m\) is the model’s expected response conditional on the subject’s random effects; \(\theta\) contains population parameters; and \(\eta_i\) represents subject-specific deviations, usually modeled as drawn from a distribution \(F_\eta\). The residual error \(\varepsilon_{ij}\) is assumed to have conditional mean zero given the design and random effects. The model also specifies its conditional variability. These assumptions define the predictions below; they do not imply that every observation will equal its predicted value.
Some workflows use “population prediction” to mean the typical-subject prediction, while others use it for the marginal mean. State which one is being compared. For a linear response in \(\eta_i\), with mean-zero random effects, these predictions may agree. For a nonlinear response, they generally need not. Marginal predictions target the expected response across subjects at a given design; typical predictions describe the subject whose random effects are zero.
Why Typical and Marginal Predictions Can Differ
The marginal prediction is an expectation over random effects, not the response evaluated at the average random effect. In general, \(E_{\eta}[m(d;\theta,\eta)]\ne m(d;\theta,E[\eta])\). Since \(E[\eta]=0\) under a mean-zero random-effects model, setting \(\eta=0\) gives the typical prediction but does not automatically give the population average.
This distinction follows directly from the conditional-mean assumption. Averaging the model equation over the random-effects distribution gives
The residual contributes zero to this mean because its conditional expectation is zero. A related decomposition clarifies why a mean check alone cannot assess subject heterogeneity:
This is the law of total variance: the population’s response variability combines variation in model-predicted responses across subjects and average within-subject residual variability. A model can get the first moment—the mean—right while allocating variability incorrectly between these components.
Worked Example: A Nonlinear Population Prediction
Suppose a hypothetical drug-response model has an effect \(m(C;\eta)=20e^\eta C/(5+C)\), where \(C\) is concentration and \(\eta\sim N(0,0.25)\). At \(C=5\), calculate the typical and marginal predictions.
Typical prediction. Set \(\eta=0\):
Marginal prediction. At \(C=5\), the response is \(10e^\eta\), so its expectation is \(10E(e^\eta)\). For a normally distributed variable with mean zero and variance \(0.25\), \(E(e^\eta)=e^{0.25/2}\). Thus
The marginal response is about 11.33 effect units, rather than 10 effect units. The difference is caused by the nonlinear exponential random effect, not by an arithmetic error or a nonzero mean for \(\eta\). A comparison with observed population means should therefore use the prediction that matches the target: marginal predictions for the expected population response, or typical predictions when explicitly describing the zero-random-effect subject.
Practical Checks Against PD Observations
Begin by aligning each observation with its prediction at the same time, concentration, covariate values, and response scale. Then examine two complementary summaries:
- Population level: Compare observed means or other relevant summaries with the corresponding population predictions across the design range. Check whether the model tracks the overall level and shape of the PD response.
- Individual level: Compare subject trajectories or subject-level summaries with individual predictions. Check whether the predictions represent differences among subjects, not merely the pooled average.
A table of observed and predicted means at selected design points can be a useful first check. A plot of subject trajectories with individual predictions can reveal whether a good pooled fit conceals systematic subject-level discrepancies. These are diagnostic comparisons, not formal tests with a universal cutoff. Their interpretation depends on the sampling design, response variability, and the intended use of the model.
Worked Example: Population Means Hide Subject-Level Misses
In a hypothetical study, three subjects have PD responses measured at three visits. The table gives observed responses and predictions from a candidate model. “Population prediction” here means the marginal prediction at each visit; “individual prediction” uses subject-specific fitted predictions. All values are in effect units.
| Subject | Observed responses | Population predictions | Individual predictions |
|---|---|---|---|
| A | 8, 10, 12 | 9, 11, 13 | 8, 10, 12 |
| B | 12, 14, 16 | 9, 11, 13 | 10, 12, 14 |
| C | 7, 9, 11 | 9, 11, 13 | 9, 11, 13 |
At the first visit, the observed mean is \((8+12+7)/3=9.00\) effect units; at the second it is \((10+14+9)/3=11.0\); and at the third it is \((12+16+11)/3=13.0\). These equal the population predictions at all three visits. The model therefore reproduces the observed pooled mean trajectory in this small example.
That agreement does not mean the individual patterns are all captured. For subject B, the observed-minus-individual-prediction differences are \(12-10=2\), \(14-12=2\), and \(16-14=2\) effect units. For subject C, they are \(-2\) at every visit. Subject A’s individual predictions match exactly. The pooled mean agreement hides these opposing subject-level discrepancies.
For a simple numerical summary, calculate the root mean squared difference between observations and individual predictions across the nine observations:
This summary does not establish that the model is adequate or inadequate by itself. The example shows why it is useful to inspect subject-level patterns in addition to the population mean: here, the model follows the average trajectory while its individual predictions leave persistent offsets for two subjects.
Interpreting Discrepancies During Model Building
A discrepancy is a clue about the model, not an automatic instruction to add a parameter. If observed population means depart from the population prediction in a consistent way across time or concentration, the candidate’s central response structure may be insufficient. If pooled means agree but individual predictions miss subjects in persistent directions, the candidate may not adequately represent between-subject differences, or the data may not support reliable subject-specific estimates. A mismatch can also reflect sparse observations, measurement variability, or a design that provides little information about the relevant PD feature.
Check whether the pattern is coherent across relevant observations before interpreting it. For example, a single unusual value is weaker evidence of a structural problem than a repeated mismatch at similar times or concentrations. Also verify that predictions use the observed covariates and dosing or concentration history, that the units and response transformations agree, and that the plotted prediction is on the same scale as the observed response.
Individual predictions require special caution because \(\hat{\eta}_i\) is estimated using the subject’s observations. This makes the comparison in-sample: the same data help estimate the subject-specific prediction and then appear in the check. Close agreement is therefore not independent confirmation of predictive performance. Sparse individual data and strong shrinkage toward the population distribution can also make individual predictions look more alike than the subjects’ underlying responses. Prediction checks should be considered alongside model structure, uncertainty, and the study design.
How These Checks Complement Other Approaches
Population and individual prediction checks have different targets and limitations. They are most useful as interpretable model-building views rather than as replacements for other diagnostics.
| Approach | Main question | Important limitation |
|---|---|---|
| Population prediction check | Does the model represent the expected PD response across the population at the observed design points? | Agreement in means can conceal subject-level misfit and incorrect variability. |
| Individual prediction check | Do fitted subject-specific predictions track observed subject patterns? | Predictions use subject data and can appear reassuring even when independent predictive performance is unknown. |
| Formal fit comparisons | Does a candidate improve a specified fit criterion relative to another model? | A better criterion does not, by itself, show that predictions are adequate across the design or for individual subjects. |
| Simulation-based checks | Are observed data features plausible under repeated data generated from the fitted model? | Results depend on the simulation design and do not remove the need to inspect scientifically important patterns. |
Prediction checks are also limited by the observations available. A model cannot be meaningfully checked at design regions with no supporting data, and a few sparse observations per subject may not distinguish individual response differences from residual variation. The checks do not prove a causal mechanism, establish external predictive validity, or identify a unique correct model. They help determine whether the candidate is a reasonable representation of the observed PD data and where further scrutiny is warranted.
Label it as typical, marginal population, or individual, and state the response scale and design conditions.
Compare observed summaries with predictions at matched times, concentrations, and covariate values.
Inspect individual trajectories or summaries for systematic discrepancies that pooled means could conceal.
Consider design support, observation variability, and the in-sample nature of fitted individual predictions before changing the model.
Check Your Understanding
Answer each question, then compare with the concise solutions.
- In a mean-zero random-effects model, when does the typical-subject prediction equal the marginal population prediction?
- A nonlinear model has random effects with mean zero. Is it valid to conclude that the marginal prediction equals the prediction at random effect zero? Why or why not?
- A candidate model matches observed means at every visit, but subject-specific predictions systematically overpredict some subjects and underpredict others. What does this suggest, and what does it not establish?
- Why should close agreement between observed data and fitted individual predictions not be treated as independent validation?
Solutions. (1) They agree when averaging the model response over the random-effects distribution gives the same result as evaluating the response at zero random effect; this holds for a response linear in the random effect under the stated mean-zero assumption, but not generally for nonlinear responses. (2) No. In general, \(E_\eta[m(d;\theta,\eta)]\ne m(d;\theta,E[\eta])\), so the marginal and typical predictions can differ. (3) The pooled central tendency may be represented while subject heterogeneity or individual predictions are not adequately captured. This pattern alone does not prove which model component is responsible or that adding a random effect will solve it. (4) The subject’s observations contribute to estimating the subject-specific random effects used in those predictions, so the comparison reuses the data and is in-sample.