Unconditional and Conditional Predictive Checks

Two Predictive Questions

A nonlinear mixed-effects model describes both how outcomes vary with predictors and how they vary between subjects. A predictive check can therefore ask whether the model reproduces the distribution of outcomes for a new sample of subjects, or whether it reproduces observations for the particular subjects already studied. These are different questions: one includes variation in subject effects, while the other typically holds estimated subject effects fixed.

The distinction matters when a model appears to fit each subject’s observations well but does not capture how much subjects differ from one another. It also matters in the opposite direction: a model may reproduce the overall population distribution while missing the time course within individual subjects.

Definition: Let \(Y_{ij}\) be the observation for subject \(i\) at observation \(j\), with design variables such as time, dose, and covariates denoted by \(d_i\). A nonlinear mixed-effects model specifies a conditional distribution for \(Y_i=(Y_{i1},\ldots,Y_{in_i})\) given subject effects \(\eta_i\) and population parameters \(\theta\). An unconditional predictive check simulates new subject effects from their population distribution and then simulates observations. A conditional predictive check holds subject effects fixed, commonly at their estimated values, and simulates observations conditional on those values.

A common model representation is

Y_{ij}=f(d_{ij};\theta,\eta_i)+\epsilon_{ij}, \qquad \eta_i\sim N(0,\Omega).

Here \(f\) is the structural model, \(\theta\) contains population parameters, \(\eta_i\) is the vector of subject-specific random effects, and \(\Omega\) is their covariance matrix. The observation error \(\epsilon_{ij}\) has a specified distribution, potentially with variance depending on the prediction. For the basic procedures below, the model and its parameters are treated as specified, the observation schedule and other design variables are held at their observed values, and the error model used to simulate outcomes is the one being evaluated.

In practice, many checks use fitted parameter estimates \(\hat{\theta}\) and \(\hat{\Omega}\) rather than drawing new parameter values. A plug-in conditional check also uses estimated subject effects, often empirical Bayes estimates (EBEs), denoted by \(\hat{\eta}_i\). These are estimates informed by the subject’s data; fixing them is not the same as conditioning on a known, true random effect. A fully conditional predictive distribution could instead account for uncertainty in \(\eta_i\) given the data, but that is a different simulation procedure from the usual EBE-based check.

How the Simulations Differ

For an unconditional check, keep each subject’s observed design, but draw a fresh random effect for that subject from the fitted population distribution. Then simulate observations using the structural and observation-error models. Repeating this creates replicate data sets for a population with the same design. The check asks whether the fitted model can generate data resembling the observed cohort, including its between-subject variability.

For a plug-in conditional check, use the same design but fix subject \(i\)’s effect at \(\hat{\eta}_i\). Simulate new observation errors and outcomes around that subject-specific prediction. The check asks whether the model’s within-subject predictions, given those estimated effects, resemble the observations. Since the estimates were obtained using the observed data, a good conditional fit is not independent confirmation that the model correctly describes the population distribution.

Specify the target.
Choose whether the question concerns a new population of subjects or observations conditional on estimated effects for the observed subjects.

Hold the design fixed.
Use the observed dosing, covariates, and observation times so the comparison isolates model predictions under the study design.

Simulate according to the target.
Draw new \(\eta_i\) for an unconditional check; fix \(\eta_i=\hat{\eta}_i\) for a plug-in conditional check. Simulate the observation error in either case.

Compare relevant features.
Compare observed and replicated distributions or patterns that answer the chosen question, such as typical values, spread, or time-course behavior.

The distinction follows directly from the law of total variance. For a scalar outcome with finite variance, write its conditional mean as \(m(\eta)=E(Y\mid\eta)\) and its conditional variance as \(v(\eta)=\operatorname{Var}(Y\mid\eta)\). Then

\operatorname{Var}(Y) = E\{\operatorname{Var}(Y\mid\eta)\} + \operatorname{Var}\{E(Y\mid\eta)\} = E\{v(\eta)\}+\operatorname{Var}\{m(\eta)\}.

The first term captures average within-subject variability; the second captures variability in the conditional mean induced by the distribution of subject effects. An unconditional check includes both sources. A check fixing effects at estimated values mainly assesses the first source, conditional on those values. This decomposition explains why the two checks need not show the same spread.

Worked Example: A Conditional Concentration Prediction

Consider a hypothetical concentration model at time \(t=4\) hours:

C_i(t)=10\exp(-0.1t+\eta_i)+0.15, \qquad \hat{\eta}_i=0.2.

Assume the displayed prediction is the conditional mean, with the additive \(0.15\) representing the specified observation-error contribution to the mean. For this conditional prediction, hold the estimated effect at \(0.2\). Substitution gives

\hat{C}_i(4) =10\exp(-0.1(4)+0.2)+0.15 =10\exp(-0.2)+0.15.

Since \(\exp(-0.2)\approx0.8187\), the prediction is \(10(0.8187)+0.15\approx8.337\) concentration units. The conditional check simulates observations around this subject-specific prediction using the specified error distribution. For an unconditional check, the effect would instead be drawn anew from the fitted distribution, for example \(\eta_i^{\mathrm{rep}}\sim N(0,\hat{\Omega})\), before calculating the prediction. The resulting prediction varies between replicates because both the draw and the observation error can vary.

Worked Comparison Under a Normal Model

A simple normal random-intercept model makes the difference in predictive spread explicit. Suppose, for one observation per subject,

Y_i=\theta+\eta_i+\epsilon_i, \qquad \eta_i\sim N(0,4), \qquad \epsilon_i\sim N(0,1),

with independent random effects and errors, and suppose \(\theta=10\) is treated as known for this illustration. The normal assumptions and independence imply that an unconditional observation has mean \(10\) and variance \(4+1=5\). A plug-in conditional observation instead has mean \(10+\hat{\eta}_i\) and variance \(1\), because the effect is held fixed.

Worked Example: Comparing Unconditional and Conditional Intervals

For a new subject, the unconditional predictive distribution is \(Y_i\sim N(10,5)\). A central 95% normal predictive interval is approximately the mean plus or minus \(1.96\) standard deviations:

10\pm1.96\sqrt{5} \approx10\pm4.383 =[5.617,\ 14.383].

The interval is in the outcome’s units. It includes both between-subject and observation-level variation. Now consider an observed subject with \(\hat{\eta}_i=1\). Conditional on this fixed estimate, the predictive mean is \(10+1=11\), and the standard deviation is \(\sqrt{1}=1\). Thus the corresponding plug-in conditional interval is

11\pm1.96(1)=[9.04,\ 12.96].

The narrower conditional interval does not show that the subject has no between-subject variability. It reflects that this check holds the estimated effect fixed and therefore does not resample that source of variation. Neither interval is a confidence interval for \(\theta\); each describes simulated outcomes under its specified predictive procedure.

Reading the Checks Together

The checks are most useful when their summaries are chosen to match their predictive questions. Unconditional simulations can reveal whether the model reproduces the population-level distribution, including between-subject spread. Conditional simulations can reveal systematic deviations around individual fitted trajectories, such as persistent time-course patterns or an observation-error model that is too narrow. Graphs and numerical summaries should be interpreted as comparisons to simulated variability, not as automatic tests that certify a model.

The following hypothetical calculation illustrates why a favorable conditional comparison can coexist with concern about population spread. Suppose four subjects each have four observations, and their observed subject means are 8, 9, 11, and 12 units. In the random-intercept model above, the variance of a subject mean is \(4+1/4=4.25\), not \(1.25\), because the random-intercept variance is 4. Its predictive standard deviation is \(\sqrt{4.25}\approx2.062\). The observed sample mean is 10 units, and its sample standard deviation is

s=\sqrt{\frac{(8-10)^2+(9-10)^2+(11-10)^2+(12-10)^2}{4-1}} =\sqrt{\frac{10}{3}} \approx1.826\text{ units}.

This observed spread is not, by itself, evidence of a problem: four subjects provide little information about the distribution of subject effects, and an appropriate comparison would examine the spread against replicated unconditional samples. Conditional simulations, by contrast, use each subject’s estimated effect and principally compare observation-level deviations around the fitted subject-specific values. The model can therefore track these four individual means conditionally while the population distribution still warrants scrutiny.

Worked Example: Why Estimated Effects Can Make Conditional Fit Look Good

Use the same hypothetical model, now treating the four subject means above as data. Suppose the population mean is fixed at 10 and the model variances are \(\operatorname{Var}(\eta_i)=4\) and \(\operatorname{Var}(\bar{\epsilon}_i)=1/4\). In this normal model, the conditional expectation of the random intercept given a subject mean is

E(\eta_i\mid\bar{Y}_i) = \frac{4}{4+1/4}(\bar{Y}_i-10) = \frac{16}{17}(\bar{Y}_i-10).

The multiplier \(16/17\approx0.9412\) is the shrinkage factor for this particular model. Thus the estimated effects are approximately \(-1.882,\ -0.941,\ 0.941,\ 1.882\), giving fitted subject means of \(8.118,\ 9.059,\ 10.941,\ 11.882\). Each is close to its observed mean because the data were used to estimate the effects. A conditional check can consequently show small deviations at the subject-mean level without demonstrating that the model’s population distribution is adequate. The unconditional check, which draws effects from the assumed \(N(0,4)\) distribution, addresses that population question instead.

Diagnostics, Alternatives, and Limitations

A discrepancy confined to unconditional checks suggests examining population-level features such as the distribution of subject effects, covariate structure, or between-subject variability. A discrepancy also present in conditional checks points toward features that remain after accounting for estimated subject effects, such as the structural time course or observation-error model. These are clues, not one-to-one diagnoses: parameter estimation, sparse data, and the selected summary can influence both checks.

A visual predictive check summarizes observed and simulated outcome distributions over time or another predictor. A prediction-corrected visual predictive check adjusts for variation in typical predictions before comparing such summaries. NPDE instead transforms predictive deviations using the simulated predictive distribution and its covariance structure. These approaches emphasize different summaries and should not be treated as interchangeable versions of the unconditional-versus-conditional distinction. In each case, state whether random effects are resampled or held fixed; that choice determines which predictive question the simulation addresses.

Important limitations include the following:

Plug-in estimates hide uncertainty. Treating \(\hat{\theta}\), \(\hat{\Omega}\), or \(\hat{\eta}_i\) as fixed can make predictive variation appear smaller than it would if parameter or effect uncertainty were propagated.
Conditional checks reuse the observations. The EBEs are informed by the same subject data being checked, so conditional agreement can be optimistic, particularly when there are few observations per subject.
Unconditional checks mix sources of variation. A mismatch may arise from the random-effects distribution, the structural model, the observation-error model, or their combination. A discrepancy alone does not identify its cause.
Design still matters. Simulations should preserve the relevant observed design. Otherwise, differences in dose, sampling times, or covariates can be mistaken for model misfit.

A frequent mistake is to say that a conditional check evaluates the model “for the observed subjects,” then interpret it as a validation of the population random-effects distribution. That conclusion does not follow: fixing EBEs removes the resampling of that distribution, and the EBEs already reflect the observed outcomes. Use an unconditional check when the target is new subjects from the modeled population, and a conditional check when the target is observations given subject-specific effect estimates. Report which procedure was used and what was held fixed.

Key takeaway: Unconditional checks ask whether the model can reproduce a cohort, including between-subject variation. Plug-in conditional checks ask whether simulated observations resemble the data around estimated subject effects. Their differences are informative precisely because they condition on different things.

Check Your Understanding

Answer each question, then compare with the short solutions.

In an unconditional predictive check, which quantity is newly drawn for each subject: the observed design, the random effect, or both?
For \(Y_i=\theta+\eta_i+\epsilon_i\), with independent effects of variance 9 and errors of variance 4, what is the unconditional variance of one observation?
Why can an EBE-based conditional check look good even if the population random-effects distribution is misspecified?
For the normal model with \(\theta=10\), \(\operatorname{Var}(\eta)=4\), and \(\operatorname{Var}(\epsilon)=1\), find the conditional predictive mean and standard deviation when \(\hat{\eta}= -0.5\).

Solutions. (1) The random effect is newly drawn; the observed design is held fixed. (2) The unconditional variance is \(9+4=13\), by independence. (3) The estimated effects use the subject’s observed data and are held fixed, so the check does not directly test whether new effects drawn from the population distribution have the right distribution. (4) The conditional mean is \(10-0.5=9.5\), and the conditional standard deviation is \(\sqrt{1}=1\); the random-effect variance is not included because the effect is fixed for this check.