Why Missing Records Change a Predictive Check
A predictive check compares observed data with data generated under a fitted model. When some planned measurements are absent, the comparison also depends on which records are available. A model may describe concentrations well at observed visits but fail to reproduce when records are missing or when subjects stop contributing measurements. Conversely, an apparent mismatch in recorded concentrations may reflect selective observation rather than a misspecified population outcome distribution.
The distinction is important: simulating outcomes at the times where the data happen to be observed is not the same as simulating which outcomes would have been observed. The first targets outcomes at those record locations; the second also targets the observation and dropout process. Neither comparison is universally preferable. The right choice depends on the question and on how observation relates to outcomes.
A useful formal description separates the outcome model from the observation mechanism. Write \(f(Y\mid X;\theta)\) for the outcome distribution given the design and covariates \(X\), with NLME parameters \(\theta\), and \(g(R\mid Y,X;\phi)\) for the distribution of the observation mask. The joint model is
In an NLME model, \(f\) includes the specified population model, between-subject variation, and residual variation. Whether \(g\) depends on outcomes is central to interpreting a predictive check. The factorization is a modeling statement, not a claim that the two components are independent.
Missingness Assumptions and the Target of the Check
The standard missingness terms describe how the probability of being observed relates to outcomes. Under missing completely at random (MCAR), observation does not depend on outcomes or covariates. Under missing at random (MAR), observation may depend on observed covariates or observed outcome history, but not on the unobserved outcome after conditioning on that observed information. Under missing not at random (MNAR), observation can still depend on an outcome that is itself missing, even after conditioning on observed information. These are assumptions about the data-generating process; a mask alone generally cannot establish which assumption holds.
If the mask is independent of outcomes under the model and design, holding it fixed can provide a useful comparison of outcomes at the observed record locations. But a fixed mask does not, by itself, make simulated outcomes draws from the outcome distribution conditional on that mask. If observation selects records according to outcome values, then the observed outcomes are selected, and the conditional distribution can differ from the population outcome distribution. For MAR mechanisms, conditioning on appropriate observed history may account for selection; simply reusing the mask without that conditioning need not do so.
A joint predictive check instead generates outcomes and then generates a mask from a specified observation model. It can compare both recorded outcomes and the amount or timing of observation. Under MCAR this may add little for outcome calibration, but it still checks whether the model reproduces missing-record patterns. Under MAR or MNAR, simulating the mechanism can be essential to reproduce the distribution of recorded data. For MNAR, credible joint simulation requires substantive assumptions about the observation mechanism, which are often weakly identified from observed data alone.
Worked Example: Outcome-Dependent Record Availability
Suppose a hypothetical population has a concentration \(Y\) of 1 mg/L or 9 mg/L, each with probability 0.5. A record is observed with probability 0.875 when \(Y=1\) and probability 0.125 when \(Y=9\). Calculate the population mean and the expected mean among recorded values.
The population mean is
The overall probability of observing a record is
Among observed records, the probability of \(Y=1\) is \(0.5(0.875)/0.5=0.875\), and the probability of \(Y=9\) is \(0.5(0.125)/0.5=0.125\). Thus
If outcomes are simulated from the population model at a fixed set of record locations without conditioning on this selection, their expected mean is 5 mg/L, not 2 mg/L. Holding the observed mask fixed does not repair that discrepancy. A joint simulation that generates \(Y\) and then \(R\) according to the stated probabilities yields a mean of 2 mg/L among simulated recorded values, on average. This example is MNAR because observation depends on the outcome itself.
Two Predictive-Check Strategies
The two strategies answer related but distinct questions. In a fixed-mask outcome check, use the observed record locations and simulate outcomes for those locations under the fitted outcome model. Compare observed and simulated summaries at matched times, covariates, or other design features. This asks whether the outcome model predicts values at those locations under the simulation conditioning used. If missingness is outcome-dependent, the simulation must also represent the relevant selection or condition on it appropriately before the comparison can be interpreted as conditional on the mask.
In a joint observation-process check, simulate outcomes and then simulate \(R\) from an observation model. Apply the simulated mask and calculate summaries using the simulated observed records, as for the real data. This can assess record counts, dropout timing, and outcome summaries among records that would be available. The mask mechanism must be defined at the relevant level—for example, whether a missed visit is intermittent or whether dropout prevents all later records.
Worked Example: Fixed Record Locations Under MCAR
In a hypothetical study, 12 subjects each have a planned concentration measurement at 4 hours. Four records are missing completely at random, so 8 values are observed. Suppose their observed mean is 6.0 mg/L. For illustration, assume the fitted model predicts \(Y\sim N(5, 2^2)\) mg/L at 4 hours, independently across subjects for this simple calculation. Under MCAR, checking the eight observed locations is appropriate for comparing their mean to the outcome model; the missingness does not preferentially select high or low concentrations.
For one simulated replicate at those eight locations, suppose the values are 4, 5, 6, 3, 7, 4, 5, and 6 mg/L. Their mean is
The observed mean, 6.0 mg/L, is 1.0 mg/L above this replicate's mean. One replicate is not enough to judge model fit: many simulated replicates are needed to show the predictive distribution and assess how unusual the observed summary is within it. The fixed-mask comparison addresses outcome values at the eight observed locations. It does not assess whether four missing records is plausible; that requires an observation-process check.
A short justification clarifies when the simplest fixed-mask comparison has a direct interpretation. If, conditional on the design, \(R\) is independent of \(Y\), then selection by the mask does not alter the outcome distribution:
for any outcome set \(A\), because \(P(R=1\mid Y,X)=P(R=1\mid X)\). This supports comparison with the outcome model at the observed locations under the stated independence. If the observation probability varies with \(Y\), that cancellation does not hold. If it depends on observed history, a valid conditional comparison may require conditioning or simulating that history and its role in observation; fixing only the final mask is not a substitute.
Worked Example: Checking Simulated Dropout Patterns
Suppose a hypothetical study follows 10 subjects at four occasions. Each subject is observed at occasion 1, and after each occasion a subject who is still being followed drops out with probability 0.20, independently of outcomes and other subjects. Dropout is permanent. Calculate the expected number observed at each occasion under this mechanism.
A subject is observed at occasion 1 with probability 1. At later occasions the subject must avoid dropout after every prior occasion. Therefore the probabilities of contributing a record are \(1\), \(0.8\), \(0.8^2\), and \(0.8^3\). Multiplying by 10 gives expected record counts
Suppose the actual hypothetical record counts are 10, 8, 5, and 2. The final two counts are lower than these expectations, but a single dataset need not equal its expectation. A predictive check would simulate many dropout patterns under the mechanism, calculate the four counts in each replicate, and compare the observed trajectory with their simulated distribution. If the observed late counts repeatedly fall in the lower tail, that is evidence the specified dropout mechanism may not reproduce the observed pattern. It is not, by itself, evidence that the concentration outcome model is wrong.
MAR, Conditioning, and a Critical Distinction
Suppose dropout after the first occasion depends on the observed first concentration. Under MAR, the dropout probability can depend on that recorded value, while the later unobserved concentration does not additionally determine dropout after conditioning on the observed history. To predict later observed values, one must preserve the relation between prior observed outcomes, dropout, and later outcomes. One valid strategy is to simulate histories sequentially: generate an outcome, generate the next observation indicator using the observed history, and continue according to the model. A fixed-mask calculation can also be designed to condition on the relevant observed history, but the mask by itself does not supply that conditioning.
Worked Example: MAR Selection Through an Observed Prior Value
Consider a hypothetical two-occasion setting. At occasion 1, \(Y_1\) is 2 or 8 mg/L with equal probability. At occasion 2, let \(Y_2=Y_1+\epsilon\), where \(\epsilon\) is independently \(-1\) or \(1\) mg/L with equal probability. The occasion-2 record is observed with probability 0.8 if \(Y_1=2\) and 0.2 if \(Y_1=8\). This mechanism depends on observed history, not on the unobserved \(Y_2\), so it is MAR in this example.
The population mean of \(Y_2\) is \(E(Y_1)+E(\epsilon)=5+0=5\) mg/L. The probability of an occasion-2 record is \(0.5(0.8)+0.5(0.2)=0.5\). Among recorded occasion-2 values, the probabilities of prior values 2 and 8 are respectively \(0.5(0.8)/0.5=0.8\) and \(0.5(0.2)/0.5=0.2\). Since the conditional mean of \(Y_2\) is \(Y_1\), the mean among recorded values is
Unconditionally simulating \(Y_2\) at the observed number of occasion-2 locations would center simulated values at 5 mg/L, not 3.2 mg/L. A sequential joint simulation generates \(Y_1\), then \(R_2\) using \(Y_1\), then \(Y_2\) according to its conditional model, and retains \(Y_2\) when \(R_2=1\). Alternatively, a fixed-location simulation must condition appropriately on the observed histories and selection structure. The fact that the mechanism is MAR does not make an unconditional outcome draw at a fixed mask representative of the recorded subset.
Diagnostics, Alternatives, and Limitations
A useful check makes its target explicit and uses consistent denominators. For outcome summaries, state whether the calculation uses all simulated outcomes, only simulated records, or values at the actual observed locations. For observation summaries, compare counts and dropout times as well as the fraction of records missing at each planned occasion. For both, use comparable strata such as nominal time or dose when those define materially different predictions. Avoid treating missing values as zero: zero is an outcome value, whereas a missing record provides no recorded value.
Typical warning signs include simulated record counts that are systematically too high or too low, dropout occurring too early or too late, and outcome distributions among simulated records that differ from observed records even when unconditional simulated outcomes appear plausible. Such patterns can reflect a poor outcome model, an inadequate observation model, or both. A predictive check diagnoses mismatch with the full specification used; it generally cannot identify which component is responsible without additional information or targeted checks.
| Approach | Main question | Key condition or limitation |
|---|---|---|
| Outcome simulations at observed locations | Are outcomes predicted at these record locations? | Selection must be irrelevant or represented by appropriate conditioning; a fixed mask alone is insufficient when selection depends on outcomes. |
| Joint outcome and mask simulations | Does the model reproduce recorded outcomes and the observation pattern? | Requires a plausible observation model; MNAR assumptions may not be identified from observed data alone. |
| Observation-pattern summaries | Are record counts and dropout patterns reproduced? | Checks the observation process but does not establish that the outcome model is adequate. |
The joint approach is more complete for questions about the observed dataset, but it is also more assumption-dependent. A fitted outcome model cannot, on its own, predict outcome-dependent dropout. Conversely, a flexible observation model cannot correct a misspecified outcome distribution. When only an outcome-model check is intended, report clearly that it is evaluated at observed locations and explain the conditioning assumptions behind interpreting it.
Common Pitfall
- Wrong reasoning: “The observed mask is fixed, so simulated values have the distribution of outcomes conditional on being observed.”
- Correction: Fixing the mask identifies locations for comparison; it does not condition simulated outcomes on the event that those records were observed. Under outcome-dependent selection, the observed distribution can differ from the population distribution. Simulate the observation mechanism or condition on it appropriately for the stated target.
- Wrong reasoning: “The observed and simulated record counts differ, so the concentration model must be wrong.”
- Correction: Record counts primarily assess the observation mechanism. A mismatch may motivate investigation of dropout assumptions, data processing, or the design before attributing it to the outcome model.
Check Your Understanding
Answer each question, then compare with the solutions.
- Under MCAR, what does an outcome-only check at the observed record locations assess, and what does it leave unchecked?
- Why can holding an observed mask fixed fail to reproduce the distribution of outcomes among recorded values under MNAR?
- A dropout model has a 0.25 probability of dropout after each occasion, independently of outcomes, with permanent dropout. Out of 20 subjects observed at occasion 1, how many records are expected at occasions 2 and 3?
- Under MAR, does the label alone guarantee that unconditional simulated outcomes at a fixed mask represent the recorded values? Explain.
Solutions. (1) Under MCAR, it checks outcome predictions at observed locations because selection does not depend on outcomes; it does not check whether the number or timing of missing records is reproduced. (2) If observation probability depends on the outcome, the distribution conditional on \(R=1\) is selected and can differ from the population distribution. (3) The probabilities of contributing at occasions 2 and 3 are \(0.75\) and \(0.75^2=0.5625\), giving \(20(0.75)=15\) and \(20(0.5625)=11.25\) expected records. (4) No. MAR allows observation to depend on observed history, which can associate the mask with outcome trajectories. The simulation must represent or appropriately condition on that history and selection process.