Simulation Checks For Informative Observation Mechanisms

Why Simulate an Observation Mechanism?

A longitudinal analysis uses measurements collected at visits, but the visits themselves may depend on a patient’s evolving health. If people are monitored more often when their biomarker is high, a summary over recorded measurements can describe the biomarker values at visits rather than the population trajectory over time. Simulation offers a direct check: specify a population trajectory and a visit mechanism, so the target is known, and see whether an analysis recovers it.

A useful simulation does not merely generate a large table of plausible-looking records. It makes the observation mechanism explicit, preserves the distinction between a person’s latent trajectory and the measured values, and defines a target that can be calculated from the generating model. Comparisons between schedules are meaningful only when the simulated population and target stay fixed while the schedule changes.

Specify the Target and the Generating Process

Begin by stating what the analysis is meant to estimate. For example, the marginal mean trajectory is \(\mu(t)=E\{Y_i(t)\}\), where \(Y_i(t)\) is subject \(i\)’s latent biomarker at time \(t\). The marginal linear slope is the change in this population mean per unit of time, when a linear mean trajectory is part of the model. These are population summaries over subjects and time; they are not automatically equal to summaries of the values recorded at visits.

A simulation should specify at least four pieces: the population distribution of trajectories, the visit process conditional on those trajectories, the measurement model at visits, and the study’s time window or censoring rule. For instance, let subjects have random intercepts \(b_{0i}\) and random deviations \(b_{1i}\) from a mean slope of \(0.5\) biomarker units per year:

The population mean is \(\mu(t)=0.5t\), and the target marginal slope is \(0.5\) biomarker units per year. At visit time \(t_{ij}\), suppose the recorded value is \(y_{ij}=Y_i(t_{ij})+\epsilon_{ij}\), with independent measurement errors \(\epsilon_{ij}\sim N(0,0.3^2)\). A visit mechanism can then be defined through the conditional intensity \(\lambda_i(t)\), the instantaneous rate of visits per year given the trajectory.

Setting \(\gamma=0\) gives a constant rate of \(1.2\) visits per year, independent of the trajectory. Choosing \(\gamma=0.7\) makes the rate higher when the latent biomarker is above its population mean. The exponent is dimensionless: \(\gamma\) has units per biomarker unit. Conditional on the trajectory, visits can be generated as an inhomogeneous Poisson process with this intensity. The visit process should be simulated over the full follow-up window, not just at times when a measurement happens to be recorded.

Why Outcome-Dependent Visits Can Change a Summary

A simple calculation shows what a simulation should detect. At a fixed time, suppose the latent outcome is \(Y\sim N(\mu,\sigma^2)\), and its visit intensity is proportional to \(\exp\{\gamma(Y-\mu)\}\). A randomly selected visit is more likely to come from a subject with a larger intensity. Consequently, the outcome distribution among visits is the population distribution tilted by this exponential factor.

For completeness, the normal moment-generating function gives \(E\{\exp(\gamma Y)\}=\exp(\gamma\mu+\gamma^2\sigma^2/2)\). Differentiating this expression with respect to \(\gamma\) gives \(E\{Y\exp(\gamma Y)\}=(\mu+\gamma\sigma^2)\exp(\gamma\mu+\gamma^2\sigma^2/2)\). The common factor from centering by \(\mu\) cancels in the ratio, yielding the stated visit-weighted mean. This result is a fixed-time illustration; it is not a formula for every longitudinal estimator or every visit mechanism.

Run and Evaluate the Simulation

For each replicate, draw a new sample of subjects and their random effects, generate each subject’s visit process over \(0\leq t\leq 2\), and generate noisy measurements at those visits. Apply the same analysis plan under at least two schedules: one with \(\gamma=0\), and one with a prespecified nonzero \(\gamma\). Keep the population trajectory, measurement error, follow-up window, and sample size unchanged between schedules. Otherwise, a difference in results cannot be attributed clearly to informative observation.

An inhomogeneous Poisson process can be generated by thinning: propose events from a rate that bounds \(\lambda_i(t)\) over the interval, then retain each proposal with probability equal to the actual intensity divided by that bound. The bound must be valid over the interval; a bound that is too low makes the simulated process incorrect. Alternatively, generate event times from the integrated intensity when its inverse is available. Check that visit times fall within follow-up, that rates are nonnegative, and that the empirical event counts agree with the intended process.

For replicate \(r\), let \(\hat\theta_r\) estimate a known target \(\theta\), such as the marginal slope. Across \(R\) independent replicates, estimate bias by \(\overline{\hat\theta}-\theta\), where \(\overline{\hat\theta}=R^{-1}\sum_{r=1}^R\hat\theta_r\). The empirical standard deviation of the replicate estimates describes their variability. If each replicate produces a confidence interval, coverage is the proportion of intervals containing \(\theta\). Bias, variability, and coverage answer different questions: a low-variability estimator may still be systematically wrong, and a small estimated bias does not ensure accurate interval coverage.

Checks, Comparisons, and Failure Modes

Simulation conclusions are only as credible as the generator and its checks. First run a noninformative benchmark: when visits are independent of trajectories and the analysis assumptions otherwise hold, an analysis intended for that setting should recover its target within Monte Carlo error. Then increase the number of subjects and replicates separately. Increasing subjects addresses the behavior of the estimator for larger studies; increasing replicates makes the simulation’s performance summaries less noisy. Neither change repairs a misspecified or incorrectly coded generating process.

Compare methods that answer the same target question. A pooled analysis gives greater representation to subjects who contribute more visits. An analysis giving subjects equal total weight can reduce that particular imbalance, but it does not automatically remove within-subject selection over time. Inverse-intensity weighting is a useful comparator when its assumptions hold; an oracle version using the true intensity is a simulation benchmark, while an applied version must estimate weights from available data and account for uncertainty or misspecification. These approaches are not interchangeable merely because each produces a slope estimate.

Useful diagnostics include plotting visit counts against simulated latent outcomes, checking rates over time, and comparing empirical fixed-time visit means with the selection calculation when its assumptions apply. Verify that the noninformative and informative schedules actually differ in the intended way. Also inspect extreme weights, empty or sparse regions of follow-up, and the frequency of subjects with too few visits for the planned analysis. Such problems can reveal weak support for the target or unstable estimation, even when the code runs without errors.

• Mechanism coded with the wrong sign. If high values should increase visits, confirm that the intensity increases as \(Y_i(t)-\mu(t)\) increases.
• Visits generated from measured values when the intended mechanism is latent. Measurement error changes the dependence being simulated. Use the variable specified by the scientific mechanism.
• Only reporting the average estimate. Report variability and interval coverage as well; averages can hide unreliable uncertainty estimates.
• Treating oracle weights as an applied solution. True intensities may depend on unobserved quantities. State what information an implementable analysis would actually use.

A simulation is conditional on its assumptions. It cannot establish that a chosen visit mechanism is the real one, and good performance under one mechanism does not guarantee good performance under another. Vary plausible strengths of outcome dependence, measurement error, follow-up, and model misspecification. When an analysis performs poorly, report which features of the simulated setting caused the failure rather than presenting one performance number as universal.