Why Compare Simulated Data With Observed Data?
A simulation can run without coding errors and still generate data that do not resemble the trial population it is intended to represent. For example, the simulated average age may look plausible while the age distribution has too few older participants; the overall dropout rate may agree with the observed rate while dropout among high-risk participants is substantially different. Comparing distributions helps reveal these problems before simulation results are used to inform a decision.
The goal is not to make every simulated sample identical to one observed trial. Observed data are themselves a finite sample, and simulation replicates vary by chance. Instead, the comparison asks whether the simulation reproduces relevant features of the empirical data closely enough for its intended use. A discrepancy should be interpreted in context: its size, where it occurs in the distribution, whether it is clinically or operationally important, and whether it is larger than expected from sampling variability.
The observed trial is an empirical reference, not necessarily the true population distribution. The simulation may represent a target population broader than the observed sample, or the observed trial may have selection criteria that the simulated population does not yet apply. Before comparing numbers, establish what population each data set describes and whether the simulation is intended to reproduce the observed trial or to represent a different target.
Choose Comparisons That Show Distributional Shape
A mean and standard deviation are useful but cannot describe a whole distribution. Similar means can coexist with different skewness, spread, or tail behavior. For continuous variables, useful displays include overlaid histograms, density plots, empirical cumulative distribution functions (ECDFs), and quantile plots. For categorical variables, compare category proportions, including rare categories that may be hidden by a single overall summary.
An ECDF gives the fraction of observations at or below each value. If \(F_{\mathrm{obs}}(x)\) is the observed ECDF and \(F_{\mathrm{sim}}(x)\) is the simulated ECDF, their vertical separation at a value \(x\) is the difference between the two proportions at or below \(x\). The largest vertical separation, \(D=\sup_x |F_{\mathrm{sim}}(x)-F_{\mathrm{obs}}(x)|\), is a compact descriptive measure of distributional discrepancy. It identifies the greatest gap in cumulative proportions, but it does not say whether the difference matters clinically, and it can be sensitive to sample size when used as a formal test statistic.
Quantiles add a direct interpretation. Compare, for example, the 10th, 50th, and 90th percentiles to examine lower-tail, central, and upper-tail behavior. For covariates, also inspect clinically important cut points and subgroup frequencies. For longitudinal outcomes or exposure, compare distributions at the same visit or time point; a match at one visit does not establish a match across the trajectory.
Quantifying Differences Without Treating Them as Automatic Failures
A standardized mean difference expresses a mean discrepancy relative to a reference scale. One simple choice for a one-sample comparison is the observed standard deviation: \(d=(\bar{x}_{\mathrm{sim}}-\bar{x}_{\mathrm{obs}})/s_{\mathrm{obs}}\). This is dimensionless, while the raw difference remains important for interpretation in the variable’s original units. The choice of scale should be stated; there is no universal cutoff that makes a distribution acceptable for every MIDD purpose.
Sampling variability also matters. If the observed and simulated values are independent samples and the comparison is of their means, the variance of the difference of means is the sum of the two variances of the means. Thus an estimated standard error is \(SE(\bar{x}_{\mathrm{sim}}-\bar{x}_{\mathrm{obs}})=\sqrt{s_{\mathrm{sim}}^2/n_{\mathrm{sim}}+s_{\mathrm{obs}}^2/n_{\mathrm{obs}}}\). This calculation describes uncertainty in the difference under those sampling assumptions; it does not define an acceptable mismatch. In particular, a very large simulated sample can make a small discrepancy precisely estimated without making it consequential.
Worked Example: Comparing a Baseline Covariate
Suppose a hypothetical observed trial includes 200 participants with mean age 58 years and standard deviation 10 years. A simulated sample includes 1,000 participants with mean age 60 years and standard deviation 10 years. Assume for this descriptive calculation that the two samples are independent and that the age definitions and eligibility criteria match.
The raw mean difference, simulated minus observed, is \(60-58=2\) years. Standardizing by the observed standard deviation gives \(d=(60-58)/10=0.200\). The simulated mean age is therefore 2 years higher, or 0.200 observed standard deviations higher.
The estimated standard error of the mean difference is \(SE=\sqrt{10^2/1000+10^2/200}=\sqrt{0.1+0.5}=\sqrt{0.6}=0.775\) years. The difference is about \(2/0.775=2.58\) standard errors from zero under the stated sampling model. This is a signal to inspect the comparison, not a decision rule by itself. The simulated sample’s age quantiles and age-category frequencies should also be checked: a mean difference alone cannot show whether the mismatch is concentrated among older participants or reflects a general shift.
Worked Checks for Exposure, Outcomes, and Missingness
Worked Example: Checking the Exposure Distribution
Consider hypothetical individual exposure summaries, area under the concentration-time curve (AUC) in mg·h/L. The observed trial has 200 participants, with median AUC 120, 10th percentile 65, and 90th percentile 210 mg·h/L. The simulated sample has 1,000 participants, with corresponding values 125, 68, and 245 mg·h/L.
The median difference is \(125-120=5\) mg·h/L. The lower-tail difference is \(68-65=3\) mg·h/L, while the upper-tail difference is \(245-210=35\) mg·h/L. If the observed interquartile range is 70 mg·h/L, the upper-tail gap is half that scale: \(35/70=0.500\). This comparison suggests that central exposure is fairly similar but the simulated upper tail may be too high.
The next diagnostic is an overlaid ECDF or quantile plot, with attention to high AUC values and the simulated covariate or dosing patterns that produce them. The comparison is meaningful only if AUC was calculated over the same interval and under comparable rules for dose interruptions and missing concentration measurements. Similar medians alone would not establish that the exposure distribution is adequate.
Worked Example: Comparing an Outcome Distribution
Suppose a hypothetical continuous outcome is change from baseline at week 12, in points, with negative values indicating improvement. The observed trial has mean change \(-4.0\), standard deviation 6.0, 10th percentile \(-12\), and 90th percentile 4. The simulated sample has mean change \(-3.0\), standard deviation 8.0, 10th percentile \(-15\), and 90th percentile 6. Assume both summaries use the same outcome definition and week-12 population.
The mean difference is \(-3.0-(-4.0)=1.0\) point: the simulated average improvement is 1 point smaller. The standardized difference using the observed standard deviation is \(1.0/6.0=0.167\). However, the simulated standard deviation is 2 points larger, and its 10th-to-90th percentile span is \(6-(-15)=21\) points, compared with \(4-(-12)=16\) points observed. The distribution is more dispersed in simulation, even though the means are close.
A plot should show whether the extra spread is roughly symmetric or comes from one tail. If week-12 outcomes are missing for some participants, also check that the comparison uses a clearly specified population—for example, all randomized participants with a defined imputation or outcome-generation rule—rather than silently comparing different subsets.
Worked Example: Finding a Missingness Pattern Hidden by the Overall Rate
In a hypothetical observed trial, 40 of 200 participants are missing the week-12 outcome, for a rate of \(40/200=0.200\), or 20%. The simulated trial has 180 missing outcomes among 1,000 participants, giving \(180/1000=0.180\), or 18%. The overall rates differ by \(-0.020\), or minus 2 percentage points.
Now divide participants by baseline severity. In the observed trial, 32 of 80 participants with high severity are missing, a rate of \(32/80=0.400\); among 120 with low severity, 8 are missing, a rate of \(8/120=0.0667\). In simulation, the corresponding rates are \(130/400=0.325\) for high severity and \(50/600=0.0833\) for low severity.
The overall rates appear close, but the high-severity missingness rate is 7.5 percentage points lower in simulation, while the low-severity rate is 1.67 percentage points higher. This suggests that the simulated missingness mechanism may not reproduce its dependence on baseline severity. Inspect rates by additional relevant factors and visit, rather than relying on the marginal rate alone. These descriptive comparisons do not establish why outcomes are missing; they show where the simulated pattern differs from the observed one.
A Practical Comparison Workflow
Confirm population, analysis group, variable definition, units, time point, and handling of missing values. Record any differences that prevent a direct comparison.
Use means and standard deviations where appropriate, but add quantiles, category frequencies, or conditional missingness rates when the tails, subgroups, or patterns matter.
Use consistent axes and scales. Examine distribution shape and subgroup or time-specific discrepancies, not just whether plotted averages overlap.
Report raw differences with units and, when useful, standardized differences. Check whether a discrepancy could reflect finite-sample variation and whether it is important for the simulation’s purpose.
State which feature differs, how it may affect the intended decision, and whether the simulation should be revised or the limitation made explicit.
Deriving the Uncertainty of a Mean Difference
The standard error formula used above follows directly from independence. Let \(\bar{X}_{\mathrm{sim}}\) and \(\bar{X}_{\mathrm{obs}}\) be sample means from independent samples with variances \(\sigma_{\mathrm{sim}}^2\) and \(\sigma_{\mathrm{obs}}^2\). The variance of each sample mean is its population variance divided by its sample size. For independent random variables, the variance of a difference is the sum of the variances:
Replacing the unknown variances by the sample variances gives the estimated standard error stated earlier. This justification relies on independent observations within each sample as well as independence between samples. If the simulated values are clustered, repeated within participants, or pooled in a way that creates dependence, the simple formula may understate or misrepresent uncertainty. Moreover, it captures sampling variation in means, not uncertainty about whether the model structure itself is appropriate.
Diagnostics, Alternatives, and Limitations
A useful comparison combines visual and numerical evidence. A histogram can reveal skewness or multiple modes but depends on bin choices; an ECDF avoids binning but can be visually crowded with very large samples. Quantiles are interpretable but summarize only selected points. A single scalar discrepancy can simplify tracking across scenarios, yet may conceal which region of the distribution differs. Use more than one view when a variable is decision-relevant.
Formal two-sample tests are an alternative when a specific inferential question is appropriate, but they should not replace descriptive assessment. Their results depend on sample sizes, and a large simulation sample can make a practically small difference statistically detectable. Conversely, a small observed sample may have little power to reveal a meaningful tail discrepancy. The purpose here is model checking against empirical features, not declaring two distributions identical.
The observed data may be affected by measurement error, trial selection, protocol deviations, or chance. Agreement with one empirical trial therefore does not prove that the simulation captures the target population or will predict a future trial. If a discrepancy appears, consider whether it reflects a coding or alignment issue, an incorrect input distribution, or a limitation of the observed reference. Revise only for a reasoned model or data issue; do not tune the simulation solely to make every observed summary match.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Observed mean age is 50 years with standard deviation 8 years; simulated mean age is 52 years. What is the standardized mean difference using the observed standard deviation, and what does it mean?
- Why can a close simulated and observed median AUC fail to establish that exposure distributions agree?
- Observed missingness is 15% and simulated missingness is also 15%. Name one comparison that could still reveal a mismatch.
- In one sentence, explain why a very small p-value from a large simulated sample does not by itself establish a practically important discrepancy.
Solutions. 1. \(d=(52-50)/8=0.250\). The simulated mean is 0.250 observed standard deviations higher; the raw difference is 2 years. 2. The tails or spread can differ despite equal medians, so compare quantiles and a distribution plot as well. 3. Compare missingness rates by baseline risk group, treatment, or visit; for example, the overall rate can match while the high-risk rate differs. 4. A large sample can detect a small difference precisely, while practical importance depends on its size, units, and relevance to the decision.