Why Scenario Fidelity Needs Its Own Audit
A simulation can be reproducible, computationally reliable, and still fail to represent the scenario its designers intended. A modality may have the right mean but the wrong spread; two modalities may have realistic marginal distributions but too little dependence; an overall missingness rate may conceal a reversed group pattern; or the outcome prevalence may look right while the simulated effect is too weak. Scenario fidelity is the agreement between a scenario’s specified data-generating properties and the properties of data produced by its generator.
The key comparison is not “does this finite data set equal its target?” Random generation creates sampling variation even when the generator is correct. Instead, ask whether differences from the target are compatible with that variation, and whether the diagnostic checks the feature that the design actually specified. A useful audit compares several complementary summaries, then traces notable discrepancies back to the relevant part of the generator.
Write \(b=1,\ldots,B\) for independent simulated replicates in a scenario. Let \(X^{(k)}\) denote the data for modality \(k\), \(R^{(k)}=1\) indicate that a modality value is missing, and \(Y\) denote an outcome. A target can describe a population distribution, a relationship among variables, or an explicit mechanism such as \(P(R^{(k)}=1\mid G)\). State which one it is before choosing a diagnostic. A target for an overall mean, for example, does not by itself specify a target for every subgroup mean.
From Target to Diagnostic
For each target, choose a summary that corresponds directly to it. For a marginal distribution, inspect location, spread, and relevant quantiles, not just the mean. For dependence, inspect the targeted correlations or other dependence summaries, including the particular pairs of modalities or variables specified by the design. For missingness, compare rates overall and within the groups, modalities, or covariate ranges that determine the missingness mechanism. For an outcome, examine both its marginal distribution and the conditional relationship used to generate it.
The target may be a population property, whereas the diagnostic is computed from a finite simulated sample. If \(T_b\) is a summary from replicate \(b\), its average across replicates is \(\bar{T}=B^{-1}\sum_{b=1}^{B}T_b\). When replicates are independent and have finite variance, the estimated Monte Carlo standard error of \(\bar{T}\) is \(s_T/\sqrt{B}\), where \(s_T\) is the sample standard deviation of the replicate summaries. This measures simulation error in estimating the average summary; it does not measure all uncertainty in an individual data set.
For a sample mean from independent observations with population standard deviation \(\sigma\), the variance of the mean in one replicate of size \(n\) is \(\sigma^2/n\). Averaging \(B\) independent replicate means divides that variance by \(B\). Thus,
In practice, the replicate-to-replicate standard deviation can estimate the needed variation without knowing \(\sigma\). This comparison is meaningful only when the replicates are generated independently as intended and the summary is defined consistently across them. A standardized discrepancy, such as target difference divided by estimated Monte Carlo standard error, helps describe scale; it is not a universal cutoff or a substitute for checking the generator.
Worked Example: Checking a Marginal Mean
Suppose a scenario generates \(n=200\) independent observations per replicate from a distribution with target mean \(0\) and standard deviation \(1\). Across \(B=100\) hypothetical replicates, the average of the 100 sample means is \(-0.012\), and their sample standard deviation is \(0.071\). Check the mean target.
The Monte Carlo standard error of the average sample mean is \(0.071/\sqrt{100}=0.0071\). Its discrepancy from the target, in Monte Carlo standard-error units, is \[ (-0.012-0)/0.0071\approx -1.69. \]
A discrepancy of about \(-1.69\) Monte Carlo standard errors is not, by itself, strong evidence of a generator defect. As a consistency check, the expected standard deviation of the sample mean in a single replicate is \(1/\sqrt{200}\approx0.0707\), close to the observed replicate standard deviation \(0.071\). The pattern is compatible with the stated independent, unit-variance design. This check does not establish that the entire marginal distribution is correct; quantiles or distribution plots could still reveal skewness or tail problems.
Check Dependence, Not Only Margins
In multi-omics settings, dependence is often a central part of the scenario. Matching each modality’s mean and variance does not ensure that cross-modality relationships are correct. Two variables can have identical marginal distributions under very different joint distributions. Therefore, use pairwise or multivariate diagnostics that correspond to the design: for example, cross-modality correlations, within-group correlations, or dependence between a latent quantity and observed features. Check the same scale and conditioning structure used to define the target.
Worked Example: A Cross-Modality Correlation Is Too Weak
A design specifies standard-normal variables in two modalities with target correlation \(0.60\). In 40 hypothetical replicates of \(n=250\) paired observations, the replicate sample correlations have average \(0.41\) and sample standard deviation \(0.060\). The Monte Carlo standard error of their average is \(0.060/\sqrt{40}\approx0.00949\). The standardized discrepancy is \[ (0.41-0.60)/0.00949\approx -20.0. \]
This difference is far larger than the replicate-to-replicate Monte Carlo variation in this diagnostic and merits investigation. For example, the generator may use \(X^{(2)}=0.40X^{(1)}+\sqrt{1-0.40^2}\,\varepsilon\), where \(X^{(1)}\) and \(\varepsilon\) are independent standard-normal variables. This construction gives each variable variance \(1\), but their correlation is \(0.40\), not \(0.60\). Replacing the coefficient \(0.40\) with \(0.60\), and the noise multiplier with \(\sqrt{1-0.60^2}=0.80\), gives the intended correlation under those assumptions. The marginal checks alone would not have exposed the original error.
Dependence diagnostics should also be matched to the structure of the target. A single overall correlation can conceal group-specific differences or nonlinear dependence. If the scenario specifies a block structure, inspect correlations within and between blocks. If a latent variable is meant to drive several modalities, check whether the simulated modalities retain the intended shared structure. A scatterplot or heat map can reveal patterns that one numeric summary misses, but visual inspection should complement, not replace, a quantitative comparison to explicit targets.
Audit Missingness at the Intended Level
An overall missingness rate is insufficient whenever the design makes missingness depend on modality, group, outcome, or observed covariates. Let \(R^{(k)}=1\) denote a missing value in modality \(k\). If the design specifies \(P(R^{(k)}=1\mid G=g)\), compare the simulated rate within each group \(g\) to that conditional target. Also verify which variables are used to generate missingness: a correct marginal rate does not confirm that the intended mechanism was implemented.
Worked Example: The Overall Missingness Rate Hides a Reversal
A hypothetical design has two equally sized groups. The target missingness probabilities are \(0.10\) in group A and \(0.30\) in group B, so the target overall rate is \[ (0.10+0.30)/2=0.20. \] Across 25 replicates, there are 2,500 observations per group. The generator produces 350 missing values in A and 650 in B.
The simulated rates are \(350/2500=0.14\) in A and \(650/2500=0.26\) in B. The overall rate is \((350+650)/5000=0.20\), exactly the target overall rate, but the conditional rates are each \(0.04\) away from their targets in opposite directions. Under independent Bernoulli missingness at the specified probabilities, the standard errors for the pooled group rates are \[ \sqrt{0.10(0.90)/2500}=0.0060 \quad\text{and}\quad \sqrt{0.30(0.70)/2500}\approx0.00917. \]
The respective conditional discrepancies are about \(0.04/0.0060=6.67\) and \(-0.04/0.00917\approx-4.36\) standard errors. The exact overall match therefore does not establish fidelity: the group pattern is inconsistent with the target mechanism and should be traced to the missingness code. These calculations assume independent missingness indicators within each group; if the design includes clustering, the independent-binomial standard errors are not appropriate.
Check the Outcome Mechanism as Well as Prevalence
Outcome diagnostics should reflect the outcome-generating mechanism specified for the scenario. For a binary outcome, compare both the overall event proportion and conditional event proportions or contrasts across relevant predictors. For a continuous outcome, examine its marginal distribution and its conditional mean or other specified response pattern. A correct overall outcome distribution can coexist with an incorrect relationship between the outcome and the features that methods are meant to analyze.
Worked Example: Correct Prevalence, Attenuated Outcome Contrast
Suppose \(X\) is binary with equal numbers in its two groups, and the target event probabilities are \(P(Y=1\mid X=0)=0.20\) and \(P(Y=1\mid X=1)=0.60\). Across 20 hypothetical replicates there are 20,000 observations in each group. The generator produces 5,000 events for \(X=0\) and 11,000 for \(X=1\).
The simulated conditional event proportions are \(5000/20000=0.25\) and \(11000/20000=0.55\). The observed risk difference is \(0.55-0.25=0.30\), whereas the target risk difference is \(0.60-0.20=0.40\). Yet the overall event proportion is \((5000+11000)/40000=0.40\), equal to the target overall proportion \((0.20+0.60)/2=0.40\).
Under independent Bernoulli outcomes at the target probabilities, the standard error of the difference in sample proportions is \[ \sqrt{\frac{0.20(0.80)}{20000}+\frac{0.60(0.40)}{20000}} =\sqrt{0.000020}\approx0.00447. \] The observed risk difference is \(0.10/0.00447\approx22.4\) such standard errors below its target. This is a strong diagnostic signal of an attenuated conditional effect, even though prevalence matches. The calculation assumes independent observations within each group and the specified group sizes.
Diagnose, Revise, and Recheck
A discrepancy is most useful when it narrows down what to inspect. If a marginal mean is wrong, check parameter values, centering, scaling, and transformation order. If margins look right but dependence does not, inspect shared latent variables, noise terms, and correlation or covariance construction. If missingness is wrong only in subgroups, examine group coding and the conditional probability calculation. If outcome prevalence matches but conditional effects do not, inspect the link function, coefficients, and predictors passed to the outcome generator.
After changing the generator, rerun the relevant diagnostics and the broader audit. A local fix can alter other properties: changing a noise variance may correct a correlation while changing a marginal variance, for instance. Keep the targets and diagnostic definitions explicit so that the revised data are judged against the same design. If a target is intentionally approximate, state its tolerance and why that tolerance is appropriate; do not silently relabel an unintended discrepancy as part of the design.
Specify whether each target is marginal, conditional, or joint, and name the modality, group, or outcome it concerns.
Use replicate-level summaries where possible, and include plots or subgroup checks when a single number could hide structure.
Use sampling and Monte Carlo variability appropriate to the design, including dependence or clustering when present.
Locate the generator component implicated by the discrepancy, make a targeted change, and repeat checks for both the affected property and properties the change could alter.
Alternatives, Limitations, and Common Pitfalls
| Diagnostic approach | What it contributes | What it can miss |
|---|---|---|
| Targeted numerical summaries | Direct, reproducible comparisons for means, rates, correlations, or contrasts | Features not represented by the chosen summaries |
| Plots and distribution overlays | Shape, outliers, nonlinear patterns, and mismatches across ranges | Small discrepancies can be hard to judge consistently by eye |
| Conditional or subgroup checks | Mechanism-specific patterns hidden by an overall summary | Unexamined groups or interactions may still be wrong |
| Automated invariants and unit checks | Fast detection of impossible values, dimensions, and coding errors | A generator can satisfy invariants while producing the wrong intended distribution |
No single diagnostic proves scenario fidelity. Finite sets of summaries cannot uniquely establish a full joint distribution, and a limited number of replicates can make Monte Carlo estimates noisy. Diagnostics must also reflect the design’s dependence structure: formulas based on independent observations can understate variability when observations are clustered or otherwise dependent. For unusual or complex targets, simulate a large reference sample from a trusted implementation or derive a reference distribution, then compare the production generator against it.
A common mistake is to see an overall rate or mean that matches and conclude that the generator is correct. That reasoning fails because aggregation can cancel opposing subgroup errors, and marginal agreement says little about dependence. The correction is to compare the properties the design actually specifies at the same level of conditioning. A related mistake is to treat every finite-sample discrepancy as a bug. Instead, examine its size relative to the expected variation, its pattern across replicates, and whether it points to a specific unintended property. Diagnostic thresholds are screening aids, not universal guarantees.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A target mean is \(5\). Across 64 independent replicates, the average replicate mean is \(5.016\), and the standard deviation of replicate means is \(0.032\). Calculate the Monte Carlo standard error and the standardized discrepancy.
- Why can matching the separate means and variances of two modalities fail to establish that their simulated joint behavior is correct?
- Two equally sized groups have target missingness rates \(0.10\) and \(0.30\). Explain why an overall rate of \(0.20\) is not enough to verify fidelity.
- A binary outcome has target probabilities \(0.2\) and \(0.6\) in two equally represented predictor groups. What is the target overall event rate and target risk difference?
Solutions. (1) The Monte Carlo standard error is \(0.032/\sqrt{64}=0.004\). The standardized discrepancy is \((5.016-5)/0.004=4.0\), a notable signal to investigate, not proof by itself of a defect. (2) Marginal means and variances do not specify dependence; for example, the cross-modality correlation could be too weak or too strong while both margins remain correct. Check the targeted joint summaries. (3) The two conditional rates can be wrong in opposite directions and still average to \(0.20\). Compare each group’s simulated rate with its own target. (4) The target overall rate is \((0.2+0.6)/2=0.4\); the target risk difference is \(0.6-0.2=0.4\).