Validate Sensitivity Code With Simulated Data

Why Validate Sensitivity Code With Simulation?

Sensitivity analyses depend on a chain of computations: generating or encoding missingness patterns, filling unobserved outcomes according to a chosen rule, and calculating an estimand from the completed data. A small implementation error at any step can produce a plausible-looking result. Simulation provides a way to check those steps against data for which the generating mechanism and, often, the target answer are known.

The aim is not to show that a sensitivity assumption is true for a real study. Instead, a simulation test asks whether the code does what its specification says. This is most informative when checks are targeted: verify a pattern label on a small hand-checkable data set, compare a simple imputation with its arithmetic answer, and study repeated estimates under a mechanism with a known target.

Definition: A simulation-based validation check generates data from a specified mechanism with known parameters, applies the analysis code being tested, and compares its output with a known value, a hand-calculated result, or a distribution implied by that mechanism. Passing these checks supports correct implementation for the tested cases; it does not establish that the mechanism describes a particular study.

Specify the Data-Generating Mechanism First

Let $i=1,\ldots,n$ index participants and $j=1,\ldots,T$ index scheduled visits. Let $A_i$ denote treatment assignment, $Y_{ij}$ the full-data outcome that would be defined at visit $j$, and $R_{ij}=1$ if that outcome is observed and $R_{ij}=0$ otherwise. In a monotone-dropout simulation, a participant observed through visit $j$ may drop out before visit $j+1$, but cannot return after dropout.

For example, a Gaussian outcome model might be $Y_{ij}=\beta_0+\beta_A A_i+\beta_j+\epsilon_{ij}$, with errors generated from a specified joint distribution across visits. The dropout mechanism must also be stated. One sequential example sets $R_{i1}=1$ for every participant and, for visits $j=2,\ldots,T$, uses

P(R_{ij}=0\mid R_{i,j-1}=1,Y_{i,j-1},A_i) =\operatorname{logit}^{-1}(\gamma_0+\gamma_A A_i+\gamma_Y Y_{i,j-1}),

where $\operatorname{logit}^{-1}(x)=1/(1+\exp(-x))$. If $R_{i,j-1}=0$, set $R_{ij}=0$; otherwise draw the dropout indicator using the stated probability. This rule defines dropout only from visit 2 onward, avoiding an undefined prior visit. A different initial-visit dropout rule would need to be specified if dropout at visit 1 were allowed.

The simulation specification should make clear the outcome distribution, dependence across visits, treatment allocation, dropout rule, and analysis target. Keep full outcomes $Y_{ij}$ separate from observed outcomes: the latter are $Y_{ij}$ when $R_{ij}=1$, and missing otherwise. That separation lets the test code compare its observed-data handling with the known full data.

Check Pattern Coding With Hand-Verifiable Data

A pattern-coding function should map each participant's observation indicators to the intended pattern. For monotone dropout, define $D_i$ as the last visit observed. With $R_{i1}=1$, $D_i$ is the largest $j$ such that $R_{ij}=1$. Thus $D_i=T$ means all scheduled outcomes are observed. The encoded pattern should agree with the entire indicator sequence, not merely with a count of observed visits; this distinction matters when intermittent missingness is possible.

Worked Example: Verify Monotone Pattern Labels

Suppose $T=4$, and a small hypothetical test data set has the following indicator sequences. Under monotone dropout, compute $D_i$ for each participant and the count in each pattern.

Participant	$(R_{i1},R_{i2},R_{i3},R_{i4})$	$D_i$
1	(1, 1, 1, 1)	4
2	(1, 1, 1, 1)	4
3	(1, 1, 1, 0)	3
4	(1, 1, 0, 0)	2
5	(1, 0, 0, 0)	1
6	(1, 1, 1, 1)	4
7	(1, 1, 0, 0)	2
8	(1, 1, 1, 0)	3

The pattern counts are $n_1=1$, $n_2=2$, $n_3=2$, and $n_4=3$. Their sum is $1+2+2+3=8$, which matches the number of participants. A validation test can assert both the eight individual labels and these counts. It should also reject an invalid sequence such as $(1,0,1,0)$ if the function is specified only for monotone dropout: that sequence contains a return to observation.

In repeated simulations, compare the observed pattern frequencies with their probabilities under the programmed dropout mechanism. Because these frequencies vary randomly, they should not be required to equal their expectations exactly. A large systematic mismatch can indicate reversed indicator coding, an incorrect dropout probability, or a problem with the sequential update.

Test Imputation Rules Against Their Arithmetic

A focused imputation test uses a small data set for which both the initial imputations and the rule applied to them are explicit. For a delta adjustment, a common form is to add a chosen offset $\delta$ to specified missing outcomes after drawing them under a reference imputation model. The test should confirm which outcomes are adjusted, the sign and units of $\delta$, and that observed outcomes remain unchanged.

Worked Example: Check a Delta Adjustment

Consider four hypothetical participants' final-visit outcomes in one treatment group. Two are observed, with values 8 and 10 points. Two are missing; suppose the reference imputation code returns 7 and 9 points for them. Apply a delta adjustment of $\delta=-2$ points to the missing outcomes only.

Before adjustment, the completed values are $8,10,7,9$, so their mean is

\bar{Y}^{\mathrm{MAR}}=\frac{8+10+7+9}{4}=\frac{34}{4}=8.5\text{ points}.

After adjustment, the missing values are $7+(-2)=5$ and $9+(-2)=7$. The observed values remain 8 and 10, giving

\bar{Y}^{\delta}=\frac{8+10+5+7}{4}=\frac{30}{4}=7.5\text{ points}.

The mean changes by $-1$ point, not by $-2$ points, because only two of four outcomes received the adjustment. In general, if $m$ of $n$ final outcomes receive the same adjustment, the completed-data mean changes by $(m/n)\delta$. Here $(2/4)(-2)=-1$ point. A useful test checks this identity as well as the individual adjusted values.

Compare Repeated Estimates With a Known Target

For an estimator check, repeatedly generate data under a mechanism where the target is known and run the estimator implementation on each replicate. The Monte Carlo average will not generally equal the target exactly. Its discrepancy should be assessed relative to Monte Carlo error rather than judged by exact equality.

If $B$ independent simulation replicates give estimates $\hat{\theta}_1,\ldots,\hat{\theta}_B$, their average is $\bar{\hat{\theta}}=B^{-1}\sum_{b=1}^B\hat{\theta}_b$. If their sample standard deviation is $s_{\hat{\theta}}$, the estimated Monte Carlo standard error of that average is $s_{\hat{\theta}}/\sqrt{B}$. This quantifies simulation noise in the estimated average; it is not the standard error of an estimate from one study.

Justification: If the replicate estimates are independent with variance $\sigma_{\hat{\theta}}^2$, then the variance of their average is $$ \operatorname{Var}(\bar{\hat{\theta}})=\operatorname{Var}\left(\frac{1}{B}\sum_{b=1}^B\hat{\theta}_b\right) =\frac{1}{B^2}\sum_{b=1}^B\operatorname{Var}(\hat{\theta}_b) =\frac{\sigma_{\hat{\theta}}^2}{B}. $$ Replacing the unknown replicate standard deviation $\sigma_{\hat{\theta}}$ with $s_{\hat{\theta}}$ gives the estimated Monte Carlo standard error.

Worked Example: Validate a Complete-Data Treatment Contrast

Generate $B=10{,}000$ independent hypothetical trials, each with 100 participants per arm. Suppose the final outcome is $Y_{iT}=\mu+\tau A_i+\epsilon_i$, where $A_i=0$ or 1, $\epsilon_i\sim N(0,4)$, and errors are independent across participants. The target difference in population means is $\tau=2$ points. With no missing outcomes, the estimator is the treatment-group sample mean minus the control-group sample mean.

For one replicate, the variance of this difference is $4/100+4/100=0.08$ square points, so its standard deviation is $\sqrt{0.08}=0.2828$ points. Suppose the hypothetical simulation output has mean estimate 2.003 points and standard deviation 0.2828 points across replicates. The Monte Carlo standard error of the average is

SE_{\mathrm{MC}}=\frac{0.2828}{\sqrt{10{,}000}}=\frac{0.2828}{100}=0.002828\text{ points}.

The average is $2.003-2=0.003$ points above the target, or $0.003/0.002828=1.061$ Monte Carlo standard errors. That discrepancy is compatible with ordinary simulation variation and does not, by itself, suggest a coding error. A large discrepancy that persists as $B$ increases would be more concerning. Also inspect the replicate standard deviation: a coding error can affect variability even when the average happens to be close to 2.

Use Limiting Cases and Diagnose Failures

Expected limiting cases are especially powerful because they often yield exact or nearly exact checks. Run the analysis code with settings under which a more complicated procedure should reduce to a simpler one. For instance, when the simulated dropout probability is zero, every outcome should be observed and the analysis should agree with its complete-data version. When a delta adjustment is zero, the adjusted imputations should equal the reference imputations. These are software checks, not claims about what missing-data mechanism is plausible in practice.

Freeze a small case.
Use a fixed, explicitly stated data set to test pattern labels and hand-calculated imputation outputs.

Check invariants.
Verify that pattern counts sum to $n$, observed outcomes are not altered by a missing-value adjustment, and invalid indicator sequences are flagged when the assumed pattern structure excludes them.

Check a known target.
Repeat simulations under a mechanism with a known estimand; compare average estimates with the target using the Monte Carlo standard error.

Check a limit and inspect variation.
Set dropout or adjustment to zero where appropriate, then examine both average estimates and their spread across replicates.

Warning signs include a treatment contrast with the wrong sign, a pattern count larger or smaller than $n$, an observed value that changes during imputation, and a Monte Carlo discrepancy that grows rather than shrinks as more replicates are run. For independent replicates, Monte Carlo error typically decreases at rate $1/\sqrt{B}$; doubling $B$ does not halve it. Investigate failures by tracing a small replicate through the generated full outcomes, indicators, encoded pattern, imputed outcomes, and final estimate.

Simulation checks complement, rather than replace, other validation approaches. Hand-calculated unit tests are precise for small cases but cannot establish behavior across a broad range of inputs. Analytic calculations provide strong benchmarks where available, but may not cover a complex implementation. Simulations exercise the full workflow under chosen mechanisms, but can miss untested cases and can themselves contain errors if their data-generating code is wrong. Independent implementations of simple checks and comparison with analytic expectations reduce that risk.

Common Mistakes and Limitations

Treating a close Monte Carlo average as proof of correctness. Different errors can offset in the average. Check individual outputs, variability, pattern frequencies, and limiting cases too.
Using exact equality for random results. Random estimates fluctuate. Compare their discrepancy with Monte Carlo error, while reserving exact assertions for deterministic outputs and invariants.
Testing only the mechanism used to write the code. A simulation and an analysis function can share the same mistaken convention. Use hand-calculated cases and independently check key probabilities or targets.
Confusing implementation validation with model validation. Code that correctly applies a specified sensitivity assumption does not show that the assumption is correct for observed participants.

A simulation can only probe mechanisms, parameter values, sample sizes, and edge cases that are actually tested. It does not remove uncertainty about unobserved outcomes, nor does a successful run guarantee correct behavior for every configuration. Treat the checks as evidence about the tested implementation, and retain the mechanism, target, and expected result alongside each test.

Check Your Understanding

Answer each question, then compare with the solutions below.

For $T=3$, what is $D_i$ for the monotone sequence $(1,1,0)$? Why is $(1,0,1)$ invalid under a monotone-dropout specification?
There are 20 final outcomes, of which 5 are adjusted by $\delta=-4$ points. What is the change in the completed-data mean?
A simulation has $B=400$ independent replicates and $s_{\hat{\theta}}=0.5$. What is the estimated Monte Carlo standard error of the average estimate?
Name one check that tests a deterministic coding rule and one check that assesses a random estimator.

Solutions. (1) $D_i=2$, the last observed visit. The sequence $(1,0,1)$ returns to observation after dropout, contrary to the monotone rule. (2) The change is $(5/20)(-4)=-1$ point. (3) $0.5/\sqrt{400}=0.5/20=0.025$ in the estimator's units. (4) A hand-calculated pattern label or zero-delta equality is a deterministic check; comparing a repeated-simulation average with a known target in Monte Carlo standard errors assesses a random estimator.