Why Simulate When the Assignment Mechanism Is Known?
In observed data, the true treatment-assignment probabilities are unavailable. A diagnostic may show that fitted weights are extreme or that weighted covariates remain imbalanced, but it cannot by itself establish what the diagnostic would show if the treatment model were correct. A simulation supplies a known assignment mechanism: it lets us deliberately create correct and incorrect models, then check whether the diagnostics distinguish them.
The aim is not to make a simulation reproduce every feature of a clinical study. It is to isolate a specific challenge, such as an omitted interaction or near-violated positivity, and see whether a proposed diagnostic responds. A useful design specifies the data-generating mechanism, the candidate models, the diagnostic, and a prespecified measure of diagnostic performance.
Population Balance as an Oracle Benchmark
For binary treatment, define the unstabilized inverse-probability weight based on a candidate probability \(q\) by \(W=A/q(X)+(1-A)/(1-q(X))\). If \(q\) equals the true probability \(p\), weighting balances covariate means in the population. This gives a benchmark against which simulation results can be checked.
Proof. Conditional expectation gives \(E[A\mid X]=p(X)\). Therefore,
Taking expectations yields numerator \(E[X]\) and denominator \(1\) for the treated group. For the control group, \(E[1-A\mid X]=1-p(X)\), so the corresponding conditional expectations are \(X\) and \(1\). The integrability condition ensures the numerators are finite; the denominators are both one. This proves the identity. \(\square\)
For independent, identically distributed observations, the sample weighted means based on the true \(p\) converge in probability to these population means by the law of large numbers and the continuous mapping theorem, provided the relevant weighted terms are integrable. Thus, small samples need not balance exactly even when the model is correct.
Deliberate Misspecification and What It Changes
A candidate probability \(q(X)\) that differs from \(p(X)\) generally produces different weighted covariate distributions. Under the conditions \(0<q(X)<1\) almost surely and finite expectations for the ratios below, the treated weighted mean is
This follows by conditioning on \(X\), as \(E[A X/q(X)\mid X]=Xp(X)/q(X)\). The control counterpart is \(E[X\{1-p(X)\}/\{1-q(X)\}]\) divided by \(E[\{1-p(X)\}/\{1-q(X)\}]\). Unless the two resulting means agree, population balance fails under \(q\).
This is a useful second benchmark: if a chosen misspecification is known to produce imbalance in the population, a simulation can test whether the proposed sample diagnostic detects it. But not every misspecification must create a detectable marginal mean difference. A misspecified probability may still balance a particular covariate by coincidence, while failing for another feature of the covariate distribution.
Worked Example: An Omitted Interaction in the Assignment Model
Let \(X_1\) and \(X_2\) be independent Bernoulli variables, each with probability \(1/2\). Generate treatment using \(p(X_1,X_2)=\operatorname{expit}(-2+1.2X_1+0.8X_2+1.5X_1X_2)\), where \(\operatorname{expit}(z)=1/(1+e^{-z})\). A candidate logistic model omits the interaction and uses \(q(X_1,X_2)=\operatorname{expit}(-2+1.2X_1+0.8X_2)\).
The four covariate combinations have probabilities \(1/4\). The assignment probabilities are approximately \(0.1192, 0.3100, 0.2315,\) and \(0.8176\), for \((0,0),(1,0),(0,1),(1,1)\), respectively. The candidate probabilities are \(0.1192, 0.3100, 0.2315,\) and \(0.5000\). Only the \((1,1)\) cell differs in this setup: the true model assigns it treatment with probability about \(0.8176\), while the candidate assigns probability \(0.5\).
A controlled check can generate, for example, \(n=1{,}000\) observations per replicate from this covariate and treatment mechanism. In each replicate, fit both the correctly specified interaction model and the omitted-interaction model, compute the weights, then record maximum weights, weight quantiles, and weighted standardized differences for \(X_1\), \(X_2\), and their product. Repeat the entire procedure across many independent replicates. The simulation does not predetermine that every diagnostic must flag the omitted interaction; rather, it measures how often, and at what sample sizes, each specified diagnostic does so.
A useful extension is to fit the candidate model anew in every replicate. This evaluates the practical fitting procedure, including estimation variability. Holding fitted coefficients fixed instead isolates the effect of the functional-form error. These answer different questions and should not be conflated.
Worked Calculations: Balance and Positivity
Worked Example: A Misspecified Probability Can Leave Large Imbalance
Let \(X\sim\operatorname{Bernoulli}(1/2)\), with true probabilities \(p(1)=0.8\) and \(p(0)=0.2\). Suppose the candidate model uses \(q(X)=0.5\) for both values. The true model has valid overlap, since both treatment probabilities lie strictly between zero and one.
Among treated individuals, \(P(X=1\mid A=1)=P(X=1,A=1)/P(A=1)\). The numerator is \(0.5(0.8)=0.4\), and \(P(A=1)=0.5(0.8)+0.5(0.2)=0.5\), so the treated prevalence of \(X=1\) is \(0.4/0.5=0.8\). Because \(q=0.5\), all treated weights are \(2\), so their normalized weighted mean remains \(0.8\).
For controls, \(P(X=1,A=0)=0.5(1-0.8)=0.1\), while \(P(A=0)=0.5\). The control prevalence is \(0.1/0.5=0.2\), and all control weights are also \(2\), so their normalized weighted mean is \(0.2\). The weighted difference is \(0.8-0.2=0.6\). The marginal standard deviation of \(X\) is \(\sqrt{(1/2)(1/2)}=0.5\); thus the absolute standardized difference is \(0.6/0.5=1.2\). In this example, the candidate model fails conspicuously on a marginal balance diagnostic.
Worked Example: Checking the Arithmetic in One Simulated Replicate
Suppose a simulated replicate has weighted treated total \(13\) and weighted treated sum of \(X_1\) equal to \(10\). Its weighted control total is \(27\), with weighted control sum of \(X_1\) also equal to \(10\). These are sums after applying the candidate inverse-probability weights.
The treated weighted mean is \(10/13=0.7692308\), and the control weighted mean is \(10/27=0.3703704\). Their difference is \((10/13)-(10/27)=(270-130)/351=140/351=0.3988604\), which rounds to \(0.3989\). If the marginal standard deviation of \(X_1\) is \(0.5\), the absolute standardized difference is \((140/351)/0.5=280/351=0.7977208\), which rounds to \(0.7977\). Reporting \(0.3988\) or \(0.7976\) would be inconsistent with the exact fractions.
This replicate is one draw, not a population result. A simulation study should examine the distribution of such diagnostics across replicates, rather than draw conclusions from a single sample.
Worked Example: A Positivity Challenge at a Rare Treatment Probability
Suppose half the population belongs to a covariate stratum with \(p(X)=0.01\), and the remaining half has \(p(X)=0.5\). Under the true model, a treated individual in the rare-treatment stratum receives weight \(1/0.01=100\). In the other stratum, a treated individual receives weight \(1/0.5=2\). The large weight is a direct consequence of the mechanism, not necessarily a fitting error.
With \(n=200\) independent observations, the probability of observing no treated individual from the rare-treatment stratum is \((1-0.5\times0.01)^{200}=0.995^{200}\), approximately \(0.367\). Thus, in about 37% of such samples that stratum would contribute no treated observation, even though the population treatment probability is positive. This shows why a diagnostic should record both fitted probabilities and realized support; finite samples can make a practically serious overlap problem visible even when the theoretical probability is not zero.
Designing and Reading a Simulation Check
A disciplined simulation varies one feature at a time when the aim is to identify which diagnostic detects which failure. A compact design can include a correctly specified model, an omitted interaction, a misspecified nonlinear term, and a low-probability stratum. The covariate distribution and treatment mechanism should be stated explicitly, and sample sizes should reflect the range in which the diagnostic is intended to be used.
Write down the distribution of \(X\) and the known \(p(X)\). Check that each simulated probability is strictly between zero and one unless a structural zero or deterministic treatment is intentionally being studied.
Define candidate models \(q(X)\) that are correct or deliberately omit a feature. State whether each model is fixed or re-estimated in each replicate.
For example, record maximum and upper-quantile weights, weighted standardized differences, and the number of fitted probabilities near zero or one. Define thresholds before inspecting results.
Across independent replicates, report the frequency of threshold exceedance and the distribution of each diagnostic, stratified by model, sample size, and mechanism.
If a diagnostic is intended to flag a particular misspecification, its simulation performance can be summarized by the proportion of misspecified replicates in which it flags. For a correctly specified model, the proportion flagged estimates its false-alarm frequency under that simulation setup. Neither quantity is universal: both depend on the chosen mechanism, sample size, model-fitting procedure, and threshold.
For a Monte Carlo estimate \(\hat r=K^{-1}\sum_{k=1}^{K}I_k\), where \(I_k\) indicates whether replicate \(k\) triggered a prespecified flag, the Monte Carlo standard error is \(\sqrt{\hat r(1-\hat r)/K}\). For instance, with \(K=1{,}000\) replicates and \(\hat r=0.20\), this is \(\sqrt{0.2(0.8)/1000}\approx0.0126\). This quantifies simulation error in the estimated flag frequency; it does not quantify uncertainty about whether the simulated mechanism represents a real application.
Diagnostics, Comparisons, and Limitations
Compare sample diagnostics with an oracle benchmark whenever possible. The known \(p(X)\) supplies oracle weights, while the candidate \(q(X)\) supplies the weights under evaluation. If the oracle weights also look unstable, the mechanism itself may have weak overlap. If oracle weighting behaves well but candidate weighting repeatedly produces residual imbalance, the candidate specification is implicated in that designed setting.
A small exact calculation is often the clearest first check: it can expose algebraic mistakes and show whether the intended misspecification actually changes a target balance quantity. Simulation then adds finite-sample variation, model fitting, and more complicated covariate structures. By contrast, diagnostics applied to observed data assess the observed application but cannot reveal the unobserved true assignment mechanism. These approaches answer complementary questions; none substitutes for the others.
- False reassurance: A small marginal standardized difference does not prove that the treatment model is correct. A diagnostic may miss a misspecification or a feature not included in its summaries.
- False alarms: A correct model can produce imperfect sample balance by chance, particularly with small samples or rare strata. Simulations estimate how often this occurs under the specified setup.
- Positivity confusion: A large weight can arise from a genuinely small treatment probability, a poorly estimated probability, or both. Known-mechanism simulations can separate these causes by comparing true and fitted probabilities.
- Overgeneralization: A model that performs well for one covariate distribution and mechanism need not perform well for another. Sensitivity scenarios should be chosen to represent plausible departures relevant to the application.
The principal limitation is that simulation conclusions are conditional on the data-generating choices. A carefully designed check can show that a diagnostic detects a specified failure under specified conditions; it cannot guarantee detection of every form of misspecification. In longitudinal settings, treatment probabilities depend on histories and weights involve treatment probabilities over time. The same design logic applies, but a one-time binary-treatment simulation does not establish performance for every longitudinal mechanism.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If a correct propensity model produces a nonzero sample standardized difference, does that contradict the population balance identity? Explain.
- For \(X\sim\operatorname{Bernoulli}(1/2)\), \(p(1)=0.8\), and \(p(0)=0.2\), calculate the population treated and control weighted means when \(q(X)=0.5\), and give the standardized difference.
- In \(K=400\) simulation replicates, a diagnostic flags 100 replicates. Calculate the estimated flag frequency and its Monte Carlo standard error.
- Why should a simulation compare oracle weights with weights based on the fitted candidate model?
Solutions. (1) No. The identity concerns population weighted means; random samples fluctuate, and sample weighted means need not match exactly. (2) The treated weighted mean is \(0.8\), and the control weighted mean is \(0.2\), since the candidate weights are constant within each group. Their difference is \(0.6\); dividing by the marginal standard deviation \(0.5\) gives an absolute standardized difference of \(1.2\). (3) The estimated frequency is \(100/400=0.25\). Its Monte Carlo standard error is \(\sqrt{0.25(0.75)/400}=\sqrt{0.00046875}\approx0.0217\). (4) Oracle weights isolate behavior under the known mechanism; comparing them with candidate weights helps distinguish instability inherent in the mechanism from problems introduced by estimation or misspecification.