Why Compare Rules in a Simulation?
No clipping, a fixed upper bound, and a quantile-based upper bound can produce different amounts of bias and variability. Which rule performs best depends on the data-generating process, the target, the sample size, and the particular threshold. A simulation makes these dependencies visible under controlled conditions; it does not establish that one rule is best for every analysis.
The comparison should hold the target, data-generating process, estimator, and replicate datasets constant across rules. Otherwise, a difference in performance could be due to something other than clipping. The earlier material on the mean squared error trade-off for truncation motivates the criteria used here; the focus now is how to design and interpret their simulation comparison.
Specify the Rules Before Running the Simulation
For clarity, consider upper clipping of positive inverse-probability weights. If the untruncated weight for an observation is \(W_i\), a cap \(c\) replaces it by \(W_i^{(c)}=\min(W_i,c)\). No clipping leaves \(W_i\) unchanged. A fixed-bound rule uses a prespecified \(c\); a quantile rule uses a prespecified quantile level \(q\) and caps at the empirical \(q\)-quantile of the weights in that dataset. State whether the quantile is computed over all eligible weights or only positive weights, and state the quantile convention used by the software. Those choices are part of the method.
A concrete design could generate baseline \(L\sim\operatorname{Bernoulli}(0.5)\), treatment \(A\) with \(P(A=1\mid L=0)=0.8\) and \(P(A=1\mid L=1)=0.1\), and potential outcomes \(Y^1=1+2L+\epsilon\), where \(\epsilon\sim N(0,1)\) is independent of \(L,A\). The target treated-regime mean is \(E[Y^1]=0.5(1)+0.5(3)=2\). For treated observations, the inverse-probability weight is \(1/P(A=1\mid L)\). Compare the untruncated weight, a fixed cap such as \(c=4\), and a stated empirical quantile cap. The low treatment probability in one stratum creates large weights; because the outcome mean also varies with \(L\), clipping can change the weighted representation of the target population.
Specify the target, treatment and covariate distributions, outcome model, and any censoring or missingness mechanisms relevant to the estimator.
Record the fixed cap or quantile level, the weights to which it applies, and whether truncation occurs before or after any stabilization.
For each simulated dataset, calculate all three estimates from the same underlying observations. This pairing makes rule comparisons less affected by random differences between datasets.
Report bias, variance, and MSE by sample size, together with the number of replicates, Monte Carlo uncertainty, and indicators of extreme weights or failed fits.
Metrics and Their Relationship
The population identity behind the comparison is
To verify the identity, write \(\hat{\psi}-\psi=(\hat{\psi}-E[\hat{\psi}])+(E[\hat{\psi}]-\psi)\), square both sides, and take expectations. The cross term is zero because \(E[\hat{\psi}-E[\hat{\psi}]]=0\). The first squared term has expectation \(\operatorname{Var}(\hat{\psi})\), and the final term is the squared bias. The finite-second-moment condition ensures these expectations exist. The empirical identity follows by the same algebra, replacing expectation by the average over the \(R\) replicate estimates. \(\square\)
This decomposition is useful for interpretation: a rule can reduce MSE by reducing variance enough to compensate for added bias. It does not imply that the empirical MSE in a finite simulation will rank rules identically in a new study; simulation results themselves have Monte Carlo error.
Worked Example: Checking Bias, Sample Variance, and MSE
Suppose the target is \(\psi=2\), and four replicate estimates under no clipping are \(2.00, 2.05, 2.15, 2.20\). The mean is \(\bar{\hat{\psi}}=(2.00+2.05+2.15+2.20)/4=2.10\), so the estimated bias is \(2.10-2=0.10\), not zero.
The deviations from the mean are \(-0.10,-0.05,0.05,0.10\). Their squares sum to \(0.0100+0.0025+0.0025+0.0100=0.0250\). The sample variance, with the usual divisor \(R-1=3\), is \(0.0250/3=0.008333\).
The squared errors from the target are \(0,0.0025,0.0225,0.0400\), which sum to \(0.0650\). Therefore the empirical MSE is \(0.0650/4=0.01625\). Using the variance convention in the definition box, the replicate variance is \(0.0250/4=0.00625\), and \(0.00625+0.10^2=0.01625\), as required. This example also shows why the sample variance with divisor \(R-1\) should not be substituted directly into that empirical identity.
Worked Example: Comparing Three Rules on Matched Replicates
For a small illustration with target \(2\), suppose four matched datasets produce the following estimates:
| Replicate | No clipping | Fixed cap | Quantile cap |
|---|---|---|---|
| 1 | 1.80 | 1.90 | 1.70 |
| 2 | 2.00 | 2.00 | 2.00 |
| 3 | 2.20 | 2.10 | 2.10 |
| 4 | 2.40 | 2.20 | 2.20 |
For no clipping, the mean is \(2.10\), giving bias \(0.10\). Squared deviations from that mean sum to \(0.01+0.01+0.01+0.09=0.12\), so the variance with divisor \(R=4\) is \(0.0300\). Squared errors from \(2\) sum to \(0.04+0+0.04+0.16=0.24\); the MSE is \(0.0600=0.0300+0.10^2\).
For the fixed cap, the mean is \(2.05\), so bias is \(0.05\). Squared deviations from \(2.05\) sum to \(0.0225+0.0025+0.0025+0.0225=0.0500\), giving variance \(0.0125\). Squared errors from the target also sum to \(0.0500\), so MSE is \(0.0125=0.0125+0.05^2\).
For the quantile cap, the mean is \(2.00\), so bias is zero. Squared deviations from the mean sum to \(0.09+0+0.01+0.04=0.14\), giving variance \(0.0350\); the squared errors sum to the same \(0.14\), so MSE is \(0.0350\). In this illustration the fixed cap has the lowest MSE, even though the quantile cap has zero bias. Four replicates are far too few for a reliable ranking; the calculations demonstrate the metrics, not a general performance claim.
Sample Size, Monte Carlo Error, and Interpretation
The next table is an illustrative simulation summary, not a universal numerical prediction. Each entry is a Monte Carlo estimate; variance and MSE use divisor \(R\), and the displayed values satisfy MSE = variance + squared bias.
| Sample size | Rule | Bias | Variance | MSE |
|---|---|---|---|---|
| 100 | No clipping | 0.04 | 0.0900 | 0.0916 |
| 100 | Fixed cap | 0.08 | 0.0450 | 0.0514 |
| 100 | Quantile cap | 0.02 | 0.0600 | 0.0604 |
| 500 | No clipping | 0.02 | 0.0250 | 0.0254 |
| 500 | Fixed cap | 0.07 | 0.0180 | 0.0229 |
| 500 | Quantile cap | 0.01 | 0.0200 | 0.0201 |
For example, at \(n=100\), the fixed-cap MSE is \(0.0450+0.08^2=0.0514\), lower than the quantile-cap MSE \(0.0600+0.02^2=0.0604\). At \(n=500\), the quantile-cap MSE is \(0.0200+0.01^2=0.0201\), lower than the fixed-cap MSE \(0.0180+0.07^2=0.0229\). The apparent ranking changes with sample size in this illustration. A simulation should report each component rather than only declaring a winner: a small MSE can conceal meaningful bias, and a small bias can coexist with high variance.
For independent replicates and finite variance, the Monte Carlo standard error of the average estimate is estimated by \(s/\sqrt{R}\), where \(s\) is the sample standard deviation of the replicate estimates. For MSE, define \(Z_r=(\hat{\psi}_r-\psi)^2\). If \(E[Z_r^2]\) is finite, the Monte Carlo standard error of the average squared error is estimated by \(s_Z/\sqrt{R}\), where \(s_Z\) is the sample standard deviation of the \(Z_r\)'s. This follows because independent replicates give \(\operatorname{Var}(R^{-1}\sum_r Z_r)=\operatorname{Var}(Z_r)/R\); taking the square root gives the standard error. The finite fourth moment of \(\hat{\psi}_r-\psi\) is sufficient for \(E[Z_r^2]\) to be finite. A large number of simulated datasets reduces Monte Carlo error; increasing the study sample size \(n\) does not, by itself, do so.
Worked Example: Separating Sampling Variation From Monte Carlo Error
Suppose \(R=1{,}000\) replicate estimates have sample standard deviation \(0.30\), and the replicate squared errors have sample standard deviation \(0.12\). The estimated Monte Carlo standard error of the mean estimate is \(0.30/\sqrt{1{,}000}\approx0.0095\). The Monte Carlo standard error of the estimated MSE is \(0.12/\sqrt{1{,}000}\approx0.0038\).
The first number describes uncertainty in the simulated average estimate; the second describes uncertainty in the simulated average squared error. Neither is the estimator's sampling standard deviation: that quantity is approximately \(0.30\) here. Confusing the standard deviation across datasets with the Monte Carlo standard error of their average understates uncertainty by a factor of about \(\sqrt{1{,}000}\).
Diagnostics, Alternatives, and Limitations
A useful comparison records more than the three summary metrics. Check the weight distribution and the fraction of observations affected by each cap; report the realized quantile threshold across replicates; record convergence failures and undefined estimates; and examine whether errors are driven by a small number of datasets. If a quantile threshold barely changes as \(n\) changes, verify that the quantile is calculated over the intended set of weights. If clipping does not change either estimates or weight summaries, the cap may not be binding. These checks can reveal an implementation error or a rule that is irrelevant in the simulated setting.
The main alternatives answer different questions. No clipping preserves the original weighting rule but may retain high variability. A fixed cap is transparent and directly comparable across sample sizes, though it can bind at different rates as the weight distribution changes. Quantile clipping adapts its numerical threshold to each dataset, but the resulting cap is random and its meaning depends on the quantile convention and weight distribution. A threshold selected to optimize estimated MSE is another approach discussed in the earlier material on choosing a threshold by estimated MSE; when included as a comparator, its selection process must be repeated within each simulated dataset, rather than treated as if the selected threshold were known in advance.
A simulation conclusion is conditional on its data-generating process, sample sizes, threshold choices, estimator, and number of replicates. It cannot demonstrate robustness to unmodeled mechanisms, nor does a lower simulated MSE guarantee better performance in a particular observed dataset. Report the rules and settings precisely, show uncertainty in the Monte Carlo summaries when relevant, and avoid choosing a rule solely because it wins in one scenario.
Common Pitfalls
- Using different datasets for different rules. Random dataset differences then contaminate the comparison. Apply every rule to each same simulated dataset.
- Calling an empirical quantile cap “the same threshold” across sample sizes. Its realized numerical value can vary by replicate and by sample size; report that behavior.
- Reporting only variance or only bias. Neither alone determines MSE. Show all three, and check that their arithmetic is consistent.
- Confusing sample variance with the simulation variance convention. The usual sample variance divides by \(R-1\); the empirical MSE decomposition stated here uses variance with divisor \(R\).
- Treating simulated success as a universal recommendation. A ranking is evidence about the specified scenario, not all possible studies.
Check Your Understanding
Answer each question, then check the worked solutions below.
- For target \(5\), replicate estimates are \(4,5,7\). Calculate the empirical bias, variance with divisor \(R\), and MSE.
- Why should all truncation rules be applied to the same simulated datasets?
- If a rule has bias \(0.10\) and variance \(0.04\), what is its population MSE?
- What does the Monte Carlo standard error of an estimated MSE quantify, and what moment condition is sufficient for it to be finite?
Solutions. (1) The mean is \(16/3\), so bias is \(1/3\). Squared deviations from the mean sum to \(14/3\), giving variance \((14/3)/3=14/9\). Squared errors from \(5\) are \(1,0,4\), so MSE is \(5/3\); indeed \(14/9+(1/3)^2=15/9=5/3\). (2) Matching controls random differences between datasets, making differences in estimates attributable to the rules more directly. (3) MSE is \(0.04+0.10^2=0.05\). (4) It quantifies simulation-to-simulation uncertainty in the average squared error across the finite set of replicates. A finite fourth moment of the estimation error is sufficient, because the squared error must have finite variance.