Why Variance Reduction Must Be Compared With Cost
A simulation method can produce more precise estimates per replication and still be a poor choice if each replication takes much longer. Conversely, a modest reduction in variance can be valuable when the method is inexpensive. A useful comparison therefore asks two questions: how much variance is removed, and what computational cost is required to remove it?
This tutorial compares simulation methods using variance reduction factors and cost-adjusted efficiency. The comparisons assume that the methods estimate the same quantity without bias. In practice, the variance and time inputs may come from a pilot study; the resulting comparison is only as reliable as those inputs.
Variance Reduction and Cost-Adjusted Efficiency
Choose a crude Monte Carlo estimator as the baseline, indexed by \(0\). Its variance per replication is \(\sigma_0^2\), and its average cost per replication is \(c_0\). The variance reduction factor (VRF) of method \(j\), at equal numbers of replications, is
A VRF greater than 1 means the new method has lower variance per replication. For example, a VRF of 4 means one replication has one-quarter of the baseline variance. It corresponds to a factor of 2 reduction in standard error at equal replication counts, because standard error depends on the square root of variance.
Equal replication counts do not necessarily mean equal computational work. If a total budget of \(B\) time units allows approximately \(n_j=B/c_j\) replications, then
Thus the variance per unit of computation is proportional to \(\sigma_j^2c_j\). Comparing that product with the baseline gives the cost-adjusted efficiency gain:
An efficiency gain above 1 means that, for a sufficiently large fixed time budget, the method has lower estimator variance than the baseline. The approximate ratio of baseline variance to method variance at that budget is the efficiency gain. The ratio of standard errors is its square root.
Why the Cost Adjustment Works
The adjustment follows directly from the variance of an average. With budget \(B\), a method costing \(c_j\) per replication can run about \(B/c_j\) independent replications. Substitute this count into the estimator variance:
For \(n_j\) independent replications, \(\operatorname{Var}(\hat{\theta}_j)=\sigma_j^2/n_j\).
Ignoring setup overhead, a budget \(B\) buys \(n_j\approx B/c_j\) replications.
Then \(\operatorname{Var}(\hat{\theta}_j)\approx \sigma_j^2c_j/B\), so dividing the baseline variance by this variance gives \(\sigma_0^2c_0/(\sigma_j^2c_j)\).
The approximation is useful for planning, but it does not make a biased estimator acceptable: all compared methods must target the same \(\theta\). It also presumes independent replications and representative estimates of both variance and cost. When cost per replication is variable, \(c_j\) should represent its average under the intended implementation.
Worked Example: A Control Variate With Higher Run Time
A hypothetical one-year, non-life policy has no mortality contingency and is not discounted; use an effective annual interest rate \(i=0\%\). Its ground-up loss \(X\) follows a Gamma distribution with shape 2 and scale 500 dollars, so \(E[X]=1{,}000\) dollars and \(\operatorname{Var}(X)=500{,}000\) dollars squared. The target is \(E[X]\). A crude estimator has per-draw variance \(500{,}000\) and takes 1.0 millisecond per draw. A control-variate estimator, using a control with known expectation, is unbiased and has pilot-estimated per-replication variance \(125{,}000\); it takes 1.4 milliseconds per replication. Compare their efficiency.
Method. Calculate the equal-replication VRF first, then adjust for the relative run time. The supplied variances are per replication, so no additional division by sample size is needed in this comparison.
The variance reduction factor is
The control-variate replication costs \(1.4/1.0=1.4\) times as much as a crude draw. Therefore,
The method has about 2.86 times the precision efficiency per unit of computation, in the sense that its estimator variance is about \(1/2.8571\) of the crude estimator’s variance at the same sufficiently large time budget. Its standard error is about \(\sqrt{2.8571}=1.690\) times smaller. For a 14-millisecond budget, the crude method runs 14 replications, giving variance \(500{,}000/14=35{,}714.29\). The control-variate method runs 10 replications, giving variance \(125{,}000/10=12{,}500\). The ratio is \(35{,}714.29/12{,}500=2.8571\), which checks the efficiency calculation. Variances are in dollars squared; results are rounded here to two decimal places where applicable.
Common mistake. Multiplying the VRF by the 1.4 cost ratio would give 5.6, incorrectly treating extra computation as a benefit. The correction is to multiply by the inverse cost ratio, \(1.0/1.4\).
Importance Sampling: A Large Gain for a Rare Event
Importance sampling changes the simulation distribution to produce more observations in a region that matters to the target. Each simulated output is reweighted by the likelihood ratio, so that its expectation remains the target under the original model. Greater variance reduction is not guaranteed: the weights can themselves create high variability. The comparison must use the variance of the weighted estimator, not just the frequency of event observations.
Worked Example: Cost-Adjusted Importance Sampling
A hypothetical one-year contract pays a fixed benefit if a severe loss occurs. There is no mortality contingency and no discounting; the effective annual interest rate is \(i=0\%\). Under the original model, loss \(X\) is 10,000 dollars with probability 0.01 and zero otherwise. The target is the severe-loss probability. A crude replication is \(I=\mathbf{1}_{\{X=10{,}000\}}\). Under an importance-sampling proposal, the severe loss is simulated with probability 0.20; the likelihood ratio on that event is \(0.01/0.20=0.05\). The method takes 1.5 milliseconds per draw, compared with 1.0 millisecond for crude simulation. Compare the methods.
Crude method. The indicator is Bernoulli with success probability 0.01, so its expectation is 0.01 and its variance per draw is
Importance-sampling method. The weighted contribution is 0.05 when the proposal generates the severe loss and zero otherwise. Its expected value is \(0.20(0.05)=0.01\), confirming unbiasedness. Its second moment is \(0.20(0.05^2)=0.0005\), so its variance is
The VRF is \(0.0099/0.0004=24.75\). Adjusting for the 1.5-times higher cost gives
For a 150-millisecond budget, crude simulation runs 150 draws and has estimator variance \(0.0099/150=0.000066\). Importance sampling runs 100 draws and has estimator variance \(0.0004/100=0.000004\). Their ratio is 16.5, as predicted. The standard errors are approximately 0.0081 and 0.0020, respectively, when probabilities are reported as proportions. Values are rounded to four decimal places for the standard errors.
Common mistake. Counting the proposal’s 20% severe-loss frequency as though it were the estimated probability would produce a biased answer. The severe-loss contribution must be weighted by 0.05; the proposal distribution is a simulation device, not the original model.
Setup Costs and Small Budgets
The simple efficiency formula treats cost as proportional to the number of replications. Some methods also require a fixed setup cost, such as implementing a specialized proposal distribution or preparing a stratification scheme. If setup costs \(s_j\) time units, a budget \(B\) supports approximately \((B-s_j)/c_j\) replications, provided \(B>s_j\). The resulting variance is approximately
The method with the best large-budget efficiency need not be best for a short run. With setup overhead, compare the actual variances at the intended budget, respecting whole-number replication counts. A fixed cost becomes less important as the budget grows.
Worked Example: When Antithetic Variates Pay Off
For a hypothetical one-year benefit calculation, take the effective annual interest rate to be \(i=0\%\), with no mortality contingency. Let \(U\sim\operatorname{Uniform}(0,1)\), and let the simulated payment be \(Y=U^2\) thousand dollars. The target is \(E[Y]\). A crude replication averages two independent simulated payments. An antithetic replication averages \(U^2\) and \((1-U)^2\). Both methods use two uniform draws and cost 2 milliseconds per replication. The antithetic implementation also has a 20-millisecond setup cost. Compare the estimator variances for budgets of 22 and 30 milliseconds.
Crude replication variance. Since \(E[U^2]=1/3\) and \(E[U^4]=1/5\),
A crude replication averages two independent values, so its variance is \((4/45)/2=2/45\).
Antithetic replication variance. Its replication estimate is
For uniform \(U\), \(E[(U-1/2)^2]=1/12\) and \(E[(U-1/2)^4]=1/80\). The added constant does not affect variance, so
The antithetic method therefore has a per-replication VRF of \((2/45)/(1/180)=8\). But its setup cost must also be considered.
At 22 milliseconds, crude simulation runs \(22/2=11\) replications, giving variance \((2/45)/11=2/495\approx0.004040\). Antithetic simulation has \(22-20=2\) milliseconds left, allowing one replication; its variance is \(1/180\approx0.005556\). The crude method is better at this budget.
At 30 milliseconds, crude simulation runs 15 replications, giving variance \((2/45)/15=2/675\approx0.002963\). Antithetic simulation runs \((30-20)/2=5\) replications, giving variance \((1/180)/5=1/900\approx0.001111\). The variance ratio is \((2/675)/(1/900)=2.6667\), so antithetic simulation is better at this budget. Variances are in squared thousands of dollars and are rounded to six decimal places.
Common mistake. Using the equal-replication VRF of 8 to claim the antithetic method is always eight times more efficient ignores its setup cost. Compare the number of replications each budget actually buys.
Making a Defensible Method Choice
A sound comparison starts by fixing the estimand and the computational unit. A “replication” for one method may be a paired set of draws, while for another it may be a full stratified run. Define each unit so that its output is one unbiased estimate of the same quantity. Then estimate variance across independent units and measure run time under comparable computing conditions.
| Comparison | What it answers | What it leaves out |
|---|---|---|
| Variance reduction factor | How much lower is variance at equal replication counts? | Different replication costs |
| Cost-adjusted efficiency | How much lower is variance at equal large computational budgets? | Fixed setup cost and small-budget integer effects |
| Budget-specific variance | Which method is more precise for the available budget? | It depends on that particular budget and cost estimate |
Pilot estimates can be noisy, especially when outputs are heavy-tailed or rare-event weights vary greatly. Report the variance and timing inputs alongside the comparison, and avoid treating a small estimated advantage as conclusive. Also check that comparisons use the same target, units, and implementation conditions. A lower variance for an estimator of a different quantity is not a valid gain.
The central decision rule is simple: use the VRF to describe variance reduction at equal replication counts, use cost-adjusted efficiency for a large-budget comparison with approximately proportional costs, and calculate budget-specific variances when setup costs or short runs matter.
Check Your Understanding
For each question, assume the estimators are unbiased and their stated replication variances and costs are applicable to the comparison.
- A baseline has variance 0.16 per replication and cost 2 milliseconds. A new method has variance 0.04 and cost 4 milliseconds. Find its VRF and cost-adjusted efficiency. (Answer: VRF \(=4\); efficiency \(=2\).)
- A rare-event indicator has probability 0.02 under the original model. An importance-sampling proposal generates the event with probability 0.10, giving event weight \(0.02/0.10\); non-event contributions are zero. The importance-sampling draw costs 1.2 times a crude draw. Find the two variances and the cost-adjusted efficiency. (Answer: crude variance \(=0.0196\); importance-sampling variance \(=0.0036\); efficiency \(=0.0196/(0.0036\times1.2)\approx4.537\).)
- A baseline replication has variance 0.9 and costs 3 time units. A proposed method has variance 0.6 and costs 2 time units. Calculate the cost-adjusted efficiency and state which method is preferable for a large budget. (Answer: efficiency \(=(0.9\times3)/(0.6\times2)=2.25\); the proposed method is preferable.)
- A baseline has variance 0.05 per replication and cost 1 time unit. An alternative has variance 0.01 and cost 3 time units. Calculate the VRF and efficiency; explain why the VRF alone gives the wrong choice. (Answer: VRF \(=5\); efficiency \(=5/3\approx1.667\); the alternative is still more efficient per unit cost, but not five times as efficient.)
- A baseline replication costs 2 milliseconds and has variance 0.08. An alternative has the same per-replication cost, variance 0.01, and a 10-millisecond setup cost. For a 50-millisecond budget, compare the approximate estimator variances. (Answer: baseline \(n=25\), variance \(=0.08/25=0.0032\); alternative \(n=20\), variance \(=0.01/20=0.0005\); the alternative’s variance is lower by a factor of 6.4.)