Choosing Simulation Draw Counts and Designs

Why Draw Counts and Designs Matter

A simulation-based estimator replaces an intractable quantity, such as a likelihood contribution, with a finite-draw approximation. That approximation introduces simulation error even if the data and statistical model are held fixed. Increasing the number of draws usually reduces this error, but costs computation. The design of the draws matters too: two approximations can each be noisy while their difference is much more precise if their draws are deliberately coupled.

This tutorial focuses on decisions made before and during estimation: whether to hold the number of draws fixed or increase it with the sample size, whether to use independent draws or common random numbers at different parameter values, and how to use independent replications to assess the remaining simulation error. These choices affect the accuracy and stability of estimates; they do not replace the model assumptions needed for statistical identification.

For simulated likelihood, for example, a likelihood contribution may be \(p_i(\theta)=E[h_i(\theta,U)]\), approximated by \(\hat p_{i,R}(\theta)\). The simulated log likelihood then uses \(\log\hat p_{i,R}(\theta)\). Although \(\hat p_{i,R}\) can be unbiased for \(p_i\), its logarithm generally is not unbiased for \(\log p_i\). This distinction is central when choosing a draw-count schedule.

Fixed and Increasing Draw Counts

A fixed draw count uses the same number \(R\) of draws as the sample size \(N\) grows. An increasing draw count uses \(R=R_N\), with \(R_N\) tending to infinity as \(N\) grows. Fixed counts can be adequate for exploratory computation or for an approximation whose accuracy is already known to be sufficient. But with a fixed number of draws, simulation approximation error does not automatically disappear in large samples.

For a simulated likelihood formed by taking logarithms of simulated likelihood contributions, the finite-\(R\) log transformation can create an approximation bias. Under regularity conditions such as smooth, positive likelihood contributions and suitable finite moments, this bias is often of order \(R^{-1}\). Thus, a fixed \(R\) can make the estimator converge to the maximizer of a simulation-distorted criterion rather than the maximizer of the exact likelihood. Increasing \(R_N\) is the standard way to make this approximation error vanish.

The draw count needed for consistency may be smaller than the count needed to make simulation error negligible on the usual \(N^{-1/2}\) sampling scale. Under standard smoothness conditions for simulated maximum likelihood, independent simulation across observations, and a second-order expansion of the log contributions, the per-observation simulation bias is typically \(O(R_N^{-1})\), while the random simulation component of the average criterion is typically \(O_p((N R_N)^{-1/2})\). For the estimator to have the same first-order, root-\(N\) behavior as exact maximum likelihood, the bias must be small relative to \(N^{-1/2}\). A sufficient rate condition is $$ \frac{\sqrt{N}}{R_N}\longrightarrow 0, \qquad\text{equivalently}\qquad \frac{R_N}{\sqrt{N}}\longrightarrow\infty. $$ These are representative conditions for this setting, not universal rules: the required rate depends on the estimator, the simulation design, and the order of the approximation bias. Consistency and first-order equivalence are distinct goals.

Independent Draws and Common Random Numbers

When estimating a criterion at several parameter values, one can draw new simulation inputs independently at each value or reuse the same inputs across values. Reusing the same draws is called common random numbers. For example, when computing \(g(\theta)\) and \(g(\theta')\), common random numbers use the same \(U_r\) in both \(h(\theta,U_r)\) and \(h(\theta',U_r)\). The draws should still be independent across replication index \(r\); the coupling is between parameter values, not between the draws within a simulation average.

The potential benefit comes from estimating a difference directly. If \(D_r=h(\theta,U_r)-h(\theta',U_r)\), then the common-draw approximation to \(g(\theta)-g(\theta')\) is \(R^{-1}\sum_r D_r\). Its variance depends on how similarly the two integrands respond to the same \(U_r\). When their responses are positively correlated, shared variation cancels in the difference. Independent draws at each parameter value do not capture that cancellation.

With independent draws at the two parameter values, the covariance term is zero. With common random numbers, a positive covariance reduces the variance of the difference. This is particularly useful when the parameter values are close and the simulated quantities move together as the parameter changes. It is not guaranteed to help: negative covariance can increase the variance of a difference. Common random numbers reduce simulation noise in comparisons only when the coupling produces a favorable covariance.

For a simulated likelihood, common random numbers can mean using the same draws for a given observation at every parameter value. This makes nearby criterion evaluations more comparable and can make the simulated criterion smoother. One should distinguish that use from using the same draw set for every observation: the latter can induce dependence across observations and change how simulation error averages out. The design must be matched to the estimator’s assumptions, rather than treating all reuse of draws as equivalent.

Replication for Measuring Simulation Error

An independent replication repeats the simulation calculation with a fresh, independent draw set. Replications provide a practical estimate of Monte Carlo variability: variation across replicate estimates reflects simulation noise, conditional on the data and the calculation being repeated. They are especially useful in a pilot run for determining whether a planned draw count gives acceptable numerical precision.

Suppose \(K\) independent replications each use \(R_0\) draws and produce estimates \(\hat g_1,\ldots,\hat g_K\). Their average is $$ \bar g_K=\frac{1}{K}\sum_{k=1}^{K}\hat g_k. $$ If the replicate estimates have common variance \(\tau^2\), their sample standard deviation \(s_{\mathrm{rep}}\) estimates \(\tau\), and the estimated standard error of their average is \(s_{\mathrm{rep}}/\sqrt{K}\). Averaging \(K\) independent blocks of \(R_0\) draws has the same Monte Carlo variance as pooling \(K R_0\) independent draws for a linear average. Keeping blocks separate is useful for diagnosing variability, but a small number of blocks gives a noisy estimate of the simulation standard error.

For likelihood calculations, the target of replication matters. In general, averaging simulated likelihood contributions and then taking a logarithm is not the same operation as averaging replicate log likelihoods. If the estimator is defined by a pooled simulated likelihood, use replications to assess the variability of that defined calculation, not to silently substitute a different objective. Replications diagnose numerical simulation error; they do not by themselves quantify uncertainty from drawing a new data sample.

A Practical Design Workflow

Draw-count and draw-design choices should be made with a clear target: precision of an integral, accuracy of an estimator, or comparability of criterion values. A useful workflow is:

Common Mistakes and Why the Choice Matters

• Assuming a large data set eliminates simulation error. With fixed simulation draws, the approximation bias may persist as \(N\) grows. Increasing \(N\) does not automatically increase \(R\).
• Assuming common random numbers always improve precision. Their benefit for a difference depends on covariance. If the covariance is negative, the difference can become noisier.
• Confusing simulation error with sampling error. Replicating draws conditional on one data set measures Monte Carlo variability, not the uncertainty that would arise from collecting a different sample.
• Treating a sufficient rate as a universal prescription. Conditions such as \(R_N/\sqrt{N}\to\infty\) rely on particular bias orders and regularity assumptions. Other estimators or simulation designs can require different rates.
• Reporting only the number of draws. A draw count is not a precision measure by itself. The variability of the simulated quantity, the coupling across parameter values, and how draws are allocated all matter.

These decisions matter because computational convenience can affect the estimator’s finite-sample accuracy and, under some designs, its large-sample behavior. A defensible strategy states the target, the draw schedule, and the dependence design, then assesses the remaining simulation error. The guiding principle is to spend computation where it improves the accuracy relevant to the estimation task.