Simulation Diagnostics for Estimated Models

Why Diagnose a Simulation-Based Estimate?

A simulation-based estimator replaces a quantity that is difficult to compute exactly with a numerical approximation. That approximation enters an objective, such as a simulated log likelihood or a simulated-moments criterion, which is then optimized over parameters. The reported estimate can therefore be affected by both the statistical information in the data and the numerical behavior of the simulation and optimization. A successful optimizer termination is not, by itself, evidence that the simulated objective was accurate or that the reported parameter is insensitive to simulation.

Diagnostics ask several distinct questions. Does the estimate change materially when the number of draws increases? Do independent simulation runs produce similar answers? Is the objective stable near the solution, and does the computed gradient agree with a numerical check? Are simulated probabilities or weights being driven by rare draws? These checks do not replace the estimator’s statistical theory. They help identify whether a numerical implementation is behaving plausibly enough for that theory to be relevant.

The checks below presume that the model, objective, and simulation design have been specified correctly: draws are generated from the intended distribution, the objective uses them as intended, and the optimizer is applied to that objective. A diagnostic can reveal symptoms of an implementation problem, but cannot prove that the model is correctly specified or that the estimator is identified.

Draw-Count Sensitivity

A direct check is to estimate the model with several draw counts, keeping the data, parameterization, optimizer settings, and simulation design comparable. When possible, use nested draws or common random numbers: the smaller draw set is contained in the larger set, or the same underlying random numbers are reused across parameter values and draw-count comparisons. This can make changes easier to interpret by reducing variation attributable solely to unrelated random draws.

Compare the parameter estimates, the objective values evaluated on a common high-accuracy simulation, and quantities of substantive interest. An objective evaluated with different draws at each draw count is not automatically comparable to the same precision: its difference may reflect simulation noise as well as a change in the optimizer’s solution. Also inspect whether parameters move along a weakly identified or nearly flat direction. Small objective changes can accompany noticeable parameter changes in such a direction.

Independent Replications

A different check holds the data and draw count fixed but repeats estimation with independent simulation draws. The resulting spread describes simulation variability conditional on the observed data and the chosen computational procedure. It is not the estimator’s sampling uncertainty across hypothetical new data sets. The two sources of uncertainty were distinguished in the earlier material on separating sampling and simulation uncertainty.

For \(B\) independent runs, let \(\hat{\theta}^{(b)}_R\) be the estimate from run \(b\). The replication mean and sample standard deviation are

The standard deviation \(s_{\mathrm{rep}}\) summarizes run-to-run variation. If the runs are independent and have a common distribution, \(s_{\mathrm{rep}}/\sqrt{B}\) estimates the Monte Carlo standard error of their mean, not the Monte Carlo standard error of a single-run estimate. Replications can also reveal failures that a single run hides, such as occasional convergence to a different local optimum.

Objective and Gradient Checks

A numerical optimizer can stop because an iteration limit was reached, because its step became small, or because a computed gradient appears small. Each condition concerns the implemented objective and algorithm; none alone establishes that the intended statistical objective has been maximized. Inspect the objective values around the reported solution, confirm that small parameter perturbations behave plausibly, and check whether different reasonable starting values lead to comparable solutions.

When an analytic or automatic-differentiation gradient is available, compare it with finite differences. For a scalar objective \(Q\), the central-difference approximation to its derivative at \(\theta\) is

For a vector parameter, apply this calculation one coordinate at a time, with the other coordinates held fixed. Use the same simulation draws when evaluating the objective at \(\theta+h\) and \(\theta-h\); otherwise, simulation noise can overwhelm the difference being checked. Try more than one step size. A step that is too large has truncation error, while a step that is too small can amplify rounding error or simulation noise.

The central-difference error has a useful justification. If \(Q\) has three continuous derivatives near \(\theta\), Taylor expansion gives \(Q(\theta+h)=Q(\theta)+hQ'(\theta)+h^2Q''(\theta)/2+h^3Q'''(\theta)/6+o(h^3)\), and the corresponding expansion at \(\theta-h\) has the odd-power terms with opposite signs. Subtracting and dividing by \(2h\) therefore gives \(D_hQ(\theta)=Q'(\theta)+h^2Q'''(\theta)/6+o(h^2)\). Thus the central difference has error of order \(h^2\) under this smoothness condition. This calculation does not justify an \(O(h^4)\) remainder without stronger smoothness assumptions.

Unstable Simulated Probabilities and Weights

Simulation can be especially fragile when an objective uses probabilities inside logarithms or relies on a few draws with very large weights. If a simulated probability \(\hat{p}\) is close to zero, its logarithm is highly sensitive to relative error, and a finite simulation may return zero even when the true probability is positive. Silently replacing zero by an arbitrary small constant changes the objective and can affect the estimate; any regularization or probability floor must be explicitly justified and reported.

For a simple illustration, suppose a probability is estimated by the average of \(R\) independent indicator draws with success probability \(p\). Its variance is \(p(1-p)/R\), so its relative standard error is \(\sqrt{(1-p)/(Rp)}\). This approximation presumes independent draws and the simple indicator estimator; other simulation designs require their own variance calculation. For importance sampling, normalized weights \( \tilde{w}_r=w_r/\sum_{s=1}^{R}w_s \) can be summarized by the effective sample size \(ESS=1/\sum_{r=1}^{R}\tilde{w}_r^2\). It ranges from 1 when a single draw carries all weight to \(R\) when weights are equal. It is a diagnostic summary, not a literal count of independent observations.

Putting the Diagnostics Together

No single threshold defines a universally acceptable diagnostic result. Adequacy depends on the scale and purpose of the parameters, the sensitivity of the objective to simulated quantities, and the precision required for the intended inference. A practical assessment records the draw count and design, whether draws are shared across evaluations, optimizer status and tolerances, and the variation across draw counts and independent runs. It also reports gradient discrepancies and relevant probability or weight summaries when those quantities enter the objective.

Interpret discrepancies as clues rather than as automatic evidence of a particular failure. Unstable estimates with increasing draw counts may indicate simulation error, but can also expose a flat objective or weak identification. Stable estimates do not establish correct specification. A gradient mismatch may come from coding, finite-difference step choice, or simulation noise. Low effective sample size signals concentrated weights, but does not alone quantify bias. Follow the symptom with a targeted check and document changes made to the computation.

• Draw-count sensitivity: compare estimates and objective behavior over increasing draw counts.
• Independent replications: measure run-to-run simulation variability conditional on the data.
• Optimization checks: inspect local objective behavior, starting values, and gradient agreement.
• Simulation stability: examine near-zero probabilities, zero simulated events, and weight concentration.