Defining Simulation Questions and Performance Estimands

From a Design Question to a Performance Target

A simulation comparison is useful only if it answers a clearly stated question. “Which sampling design performs better?” is not yet a statistical target: better could mean less bias, more precise estimates, intervals that cover the true parameter more often, or a greater chance of obtaining a usable fit. These criteria can favor different designs. The first task is therefore to define the quantity whose value would count as evidence about performance.

The earlier material on simulation workflows provides the broad sequence for comparing designs. Here the emphasis is narrower: translate the scientific or operational question into an estimand, a precisely defined target quantity. An estimand states what is to be summarized over repeated simulated studies, and under which design and scenario. It is not the result from a finite set of simulated replicates.

To make an estimand interpretable, state at least four things: the target parameter, the design being evaluated, the scenario that defines the conditions of repeated studies, and the performance quantity of interest. A fifth detail is often essential: the event or denominator used to define the quantity. This matters especially when analyses can fail or produce unusable results.

For a parameter whose true value in a specified scenario is \(\theta\), let \(\hat{\theta}\) denote the estimate produced by applying the stated analysis to one simulated study. The same design and scenario define a distribution of possible simulated datasets and analysis outcomes. Performance estimands describe features of that repeated-study distribution; they are not properties of one dataset alone.

Choose the Estimand That Matches the Question

A broad question such as “Are estimates accurate?” must be made more specific. Accuracy might refer to average signed error, typical spread, or the frequency with which an estimate lies close enough to the truth. These are related but not interchangeable.

Bias answers a question about systematic direction, not the size of typical errors. An estimator can have zero bias but substantial variability. A precision question can be represented by the standard deviation of \(\hat{\theta}\), with smaller values indicating less spread under the same conditions. Alternatively, the question may concern a practical tolerance, such as the probability that an estimate is within a specified distance of the truth. That probability requires the tolerance and scale to be stated.

For interval estimation, “Does the interval work?” usually refers to coverage: the probability that the interval procedure contains the true parameter over repeated studies. The interval procedure, including its construction method, is part of the definition. If failed analyses produce no interval, one must decide whether the target is coverage among intervals that were successfully produced or the probability of both producing an interval and covering the truth. Those answer different operational questions.

For an operational question such as “How often can this pipeline deliver an estimate we can use?”, define success as an event before counting outcomes. For example, success might require convergence, a finite estimate, and an estimate within a prespecified tolerance. The target is then the probability of that complete event across all attempted simulated studies. A looser criterion can be appropriate, but its interpretation must be stated.

State the Conditioning and Scope

A performance estimand is conditional on the conditions used to generate and analyze the simulated studies. It is therefore helpful to write the design and scenario into the question itself: for example, “What is the expected bias of the treatment-effect estimator for design A under scenario S?” This makes clear that the result does not automatically describe other designs or scenarios.

When several scenarios matter, two common targets are scenario-specific performance and an average across scenarios. A scenario-specific estimand reports performance separately under each defined condition. An average requires explicit weights. If scenario \(s\) has performance \(Q_s\) and weight \(w_s\), with weights summing to one, the weighted target is:

Equal weights answer the question “What is the average performance across these listed scenarios, treating each equally?” Weights based on a target population or decision context answer a different question. Neither choice is universally correct. The weights, scenarios, and interpretation should be specified rather than selected after seeing which design looks best.

Likewise, a comparison must name the contrast of interest. The difference in bias between two designs, the ratio of their standard deviations, and the difference in success probabilities are distinct comparisons. A design can improve one criterion while worsening another, so a single overall ranking needs an explicit decision rule or priority among criteria.

Worked Examples: Making Questions Operational

A Short Justification: Why Event Probabilities Are Fractions

The connection between a probability estimand and a replicate fraction follows from an indicator variable. For replicate \(r\), define \(I_r=1\) when the event of interest occurs and \(I_r=0\) otherwise. If the simulated replicates are generated under the same specified design and scenario, each indicator has expectation equal to the event probability \(q\): \(E(I_r)=1\cdot q+0\cdot(1-q)=q\). The average indicator is the observed event fraction.

Thus the fraction is a natural estimate of the probability when all \(R\) attempts contribute a defined indicator. For a success probability, each attempt has a success or failure indicator. For coverage conditional on interval production, however, attempts with no interval are excluded by definition; the resulting fraction estimates a conditional probability, not the unconditional probability of successful coverage. The denominator is part of the estimand, not a bookkeeping detail.

Diagnostics, Alternatives, and Limits

Before interpreting a performance summary, check that its operational definition matches the design question. Ask whether the target parameter and units are clear, whether the scenario is stated, whether a probability event has an unambiguous rule, and whether all relevant outcomes have a defined place in the denominator. Also check that the summary is not being interpreted beyond its scope: an estimand for one scenario does not establish performance in unexamined scenarios.

A practical comparison of estimands is useful when deciding what “better” means:

These alternatives address different failure modes. Bias alone does not measure spread; spread alone does not reveal systematic displacement. Coverage assesses the interval procedure rather than just the point estimator. A success probability can capture computational or operational usability but says nothing about the quality of a successful estimate unless quality is included in the success event. If a design decision depends on more than one concern, report multiple estimands or specify a transparent rule for combining them.

A common mistake is to define success after reviewing which replicates failed, or to remove failed replicates without stating that the target is conditional on success. The reasoning might be: “Only converged fits can be evaluated, so exclude nonconverged fits and report the success rate among the rest.” That fraction is not an overall success rate: it conditions on an intermediate event and may even omit the failures that the success criterion is meant to count. The correction is to define the success event first and use all attempted studies in the denominator for its probability. If the intended target is conditional performance among converged fits, label it explicitly as conditional and report the convergence probability separately.

Finally, an estimand is only as informative as its definitions and the conditions under which it is evaluated. A tolerance must be scientifically meaningful and expressed in the parameter’s units. A coverage target needs a specified interval procedure. An average across scenarios needs justified weights. And a simulation result estimates the chosen target under the stated assumptions; it does not guarantee performance in scenarios that were not represented.