A Simulation Workflow for Comparing Study Designs

Why Compare Designs by Simulation?

A study design is more than a sample size. It can specify who is enrolled, when observations are planned, which measurements are collected, and how treatment or dose is assigned. Two designs may also lead to different amounts of usable data or different opportunities to estimate a model parameter. Simulation provides a way to compare such designs under stated assumptions: repeatedly create trial data as if each design had been used, analyze those data using the intended method, and summarize prespecified measures of performance.

The central discipline is consistency. The design determines how data are collected; a data-generating model represents the process that produces them; and an analysis method is applied to each simulated trial. These are related but distinct parts of the workflow. A design comparison is not persuasive if one design is analyzed with a more favorable method, if the simulated data do not reflect the intended trial, or if the comparison measure is chosen only after results are seen.

The workflow answers a conditional question: given the assumptions used to generate data and the analysis method used, how do the candidate designs perform on the measures selected for comparison? It does not establish which design will perform best in every possible real-world setting. Conclusions are only as credible as the assumptions, implementation, and scope of the comparison.

Specify the Workflow Before Running It

Begin by writing down the decision the comparison is meant to inform. Then define candidate designs in terms that can be implemented, rather than relying on labels such as “sparse” or “intensive.” For example, state the planned number of participants, allocation, measurement times, and any other feature that differs. Keep features that are intended to be the same fixed across candidates.

Next, state how data will be generated for each candidate. The generating assumptions should describe the trial process relevant to the question, including the quantities treated as known for the simulation and the sources of variation represented. They are not automatically the same as the assumptions in the analysis model. Indeed, comparing performance under a difference between the generating model and the analysis model may be intentional, but the difference must be explicit.

Specify the analysis pipeline before generating results. It includes data preparation, the model or estimator to be used, and the rule for recording an analysis that does not return a usable result. Apply the same pipeline across designs unless a difference is itself part of the planned comparison. If each candidate receives a different analysis method, the comparison is of design-and-analysis strategies, not designs alone.

Finally, define the performance measures in advance. Examples include estimation error relative to a known simulated value, the proportion of trials meeting a prespecified accuracy criterion, and computational time. The measure should reflect the decision at hand. A design that produces more precise estimates may also cost more or be harder to implement; no single measure necessarily captures every trade-off.

A Simple Justification for Simulation-Based Precision Comparisons

A simple model helps show why design details can affect performance. Suppose each participant contributes one measurement \(Y_i\), with independent observations from a normal distribution with mean \(\theta\) and variance \(\sigma^2\). If the analysis estimates \(\theta\) by the sample mean, then the sample mean has expected value \(\theta\), and its variance is \(\sigma^2/n\). Thus, under these assumptions, increasing the number of independent observations reduces the sampling variance of the estimator.

The variance result follows from independence: the variance of a sum is the sum of the variances, so \(\operatorname{Var}(\sum_iY_i)=n\sigma^2\). Dividing that sum by \(n\) divides its variance by \(n^2\), giving \(n\sigma^2/n^2=\sigma^2/n\). This justification depends on the stated model and estimator; it is not a general guarantee that every design with more observations will be better on every measure. Simulation is useful when the design, data structure, or analysis is too complicated for such a simple calculation.

Apply the Analysis to Every Simulated Trial

A simulation replicate represents one possible trial under a candidate design. The analysis pipeline should produce the planned output for each replicate, such as a parameter estimate, an interval, or a decision. It should also record whether the analysis completed and whether its output met any prespecified usability rule. A failed fit is part of the design’s simulated performance; silently discarding it can make a design appear more successful than it is.

Before interpreting summaries, check that the implementation matches the plan. Confirm that each candidate has the intended number of participants and measurements, that generated values have plausible units and ranges under the assumptions, and that the analysis is receiving the correct data. Review a small number of simulated trials by hand. If the program gives surprising results, first check the data and code rather than treating the output as a scientific finding.

Compare Prespecified Measures, Not Just Appealing Outputs

For a parameter with known generating value \(\theta\), an estimate \(\hat{\theta}_r\) from replicate \(r\) can be compared with that value. Across \(R\) replicates, one simple summary is the average signed error, \(\sum_{r=1}^{R}(\hat{\theta}_r-\theta)/R\). Another is the root mean squared error, the square root of the average squared error. A team may also define a practical criterion, such as an estimate within a specified distance of \(\theta\). These summaries answer different questions and should not be interchanged.

For each measure, keep the direction of preference clear. Smaller estimation error may be desirable, while a larger proportion meeting a practical criterion may be desirable. If computational burden or feasibility matters, include an appropriate measure rather than assuming statistical performance alone decides the design. Report the number of simulated trials and describe how unusable analyses are handled so that readers can understand what the summaries represent.

Diagnostics, Alternatives, and Limitations

Warning signs include results that are identical across designs despite meaningful design differences, implausible simulated values, analysis failures that are omitted from summaries, and conclusions that change when the implementation is checked. Examine distributions of outputs as well as averages: a favorable average can hide a substantial number of extreme estimates or failed analyses. Keep a record of the assumptions and code settings needed to reproduce the comparison.

Simulation is not the only design tool. Analytical calculations can be more efficient and transparent when a simple model yields a relevant exact or approximate result, as in the standard-error calculation above. Pilot data can help inform plausible ranges of inputs, although a pilot does not remove uncertainty about those inputs. Simulation is most useful when features of the design or analysis make a direct calculation difficult, or when several outcomes and operational constraints must be examined together. These approaches can complement one another.

A simulation study cannot compensate for a poorly chosen generating model. If important sources of variation, design constraints, or analysis failures are absent, a large number of replicates only makes the results more precise for the incomplete assumptions. It is also possible to overinterpret a ranking: a candidate that performs better under one set of assumptions may not do so under another. State the scope of the conclusions, and avoid treating simulated performance as an empirical guarantee.

Common Pitfall