Why Select Sample Size by Simulated Precision?
A sample size can be statistically adequate in one simulated setting and too small in another. For example, a study may estimate clearance precisely under a typical parameter scenario but much less precisely when variability is higher or observations are sparser. Selecting a sample size by simulated precision makes the design decision explicit: specify how precise the estimates must be, simulate the candidate sizes under relevant scenarios, and choose the smallest size that satisfies the requirements.
The target must describe a quantity that can be checked. It might be a maximum standard deviation for an estimate, a maximum relative standard error, or a maximum confidence-interval width. These are different criteria and should not be treated as interchangeable. The decision also needs to specify which parameters and scenarios must meet the target. A favorable average across scenarios does not establish that every important scenario meets it.
Define the Target Before Comparing Sample Sizes
Let \(N\) denote a candidate sample size, \(s\) a simulation scenario, \(j\) a parameter, and \(R\) the number of simulation replicates. Write \(\hat{\theta}_{jrs}(N)\) for the estimate of parameter \(j\) from replicate \(r\), under scenario \(s\) and sample size \(N\). The empirical standard deviation across replicates is
This standard deviation describes the variability of the estimator across hypothetical repetitions of the design. It is not the standard error within one fitted data set, although the two quantities may be related. The simulated spread is calculated for the parameter on its stated scale and therefore has that parameter’s units.
An absolute target might require \(\hat{\sigma}_{j,s}(N)\leq c_j\), where \(c_j\) is an acceptable spread in the parameter’s units. For a positive parameter whose true value \(\theta_{j,s}\) is meaningfully away from zero, relative standard error can instead be used:
An RSE target is scale-free, but it can be misleading for parameters near zero and depends on the chosen parameter scale. For those parameters, an absolute target or a target defined on a transformation scale may be more appropriate. A confidence-interval width is another possible target, but it must be calculated using the interval procedure intended for the study. Multiplying a simulated standard deviation by 3.92 approximates an average 95% Wald interval width only when a normal approximation is reasonable and that is the interval method being evaluated.
Before running simulations, list the required parameters and scenarios, and state the target for each. A general decision rule is to select the smallest candidate \(N\) for which every required precision measure is within its limit. If one scenario is a genuine design constraint, it must meet the target even if the overall average is acceptable.
How Much Simulation Is Enough to Judge Precision?
The simulated standard deviation is itself estimated from a finite number of replicates. A design close to the precision threshold can therefore appear to pass or fail partly because of Monte Carlo error. Under the assumption that the replicate estimates are independent draws from a normal distribution with standard deviation \(\sigma\), the sample variance satisfies
Since a chi-square random variable with \(R-1\) degrees of freedom has variance \(2(R-1)\), this result gives the approximate relative Monte Carlo standard error of the estimated standard deviation:
This approximation is a diagnostic, not a substitute for checking whether the simulated estimate distribution is roughly normal. For example, with \(R=500\), the approximate relative Monte Carlo standard error is \(1/\sqrt{998}\), or about 3.17%. A precision estimate near a strict boundary should be treated with caution. One conservative rule, under the normality assumption, is to require an upper confidence bound for \(\sigma\) to meet the target rather than requiring only the point estimate to do so. If the distribution is skewed or has outliers, the chi-square calculation may not be reliable; inspect the replicate distribution and consider an appropriate simulation-based uncertainty assessment.
The number of replicates should be sufficient to distinguish a pass from a fail at the precision threshold. Earlier material on choosing replicate counts with Monte Carlo error provides the broader replicate-planning framework. Here, the essential point is that an estimated precision value is not exact merely because it came from a simulation.
A Practical Selection Procedure
Name the parameters, scenarios, precision metric, threshold, and candidate sample sizes before examining results.
Use the same data-generating assumptions, estimation procedure, and precision calculation for each candidate size. Record estimation failures rather than silently dropping them.
Calculate the chosen precision metric for every required parameter-scenario pair. Account for Monte Carlo uncertainty if a result is near its threshold.
Choose the smallest candidate meeting every requirement. If the grid is coarse or results are borderline, simulate additional sizes near the apparent boundary and verify the decision.
Worked Examples
Worked Example: An Absolute Standard-Deviation Target Across Scenarios
Suppose a hypothetical design will estimate a clearance parameter in L/h. The prespecified requirement is that its simulated standard deviation be no greater than 0.25 L/h in both a typical and a high-variability scenario. Each row below summarizes \(R=1{,}000\) simulation replicates.
| Candidate \(N\) | Typical scenario, SD (L/h) | High-variability scenario, SD (L/h) | Meets both targets? |
|---|---|---|---|
| 40 | 0.31 | 0.44 | No |
| 60 | 0.25 | 0.32 | No |
| 80 | 0.22 | 0.24 | Yes |
At \(N=40\), both standard deviations exceed 0.25 L/h. At \(N=60\), the typical scenario meets the target exactly, but the high-variability scenario does not: \(0.32>0.25\). At \(N=80\), both requirements hold, since \(0.22\leq0.25\) and \(0.24\leq0.25\). Therefore, among the sizes simulated, \(N=80\) is the smallest size meeting the joint target.
The conclusion is limited to the candidate sizes and scenarios evaluated. Because the high-variability result of 0.24 L/h is close to the boundary, the decision should also consider Monte Carlo uncertainty. Passing on the point estimate alone does not guarantee that the underlying simulated standard deviation is below the threshold.
Worked Example: Relative Precision for Two Parameters
Consider a hypothetical scenario with true clearance \(5.0\) L/h and true volume \(20.0\) L. The design target is an RSE no greater than 20% for both parameters. At \(N=100\), the simulated standard deviations are 0.80 L/h for clearance and 4.4 L for volume. The RSEs are
Clearance meets the target, but volume does not. At \(N=150\), suppose the simulated standard deviations are 0.65 L/h and 3.8 L. Then
Both parameters now meet the 20% limit. If these are the only candidate sizes evaluated and the smaller candidate fails the joint rule, select \(N=150\) among the candidates. The result does not mean every estimate from a study of 150 individuals will be within 19% of the true volume; RSE summarizes the across-replicate spread relative to the true value.
Worked Example: Using Approximate Scaling to Choose Candidate Sizes
Suppose a pilot simulation at \(N_0=50\) gives an estimated standard deviation of 0.30 units for a parameter, and the target is 0.20 units. Under a simple large-sample approximation, estimator standard deviation decreases in proportion to \(1/\sqrt{N}\). Solving for the sample size that would reach the target gives
This suggests simulating candidate sizes near 113, not choosing 113 without checking. Suppose simulations produce standard deviations of 0.218 at \(N=100\), 0.196 at \(N=125\), and 0.181 at \(N=150\). The first fails because \(0.218>0.20\); the second and third pass. The smallest passing size among these candidates is \(N=125\). The scaling calculation helped locate the boundary, while the simulation results determined the selection.
Diagnostics, Alternatives, and Limitations
Precision should be assessed alongside other design-performance measures, not used as a replacement for them. A tightly concentrated estimator can still be systematically displaced from the true value; precision alone does not establish low bias. Likewise, a standard-deviation target does not establish that confidence intervals have their intended coverage. Earlier material on bias and root mean squared error and on confidence-interval coverage addresses these distinct properties.
Check the simulated parameter-estimate distributions and the estimation-failure rate. A small number of extreme estimates can inflate the standard deviation; a multimodal or strongly skewed distribution can make a single spread summary incomplete. If failed fits are excluded from the standard-deviation calculation, the reported precision describes estimates conditional on successful fitting and may look artificially favorable. Report failures and define in advance how they affect the design decision. A design that meets its precision target only after excluding a material share of replicates may not be acceptable.
Candidate results need not decrease smoothly with \(N\). Finite simulation error, optimization instability, and changes in which replicates converge can all produce apparent irregularities. Do not assume that a passing size proves every smaller or larger size will pass, or select a size based on an untested interpolation. If results are non-monotone or near the threshold, investigate the cause, increase simulation precision where useful, and evaluate additional candidate sizes.
Averaging across scenarios is appropriate only if the design objective explicitly defines a weighted overall precision criterion. Otherwise, an average can conceal a poor result in a clinically or scientifically important scenario. For a requirement that all designated scenarios meet a limit, evaluate each one separately and use the most demanding result to determine the minimum size.
Analytical standard-error approximations, such as those based on large-sample information, can be a useful alternative for rapidly screening candidate sizes when their assumptions are credible. They may be inaccurate with sparse data, complex estimation, or challenging scenarios. Simulation directly evaluates the specified data-generating and estimation procedures, but costs computation and remains conditional on those assumptions and implementation. The approaches can complement each other: use an analytical approximation to propose a grid, then simulation to check the actual target.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A target requires a standard deviation no greater than 0.15 mg/L in two scenarios. At \(N=80\), the simulated values are 0.12 and 0.17 mg/L; at \(N=100\), they are 0.14 and 0.15 mg/L. Which candidate meets the rule?
- A positive parameter has true value 4.0 L/h and a simulated standard deviation of 0.60 L/h. Calculate its RSE and assess it against a 20% target.
- Why can a design with a simulated standard deviation below its target still be a poor choice if many replicates fail to converge?
- At \(N=60\), a pilot simulation gives a standard deviation of 0.24 units. Estimate the sample size needed for a target of 0.16 units using inverse-square-root scaling. What must happen before adopting that size?
Solutions. (1) \(N=80\) fails because \(0.17>0.15\) mg/L in the second scenario. \(N=100\) meets both requirements because \(0.14\leq0.15\) and \(0.15\leq0.15\) mg/L, so it is the passing candidate shown. (2) \(\mathrm{RSE}=100(0.60/4.0)=15.0\%\), which meets the 20% target. (3) If failed replicates are omitted, the reported spread describes only successful fits and may understate the practical difficulty of estimation. The failure rate must be reported and included in the design decision. (4) \(N_{\mathrm{approx}}=60(0.24/0.16)^2=60(2.25)=135\). This is a candidate-planning approximation, not proof that \(N=135\) meets the target; simulate and evaluate that size using the prespecified precision rule and scenarios.