Choosing Replicate Counts And Virtual-Population Size

Why Replicate Counts Must Match the Simulation Target

A simulation can be computationally large yet poorly designed. Thousands of simulated observations do not guarantee a precise estimate if they come from too few parameter draws or too few independent virtual subjects. Conversely, a simulation can waste substantial computation by repeating a level of sampling that contributes little to uncertainty in the target quantity.

The count decision begins with the estimand: the numerical quantity the simulation is intended to estimate. For example, the target might be the population mean of an exposure metric under a fixed parameter set, an average prediction over parameter uncertainty, or the distribution of an observed endpoint when residual error is included. Those targets require different sampling levels. The earlier material on separating parameter and individual variability provides the key distinction: parameter uncertainty, between-subject variation, and residual variation are not interchangeable sources of variation.

A common nested structure is to draw \(\theta_m\), the parameter vector for draw \(m\); simulate \(N\) subjects conditional on \(\theta_m\); and, if the target includes outcome-level randomness, simulate \(R\) outcomes for each subject. The total number of outcome records is \(MNR\), but the number of independent draws at each level is \(M\), \(MN\), or \(MNR\), respectively. In particular, repeated outcomes on one subject do not create new parameter draws or new virtual subjects.

State the Target and Its Sampling Levels

Before choosing counts, write down which randomness belongs in the target. If the goal is a prediction conditional on one fixed parameter estimate, drawing parameters repeatedly from an uncertainty distribution changes the target rather than improving its precision. If the goal averages predictions over parameter uncertainty, parameter draws are part of the design. Similarly, residual-error replicates are needed when the target concerns simulated observations, but may be unnecessary for a target based on an expected response when residual error has conditional mean zero and can be integrated out.

For a simple mean target with a balanced nested design, a useful planning approximation is

Here \(A\) is the variance across parameter draws of the conditional mean target; \(B\) is the variance across virtual subjects of their expected outcome, conditional on the parameter draw; and \(C\) is the residual outcome variance among replicate outcomes for a subject. The approximation assumes independent parameter draws, independent subjects conditional on each parameter draw, independent residual replicates conditional on a subject and its parameter draw, and finite component variances. It also assumes a balanced design and that the reported estimate averages over all levels relevant to the target. If the simulation has dependence, unequal allocation, or a nonlinear summary such as a quantile, this formula may not apply directly; a pilot simulation and repeated-batch diagnostics are then important.

The first term decreases by increasing \(M\); the second decreases by increasing either \(M\) or \(N\); and the third decreases by increasing any of \(M\), \(N\), or \(R\). The formula therefore clarifies why simply increasing the number of outcome records can be inefficient: if \(A\) dominates, many subjects and replicates under a small number of parameter draws will leave most of the uncertainty untouched.

A Short Justification for the Nested Variance Formula

Let \(Y_{mnr}\) be the outcome for replicate \(r\) of subject \(n\) under parameter draw \(m\), and let \(\mu(\theta_m)\) be the expected outcome averaged over subjects and residual error at that parameter draw. For each parameter draw, the sample average across its subjects and replicates estimates \(\mu(\theta_m)\). The overall estimate averages those \(M\) conditional averages.

By the law of total variance, the variance of an estimate that averages \(M\) independent parameter-level averages is the variance of a parameter-level conditional mean divided by \(M\), plus the expected conditional variance of one parameter-level average divided by \(M\). Within that conditional average, independent subjects contribute a between-subject term divided by \(N\), and independent replicate outcomes contribute a residual term divided by \(NR\). Therefore,

This derivation also states the limits of the formula. If repeated outcomes share an unmodeled time trend or are correlated, the residual contribution does not generally fall as \(1/R\). If subjects are reused across parameter draws, the draws are not independent in the required way. The counts must be interpreted alongside the actual simulation dependence structure.

Work From a Precision Goal

A target Monte Carlo standard error (the standard deviation of the simulation estimator under repeated runs of the same design) can be used to compare proposed counts. For a mean target, its approximate value is the square root of the variance formula. A precision goal should be set in context and in the units of the target, not as an arbitrary number of records. Variance components can be estimated from a pilot run, analytic calculations, or prior validated simulations. Planning estimates need not be exact, but uncertainty in them should be considered by checking more than one plausible set of components.

When there is a fixed computational budget, candidate allocations can be compared by their estimated precision. If each parameter-subject combination has approximately equal cost and residual error can be integrated out, the approximate cost is proportional to \(MN\). If each replicate requires a separate costly simulation, cost may instead be close to proportional to \(MNR\). In practice, setup and data-handling costs can matter, so a short timing run is often more informative than assuming exact proportionality.

When Within-Subject Replication Helps

Within-subject replication is most useful when the target includes outcome-level variability and the residual component is large enough to matter. It does not replace independent subjects when the subject-to-subject variation is important. For an average of \(R\) outcomes per subject, a subject’s persistent characteristics are still shared across those outcomes; only the part that genuinely varies independently between replicates is reduced by averaging.

Diagnostics, Alternatives, and Limitations

Count planning is a design decision, not a substitute for checking the resulting simulation. Examine the estimate across independent batches or progressively larger runs, and compare the observed variability with the planning approximation where appropriate. Large shifts between batches, unstable tail summaries, or a standard error that does not decrease as expected can indicate too few draws at a relevant level, dependence between draws, an incorrect variance decomposition, or failed simulations. For a mean based on independent samples, doubling a relevant count is expected to reduce that component’s standard error by about a factor of \(\sqrt{2}\), not by half.

The nested formula is most directly useful for averages with manageable variance components. Quantiles, maxima, and other nonlinear summaries can converge differently; a simple mean-based variance formula should not be treated as their precision guarantee. A pilot can compare candidate counts for the actual reported summary. When computation is expensive, alternatives include analytically integrating residual error, using a more efficient sampling design where appropriate, or allocating computation to the level that dominates uncertainty. Any alternative must preserve the intended estimand and the dependence structure of the simulation.

A practical allocation can be organized as follows:

Common pitfall: “The simulation has one million records, so the result must be precise.” That conclusion is not justified. If those records arise from one parameter draw and a small number of subjects with many repeated outcomes, parameter and subject-level uncertainty may remain large. Report \(M\), \(N\), and \(R\) separately, state which levels are independent, and explain why those counts are adequate for the target.