Choosing Virtual Population Size By Monte Carlo Error

Why Choose a Virtual Population Size?

A virtual population is finite, even when its patients are generated from a probability distribution. Its mean, phenotype proportions, and other summaries will therefore vary from one generated cohort to another. A small cohort can give an unstable summary simply because of which patients happened to be drawn. Choosing a cohort size by Monte Carlo error means planning for this simulation-to-simulation variation to be acceptably small for the summaries that matter.

This is a precision question, not a population-fit question. Increasing the number of virtual patients can reduce random error in estimated summaries, but it cannot repair a misspecified generating distribution, an unsuitable target, or a systematic selection bias. Nor does increasing the cohort size reduce the real heterogeneity among patients: it makes the cohort’s summaries more stable estimates of the distribution from which patients are generated.

Definition: Monte Carlo error is the random variation in a computed estimate caused by using a finite number of random draws. For a summary $T$ estimated from a virtual cohort of size $n$, its Monte Carlo standard error $SE$ describes the repeated-generation variability of that estimate, under the specified generation procedure.

The basic calculations below assume $n$ independent, equally weighted patients drawn from the same target distribution. Under this assumption, the standard error usually decreases in proportion to $1/\sqrt{n}$. Thus, reducing the standard error by half requires roughly four times as many patients. This slow rate makes it important to define a precision requirement before generating a very large cohort.

Planning for a Mean

Suppose the target is the mean of a continuous patient-level outcome, such as exposure in $\mathrm{mg\cdot h/L}$. Let the outcome have population mean $\mu$ and population standard deviation $\sigma$. For $n$ independent draws, the sample mean $\bar{x}$ has standard error $\sigma/\sqrt{n}$. If $\sigma$ is unknown, a pilot cohort or prior simulation can provide an estimate $s$.

For a sufficiently large cohort, the central limit theorem motivates an approximate two-sided confidence interval for the generating-distribution mean: $\bar{x}\pm z_{1-\alpha/2}s/\sqrt{n}$, where $z_{1-\alpha/2}$ is the standard normal quantile. This interval describes uncertainty in the estimated mean due to finite simulation. It is not an interval for individual patient outcomes.

\text{Approximate half-width } h = z_{1-\alpha/2}\frac{s}{\sqrt{n}}, \qquad n\geq\left(\frac{z_{1-\alpha/2}s}{h}\right)^2

The desired half-width $h$ must be stated in the outcome’s units. The calculated size is rounded up to the next whole patient, since rounding down could fail the precision requirement. This is a planning calculation: the actual standard deviation is not known exactly, and the normal approximation may be less accurate for small or strongly skewed samples.

Worked Example: Planning for a Mean Exposure

A pilot simulation of a hypothetical QSP model suggests that individual exposure has standard deviation $s=12.0\ \mathrm{mg\cdot h/L}$. The team wants the simulated cohort mean to have an approximate 95% Monte Carlo half-width no greater than $1.5\ \mathrm{mg\cdot h/L}$. Assume independent, equally weighted draws and use $z_{0.975}=1.96$.

The required size is $$ n\geq\left(\frac{1.96(12.0\ \mathrm{mg\cdot h/L})}{1.5\ \mathrm{mg\cdot h/L}}\right)^2 =\left(\frac{23.52}{1.5}\right)^2 =(15.68)^2 =245.8624. $$

Rounding up gives a planning size of 246 virtual patients. The units cancel in the ratio, so $n$ is a count. With the pilot standard deviation treated as an adequate planning value, this size targets an approximate 95% half-width of $1.5\ \mathrm{mg\cdot h/L}$ for the generating-distribution mean. It does not promise that every realized cohort will have exactly that half-width.

Planning for a Phenotype Proportion

A phenotype can be represented for each patient by an indicator $I$, equal to 1 if the phenotype is present and 0 otherwise. If its prevalence in the generating distribution is $p$, then for independent draws the count of phenotype-positive patients follows a binomial distribution with parameters $n$ and $p$. The observed proportion $\hat{p}$ has standard error $\sqrt{p(1-p)/n}$. Replacing $p$ by a planning value gives an estimated standard error; using the normal approximation, the corresponding half-width is approximately $z_{1-\alpha/2}\sqrt{p(1-p)/n}$.

h \approx z_{1-\alpha/2}\sqrt{\frac{p(1-p)}{n}}, \qquad n\geq\frac{z_{1-\alpha/2}^{\,2}p(1-p)}{h^2}

The formula is useful when a plausible planning prevalence is available. If it is not, $p(1-p)\leq 0.25$ for all $0\leq p\leq1$, with equality at $p=0.5$. Substituting $0.5$ therefore gives a conservative size for a specified absolute half-width across possible prevalences. For rare phenotypes, the normal approximation can be poor when the expected count $np$ is small; a nominally narrow normal-based interval should not be taken as evidence of reliable tail coverage.

Worked Example: Setting Precision for a Phenotype Proportion

Suppose a hypothetical phenotype has planning prevalence $p=0.080$. The desired approximate 95% Monte Carlo half-width for its proportion is $h=0.010$, or one percentage point. Assume independent, equally weighted draws and use $z_{0.975}=1.96$.

Substitution gives $$ n\geq\frac{1.96^2(0.080)(0.920)}{0.010^2} =\frac{0.28274176}{0.0001} =2827.4176. $$

The minimum whole-number planning size is therefore 2,828 patients. The numerator is $1.96^2(0.080)(0.920)=0.28274176$; dividing by $0.0001$ gives $2827.4176$, which must be rounded up. This calculation targets approximate precision for the generated proportion, not proof that the model’s phenotype prevalence is correct.

Worked Example: Planning Without a Prevalence Estimate

For a hypothetical phenotype, no reliable planning prevalence is available. The team requires an approximate 90% Monte Carlo half-width no greater than $0.020$, or two percentage points. Use $z_{0.95}=1.645$, and plan conservatively over all possible prevalence values.

Since $p(1-p)\leq0.25$, use $p=0.5$ in the size calculation: $$ n\geq\frac{1.645^2(0.25)}{0.020^2} =\frac{0.67650625}{0.0004} =1691.265625. $$

Round up to 1,692 patients. The size is conservative for the normal-approximation half-width because $0.25$ is the maximum possible Bernoulli variance. It may be more than needed if the actual prevalence is far from $0.5$. Conversely, if the phenotype is very rare, this calculation alone does not ensure that enough positive patients will be generated for downstream analyses.

Quantiles and Other Target Summaries

A quantile can require a larger cohort than a mean when the distribution has little density near the target quantile. For a continuous distribution with density $f$ that is positive and sufficiently smooth at its $q$-quantile $x_q$, the large-sample standard error of the sample quantile is approximately $$ \sqrt{\frac{q(1-q)}{n f(x_q)^2}}. $$ Here $f(x_q)$ has inverse outcome units, so the standard error has the outcome’s units. The approximation depends on a suitable density estimate near $x_q$; it becomes unstable if that density is small or poorly estimated. It is not a universal formula for every summary statistic.

Worked Example: Planning for a Median

Suppose a hypothetical outcome is approximately normal with standard deviation $20$ outcome units. Its median is at the distribution’s center, where the density is $f(x_{0.5})=1/(20\sqrt{2\pi})\approx0.01995$ per outcome unit. The goal is an approximate 95% Monte Carlo half-width of $2$ outcome units for the simulated median.

Using the quantile standard error approximation and $q=0.5$, $$ n\geq\frac{1.96^2(0.5)(0.5)}{2^2(0.01995)^2} \approx\frac{0.9604}{0.001592} \approx603.3. $$

Rounding up gives 604 patients. This is an approximate planning value conditional on the normal-distribution density assumption. In practice, the density near the median would need to be estimated or otherwise justified, and the resulting precision should be checked in generated cohorts.

Derivation, Diagnostics, and Practical Choices

The mean and proportion formulas follow from the same variance argument. If $X_1,\ldots,X_n$ are independent draws with variance $\sigma^2$, then $$ \operatorname{Var}(\bar{x}) =\operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^{n}X_i\right) =\frac{1}{n^2}\sum_{i=1}^{n}\operatorname{Var}(X_i) =\frac{\sigma^2}{n}. $$ Taking the square root gives $SE(\bar{x})=\sigma/\sqrt{n}$. For a phenotype indicator, the variance of each draw is $p(1-p)$; applying the same result to its sample mean, which is $\hat{p}$, gives $SE(\hat{p})=\sqrt{p(1-p)/n}$. A normal-approximation half-width is $z_{1-\alpha/2}$ times the standard error; solving this inequality for $n$ yields the planning formulas.

A practical plan begins with the summaries that will be reported and their acceptable Monte Carlo errors. Calculate a size for each summary using an appropriate pilot estimate or conservative bound, then choose a cohort size at least as large as the largest requirement. If precision is required for several outcomes or subgroups, check each one: an overall mean may be stable while a subgroup proportion remains noisy. Pilot estimates of variability are themselves uncertain, so a larger planning value for the standard deviation, or a prespecified safety margin, may be appropriate.

After generation, diagnose the calculation rather than relying on cohort size alone. Compare estimates from independent generated cohorts or independent batches, and check whether discrepancies are broadly consistent with the predicted standard errors. Increasing $n$ should reduce Monte Carlo standard errors at approximately the $1/\sqrt{n}$ rate under the assumptions. Large unexplained differences, lack of stabilization, or strong sensitivity to the random seed can indicate an inadequate size, a faulty implementation, or a failure of the assumptions.

Approach	Useful for	Key caution
Normal-approximation planning formulas	Quick size calculations for means and proportions when assumptions are reasonable	Approximate; may perform poorly for small samples or rare phenotypes
Exact or improved binomial intervals	Assessing uncertainty in a realized phenotype proportion, especially with small counts	Intervals for the realized proportion do not by themselves select a cohort size or validate the target prevalence
Repeated independent simulation	Empirically checking stability of summaries across generated cohorts	Requires multiple runs and does not eliminate systematic errors in the generator

The formulas should not be applied unchanged when patients are dependent, have unequal weights, or arise from a sampling design that changes the variance. Stratified sampling, for example, requires a variance calculation that reflects the strata and allocation rather than treating all draws as a simple independent sample. In those settings, use a design-appropriate standard error or repeated simulation that reproduces the actual generation procedure. A raw count of virtual patients is not always the effective amount of information for a particular summary.

Common Pitfalls and Interpretation

Confusing Monte Carlo error with patient variability. The standard deviation among individual outcomes describes heterogeneity; the standard error of a cohort mean describes simulation error in that mean.
Rounding a calculated size down. A result such as $245.86$ requires 246 patients, not 245, to meet the stated inequality.
Treating a precision target as a fit criterion. A small Monte Carlo error says the simulated summary is estimated stably; it does not say the summary matches its target or that the model is valid.
Using a rare-phenotype normal approximation without checking counts. When expected positive counts are low, the approximation may be unreliable; inspect the count and use an appropriate binomial method or direct repeated simulation.
Ignoring the generation design. Dependence or unequal weights can invalidate the simple $1/\sqrt{n}$ standard errors, so use a variance calculation that reflects how the cohort was constructed.

Key takeaway: Choose virtual population size from a stated Monte Carlo precision target, using a variance formula that matches the summary and generation design. Then check the realized error and keep simulation precision distinct from biological variability, target fit, and model uncertainty.

Check Your Understanding

Work through each question, then compare with the solutions.

A continuous outcome has pilot standard deviation $8.0$ units. What cohort size gives an approximate 95% half-width of $1.0$ unit for its mean?
For a phenotype with planning prevalence $0.20$, what is the approximate standard error of its observed proportion in $n=400$ independent patients?
Why does the conservative proportion planning calculation use $p=0.5$ when prevalence is unknown?
A simulation team doubles its cohort size. By what factor should the standard error of the mean change under the independent-draw assumptions?
Does meeting a Monte Carlo precision target establish that a virtual population matches the clinical target distribution? Explain.

Solutions. (1) $n\geq(1.96(8.0)/1.0)^2=(15.68)^2=245.8624$, so round up to 246 patients. (2) $SE(\hat{p})=\sqrt{0.20(0.80)/400}=\sqrt{0.0004}=0.0200$, or two percentage points. (3) Because $p(1-p)$ is maximized at $p=0.5$, giving the largest Bernoulli variance and a conservative normal-approximation size for a fixed half-width. (4) The standard error is multiplied by $1/\sqrt{2}\approx0.707$, a reduction of about 29.3%. (5) No. Precision concerns random variation from finite simulation; matching the clinical target concerns the appropriateness and fit of the generating distribution.