Why Parameter Uncertainty Matters
A simulated probability of target attainment (PTA) is often reported as one number for each candidate dose. That number is conditional on the population model parameters used in the simulation. Those parameters have themselves been estimated from finite data. If plausible parameter values lead to meaningfully different PTAs, a single plug-in estimate can give a false impression of precision in the dose comparison.
The key distinction is between variation among individuals under a specified population model and uncertainty about the population parameters used to define that model. Between-person variability is part of the population PTA: it describes how exposure or response differs across people. Parameter uncertainty describes how much the estimated PTA would change if the population parameters were different values still compatible with the estimation data.
The target population, covariate distribution, endpoint definition, dosing regimen, and rules for simulating individual random effects must be specified and held consistent when comparing parameter draws. Include residual error in the simulated endpoint if the target is defined on an observed measurement; do not add it automatically when the target concerns a latent model quantity. The model must also be an adequate basis for simulation, and parameter estimates and their uncertainty must be available from a successfully fitted model. Propagating uncertainty does not correct model misspecification, biased estimation, or an unrepresentative target-population distribution.
Nested Simulation: Two Different Sources of Variation
A practical approach uses an outer loop for plausible population parameter values and an inner loop for individuals simulated under each value. The outer loop can use bootstrap refits or draws from an approximate sampling distribution for the parameter estimates. The inner loop simulates the target population, including between-person random effects and any other variation required by the endpoint definition.
Obtain \(B\) plausible parameter sets \(\theta^{(1)},\ldots,\theta^{(B)}\), preserving estimated parameter correlations and respecting parameter constraints.
For each dose and each \(\theta^{(b)}\), simulate \(M\) individuals from the intended covariate distribution and the fitted between-person variability model. Calculate each individual's endpoint and whether it meets the target.
For each parameter set, estimate \(q(d;\theta^{(b)})\) as the fraction of the \(M\) simulated individuals who attain the target.
Report the central PTA estimate and a clearly defined interval across the \(B\) conditional PTA estimates. Separately assess inner-loop Monte Carlo error.
With a nonparametric bootstrap, each outer parameter set comes from refitting the model to a bootstrap resample of the original estimation data. The resulting distribution of conditional PTAs describes how the estimated PTA varies under that resampling procedure. A percentile interval uses selected quantiles of that distribution, such as its 2.5th and 97.5th percentiles. Under the assumptions that make the bootstrap appropriate, this interval estimates uncertainty due to sampling and parameter estimation. It is not a statement that there is a 95% probability the true PTA lies in the calculated interval.
An alternative is an approximate sampling distribution, often a multivariate normal approximation centered at \(\hat{\theta}\) with covariance based on the parameter-estimation covariance matrix. This can be efficient when the approximation is reasonable and the estimate is away from boundaries. However, parameters such as variances must remain in their allowable range, and parameter correlations matter. Unconstrained normal draws can produce impossible parameter values or misrepresent uncertainty near a boundary. Transformations or a bootstrap may be more appropriate in such cases.
A Short Derivation: Why the Two Uncertainties Are Not Interchangeable
For a fixed parameter set \(\theta\), define \(Y\) as an indicator equal to 1 when a simulated person attains the target and 0 otherwise. Then \(Y\mid\theta\) is Bernoulli with probability \(q(d;\theta)\). Across repeated simulated people at fixed \(\theta\), the variance is \(q(d;\theta)\{1-q(d;\theta)\}\); this is person-to-person outcome variation. If \(\theta\) varies across plausible parameter sets, the conditional PTA also varies.
The law of total variance separates these sources. The first term below is the expected individual outcome variance conditional on parameters; the second is the variance of conditional PTA across parameter sets.
This decomposition explains a common error: pooling individual outcomes simulated under many parameter sets and treating the resulting spread as ordinary between-person variability. That mixes the two terms. For dose selection, preserve each outer draw's conditional PTA; summarize the distribution of those PTAs to show parameter uncertainty.
Worked Example: A Simple Exposure Model
Consider a hypothetical one-compartment exposure calculation at a fixed regimen, with \(AUC\) in mg·h/L given by \(AUC=(D/CL)\exp(\eta)\). Here \(D\) is dose in mg, \(CL\) is typical clearance in L/h, and the individual log-exposure deviation \(\eta\) is distributed as \(N(0,0.30^2)\). Suppose the endpoint is \(AUC\geq 50\) mg·h/L and \(D=300\) mg. To isolate the calculation, suppose the outer parameter draws vary \(CL\) while the random-effects standard deviation is held at 0.30.
For a given \(CL\), the individual attains the target when \(\eta\geq \log\{50/(300/CL)\}\). Thus the conditional PTA is
where \(\Phi\) is the standard normal cumulative distribution function. Suppose three hypothetical bootstrap refits give typical clearance values of 4.5, 5.0, and 5.5 L/h. At \(CL=4.5\) L/h, typical \(AUC=300/4.5=66.67\) mg·h/L, so
At \(CL=5.0\) L/h, typical \(AUC=60.00\) mg·h/L and \(z=\log(60/50)/0.30=0.608\), giving \(q\approx0.728\). At \(CL=5.5\) L/h, typical \(AUC=54.55\) mg·h/L and \(z=\log(54.55/50)/0.30=0.290\), giving \(q\approx0.614\). The difference among these PTAs reflects the change in the conditional population PTA as the parameter value changes. The spread of individual exposures at each fixed \(CL\), represented here by \(\eta\), is already included in each conditional PTA.
These three values illustrate the calculation; they are not enough to form a reliable uncertainty interval. In an analysis, use many well-behaved parameter draws, and vary other uncertain parameters too when they are part of the fitted model.
Quantifying and Reporting PTA Uncertainty
A parameter-uncertainty interval should be calculated from the outer-loop distribution of conditional PTAs, not from a confidence interval for one model parameter alone. In general, PTA is a nonlinear function of several parameters, so uncertainty in multiple estimates and their correlations can jointly affect the result. The interval should state how parameter sets were generated, how many were usable, and which quantiles were reported.
There is also Monte Carlo error because each conditional PTA is estimated using a finite number \(M\) of simulated people. If \(M\) independent people are simulated at fixed parameters, the conditional Monte Carlo standard error is approximately
This simulation error can be reduced by increasing \(M\). It is not parameter-estimation uncertainty and should not be presented as such. Increasing \(M\) cannot compensate for an inadequate number of informative estimation-data resamples or an unreliable parameter covariance estimate.
Worked Example: Separating an Interval From Monte Carlo Error
Suppose a hypothetical analysis uses 1,000 bootstrap refits. At one dose, the 2.5th and 97.5th percentiles of the 1,000 conditional PTA estimates are 0.69 and 0.85, with median 0.78. These are reported as a bootstrap 95% interval for the estimated PTA under the stated resampling and model assumptions: 0.69 to 0.85. They describe uncertainty in the PTA estimate, not a range containing 95% of individual patients' outcomes.
For each parameter set, suppose 10,000 individuals are simulated and a representative conditional PTA is 0.78. Its inner-loop Monte Carlo standard error is
A rough 95% Monte Carlo range around that single conditional estimate is \(0.78\pm1.96(0.00414)\), or approximately 0.772 to 0.788. That range describes simulation noise for the fixed parameter set, not the 0.69-to-0.85 bootstrap interval. The two intervals answer different questions and should not be substituted for one another.
Worked Example: Applying a Pre-Specified Dose Criterion
Suppose a hypothetical dose analysis requires a PTA of at least 0.90 and uses 1,000 valid bootstrap refits. The table shows the 5th percentile, median, and 95th percentile of conditional PTA across refits, and the number of refits with conditional PTA at least 0.90.
| Dose | 5th percentile | Median PTA | 95th percentile | Refits with PTA ≥ 0.90 |
|---|---|---|---|---|
| 400 mg | 0.75 | 0.84 | 0.91 | 80 of 1,000 |
| 600 mg | 0.87 | 0.93 | 0.97 | 820 of 1,000 |
For 400 mg, the fraction of refits meeting the specified PTA criterion is \(80/1000=0.080\), or 8.0%. For 600 mg, it is \(820/1000=0.820\), or 82.0%. Under a decision rule requiring at least 90% of bootstrap refits to have conditional PTA of 0.90 or higher, neither candidate meets the rule. The 600 mg dose has stronger support by this summary, but the reported 82.0% is a fraction of bootstrap refits, not a probability that the true PTA exceeds 0.90. The decision also remains conditional on the model and simulation assumptions.
Diagnostics, Alternatives, and Limitations
Before interpreting the outer-loop PTA distribution, examine whether it is credible. Check that model refits converged, that parameter values obey model constraints, and that random-effects covariance matrices are valid. Review the distribution of each key parameter and of the conditional PTAs for skewness, extreme values, or clusters. A small set of extreme PTAs can dominate a mean; quantiles and plots across parameter draws can make such behavior visible. If many bootstrap refits fail or estimates repeatedly hit boundaries, a nominal percentile interval may be unstable rather than reassuringly precise.
The main alternatives have different trade-offs:
| Approach | What it provides | Main caution |
|---|---|---|
| Plug-in simulation at \(\hat{\theta}\) | A conditional PTA for one fitted parameter set | Does not quantify parameter-estimation uncertainty |
| Approximate parameter draws | A fast approximation to PTA uncertainty using an estimate and covariance matrix | Can be unreliable near boundaries or when the sampling distribution is strongly non-normal |
| Bootstrap refitting | A resampling-based distribution that propagates estimation variation through the full simulation | Requires adequate resamples, successful refits, and a bootstrap scheme appropriate to the data and estimation procedure |
None of these approaches includes every source of uncertainty automatically. In particular, parameter draws do not represent uncertainty about model structure, the target population, the endpoint threshold, or the adequacy of the data. Do not silently treat a parameter-uncertainty interval as if it covered those sources. State what is held fixed, what is varied, and how the interval was obtained.
Common Mistakes
- Treating outer parameter draws as extra patients. They are alternative plausible fitted population parameter sets, not independent individuals from one fixed population model. Calculate a conditional PTA within each set before summarizing across sets.
- Calling the spread of simulated exposures parameter uncertainty. Exposure variation among people at fixed parameters is between-person variability. Parameter uncertainty concerns the variation in the conditional PTA across plausible parameter values.
- Reporting a plug-in PTA as though it were precise. A large number of inner-loop simulated individuals reduces Monte Carlo error but does not remove uncertainty in estimated population parameters.
- Ignoring invalid parameter draws. Negative variance estimates or non-positive-definite covariance matrices are not usable model parameter sets. Resolve the generation or estimation problem; do not quietly retain impossible draws.
- Over-interpreting bootstrap fractions. The fraction of resamples above a PTA threshold summarizes support under the chosen bootstrap procedure. It is not the probability that the true PTA exceeds that threshold.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a nested simulation, what does the inner loop represent, and what does the outer loop represent?
- A conditional PTA is 0.70 and is estimated using 5,000 independent simulated people. Calculate its approximate Monte Carlo standard error.
- Why is a 95% bootstrap interval for PTA not an interval containing 95% of individual patients' outcomes?
- For a candidate dose, 600 of 1,000 bootstrap refits have conditional PTA at least 0.90. State the fraction and explain what it does and does not mean.
Solutions. (1) The inner loop simulates individuals and their relevant variability conditional on a parameter set; the outer loop represents uncertainty across plausible parameter estimates. (2) \(SE_{\mathrm{MC}}=\sqrt{0.70(0.30)/5000}=\sqrt{0.000042}\approx0.00648\). (3) The bootstrap interval summarizes uncertainty in the estimated conditional population PTA across resamples; individual outcomes vary within each fixed-parameter population and are not the estimand of that interval. (4) The fraction is \(600/1000=0.600\), or 60.0%. It summarizes how often the bootstrap refits yield PTA at least 0.90 under the chosen procedure; it is not a 60% probability that the true PTA is at least 0.90.