Two Procedures, Two Questions
A simulated concentration profile can vary because subjects have different random effects, because measurements contain residual error, or because the population parameters themselves are uncertain. These are distinct sources of variation. A simulation that holds fitted population parameters fixed can represent some of them, but it cannot show how much the estimated parameters would change if the study were repeated. To assess that change, the model must be refitted to resampled or simulated data.
The distinction matters in population pharmacokinetics (PopPK) because the target may be a feature of the fitted model, such as typical clearance, or a future-data quantity, such as the distribution of trough concentrations in a specified population. The procedure should be chosen for the target, not simply because one procedure produces a wider or narrower distribution.
Here \(\theta\) denotes the model's population parameters, which can include typical PK parameters, parameters describing between-subject variability, and residual-error parameters. Individual-level random effects and residual errors are random quantities generated under the model; they are not the same as uncertainty about the fitted population parameters. A fixed-parameter simulation can generate new random effects and residual errors while keeping \(\hat{\theta}\) unchanged.
What Each Procedure Captures
In a subject-level nonparametric bootstrap, subjects are sampled with replacement, retaining each sampled subject's observations together. The model is refitted to each resampled dataset. In a parametric refit bootstrap, new data are simulated from the fitted model under the study design, and the model is refitted to each simulated dataset. These procedures differ in how replicate datasets are generated, but both use refitting to obtain replicate parameter estimates.
By contrast, a fixed-parameter simulation generates data under the fitted model and calculates a target without re-estimating the population parameters. It can represent variability in simulated outcomes conditional on the fitted parameters, including random effects and residual error if those components are simulated. It does not produce a sampling distribution of \(\hat{\theta}\): by construction, the population parameters remain at their fitted values in every replicate.
| Procedure | What varies across replicates? | What can it assess? | What it does not directly assess |
|---|---|---|---|
| Subject-level bootstrap with refitting | Which observed subjects are sampled, and the resulting fitted parameters | Sampling variability in fitted parameters under the empirical subject distribution | Variation outside the empirical distribution, unless represented by the data or design |
| Parametric bootstrap with refitting | Simulated study data and the resulting fitted parameters | Sampling variability in fitted parameters under the assumed model and design | Uncertainty due to model misspecification not represented in the simulation |
| Fixed-parameter simulation | Simulated outcomes, random effects, and measurement errors, as specified | Outcome variability conditional on the fitted parameters | Sampling variability of the fitted population parameters |
The table describes the usual procedures, not a claim that one source of variability is always absent from a particular analysis. For example, a refit bootstrap also produces variable simulated or resampled data, but its parameter estimates vary because the model is fitted anew. A fixed-parameter simulation may be nested inside a procedure that also varies parameter values; once parameter values are varied, however, it is no longer a simulation that holds them all fixed.
Worked Example: Does a Fixed-Parameter Run Estimate Clearance Uncertainty?
Suppose a hypothetical one-compartment model has a fitted typical clearance of \(\hat{CL}=8.0\) L/h. A researcher simulates 1,000 new study datasets using that fitted model, with between-subject random effects and residual error, but does not refit the model. What can the replicate results say about typical clearance?
In every replicate, the population typical clearance used to generate the data remains \(8.0\) L/h. The random effects can make individual clearances differ from that value, and residual error can change measured concentrations, but neither feature changes the fixed population parameter used by the simulation.
If the researcher reports the generating typical clearance across replicates, its replicate standard deviation is \(0\) L/h: every value is \(8.0\) L/h. That zero is imposed by the procedure, not evidence that clearance is estimated without uncertainty. To study how typical clearance estimates vary between studies, the researcher must fit the model to each replicate and examine the resulting \(\hat{CL}^*\) values.
The fixed-parameter runs can still answer a different question, such as how much simulated concentration varies among new subjects and measurements conditional on a typical clearance of \(8.0\) L/h and the other fitted parameters.
Why Refitting Adds Parameter-Estimation Variability
A refit bootstrap treats the fitted parameter as an output of an analysis procedure. In each replicate, a changed dataset is passed through the same fitting procedure, so the parameter estimate can change. The spread of estimates reflects the combined effect of the chosen data-generation or resampling scheme, the study design, and the model-fitting procedure. It is not a universal measure of every kind of uncertainty about the model.
The distinction can be expressed using the law of total variance. Suppose \(\Theta\) represents uncertainty about population parameters and \(Y\) is a future outcome. Under a specified probability model,
The first term is average outcome variability conditional on parameter values; the second is variability in the expected outcome as parameter values vary. A fixed-parameter simulation at \(\hat{\theta}\) represents conditional outcome variation at that one parameter value. It does not include the second term. This identity is exact for a distribution over \(\Theta\); a bootstrap distribution of fitted estimates is a frequentist device for assessing sampling variability, not automatically a probability distribution for the true parameter.
A refit bootstrap estimates the sampling behavior of the fitting procedure under its replicate-data scheme. If the goal is a predictive distribution that includes parameter uncertainty, a suitable approach can combine parameter uncertainty with new outcome simulation. That combination must be defined explicitly: refitting alone produces replicate fitted parameters, while simulating outcomes at one fixed estimate alone does not propagate their variability.
Worked Example: Separating Conditional and Parameter-Driven Variability
Consider a hypothetical prediction of concentration at a specified time. Across simulations at fixed fitted parameters, suppose the conditional standard deviation is \(0.80\) mg/L. Suppose an additional parameter-uncertainty analysis estimates a standard deviation of the conditional mean prediction of \(0.60\) mg/L. For illustration, assume the two components are represented by the law of total variance above.
The conditional variance is
The variance of the conditional mean across parameter values is
Thus the total variance is \(0.64+0.36=1.00\ (\mathrm{mg/L})^2\), and the total standard deviation is \(\sqrt{1.00}=1.00\) mg/L. The fixed-parameter simulations at one fitted value represent the \(0.80\) mg/L conditional component, not the full \(1.00\) mg/L.
This calculation illustrates a variance decomposition; it does not say that the standard deviations should simply be added. Nor does it guarantee that a particular refit bootstrap will estimate exactly the second component: the replicate scheme, fitting method, and prediction target determine what its output represents.
Choosing and Checking the Procedure
Start by stating the target in operational terms. If the target is the sampling variability of a population parameter estimate, use a procedure that refits the model in every replicate. A subject-level bootstrap is tied to the observed subjects and their empirical distribution; a parametric refit bootstrap is tied to the assumed model. If the target is the distribution of new outcomes conditional on the fitted model, fixed-parameter simulation may be appropriate, provided the required random effects, residual errors, covariates, and design features are simulated.
The assumptions need to match the procedure. Subject-level resampling assumes that the resampling unit is appropriate and that the observed subjects can represent the target sampling population; repeated observations within a subject should remain together. A parametric refit bootstrap assumes the fitted model is an adequate generator for the data being simulated. Both refit approaches also depend on the fitting procedure being applied consistently to each replicate. Fixed-parameter simulation assumes the chosen fitted values and simulated random components are suitable for the conditional question being asked.
Decide whether the target is a model parameter, a fitted-model prediction, or the variability of future observed outcomes.
For a bootstrap, identify the resampling unit or data-generating model. For fixed-parameter simulation, state which random effects, errors, covariates, and design features vary.
To obtain replicate estimates of population parameters, refit the model in every replicate. To assess outcomes conditional on \(\hat{\theta}\), keep those parameters fixed and label the result accordingly.
Confirm that reported variability has the units and interpretation of the estimand, and that simulated or resampled data preserve the relevant study structure.
Worked Example: Comparing Two Sets of Replicates
In a hypothetical analysis, a parametric refit bootstrap produces five typical-clearance estimates: \(7.2,\ 8.1,\ 7.6,\ 9.0,\ 8.1\) L/h. A separate fixed-parameter simulation uses \(\hat{CL}=8.0\) L/h in every replicate and reports five simulated trough concentrations: \(4.8,\ 5.1,\ 4.9,\ 5.2,\ 5.0\) mg/L. These are different targets and should not be compared as if they were two estimates of the same uncertainty.
For the refit bootstrap, the mean of the five clearance estimates is
Their sample standard deviation is
This standard deviation summarizes the spread of these replicate clearance estimates. It is based on only five illustrative replicates and is not offered as a stable uncertainty estimate. In the fixed-parameter run, typical clearance is \(8.0\) L/h by construction; the varying trough concentrations instead describe simulated outcome variability conditional on the fixed parameters and the components that were simulated.
Diagnostics, Limitations, and a Common Mistake
A useful diagnostic is to check whether each replicate output could vary under the procedure as implemented. If every fixed-parameter replicate reports the same population parameter, that is expected; presenting that lack of variation as evidence of precise estimation is a mistake. Conversely, if a purported refit bootstrap produces identical parameter estimates in every replicate, check whether refitting actually occurred, whether datasets changed, and whether the fitting code returned replicate-specific estimates.
For either procedure, inspect whether the simulated or resampled data preserve the study design and the subject-level structure relevant to the target. Fixed-parameter simulations can be misleading if random effects or residual error are omitted when the target concerns observed outcomes. Parametric refit bootstraps can be misleading if the generating model fails to represent important features of the data. Subject-level bootstraps can be limited when the observed sample poorly represents the target population or contains few subjects in important subgroups. Refitting also makes the result dependent on the fitting procedure; results from replicate fits should be interpreted in light of the same model and estimation approach used for the original data.
Common mistake: “The fixed-parameter simulations show a narrow distribution of concentrations, so typical clearance is precisely estimated.” This conclusion confuses conditional outcome variability with sampling uncertainty in an estimated parameter. The correction is to state that the concentration distribution is conditional on the fitted parameters, then use a refit procedure if the target is the sampling variability of typical clearance.
Neither approach automatically captures every form of uncertainty. In particular, a bootstrap under one chosen model does not by itself quantify uncertainty from choosing among different model structures. A fixed-parameter simulation does not propagate uncertainty in the fitted parameters unless parameter values are varied as part of a separate, explicitly defined procedure. These limitations make precise labeling of the target and procedure essential.
Check Your Understanding
Answer each question, then compare with the solutions.
- A researcher simulates new subjects using fitted population parameters, random effects, and residual error, but never refits the model. Does the simulation estimate the sampling variability of typical clearance?
- For estimating the sampling variability of a population parameter, what essential step distinguishes a refit bootstrap from fixed-parameter simulation?
- A fixed-parameter simulation produces a narrow distribution of predicted concentrations. Name one conclusion it supports and one conclusion it does not support by itself.
- Why should repeated observations from one subject generally remain together when that subject is selected in a subject-level bootstrap?
Solutions. (1) No. Typical clearance remains fixed at its generating value, so the simulation does not produce replicate clearance estimates. (2) Fit the model anew to each resampled or simulated replicate dataset and retain the resulting parameter estimates. (3) It supports a statement about outcome variability conditional on the fitted parameters and the simulated random components; by itself, it does not establish that the population parameters are estimated precisely. (4) Those observations belong to the same sampling unit and may be correlated; separating them would alter the within-subject data structure represented by the bootstrap.