Simulation-Based Comparison of Estimation Strategies

Why Compare Estimation Strategies by Simulation?

Sequential and simultaneous PK/PD analyses can differ in how they carry information from concentration data into the PD parameter estimates. Their relative performance depends on the model, sampling design, noise, and parameter values. A simulation study makes those conditions explicit: generate data under a stated model, fit each strategy to the same simulated data, and summarize how closely and reliably each recovers a defined target.

A simulation does not establish that one method is universally superior. It estimates performance under the scenarios that were simulated. A useful comparison therefore begins with a precise estimand, uses realistic study designs, and reports uncertainty and failures as well as the favorable results.

Specify the Simulation Before Fitting

A practical design has four linked parts: the data-generating process, the study design, the estimation procedures, and the performance measures. Write these down before running the simulation. Otherwise, it is easy to change definitions after seeing the results or to compare methods that do not answer the same question.

For a simple hypothetical example, let subject \(i\)'s concentration after an intravenous dose be described by a one-compartment model:

Suppose observed concentrations have log-normal measurement error, so that \(\log C_{ij}^{obs}=\log C_i(t_{ij})+\epsilon_{ij}\), with \(\epsilon_{ij}\sim N(0,\sigma_{PK}^2)\). A simple increasing PD response might be \(E_i(t)=E_0+E_{\max}C_i(t)/(EC_{50}+C_i(t))\), with additive normal observation error. These equations define an illustrative data-generating process, not a universal PK/PD model. A population simulation may also include between-subject variability in PK or PD parameters. Those distributions and any correlations must be specified explicitly.

The sequential procedure first fits the PK data and then uses estimated concentrations or PK parameters in the PD analysis. The simultaneous procedure estimates PK and PD parameters together from the concentration and response data. Define the target—for example, population \(EC_{50}\)—in the same way for both procedures. If one procedure estimates a population parameter and the other reports an individual-level quantity, their numerical results are not a fair head-to-head comparison.

Use the same simulated subjects, observations, and sampling times for both methods within each replicate. This paired design removes avoidable variation due simply to one method receiving a more favorable random data set. The methods may have different fitting algorithms, but their model assumptions, likelihood contributions, parameter constraints, and rules for deriving the target should be documented.

Performance Measures and Their Interpretation

Let \(\theta\) be the true target and let \(\hat{\theta}_r\) be its estimate in replicate \(r\), for \(r=1,\ldots,R\). Calculate each summary using a clearly stated denominator. In particular, distinguish summaries conditional on successful fits from the proportion of all simulations that succeed.

Here \(\bar{\hat{\theta}}\) is the mean estimate across replicates. Bias measures systematic displacement from the truth; empirical standard error measures the across-replicate spread. If both matter, root mean squared error combines them:

The last equality follows by expanding each deviation around the replicate mean: the cross-products sum to zero because \(\sum_r(\hat{\theta}_r-\bar{\hat{\theta}})=0\). For large \(R\), RMSE is approximately the square root of squared bias plus squared empirical standard error. Lower RMSE is not automatically preferable if the estimand, failure rate, or interval behavior differs.

For a nominal 95% confidence interval, estimate coverage as the fraction of intervals containing \(\theta\). Coverage is a repeated-sampling property of the procedure under the simulated conditions; it is not the probability that a particular calculated interval contains the fixed true value. Also report interval width when relevant, since wide intervals can achieve good coverage while providing limited precision.

Convergence should be defined operationally before simulation. For example, count a fit as successful only if the optimizer reports convergence, the objective and parameter estimates are finite, and specified numerical checks (such as a usable covariance estimate) pass. Report the success proportion out of all \(R\) generated data sets. For estimate-based summaries, state whether failed fits are excluded and give the number remaining. A method that converges only on easy data sets can look artificially precise if failures disappear from the summary.

Worked Examples of Simulation Summaries

Make Sampling Scenarios Realistic and Informative

The sampling schedule is part of the simulation, not a detail to add after selecting a winner. For an IV concentration profile, early samples help characterize distribution or rapid decline when those features are in the model; later samples inform the elimination phase. A schedule with only late samples may provide little information about exposure around the PD observations. In a clinical study, PK samples may be sparse and concentrated at practical visit times, while PD measurements may occur more often or at different times.

A useful design set might include a relatively informative PK schedule and a sparse schedule representative of the intended study. Keep the PD schedule, dose, subject count, and noise assumptions clear, and vary one design feature at a time when the goal is to understand its influence. For each schedule, generate concentration and PD observations at their actual planned times rather than pretending that both measurements are collected continuously. Examine whether the simulated concentration range spans the PD response region needed to estimate the target.

A strong comparison can include a primary scenario in which the fitted model matches the data-generating model, plus sensitivity scenarios for plausible departures: greater measurement error, less informative PK sampling, or a prespecified PK-model discrepancy. These scenarios answer different questions. Correct specification evaluates performance under the assumed model; misspecification evaluates robustness to the particular departure introduced. It does not diagnose every possible error in a real analysis.

Before fitting, check that the parameter values and sampling times generate plausible concentration and response profiles, that the target is identifiable enough for the intended comparison, and that simulation errors and optimizer settings are controlled. Residual diagnostics from fitted data can reveal patterns worth representing in scenarios, but a simulation summary cannot substitute for checking the assumptions and fit of an actual analysis.

Comparison, Diagnostics, and Limitations

The sequential strategy is usually easier to modularize: PK fitting can be inspected separately, and the PD stage can use its output. Its limitation in this comparison is that plug-in quantities may fail to carry all PK estimation uncertainty into PD inference, depending on the implementation. The simultaneous strategy can account for the PK and PD likelihood contributions together, but it may be more computationally demanding and can have convergence or identifiability difficulties. Neither feature guarantees better estimates in every simulated setting.

A simulation comparison should make failure visible. Alongside numerical summaries, inspect estimate distributions, interval widths, convergence status, and whether estimates approach parameter bounds. Look for extreme estimates or systematic bias that an average alone could conceal. If numerical failures occur, repeat fits with documented alternative starting values or tolerances as a diagnostic; do not silently change the fitting procedure for only one method. Apply the same prespecified success criteria and report any changes.

Monte Carlo error also limits interpretation. Increasing \(R\) reduces the simulation noise in reported summaries, but does not repair a poorly chosen data-generating model, unrealistic sampling schedule, or incorrect estimand. A conclusion that reverses across plausible scenarios should be reported as scenario-dependent rather than compressed into a single ranking.

Common Mistake