Why the Representation of Parameter Uncertainty Matters
A simulation can assess a candidate design under a specified data-generating model. But model parameters such as clearance, residual error, or a treatment effect may not be known exactly. A design that performs well at one selected parameter estimate may perform less well at other plausible values. How parameter uncertainty is represented therefore determines what a simulation result can support.
A fixed-parameter analysis evaluates performance conditional on one specified parameter vector. A scenario analysis repeats this conditional evaluation at several selected vectors. An uncertainty-distribution analysis instead draws parameter vectors from a distribution intended to represent uncertainty, evaluates the design at those draws, and summarizes performance across them. These approaches answer related but distinct questions; none is automatically the right choice in every design study.
The earlier material on scenario matrices provides a way to organize alternative conditions. Here the emphasis is on what the parameter values in those conditions mean, and on the consequences of replacing a finite set of fixed values with draws from a probability distribution.
The distribution \(q(\theta)\) is not automatically a distribution of patient values. It describes uncertainty about the model parameters used to generate data. In a Bayesian analysis, it might be a posterior distribution conditional on observed data and the model. In a frequentist workflow, it might be represented using bootstrap estimates or another uncertainty approximation. The interpretation depends on how the distribution was constructed. A distribution chosen simply to create a range of stress-test values should not be presented as if it were a calibrated probability distribution.
Conditional and Averaged Design Questions
A fixed value answers a conditional question: “How does this design perform if the data-generating parameters equal \(\theta_0\)?” A scenario set answers several such questions. For example, it can show whether performance is robust across a specified range, or reveal a weak point at a clinically important combination of parameter values. Unless scenario weights have a defensible interpretation, a simple average across scenarios is only a summary of that selected set, not an estimate of performance averaged over parameter uncertainty.
When \(q(\theta)\) is defensible, an uncertainty-averaged performance quantity is
This is an average relative to the chosen \(q\), not a universal design property. A result such as \(\bar{g}_q=0.85\) means that the performance quantity averages to 0.85 over parameter values drawn according to \(q\). It does not mean performance is at least 0.85 for every parameter value, nor does it establish robustness in the tails of \(q\).
In practice, the integral is usually approximated with \(M\) parameter draws, \(\theta^{(1)},\ldots,\theta^{(M)}\):
If draws have unequal weights \(w_m\), with \(w_m \geq 0\) and \(\sum_{m=1}^{M}w_m=1\), the corresponding approximation is \(\sum_{m=1}^{M}w_mg(\theta^{(m)})\). The weights must be part of the intended uncertainty representation, rather than chosen after seeing which scenarios make a design look favorable.
Worked Example: Conditional Results Across a Fixed Scenario Set
Suppose a hypothetical design is evaluated using a success probability \(g(\theta)\). For illustration, the model has two parameters: clearance \(CL\), in L/h, and volume \(V\), in L. A team evaluates four prespecified combinations. The resulting success probabilities, estimated with sufficiently many simulation replicates for this example, are shown below.
| Scenario | \(CL\) (L/h) | \(V\) (L) | Estimated success probability |
|---|---|---|---|
| 1 | 4 | 18 | 0.92 |
| 2 | 4 | 22 | 0.88 |
| 3 | 6 | 18 | 0.82 |
| 4 | 6 | 22 | 0.78 |
At the fixed parameter vector \((CL,V)=(4\text{ L/h},18\text{ L})\), the estimated success probability is 0.92. Across the four scenarios, the minimum is 0.78 and the maximum is 0.92. These are conditional results, so they show how this design performs at the specified combinations.
If, solely for illustration, the four scenarios are assigned equal weights, their average is
The value 0.850 is an average over these four equally weighted points. It is not an uncertainty-averaged probability unless those points and weights are an appropriate representation of parameter uncertainty. The range from 0.78 to 0.92 also conveys information that the average alone hides.
Drawing Parameters and Summarizing Performance
A parameter-draw approach is useful when the uncertainty distribution itself is meaningful for the question. For each draw, the data-generating model uses one complete parameter vector. The design is evaluated conditional on that vector; performance is then summarized across vectors. The parameter draw is not a new patient-level random effect. It represents uncertainty about which parameter values should be used to describe the data-generating process.
A simple justification for averaging follows from the law of total expectation. Let \(Y\) be a simulated design outcome, such as an indicator that a prespecified success criterion is met. Conditional on \(\theta\), suppose \(E(Y\mid\theta)=g(\theta)\). Then
Thus, averaging conditional performance over \(q\) gives the marginal expected performance when parameter uncertainty is part of the evaluation. This interpretation requires that \(q\) be the distribution intended for the question and that the conditional simulation correctly represent outcomes under each \(\theta\).
The variation in simulated outcomes and the variation in conditional performance across parameter values are distinct. The law of total variance gives
The first term describes average outcome variation conditional on the parameters. The second describes how the conditional mean outcome changes over parameter uncertainty. Reporting only an overall average can conceal substantial variation in the second term. This decomposition is also a reminder not to describe parameter uncertainty as if it were the same thing as the spread of outcomes within a fixed-parameter simulation.
Worked Example: Averaging Performance Over Parameter Draws
Suppose a hypothetical uncertainty distribution for model parameters is represented by four equally weighted draws. At each draw, the estimated conditional probability of design success is 0.91, 0.87, 0.83, and 0.75. Treat these conditional probabilities as known for this arithmetic example; in an actual study they would themselves be estimated by simulation.
The estimated mean performance over the four draws is
For these four equally weighted draws, the sample standard deviation of the conditional success probabilities is
The mean of 0.840 describes average success probability over the specified draw distribution. The standard deviation of 0.06831 describes spread in conditional success probabilities across these draws; it is not the standard error of the average and does not include Monte Carlo error from estimating each conditional probability. Four draws are far too few to establish a stable summary for most practical uncertainty distributions. The example illustrates the calculation, not a recommended draw count.
Dependence, Constraints, and the Choice of Distribution
A parameter vector often contains related quantities. Drawing each parameter independently from its marginal distribution can produce combinations that are rare, inconsistent with the fitted model, or outside the parameter space. For example, estimates of two parameters may be strongly correlated. Ignoring that dependence changes the joint distribution of \(\theta\), and therefore can change the distribution of \(g(\theta)\) and its average.
Parameter constraints require similar care. A clearance or variance cannot be negative. If a distribution is specified on a transformed scale, such as \(\log(CL)\), draws should be made on that scale and transformed back consistently. Arbitrarily truncating draws after sampling can alter the distribution and its implied dependence. The uncertainty representation should preserve the constraints and relationships implied by the model or estimation procedure.
Worked Example: Why Parameter Correlation Changes a Summary
Consider two standardized parameter deviations, \(X\) and \(Z\), each with mean 0 and variance 1. Suppose the parameter uncertainty representation gives them correlation \(-0.80\). For a hypothetical performance-related quantity \(H=X+Z\), its variance is
Because \(\operatorname{Cov}(X,Z)=\operatorname{Corr}(X,Z)\sqrt{\operatorname{Var}(X)\operatorname{Var}(Z)}=-0.80\), the variance under the correlated representation is
If the same marginals were drawn independently, the covariance would be 0, giving \(\operatorname{Var}(H)=1+1=2.00\). Independence would therefore imply five times the variance for this sum. The calculation does not assert that a particular pharmacometric performance measure is a sum of two parameters; it shows directly why discarding parameter dependence can materially change uncertainty in a quantity that depends on both.
In an applied simulation, use joint parameter draws that retain the relevant covariance, for example draws from an estimated joint distribution or a suitable resampling procedure. Check that generated values remain within valid parameter ranges and that the draws reflect the intended source of uncertainty.
Practical Checks and Alternative Summaries
Before interpreting an uncertainty-averaged result, examine both the uncertainty representation and the resulting performance summaries. A single average should not replace inspection of conditional results. Useful checks include:
- Confirm that the parameter uncertainty distribution comes from a defensible source and matches the population, model, and purpose of the design evaluation.
- Check marginal ranges, parameter correlations, transformations, and constraints. Investigate implausible or invalid draws rather than silently discarding them.
- Inspect conditional performance over the draws, including low-performing regions and tail behavior, not only the overall mean.
- Assess whether conclusions change under reasonable alternative uncertainty distributions or fixed stress-test scenarios.
- Keep uncertainty about the parameters distinct from simulation noise. Finite simulation replicates can make each conditional performance estimate imprecise, even if the parameter draws are adequate.
The main alternatives serve different purposes. A point estimate gives a clear conditional benchmark but can be unrepresentative if uncertainty is material. A scenario set makes conditional behavior at chosen values transparent and can target clinically important extremes, but does not automatically assign probabilities to those values. Distributional averaging provides an overall summary relative to an uncertainty distribution, but may hide poor performance in parts of that distribution and depends on the distribution being credible. A strong evaluation can report an average alongside conditional ranges, selected stress cases, or a lower-tail summary when those summaries match the design question.
Parameter-draw results also have limits. They do not account automatically for uncertainty in model structure, omitted mechanisms, or the uncertainty distribution itself. A narrow distribution can produce a reassuring average while excluding important plausible values; a broad but poorly justified distribution can obscure the design question. Sensitivity analysis across defensible representations is therefore often more informative than treating one distribution as definitive.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A design has success probabilities of 0.90 and 0.70 at two fixed parameter scenarios. What does their unweighted average mean, and what does it not establish?
- In an uncertainty-draw analysis, why should correlated parameters generally be drawn jointly rather than independently?
- Suppose \(E(Y\mid\theta)=g(\theta)\). State the expression for marginal expected performance over \(q(\theta)\).
- What is the difference between variability in outcomes conditional on fixed parameters and variation in conditional mean performance across parameter draws?
- A parameter must be positive, but some independent normal draws are negative. Give two concerns with this approach.
Solutions. (1) The unweighted average is \((0.90+0.70)/2=0.80\), the mean of the two reported conditional results under equal weights. It does not establish an uncertainty-averaged success probability or performance at unlisted parameter values. (2) Independent draws erase covariance and can change the joint combinations of values, potentially distorting performance summaries or producing implausible combinations. (3) \(E(Y)=E_q\{E(Y\mid\theta)\}=E_q[g(\theta)]\). (4) Conditional outcome variability is \(\operatorname{Var}(Y\mid\theta)\); variation across parameter values in conditional mean performance is \(\operatorname{Var}_q\{E(Y\mid\theta)\}\). They are separate components of total variance. (5) Negative values violate the parameter constraint, and independent normal sampling may also fail to represent the intended marginal distribution or dependence. A constrained or transformed-scale joint distribution should be considered.