Including Parameter Uncertainty in Predictive Checks
See how uncertainty in fitted population parameters can widen, shift, or reshape predictive reference distributions, and how to account for it without confusing it with biological variability.
Intermediate
12 min read
What You'll Learn
Distinguish a predictive check conditional on fitted parameter estimates from one that propagates parameter uncertainty.
Derive predictive means and variances using the law of total expectation and variance.
Calculate how parameter uncertainty changes a predictive interval and a tail probability.
See how uncertain model parameters can induce dependence across a concentration profile.
Identify practical diagnostics, limitations, and alternative ways to construct parameter-uncertainty distributions.
Why Parameter Uncertainty Changes a Predictive Check
A predictive check compares observed data with data the fitted model could generate under a specified design. The reference distribution depends not only on the model’s residual and between-subject variability, but also on how the fitted parameters are handled. If every replicate uses exactly the fitted parameter estimates, the check is conditional on those estimates. If parameter uncertainty is propagated, replicate predictions also reflect plausible variation in the parameter values.
These are different reference distributions and answer different questions. A conditional check asks whether the data look unusual if the fitted parameter values are treated as fixed. An uncertainty-propagating check asks whether they look unusual after allowing for uncertainty in those values as well. The second distribution is often broader, but it can also change shape, dependence, or location; it is not necessarily just a wider version of the first.
Definition: Let \(\theta\) denote the vector of population-model parameters, \(D\) the observed data, and \(Y^{\mathrm{rep}}\) a replicate dataset generated under the model and the design being checked. A conditional predictive distribution uses a fixed estimate \(\hat{\theta}\), such as \(p(Y^{\mathrm{rep}}\mid\hat{\theta})\). A parameter-uncertainty-propagating predictive distribution averages predictions over a distribution \(q(\theta\mid D)\) representing parameter uncertainty:
$$
p(Y^{\mathrm{rep}}\mid D)=\int p(Y^{\mathrm{rep}}\mid\theta)\,q(\theta\mid D)\,d\theta.
$$
The meaning of \(q(\theta\mid D)\) depends on the inferential framework. In a Bayesian analysis it may be the posterior distribution of the parameters. In a likelihood-based or other frequentist analysis it may be an approximation constructed from parameter-estimation uncertainty, or from an appropriate resampling procedure. The calculations below use an explicitly specified uncertainty distribution so that the predictive consequences can be seen directly. Averaging over such a distribution does not, by itself, make a frequentist parameter a random quantity in the same sense as a Bayesian parameter.
What Is Held Fixed, and What Is Simulated?
A population PK prediction generally involves several sources of variation. The model parameters determine such features as typical clearance, the distribution of individual clearances, and residual-error parameters. Simulations may also include between-subject random effects and residual measurement error. Parameter uncertainty is a separate source: it reflects limited information about the values of the model parameters themselves.
For a conditional check, set \(\theta=\hat{\theta}\) for every replicate, then simulate the other random components required by the prediction target. For an uncertainty-propagating check, first draw a parameter vector \(\theta^{(b)}\) from \(q(\theta\mid D)\) for replicate \(b\), then simulate the remaining model components conditional on that draw. Use the same dose, sampling times, covariates, and other design features as the data being checked. Whether subject-level random effects are redrawn or conditioned on depends on whether the target is prediction for new subjects or for the observed subjects.
The distinction matters because a model can have substantial biological variability but precisely estimated population parameters, or relatively modest biological variability but poorly estimated parameters. Treating one source as if it represented the other can give a misleading reference distribution. For instance, redrawing individual random effects does not account for uncertainty in the estimated typical clearance.
A Short Derivation: Predictive Variance
The effect of parameter uncertainty follows from the law of total expectation and the law of total variance. For a scalar predictive quantity \(Y^{\mathrm{rep}}\), conditional on \(\theta\),
$$
E(Y^{\mathrm{rep}}\mid D)=E_{\theta\mid D}\left[E(Y^{\mathrm{rep}}\mid\theta)\right],
$$
$$
\operatorname{Var}(Y^{\mathrm{rep}}\mid D)
=E_{\theta\mid D}\left[\operatorname{Var}(Y^{\mathrm{rep}}\mid\theta)\right]
+\operatorname{Var}_{\theta\mid D}\left[E(Y^{\mathrm{rep}}\mid\theta)\right].
$$
The first variance term averages the within-parameter predictive variance, including whichever random effects and residual error are part of the prediction. The second is the variance, across plausible parameter values, of the conditional predictive mean. To justify the decomposition, write \(Y^{\mathrm{rep}}-E(Y^{\mathrm{rep}}\mid D)\) as the sum of \(Y^{\mathrm{rep}}-E(Y^{\mathrm{rep}}\mid\theta)\) and \(E(Y^{\mathrm{rep}}\mid\theta)-E(Y^{\mathrm{rep}}\mid D)\). The cross-product has expectation zero because the first term has conditional expectation zero given \(\theta\). Taking the expectation of the squared sum therefore gives the two variance terms above.
Thus, if uncertainty in \(\theta\) changes the conditional mean, it contributes additional predictive variance. If the conditional mean is nonlinear in \(\theta\), the integrated distribution may also be skewed or multimodal even when predictions conditional on each parameter draw are simple. Consequently, uncertainty propagation should be performed on the prediction scale through simulation or integration, rather than assumed to produce a particular distributional shape.
Worked Example: A Predictive Interval With Parameter Uncertainty
Suppose a hypothetical concentration prediction \(Y^{\mathrm{rep}}\), in mg/L, has conditional model \(Y^{\mathrm{rep}}\mid\theta\sim N(\theta,4)\). The parameter uncertainty approximation is \(\theta\mid D\sim N(10,1)\), with both variances measured in \((\mathrm{mg/L})^2\). Compare the conditional and uncertainty-propagating 95% intervals.
Conditional check. Fixing \(\theta\) at its estimate, 10 mg/L, gives \(Y^{\mathrm{rep}}\mid\hat{\theta}\sim N(10,4)\). Its standard deviation is \(2\) mg/L, so the approximate central 95% interval is
\(10\pm1.96(2)=[6.080,13.92]\) mg/L.
Uncertainty-propagating check. The conditional mean is \(\theta\), and the conditional variance is 4. The predictive mean is \(E(\theta\mid D)=10\), while the predictive variance is \(4+1=5\). In this normal-normal example, the integrated distribution is exactly \(N(10,5)\). Its standard deviation is \(\sqrt{5}=2.236\) mg/L, giving
\(10\pm1.96(2.236)=[5.617,14.38]\) mg/L.
The propagated interval is wider because it includes uncertainty in the mean parameter as well as the conditional predictive variation. The interval is a reference interval for replicate predictions under the stated model and uncertainty distribution; it is not a 95% probability statement that the fixed, unknown parameter lies inside this particular interval.
Parameter Uncertainty Can Change Tail Checks
The practical effect is not limited to interval width. Tail probabilities and apparent discrepancies can change because the observed value is compared with a different reference distribution. A point that is unusual under a narrow plug-in distribution may be less unusual after parameter uncertainty is included. The calculation must use the predictive distribution corresponding to the chosen check, not a conditional distribution paired with an uncertainty-propagating interpretation.
Worked Example: A Concentration Above a Threshold
Consider a hypothetical prediction at one sampling time. Let \(Y^{\mathrm{rep}}\mid\theta\sim N(\theta,9)\), in which the conditional residual standard deviation is 3 mg/L. Suppose the uncertainty distribution is \(\theta\mid D\sim N(10,4)\), so the standard deviation of the uncertain mean is 2 mg/L. The observed concentration is 16 mg/L. Calculate the upper-tail probability under each reference distribution.
With the parameter fixed at \(\hat{\theta}=10\) mg/L, the standardized value is
\((16-10)/3=2.000\). The conditional upper-tail probability is \(P(Z>2.000)\approx0.0228\), where \(Z\sim N(0,1)\).
After propagating uncertainty, the predictive variance is \(9+4=13\;(\mathrm{mg/L})^2\), and the predictive standard deviation is \(\sqrt{13}=3.606\) mg/L. The standardized value is \(6/\sqrt{13}=1.664\), giving an upper-tail probability of approximately \(0.0480\).
Under the conditional check, 16 mg/L is a more extreme upper-tail value. Under the uncertainty-propagating check, it is still in the upper tail, but less unusual relative to the broader predictive reference. Neither calculation changes the observed concentration; the difference comes from the reference distribution used.
Uncertain Parameters Can Induce Dependence Across a Profile
For a vector of predictions, parameter uncertainty can add covariance between sampling times or endpoints. Even if the conditional residual errors are independent, a shared uncertain parameter can move several predicted concentrations together. In vector notation, the total covariance is
$$
\operatorname{Cov}(Y^{\mathrm{rep}}\mid D)
=E_{\theta\mid D}\left[\operatorname{Cov}(Y^{\mathrm{rep}}\mid\theta)\right]
+\operatorname{Cov}_{\theta\mid D}\left[E(Y^{\mathrm{rep}}\mid\theta)\right].
$$
The second term is a covariance matrix, not merely an increment to each marginal variance. It can therefore change the joint reference distribution of a concentration profile. This is important when a discrepancy concerns a pattern across multiple times rather than one concentration in isolation.
Worked Example: A Shared Uncertain Scale Parameter
Suppose two hypothetical concentrations have conditional mean vector \((\theta,0.5\theta)\) mg/L and conditional covariance matrix equal to the \(2\times2\) identity matrix in \((\mathrm{mg/L})^2\). Let \(\theta\mid D\sim N(10,4)\), where the mean is in mg/L and the variance is in \((\mathrm{mg/L})^2\). Compare a joint discrepancy for observed values \((13,5)\) mg/L under the plug-in and integrated distributions.
The plug-in mean vector is \((10,5)\) mg/L and the plug-in covariance is the identity matrix. The observed-minus-predicted vector is \(d=(3,0)\) mg/L. Its squared Mahalanobis distance is
\(Q=d^\mathsf{T}I^{-1}d=3^2=9\).
For the integrated distribution, the mean remains \((10,5)\) mg/L. The uncertainty in \(\theta\) contributes covariance
\(4(1,0.5)^\mathsf{T}(1,0.5)\), so the total covariance is
\(\Sigma=\begin{pmatrix}5&2\\2&2\end{pmatrix}\;(\mathrm{mg/L})^2\).
Its determinant is \(5(2)-2(2)=6\), and
\(\Sigma^{-1}=\frac{1}{6}\begin{pmatrix}2&-2\\-2&5\end{pmatrix}\).
Therefore \(Q=d^\mathsf{T}\Sigma^{-1}d=9(2/6)=3\).
In this example the integrated vector is exactly bivariate normal. With the corresponding known mean and covariance, the squared Mahalanobis distance has a \(\chi^2\) distribution with 2 degrees of freedom. Its upper-tail probability is \(e^{-Q/2}\): it is \(e^{-4.5}\approx0.0111\) for the plug-in calculation and \(e^{-1.5}\approx0.2231\) for the integrated calculation. The joint discrepancy is much less extreme relative to the integrated reference. These values rely on the specified Gaussian model and known reference covariance; estimated or non-normal reference distributions need not yield this exact calibration.
Implementation, Diagnostics, and Alternatives
A practical simulation scheme keeps the predictive target explicit. For each replicate, preserve the design being evaluated, draw a parameter vector if uncertainty is to be propagated, simulate the relevant individual-level and residual components, and calculate the same summary or discrepancy used for the observed data. For a conditional check, omit the parameter-draw step and use \(\hat{\theta}\) throughout. If parameter vectors are drawn, draw the vector jointly so that estimated parameter correlations are retained; drawing each component independently can create implausible combinations and distort the predictions.
1
Specify the target. State whether predictions concern new subjects or observed subjects, and fix the dose, covariates, sampling schedule, and summary being checked.
2
Choose the parameter treatment. Use fixed estimates for a conditional check, or draw jointly from a justified parameter-uncertainty distribution for an uncertainty-propagating check.
3
Simulate and compare. Generate the remaining model variation and compare the observed summary with the reference distribution from the same simulation procedure.
Useful diagnostics include examining parameter draws for implausible values, boundary violations, extreme predictions, or strong dependence; checking that the resulting predictive summaries are stable across enough replicates; and comparing conditional with uncertainty-propagating references to see which conclusions change. A large difference may indicate that parameter estimation is a material part of predictive uncertainty. It may also reveal unstable parameter estimates, weak identifiability, or an inappropriate approximation for \(q(\theta\mid D)\). Simulation noise should be assessed separately: a noisy estimated tail probability is not evidence of parameter uncertainty.
Common failure modes include propagating uncertainty in only one parameter while ignoring important correlated parameters; sampling transformed parameters on an inappropriate scale; and treating the resulting distribution as automatically calibrated. For positive parameters, for example, a normal approximation on the natural scale can assign probability to negative values. A transformation or another justified uncertainty representation may be needed. A further mistake is to label random-effect variation as parameter uncertainty. Random effects describe differences among individuals under the model; parameter uncertainty describes uncertainty about model quantities such as typical clearance or the magnitude of between-subject variability.
Approach
Reference distribution
Useful interpretation
Main caution
Conditional plug-in
Simulate with \(\theta=\hat{\theta}\) fixed.
Model fit conditional on the reported estimates.
Can understate predictive spread when parameter uncertainty matters.
Parameter-draw prediction
Average conditional predictions over \(q(\theta\mid D)\).
Predictions that include the specified parameter uncertainty.
Results depend on whether the uncertainty distribution is well calibrated.
Resampling and refitting
Use estimates from resampled datasets to represent estimation variability, then generate predictions as specified.
Can reflect estimation behavior, including some nonlinearity.
Resampling and refitting choices affect the result and may be computationally demanding.
No single approach is universally preferable. A conditional check can be useful for isolating how the fitted model behaves at its reported estimates. Propagating parameter uncertainty is important when the intended prediction should reflect uncertainty from estimation, especially when estimates are imprecise or the prediction is sensitive to them. Neither approach repairs model misspecification: an uncertainty-propagating reference can still be wrong if the structural, random-effects, residual-error, or design assumptions are inadequate. The assumptions and uncertainty representation should therefore be reported along with the check.
Common Mistake
Wrong reasoning: “The model already simulates between-subject variability and residual error, so its predictive checks automatically include uncertainty in the fitted parameters.” These are distinct sources of variation. Simulating random effects and residual errors while fixing all parameter estimates produces a conditional reference distribution.
Correction: To include parameter uncertainty, vary the parameter vector according to a defensible uncertainty distribution as part of each predictive replicate, then simulate the other components conditional on that draw. State which components were varied and interpret the observed discrepancy against the reference distribution produced by that exact procedure.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
A conditional predictive variance is 16 concentration-units squared, and uncertainty in the conditional mean contributes variance 9. What is the integrated predictive variance and standard deviation?
Why can uncertainty in one shared parameter create correlation between two sampling-time predictions even when their conditional residual errors are independent?
A simulation varies individual random effects but holds every population parameter at its estimate. Does it propagate parameter uncertainty? Explain.
In the threshold example, why did the upper-tail probability change from about 0.0228 to about 0.0480?
Name one reason a normal approximation for parameter uncertainty may produce poor predictive checks.
Solutions. (1) The total variance is \(16+9=25\), so the standard deviation is \(\sqrt{25}=5\) concentration units. (2) The shared parameter moves both conditional means together; the covariance of those means across parameter draws contributes off-diagonal covariance to the integrated distribution. (3) No. It propagates between-subject variability, but fixed population estimates leave parameter uncertainty out. (4) Integrating over the uncertain mean increases the predictive variance from 9 to 13, changing the standardized value from \(2.000\) to \(6/\sqrt{13}=1.664\). (5) A normal approximation may put mass on impossible parameter values, such as negative clearance, or may poorly represent skewness or strong parameter correlation.