Worked Example: Uncertainty in HbA1c Response

From Model Uncertainty to an HbA1c Prediction

A treatment simulation does not produce one certain HbA1c response. Even for a fixed regimen and a defined patient group, uncertainty in glucose-insulin model parameters changes the predicted response. Patients also differ in characteristics and physiology that affect their individual trajectories. The practical question is therefore not only “What is the predicted response?” but also “How widely might responses vary, and what fraction of patients might reach a specified target?”

This tutorial follows a hypothetical 24-week treatment simulation. HbA1c is expressed as a percentage, but changes in HbA1c are reported in percentage points: a change from 8.2% to 7.0% is a decrease of 1.2 percentage points. A target-attainment probability is meaningful only after specifying the treatment regimen, time point, target, patient population, and uncertainty sources included.

Write \(\theta\) for uncertain glucose-insulin model parameters and \(\eta\) for patient-level characteristics or random effects. With data \(D\) used to estimate parameters, a predictive distribution for a new patient can be represented as

Here \(x\) denotes the patient population or covariates being predicted. The expression says to average conditional predictions over parameter uncertainty and the specified patient distribution. If the model includes residual variability in the outcome, that variability belongs in the conditional outcome distribution as well. Include it only when the prediction concerns an outcome that is meant to contain that variation; do not add it again if the simulation output already represents it.

In computation, draw \(\theta_m\) from the joint parameter uncertainty distribution and draw \(\eta_{mi}\) from the patient distribution. Simulate the treatment response for each combination. Preserving the joint parameter distribution matters: parameter estimates may be correlated, and independent marginal draws can create combinations that are implausible under the fitted model. If baseline HbA1c varies across patients, simulate baseline and response consistently, then apply the target to \(A_0+\Delta A\). A response change alone does not determine target attainment when baseline varies.

Why the Predictive Spread Has Multiple Sources

Even without residual outcome noise, a simulated response varies because both \(\theta\) and \(\eta\) vary. The following variance identity helps explain the roles of these sources. Conditional on \(\theta\), average over patient variability; then average the conditional variance and the variation in the conditional mean over parameter uncertainty:

This follows by adding and subtracting \(E(\Delta A\mid\theta,D)\) inside \(\Delta A-E(\Delta A\mid D)\). The cross-product has expectation zero conditional on \(\theta\), because \(E[\Delta A-E(\Delta A\mid\theta,D)\mid\theta,D]=0\). The first term captures variation remaining among patients at a given parameter draw; the second captures variation in the conditional mean across parameter draws. With additional random components, the conditional variance also includes their contribution and any dependence specified by the model.

The identity explains why a prediction at the mean parameter values is not generally a substitute for a predictive distribution. A nonlinear QSP model may respond differently at the average parameter vector than the average of responses over parameter draws. Further, a single “typical patient” does not represent the distribution of individual outcomes. The simulation should run the model for draws from the intended joint distributions.

Estimating Target Attainment and Its Simulation Error

For \(N\) simulated patients, define \(I_i=1\) when patient \(i\) meets the target and \(I_i=0\) otherwise. The Monte Carlo estimate of target probability is the fraction of simulated outcomes meeting the target. If the simulated outcomes are independent draws from the intended predictive distribution, the approximate Monte Carlo standard error is the binomial standard error shown below.

This standard error concerns finite simulation size: it measures how much the estimated probability would vary across repeated Monte Carlo runs of size \(N\). It does not measure the predictive variation in individual HbA1c responses. If draws are dependent—for example, because patients share an outer parameter draw—the independent-binomial formula may not apply directly. Use the simulation design, such as independent replicate runs or appropriate grouping by parameter draw, to assess numerical stability.

Joint Parameter Uncertainty: A Numerical Check

The dependence among uncertain parameters can substantially change response spread. This is especially important when parameters jointly control processes such as glucose handling and insulin action. The next calculation uses a deliberately simplified linear representation of a QSP simulation output to isolate that point. It is not a replacement for running a nonlinear mechanistic model.

Diagnostics, Alternatives, and Limitations

Before interpreting a simulated HbA1c distribution, verify that it corresponds to the intended prediction. Check the regimen, simulation time, baseline definition, patient population, and target inequality. Confirm that parameter draws preserve estimated dependence and plausible constraints, and that patient draws reflect the covariates or subgroup distribution of interest. Inspect the simulated response distribution for implausible values, unexpected truncation, excessive concentration at a boundary, or a shape that makes a reported mean and standard deviation unrepresentative. A target probability can look stable while the distribution’s tails remain poorly sampled, so inspect the number of target events and repeat the simulation with larger or independent runs when precision is inadequate.

There are several common alternatives, each answering a different question. A plug-in simulation at estimated mean parameters gives a conditional prediction for one parameter vector; it omits parameter uncertainty and may not equal the predictive mean in a nonlinear model. A delta-method approximation uses local derivatives to approximate uncertainty near a reference parameter vector; it can be computationally efficient but may be inaccurate for strongly nonlinear responses or threshold probabilities. A full Monte Carlo simulation can represent nonlinearities, joint parameter dependence, and patient heterogeneity, but it requires adequate sampling and sound inputs. A population-average response is not the same as an individual predictive distribution, and a probability for an individual target is not automatically a trial-level probability.

The output is conditional on model structure, estimated parameter uncertainty, patient distribution, and simulation assumptions. It does not account for an uncertainty source that was not represented. In particular, a very precise Monte Carlo estimate cannot repair a misspecified model or an unrepresentative patient population. Report what the simulation includes, the time point and target definition, the number and structure of draws, and the Monte Carlo precision. Interpret the result as a model-based prediction for the stated scenario, not as a guarantee about any individual patient.

Common Pitfalls

• Using independent parameter draws by default. This can destroy dependence in the fitted parameter distribution and distort response spread. Draw from the joint distribution or use a sampling method that preserves its dependence.
• Reporting the response mean as the target probability. A mean change does not say how many patients meet a threshold. Apply the target condition to each simulated outcome and count the qualifying outcomes.
• Confusing predictive spread with Monte Carlo error. The response standard deviation describes variation across predicted outcomes; the Monte Carlo standard error describes numerical uncertainty in an estimated summary such as target probability.
• Applying a change threshold without checking baseline. When baseline HbA1c varies, target attainment depends on the joint values of baseline and change. Simulate or otherwise account for both.
• Calling a model-based probability a clinical guarantee. It is conditional on the model and represented uncertainties, not a promise that a specified fraction of future patients will respond.