Why Compare Regimens on the Same Draw?
A PBPK comparison of two dosing regimens should account for uncertainty in the inputs shared by both predictions. For example, clearance, enzyme abundance, and other uncertain parameters may affect the AUC predicted under either regimen. If one regimen is evaluated using one set of parameter draws and the other regimen using a different set, differences between their predictions reflect both the regimen change and the mismatch in sampled inputs. Applying both regimens to each same draw isolates the comparison more directly.
This is a simulation pairing: each sampled set of uncertain inputs is used to generate an output under regimen A and an output under regimen B. It does not, by itself, mean that the simulations represent observed paired patients, nor does it establish a causal effect in a clinical population. The comparison describes what the PBPK model predicts across the uncertainty distribution supplied to it.
The paired differences \(D_1,\ldots,D_n\) form a simulated distribution of regimen contrasts. Its mean summarizes the average difference over the sampled uncertainty distribution; its median and quantiles describe other features of that distribution. The paired ratios \(R_i\) describe relative change on a fold scale. The direction must be stated: a positive \(D_i\) means regimen A gives a larger output than regimen B, while \(R_i>1\) means the same for a positive output.
How Pairing Improves Precision
Suppose the goal is to estimate the mean difference in model-predicted output under the input uncertainty distribution. With independent draws \(\theta_i\) from that distribution, the Monte Carlo estimator is the mean of the paired differences. Its precision depends on the variability of those differences, not simply on the separate variability of the outputs under each regimen.
Here \(s_D\) is the sample standard deviation of the paired differences. For a comparison based on independent regimen-specific draws, the variance of a difference between outputs is the sum of their variances. For paired outputs, it also depends on their covariance. The variance identity gives the reason pairing can help:
If both regimens respond similarly to the same uncertain inputs, their outputs tend to move together and their covariance is positive. Subtracting the covariance term reduces the variability of the paired differences. Pairing does not guarantee a precision gain: if the outputs have little correlation, the gain may be small; if their covariance is negative, the paired differences can be more variable than an independent comparison. The calculation should therefore use the observed paired differences rather than assume a benefit.
Generate a draw \(\theta_i\) from the intended joint uncertainty distribution, preserving any specified dependence among inputs.
Use the same \(\theta_i\) for each regimen and record the two outputs with consistent units and endpoint definitions.
Calculate \(D_i=Y_{Ai}-Y_{Bi}\), or \(R_i=Y_{Ai}/Y_{Bi}\) when a ratio is meaningful.
Report the chosen summaries of the differences or ratios and, for Monte Carlo estimates, assess their numerical stability.
Worked Example: Mean Paired AUC Difference
Suppose a hypothetical PBPK analysis samples six uncertain parameter sets. Both regimens are evaluated at every set. The resulting AUC values are in mg·h/L.
| Draw \(i\) | Regimen A AUC | Regimen B AUC | Difference \(D_i=A_i-B_i\) |
|---|---|---|---|
| 1 | 120 | 100 | 20 |
| 2 | 150 | 125 | 25 |
| 3 | 90 | 80 | 10 |
| 4 | 180 | 150 | 30 |
| 5 | 110 | 95 | 15 |
| 6 | 160 | 130 | 30 |
The mean paired difference is
The deviations from 21.67 are approximately \(-1.67,\ 3.33,\ -11.67,\ 8.33,\ -6.67,\ 8.33\). Their squared values sum to about 333.33, so
For these six draws, regimen A's simulated mean AUC exceeds regimen B's by 21.67 mg·h/L. The reported standard error describes Monte Carlo variability in estimating the mean contrast under the stated independent-draw setup; it is not the spread of individual predictions, which is represented by \(s_D\). Six draws are also too few to establish reliable tail quantiles.
The precision advantage is visible in these hypothetical data because the outputs are strongly aligned across draws. The sample variances are \(s_A^2=1150\) and \(s_B^2=676.7\), in squared AUC units. Treating the two sets of outputs as independent would give an estimated standard error for the difference of means of
The paired standard error from the actual within-draw differences is 3.333 mg·h/L. The contrast in this example is unusually strong because the same parameter changes move the two regimen outputs together. The independent-draw calculation discards that alignment and is not the appropriate standard error for the paired estimator.
Ratios and Decision-Relevant Summaries
Differences are useful when an absolute change has a clear interpretation in the endpoint's units. Ratios are useful when relative change is more meaningful, but they require positive denominators and can become unstable when regimen B's output is near zero. A ratio distribution should be formed draw by draw; the ratio of the two marginal means is generally not the mean of the paired ratios. If relative change is summarized on a multiplicative scale, \(\log(R_i)\) can also be examined, then interpreted by exponentiating a summary on the log scale.
Worked Example: Interpreting Paired AUC Ratios
In a second hypothetical analysis, five matched input draws produce the following AUC pairs, in mg·h/L: \((132,110)\), \((156,130)\), \((90,100)\), \((180,150)\), and \((126,105)\), where each pair lists regimen A then regimen B.
The paired ratios are calculated separately for each draw:
Their arithmetic mean is \((1.20+1.20+0.900+1.20+1.20)/5=1.14\), and their median is 1.20. Thus, in this finite set of draws, the mean paired ratio is 1.14 and the median paired ratio is 1.20. Four of five draws have regimen A AUC 20% higher than regimen B, while one has regimen A AUC 10% lower. These summaries describe the simulated paired ratios; they do not imply that every plausible input set favors regimen A.
The ratio of the marginal means would instead be \((132+156+90+180+126)/(110+130+100+150+105)=684/595=1.150\). That number is a different summary from the mean of the paired ratios, 1.14. State which quantity is being reported rather than treating them as interchangeable.
Worked Example: Probability of Exceeding an AUC Difference Threshold
Suppose eight hypothetical paired simulations yield differences, in mg·h/L, of \(12,18,21,24,26,29,34,\) and \(40\). A decision team asks how often regimen A's AUC exceeds regimen B's by more than 20 mg·h/L in the simulated uncertainty distribution.
Six of the eight values exceed 20, so the empirical Monte Carlo estimate is
In this sample, 75.0% of paired draws exceed the stated threshold. This is a simulation-based estimate of a probability under the specified input distribution and model. It is not the probability that the model is correct, nor does it establish a clinical benefit. With only eight draws, the estimate is coarse and should not be presented as precise.
Assumptions, Diagnostics, and Alternatives
A paired comparison is meaningful only when the two outputs at each index really correspond to the same draw from the intended joint input distribution. The model implementation must also use consistent endpoint definitions, units, simulation horizons, and numerical settings. If the model includes stochastic elements beyond the sampled parameters, decide whether the comparison targets those elements too. When appropriate, using common random numbers for both regimens can preserve pairing of simulation noise; otherwise independent stochastic noise can obscure the input-level contrast.
For the standard error formula \(s_D/\sqrt{n}\), the paired differences should be independent draws from the target distribution. If a sampling design deliberately induces dependence between draws, such as some stratified designs, the ordinary formula may not describe Monte Carlo error correctly; use an error assessment compatible with that design. The distribution of the \(D_i\) values themselves remains the relevant simulated distribution, but its numerical precision needs suitable diagnostics.
Useful checks include confirming that both regimens were run for every parameter draw, checking for missing or duplicated indices, comparing the marginal output summaries with the corresponding paired outputs, and plotting or tabulating the paired differences against the draw index. Large trends by index can signal an ordering or implementation problem, especially when draws were expected to be exchangeable. Repeat the calculation with a larger simulation run or independent reproducible runs to assess stability of the mean, spread, and decision-relevant quantiles or probabilities. For ratios, inspect small denominators and unusually large values; these may reflect genuine model behavior or an unstable comparison.
| Approach | What it summarizes | Main caution |
|---|---|---|
| Paired differences or ratios | Draw-level contrast using the same uncertain inputs for both regimens | Requires correct matching and a clearly specified contrast |
| Separate marginal summaries | Each regimen's prediction distribution considered on its own | Subtracting marginal means loses the draw-level pairing and does not describe the distribution of differences |
| Independent Monte Carlo runs | Each regimen evaluated on separate draws | Can add avoidable sampling noise to a comparison when shared inputs drive both outputs |
A useful identity is that the difference between the two marginal sample means equals the mean of the paired differences when both regimens use the same \(n\) matched draws:
This equality concerns the point estimate of the mean contrast. It does not make separate marginal intervals a valid interval for the paired difference, because those marginal intervals do not retain the covariance information. Nor does subtracting two marginal quantiles generally yield a quantile of the paired differences.
Common Pitfall and Limits
Wrong reasoning: “The 90th percentile of regimen A AUC minus the 10th percentile of regimen B AUC is a conservative 90th percentile of the paired AUC difference.” This calculation combines marginal extremes that may come from different parameter draws, so it does not preserve the pairings and has no general quantile interpretation for \(D\).
Correction: First calculate \(D_i=Y_{Ai}-Y_{Bi}\) for every matched draw; then compute the desired summary or quantile from the resulting \(D_i\) values. The paired distribution is tied to the specified joint uncertainty distribution and model. It may not include all sources of clinical variation, and a narrow Monte Carlo error for its mean does not imply that individual predictions are narrowly distributed or that the model has been validated.
Pairing is a computational comparison strategy, not a substitute for deciding which inputs are uncertain, whether their dependence is represented adequately, or whether the model is appropriate for the question. It improves precision when common input variation produces positively correlated outputs, but it cannot repair biased model assumptions, incorrect input distributions, or inadequate numerical convergence.
Check Your Understanding
Answer each question using the paired-comparison definitions and conditions above.
- For each draw, regimen A AUC is 140 mg·h/L and regimen B AUC is 115 mg·h/L. What are the paired difference and ratio, and how should each be interpreted?
- Why can using the same parameter draws reduce the Monte Carlo standard error of a mean AUC difference?
- For four paired differences of \(4, 8, 10,\) and \(18\) mg·h/L, calculate the mean difference and the empirical proportion exceeding 9 mg·h/L.
- Why is the difference between the 90th percentiles of two marginal AUC distributions not generally the 90th percentile of the paired AUC difference?
Solutions. (1) \(D=140-115=25\) mg·h/L, so regimen A is higher by 25 mg·h/L on this draw; \(R=140/115=1.217\), so regimen A is about 21.7% higher relative to regimen B. (2) When the same inputs move both outputs in the same direction, their covariance is positive; the paired-difference variance subtracts twice that covariance, reducing variability. (3) The mean is \((4+8+10+18)/4=10\) mg·h/L. Two values exceed 9, so the empirical proportion is \(2/4=0.500\). (4) Marginal percentiles need not correspond to the same parameter draw. The paired-difference percentile must be calculated from the draw-by-draw differences themselves.