Simulation-Based Comparison of Uncertainty-Aware Dose Rules

Why Compare Dose Rules by Simulation?

A dose rule is a procedure for turning available model information into a dose recommendation. Chance-constrained, robust, and risk-sensitive rules can select different doses because they treat uncertainty and harm differently. Comparing their recommendations alone is not enough: a rule that appears cautious may still have poor target attainment, and a rule with good average performance may have an unacceptable toxicity rate or vary sharply when the data or assumptions change.

Simulation makes the comparison concrete. Specify a set of plausible model scenarios, apply each rule using the information it would have at the decision point, and evaluate the resulting recommendations against outcomes generated under a stated simulation model. The comparison is only as informative as that design: using the same outcomes and metrics for each rule helps isolate differences between rules, while independent evaluation scenarios help reveal overfitting to the scenarios used to choose a dose.

Let \(d_r\) denote the dose recommended by rule \(r\). For a simulated patient or simulated patient replicate \(i\), define \(T_i(d_r)=1\) if the prespecified efficacy target is attained at that dose, and \(H_i(d_r)=1\) if the prespecified toxicity event occurs. These are simulated outcomes, not guarantees about individual patients. The target-attainment rate and toxicity rate are estimated by the corresponding sample proportions. The clinical target, toxicity definition, follow-up period, and eligible population must be held fixed across rules.

Set Up a Fair Comparison

The rules being compared should use a common candidate-dose grid, the same clinical target and toxicity definition, and the same information available at the decision time. Their decision criteria remain distinct. A chance-constrained rule selects from doses that satisfy a stated probability constraint; a robust rule evaluates performance across a prespecified set of plausible conditions, often emphasizing the least favorable condition; a risk-sensitive rule incorporates the consequences of unfavorable outcomes, for example by penalizing a lower tail of utility or assigning a larger cost to toxicity. The comparison should report the criteria used, rather than treating these labels as interchangeable.

A useful design separates two simulation tasks. In a decision stage, each rule is applied to training draws or scenarios and produces a recommendation. In an evaluation stage, those fixed recommendations are tested on fresh scenarios or simulated patients. Reusing exactly the decision-stage scenarios for evaluation can make a rule look better because it was selected to perform well on those scenarios. If the aim is to assess a rule across repeated studies, repeat the entire decision stage across simulated data sets and record the resulting recommendations.

Metrics and Monte Carlo Precision

For \(N\) simulated evaluation replicates, the estimated target-attainment rate is \(\hat p_T=\sum_{i=1}^{N}T_i(d_r)/N\), and the estimated toxicity rate is \(\hat p_H=\sum_{i=1}^{N}H_i(d_r)/N\). These rates are interpretable only with their population and simulation conditions stated. A rule can be compared on both rates without collapsing them into one score. If a utility score is also used, its components and weights should be explicit; a utility ranking cannot replace a separate safety summary.

Under independent, identically distributed Bernoulli simulation replicates for a fixed dose and scenario distribution, the Monte Carlo standard error of an estimated event rate \(\hat p\) is approximately \(\sqrt{\hat p(1-\hat p)/N}\). This quantifies finite simulation noise conditional on the simulation design. It does not quantify uncertainty about whether the assumed model, parameter distribution, or scenario set is correct.

When outcomes for two rules are generated on the same replicates, compare them as paired results. For example, define \(D_i=T_i(d_A)-T_i(d_B)\). The estimated difference in attainment is \(\bar D=N^{-1}\sum_iD_i\), and its Monte Carlo standard error is \(s_D/\sqrt{N}\), where \(s_D\) is the sample standard deviation of the replicate-level differences. Pairing is useful because patients or scenarios that are intrinsically easier or harder affect both rules, and this shared variation can cancel in the difference. The rules must still be evaluated under the same intended population and outcome-generation assumptions.

Recommendation Stability Is a Separate Outcome

Performance metrics evaluate what happens after a recommendation is made. Stability evaluates whether the rule repeatedly makes similar recommendations when the decision process is rerun under plausible variation. To assess it, repeat the decision stage across simulated data sets, parameter draws, or prespecified perturbations to assumptions. Report the frequency of each selected dose, the range or spread of recommendations, and how often no dose satisfies the rule. A small average dose difference can conceal frequent switches between clinically distinct options, so the full distribution is often more informative than one summary.

Stability is not the same as correctness. A rule that always selects one dose can be highly stable and still have poor target attainment or excessive toxicity. Conversely, a rule can have acceptable average performance but unstable recommendations near a feasibility boundary. The earlier material on diagnosing unstable dose recommendations motivates examining this behavior; here, stability is one component of a common simulation comparison.

Interpreting the Comparison and Diagnosing Failure

A useful report puts the principal metrics side by side rather than declaring a single winner from one favorable quantity. For example, if a decision maker prespecifies a maximum acceptable toxicity rate, show whether each rule's estimated rate and its simulation uncertainty are compatible with that limit, alongside target attainment and recommendation stability. If rules are compared by a utility score, show the score definition and retain the separate attainment and toxicity results so readers can see which outcomes drive the ranking.

Simulation results can mislead when the evaluation conditions are too narrow, when the same scenarios are used both to choose and assess doses, or when a large number of replicates are mistaken for evidence that the underlying model is valid. Check that simulated covariate distributions and parameter ranges are clinically defensible, that the outcome definitions match the decision question, and that the dose grid is sufficiently fine near important thresholds. Compare results across reasonable scenario sets, including assumptions that were not used to select the dose, when such variation is relevant.

The comparison also depends on the simulation architecture. Independent evaluation replicates support the simple Monte Carlo standard-error calculation above; dependent draws require accounting for their dependence. If several outcomes are generated per simulated patient, do not count them as independent patients without justification. For a paired comparison, preserve the pairing when estimating differences. Finally, a high stability frequency can be an artifact of a coarse dose grid, while apparent instability may reflect tiny numerical changes across a hard constraint boundary. Report enough detail to distinguish these cases.

Simulation is especially useful when analytic comparison is difficult and the rules respond differently to joint efficacy, toxicity, parameter uncertainty, or heterogeneity. It does not remove the need to choose defensible assumptions. The most informative conclusion is conditional: under the specified population, model scenarios, outcome definitions, and simulation design, one rule had a stated pattern of attainment, toxicity, and stability. That conclusion should be revisited if those conditions change.