Dose Ranking and Probability of Superiority

Why Rank Regimens by Paired Outcomes?

Two regimens can have similar average benefit-risk utility yet perform differently for particular members of the target population. A population-level ranking can therefore ask a direct question: for a randomly drawn virtual subject, how often does regimen A produce a better outcome than regimen B? This is a probability-of-superiority question. It complements comparisons of average utility or target-attainment rates, but it does not answer the same question.

The comparison is most interpretable when each simulated subject receives both regimens in a paired calculation. The subject’s covariates and other shared sources of variation are held consistent across the two simulated outcomes. The pairing represents a comparison of the regimens for the same target-population member; it is not an assertion that one person can receive both regimens simultaneously.

The outcome \(U\) may be a utility combining efficacy and safety, or a binary target indicator such as attainment of a prespecified pharmacodynamic target. It must be defined before simulation, and both regimens must be assessed on the same scale. For a chosen margin \(\delta\geq 0\), one may instead estimate \(P(U_A-U_B>\delta)\); this is the probability that A exceeds B by more than that margin, not merely the probability of any strict win.

The joint specification matters. Marginal distributions of \(U_A\) and \(U_B\) alone do not determine \(P(U_A>U_B)\): different ways of pairing those same marginal outcomes can yield different probabilities. A paired analysis should therefore use a scientifically defensible coupling, commonly the same virtual subject’s covariates, random effects, and shared stochastic components under both regimens. Any regimen-specific residual or event variability should be simulated according to the model’s assumptions, not forced to be identical just to create a pair.

Estimating Superiority From Paired Simulations

Suppose \(M\) independent virtual subjects are simulated, with outcomes \((U_{Ai},U_{Bi})\) for subject \(i\). Define an indicator equal to 1 when A strictly wins and 0 otherwise. The Monte Carlo estimator of \(p_A\) is the number of A wins divided by \(M\). The same counting procedure estimates \(p_B\) and \(p_T\). These are estimates of probabilities under the specified model and target population; they are not the proportion of real patients known to benefit.

The estimated net probability advantage is \(\widehat p_A-\widehat p_B\). It can be calculated directly from the paired outcomes: assign \(Z_i=1\) for an A win, \(Z_i=-1\) for a B win, and \(Z_i=0\) for a tie. Then the net advantage estimate is the sample mean of the \(Z_i\). This coding makes explicit that ties contribute neither a win nor a loss.

A short justification gives the Monte Carlo standard error. Under independent, identically distributed paired simulation draws, each A-win indicator is Bernoulli with probability \(p_A\). Its variance is \(p_A(1-p_A)\), so the variance of the mean of \(M\) such indicators is \(p_A(1-p_A)/M\). Replacing the unknown probability by its estimate gives the usual estimated Monte Carlo standard error.

This standard error measures simulation noise conditional on the model and simulation design. It does not measure uncertainty in model parameters, uncertainty about the target covariate distribution, or uncertainty about whether the model is adequate. Those sources require additional analyses and should not be represented as if they were reduced merely by increasing \(M\).

Utility, Ties, and the Meaning of the Ranking

For a continuous utility, exact ties may be uncommon, but they can occur through rounding, discrete outcome components, or a utility rule that maps a range of outcomes to the same value. For binary target indicators, ties are often frequent. Report the tie probability rather than silently excluding ties. If a conditional win rate among non-ties is also useful, define it explicitly: when \(p_A+p_B>0\), it is \(p_A/(p_A+p_B)\). That conditional quantity can appear large even when most simulated subjects tie, so it should accompany—not replace—the unconditional win and tie probabilities.

The scale also matters. A strictly increasing transformation of continuous outcomes preserves their ordering and thus preserves strict wins and losses. A strictly decreasing transformation reverses the ordering. A non-strictly increasing transformation may turn strict wins into ties. Changing target thresholds, utility weights, or tie handling can therefore change the ranking; the ranking is not an intrinsic property of the dose labels alone.

Monte Carlo Precision and a Net Comparison

The number of simulated pairs determines how precisely the probabilities are computed, conditional on the simulation model. For an estimated A-win probability, the standard error formula above applies to the paired A-win indicators. For the net advantage, use the variability of the paired values \(Z_i\); this preserves the information that wins, losses, and ties occur within the same simulated pair.

Assumptions, Diagnostics, and Alternatives

A probability-of-superiority result depends on several conditions. The simulated subjects should represent the intended target population; the NLME model and any pharmacodynamic or utility rules should generate outcomes appropriate to each regimen; and the pairing should represent a defensible same-subject comparison. The simulation replicates used for the standard error should be independent draws from the specified simulation process, or the uncertainty calculation should account for any dependence introduced by the simulation method.

Before interpreting a ranking, check that the same endpoint definition, time horizon, target threshold, and utility direction were used for both regimens. Inspect the counts of A wins, B wins, and ties, not only a single summary. Check that simulated covariates and random effects cover the intended population and that regimen-specific outcomes respond plausibly to changes in dose and exposure. Repeat the comparison with more simulation draws to see whether Monte Carlo error is small enough for the decision. If the ranking changes under reasonable parameter or structural assumptions, that is model sensitivity, not Monte Carlo noise alone.

Probability of superiority is one choice among complementary summaries. Average utility compares the magnitude of outcomes and can distinguish regimens with the same win frequency. Marginal PTA compares each regimen’s population attainment rate but does not by itself describe which subjects attain under one regimen and not the other. A paired win probability emphasizes frequency of individual-level advantage but can ignore how large a win is. A decision should therefore use the summary that matches the question, and may report several with clear definitions rather than treating any one as universally best.

A common mistake is to claim that the regimen with the higher mean utility must also have the higher probability of superiority. This is false: a few large gains can raise the mean while the regimen loses more often. Another mistake is to compute wins after independently simulating each regimen’s population and pairing rows arbitrarily. That pairing can change the estimated superiority probability because the joint distribution changes. Simulate both regimens for the same virtual subject under a stated, scientifically justified dependence structure, and report the tie rule and estimand.