Stochastic Multi-Criteria Acceptability Analysis

Why propagate weight distributions?

A multi-criteria benefit-risk model combines attribute-level scores using weights that represent the relative importance of those attributes. If the weights are uncertain or vary across plausible preference profiles, a single selected weight vector can hide how strongly the treatment ranking depends on that choice. Stochastic multi-criteria acceptability analysis makes this dependence visible by propagating a specified distribution of weights through the model and recording how often each treatment ranks first.

The result is a probability conditional on the model, the treatment scores, and the chosen weight distribution. It describes how often a treatment is top-ranked across the represented weight profiles. It is not, by itself, a probability of clinical success, a probability that the treatment is best for every patient, or evidence that a preference distribution is correct.

Definition: For treatment $i$, let $U_i(\mathbf{w})$ be its multi-criteria score at weight vector $\mathbf{w}$, and let $\mathbf{W}$ have a specified joint distribution of plausible weights. The rank acceptability for first rank is $$ \phi_i^{(1)}=P\bigl(U_i(\mathbf{W}) \geq U_j(\mathbf{W})\ \text{for every treatment } j\bigr). $$ It is the probability that treatment $i$ has the highest score under that distribution and model. A rule for resolving ties is needed if ties can occur with positive probability.

Model and assumptions

For an additive multi-criteria model with $K$ attributes and $J$ treatments, write the score for treatment $i$ as

U_i(\mathbf{w})=\sum_{k=1}^{K} w_k u_{ik}, \qquad w_k\geq 0, \qquad \sum_{k=1}^{K}w_k=1.

Here $u_{ik}$ is the treatment’s score on attribute $k$, expressed on a scale where larger values are preferred, and $w_k$ is that attribute’s relative weight. The attribute scores must be defined on scales suitable for the chosen aggregation model. The same treatment scores and weight vector must be used to compare all treatments in a draw. If the specified multi-criteria model uses a different utility function, use that function in place of the additive equation; the ranking procedure is otherwise the same.

The analysis requires a defensible joint distribution for $\mathbf{W}$. Every draw must produce nonnegative weights that sum to one. Dependence between weights matters: increasing the relative importance of one attribute can require decreasing the relative importance of others. The distribution should represent the intended source of preference variation or uncertainty, such as a defined set of plausible preference profiles. It is not enough to specify separate marginal distributions that do not produce valid weight vectors.

For this tutorial, treatment scores are treated as fixed inputs and only the weights are propagated. Thus, the resulting probabilities describe sensitivity to the specified weights conditional on those scores. They do not incorporate uncertainty in clinical outcome estimates or other model inputs. This distinction matters: changing what is uncertain changes what the reported probability means.

Worked Example: An Exact Two-Attribute Calculation

Suppose two hypothetical treatments, A and B, have fixed scores on two attributes. A has scores $0.9$ and $0.2$; B has scores $0.5$ and $0.8$. Let $W_1$ be uniformly distributed from 0 to 1 and let $W_2=1-W_1$. The weights are therefore nonnegative and sum to one for every draw. Find the probability that each treatment ranks first.

Using the additive model, the scores at a given $W_1$ are

U_A=0.9W_1+0.2(1-W_1)=0.2+0.7W_1, \qquad U_B=0.5W_1+0.8(1-W_1)=0.8-0.3W_1.

A ranks first when $U_A>U_B$. Subtracting gives $U_A-U_B=W_1-0.6$, so A ranks first exactly when $W_1>0.6$. Since $W_1$ is uniform on the unit interval, $P(W_1>0.6)=1-0.6=0.4$. B ranks first when $W_1<0.6$, with probability $0.6$. The equality at $W_1=0.6$ has probability zero under this continuous distribution, so the tie rule does not affect these results.

The calculation is conditional on the stated scores and weight distribution. For example, it does not say that 40% of patients will respond to A; it says that A is top-ranked for 40% of the weight values represented by this particular distribution.

Estimating rank acceptability by simulation

When the ranking boundary is difficult to calculate directly, simulate $B$ weight vectors from the joint distribution. For every draw, calculate every treatment’s score, determine the top-ranked treatment, and increment its count. Use the same weight draw for all treatments so that the comparison within a draw reflects one coherent preference profile.

Specify the model and weight distribution.
State the score function, the fixed treatment scores, the source and support of the weight distribution, and how ties will be handled.

Draw valid weight vectors.
For each simulation draw $b$, sample $\mathbf{w}^{(b)}$ from the joint distribution, ensuring its components are nonnegative and sum to one.

Score and rank treatments.
Calculate $U_i(\mathbf{w}^{(b)})$ for every treatment $i$, then record which treatment or treatments attain the highest score.

Summarize the ranks.
Divide each treatment’s number of first-place draws by $B$. Report the resulting estimates with the simulation size and enough information to interpret the weight distribution.

Let $I_{ib}$ equal 1 if treatment $i$ ranks first on draw $b$, and 0 otherwise, using a stated tie rule. The estimated first-rank probability is

\widehat{\phi}_i^{(1)}=\frac{1}{B}\sum_{b=1}^{B} I_{ib}.

If the draws are independent samples from the specified distribution and there are no ties, each $I_{ib}$ is a Bernoulli variable with probability $\phi_i^{(1)}$. Its mean is $\phi_i^{(1)}$ and its variance is $\phi_i^{(1)}(1-\phi_i^{(1)})$. Independence across draws therefore gives

E\!\left[\widehat{\phi}_i^{(1)}\right]=\phi_i^{(1)}, \qquad \operatorname{Var}\!\left(\widehat{\phi}_i^{(1)}\right) =\frac{\phi_i^{(1)}(1-\phi_i^{(1)})}{B}.

This establishes that the simulation estimate is unbiased under those sampling conditions and that its Monte Carlo standard error can be estimated by

\widehat{SE}_{\mathrm{MC}}\!\left(\widehat{\phi}_i^{(1)}\right) = \sqrt{\frac{\widehat{\phi}_i^{(1)}(1-\widehat{\phi}_i^{(1)})}{B}}.

This standard error measures simulation noise, not uncertainty about whether the chosen scores, model, or preference distribution adequately represent the decision problem.

Worked Example: Ranking Treatments Across Weight Draws

Consider three hypothetical treatments with fixed attribute scores shown below. Six valid weight vectors are used as equally weighted illustrative draws. For each draw, the score is the sum of each weight multiplied by the corresponding attribute score.

Treatment	Attribute 1	Attribute 2	Attribute 3
A	0.8	0.3	0.6
B	0.6	0.7	0.4
C	0.5	0.5	0.9

For the first weight vector, $(0.6,0.3,0.1)$, the scores are $U_A=0.6(0.8)+0.3(0.3)+0.1(0.6)=0.63$, $U_B=0.6(0.6)+0.3(0.7)+0.1(0.4)=0.61$, and $U_C=0.6(0.5)+0.3(0.5)+0.1(0.9)=0.54$. A ranks first.

The same calculation for each vector gives these scores and winners:

Weight vector	Score A	Score B	Score C	First-ranked
(0.6, 0.3, 0.1)	0.63	0.61	0.54	A
(0.2, 0.6, 0.2)	0.46	0.62	0.58	B
(0.2, 0.2, 0.6)	0.58	0.50	0.74	C
(0.4, 0.4, 0.2)	0.56	0.60	0.58	B
(0.5, 0.1, 0.4)	0.67	0.53	0.66	A
(0.1, 0.3, 0.6)	0.53	0.51	0.74	C

Each treatment is first in two of the six draws. Thus $\widehat{\phi}_A^{(1)}=\widehat{\phi}_B^{(1)}=\widehat{\phi}_C^{(1)}=2/6=0.333$, to three significant figures. These are exact proportions for this small, specified set of draws, not reliable estimates of an unspecified continuous distribution. The example shows the bookkeeping; a real simulation should draw enough weight vectors from its stated joint distribution to achieve adequate Monte Carlo precision.

Simulation precision, interpretation, and diagnostics

Worked Example: Reporting Monte Carlo Precision

Suppose a simulation uses $B=1{,}000$ independent weight draws from a specified joint distribution. Treatment A ranks first on 420 draws, B on 350, and C on 230. There are no ties, so the counts sum to 1,000 and the estimated first-rank probabilities are 0.420, 0.350, and 0.230.

For A, the estimated Monte Carlo standard error is

\widehat{SE}_{\mathrm{MC}}= \sqrt{\frac{0.420(1-0.420)}{1{,}000}} =\sqrt{0.0002436} =0.0156.

Thus the simulation’s numerical error for A’s estimated rank acceptability is about 0.0156, or 1.56 percentage points, under the independent-draw calculation. For B it is $\sqrt{0.350(0.650)/1{,}000}=0.0151$, and for C it is $\sqrt{0.230(0.770)/1{,}000}=0.0133$. Increasing the number of draws reduces this simulation error, but does not repair a poorly chosen preference distribution or an inappropriate score model.

With no ties, the first-rank probabilities across all treatments sum to one, because every draw has exactly one winner. A sum different from one can signal a coding or reporting problem. It can also arise if a tie is split or counted in a nonstandard way; in that case, describe the tie rule and ensure the reported quantities have the intended interpretation.

Useful checks include confirming that all simulated weights are nonnegative and sum to one, that the score calculation uses the intended direction of preference for every attribute, and that each draw ranks all treatments using the same weights. Compare results across larger simulation runs or independent random seeds to see whether estimates are stable at the precision needed. A large change in estimated rank acceptability between runs suggests that more draws may be needed.

The principal alternative is to report a single ranking at a fixed, agreed weight vector. This is transparent for a specific preference profile, but says little about how the ranking changes across other plausible profiles. Scenario analysis evaluates a small, deliberately selected set of weight vectors. It can be easier to explain and helps show which profiles favor which treatment, but it does not estimate probabilities unless the scenarios are assigned defensible probabilities. Stochastic analysis summarizes a full specified distribution, but its probability estimates depend on that distribution and can conceal important subgroups or patterns if reported alone.

Rank acceptability is also different from ranking treatments by their average score. A treatment can be first in many draws by a small margin, while another can have a high average score but win less often. The first-rank probability answers how frequently a treatment wins under the specified weights; it does not measure the size of its advantage or the consequences of choosing it. Report the treatment scores or complementary summaries when those details matter to the decision.

Common pitfalls and limitations

Using invalid separate weight distributions. Independently sampling each weight can produce vectors that do not sum to one. Specify and sample a joint distribution on valid weight vectors instead.
Treating a rank probability as a clinical probability. A first-rank estimate of 0.70 means the treatment ranks first for 70% of the represented weight draws, conditional on the model. It does not mean a 70% chance of response or a 70% chance that the treatment is best in an absolute sense.
Confusing simulation precision with model credibility. A very small Monte Carlo standard error only shows that the simulation has estimated the chosen distribution’s rank probabilities precisely. It does not validate the input scores, utility model, or weight distribution.
Ignoring ties or model boundaries. Discrete weights, rounded scores, or structurally identical treatments can create ties. Predefine a tie rule and check whether it changes the reported probabilities.

The analysis is conditional on the weight distribution chosen by the analyst. Different distributions, including different dependence structures, can produce different rank acceptabilities even when the treatment scores stay fixed. A probability summary should therefore be accompanied by a clear account of how weights were specified and by sensitivity checks for reasonable alternatives. Stochastic ranking is a decision aid, not a substitute for examining the assumptions and consequences behind the model.

Check Your Understanding

Work through each question, then compare your reasoning with the solutions.

Two treatments have scores A $(0.7,0.4)$ and B $(0.5,0.8)$ on two attributes. If $W_1$ is uniform from 0 to 1 and $W_2=1-W_1$, what is the probability that A ranks first?
In 200 independent weight draws, treatment A ranks first 110 times and B ranks first 90 times, with no ties. Calculate their estimated first-rank probabilities and A’s Monte Carlo standard error.
Why is it generally invalid to draw two attribute weights independently from uniform distributions on 0 to 1 and use them directly in an additive model requiring weights to sum to one?
An analysis reports that treatment A has a first-rank probability of 0.70. State what this means and one conclusion it does not support.

Solutions. (1) $U_A=0.4+0.3W_1$ and $U_B=0.8-0.3W_1$, so A wins when $0.6W_1>0.4$, or $W_1>2/3$. Its rank acceptability is $1/3=0.333$. (2) A’s estimate is $110/200=0.550$; B’s is $90/200=0.450$. A’s Monte Carlo standard error is $\sqrt{0.550(0.450)/200}=\sqrt{0.0012375}=0.0352$. (3) The independent draws need not be nonnegative weights summing to one; they can produce invalid vectors for the stated model. Use a joint distribution supported on valid weight vectors. (4) Under the specified model and weight distribution, A ranks first in 70% of the represented weight draws. This does not imply that A has a 70% chance of clinical success or is best for 70% of patients.