Separating Parameter and Preference Uncertainty

Why Separate the Sources of Uncertainty?

A benefit-risk result can be uncertain for different reasons. Clinical outcome parameters—such as treatment effects or event probabilities—may be uncertain because the available evidence is limited. Preference weights may also be uncertain because patients or decision makers have not specified their trade-offs precisely. Combining these sources into one distribution can summarize overall uncertainty, but it does not show which source contributes more.

Separating the sources can help clarify what additional information would be most useful. More clinical data may reduce uncertainty about outcome parameters; better preference elicitation may reduce uncertainty about weights. These are different activities, and neither necessarily resolves the other. The distinction is also important when interpreting a result: uncertainty about the average preference weight is not the same as genuine variation in preferences among individuals.

The result \(D\) could be an incremental expected utility or another defined score. The decomposition describes uncertainty in that chosen result under the specified distributions for \(\theta\) and \(w\). It does not, by itself, establish that a treatment causes a clinical outcome, nor does it account for sources of uncertainty omitted from those distributions.

The Variance Decomposition

Suppose the joint uncertainty distribution for \((\theta,w)\) is specified. Conditional on \(\theta\), define the mean result over preference weights as \(m(\theta)=E[D\mid\theta]\). The conditional variance \(\operatorname{Var}(D\mid\theta)\) measures the remaining variation in the result when \(\theta\) is held fixed and preference uncertainty is varied.

The first term is the variation, across clinical parameter draws, in the result averaged over preference uncertainty. Under this conditioning order, it is the parameter-uncertainty component. The second term is the average variation across preference draws at fixed clinical parameters, so it is the preference-uncertainty component. These labels describe the chosen decomposition, not an intrinsic property of the result independent of how the joint distribution is specified.

This identity follows by adding and subtracting the conditional mean. Write \(D-E[D]=\{D-E[D\mid\theta]\}+\{E[D\mid\theta]-E[D]\}\). When the square is taken and averaged, the cross-product has expectation zero: conditional on \(\theta\), \(E[D-E[D\mid\theta]\mid\theta]=0\), while \(E[D\mid\theta]-E[D]\) is fixed. The two remaining squared terms give the two components.

The identity applies whether \(\theta\) and \(w\) are independent or dependent, provided the conditional distribution of \(w\mid\theta\) is the one used to define the simulation. If they are dependent, preference draws must come from the appropriate conditional distribution; sampling weights independently would instead impose a different uncertainty model. If they are independent, the same preference distribution can be used at each outer draw.

Implementing Nested Simulation

Draw \(K\) outer values \(\theta_k\) from the specified distribution for clinical parameters. For each outer draw, generate \(L\) inner preference draws \(w_{k\ell}\), from the specified distribution of \(w\mid\theta_k\), and evaluate \(D_{k\ell}=D(\theta_k,w_{k\ell})\). The inner results estimate the conditional mean and variance for that outer draw.

With equally weighted independent outer draws, the average of the \(\widehat v_k\) values estimates the preference component. The sample variance of \(\widehat m_k\) estimates the parameter component, but contains some inner simulation noise: each \(\widehat m_k\) is only an estimate of \(E[D\mid\theta_k]\). When \(L\) is the same for all outer draws, a common correction is

where \(s_{\widehat m}^{\,2}\) is the sample variance of the \(K\) conditional-mean estimates. The subtraction accounts for the estimated variance of the inner-simulation error in each \(\widehat m_k\). With enough inner draws, that correction may be small. A negative corrected estimate can occur from Monte Carlo error, especially when \(K\) or \(L\) is small; it is a warning to increase simulation precision, not evidence that a true variance component is negative.

Decomposing a Decision Probability

A decision maker may focus on whether the result is positive rather than on its numerical variance. Define \(Y=1\) when \(D>0\), and \(Y=0\) otherwise. For each clinical draw, the inner simulation estimates \(p(\theta)=P(D>0\mid\theta)\). The overall probability of a positive result is \(E_{\theta}[p(\theta)]\).

The same total-variance identity can be applied to this indicator. Since a binary variable with success probability \(p\) has variance \(p(1-p)\), the between-parameter component is \(\operatorname{Var}_{\theta}(p(\theta))\), and the preference component is \(E_{\theta}[p(\theta)(1-p(\theta))]\). This separates variation in conditional decision probabilities across clinical draws from uncertainty about the decision across preference draws at fixed clinical parameters. It is a decomposition of the variance of \(Y\), not a decomposition of the probability itself into additive source-specific probabilities.

Assumptions, Diagnostics, and Alternatives

A useful decomposition depends on a well-defined uncertainty model. The clinical draws should represent uncertainty in the outcome parameters relevant to \(D\), and the preference draws should represent uncertainty in the specified weights. If weights must satisfy constraints—such as nonnegativity and a fixed sum—the simulation must respect those constraints. Preference uncertainty should not be created by independently sampling weights in a way that produces impossible combinations.

Nested simulation also requires a clearly stated conditioning order. The decomposition above conditions on \(\theta\) first and averages over \(w\mid\theta\). Reversing the order gives a mathematically valid alternative decomposition, but the assigned components can differ when the result combines parameters and weights non-additively, or when the sources are dependent. Therefore, a report should state which order and joint distribution were used. The components are an attribution under that model, not unique causal shares of uncertainty.

Check that the component estimates and the total result are stable when the number of outer and inner draws increases. Small \(L\) can make conditional means noisy; small \(K\) can make between-draw estimates unstable. For decision probabilities, too few inner draws can also make estimated conditional probabilities jump between zero, one, and intermediate values. Examine the conditional results, not only their pooled summary, and verify that the treatment of ties at the decision threshold is explicit.

A single simulation that samples \((\theta,w)\) jointly can estimate the distribution of \(D\) and its overall variance, but does not automatically identify the two components. One-at-a-time sensitivity analysis varies one input while holding others fixed; it is useful for exploring scenarios, but generally does not quantify the variance contributions under a joint uncertainty distribution. Nested simulation provides that variance-based attribution under its stated model, at the cost of more computation and dependence on the chosen distributions.

A further limitation is that a variance decomposition can conceal the shape of the result distribution. Two sources can contribute the same variance while affecting tail probabilities or the chance of crossing a decision threshold differently. Report the overall distribution or relevant decision summaries alongside the components, and describe which uncertainty distributions and result scale were used. The decomposition is conditional on those choices and does not repair misspecification of the clinical or preference models.