Propagating Parameter Uncertainty Into Trial Outcomes

Why Parameter Uncertainty Changes Trial Predictions

A trial simulation can represent variation among participants while holding every model parameter fixed. Its results then answer a conditional question: what outcomes does the model predict if these parameter values are correct? That is useful, but it does not answer how likely the trial is to meet a decision criterion when the parameter values themselves are uncertain.

For example, a model may predict a high response rate at one plausible treatment effect and a low response rate at another. Averaging those predictions requires more than substituting the average treatment effect into the model: the probability of a trial-level event is often a nonlinear function of the parameters. The parameter distribution and the trial outcome model must therefore be propagated together.

Definition: Let $\boldsymbol{\theta}$ denote the uncertain model parameters, let $q(\boldsymbol{\theta})$ be a specified distribution representing uncertainty about them, and let $A$ be a trial outcome or decision event. A fixed-parameter decision probability is $P(A\mid\boldsymbol{\theta})$ at stated parameter values. The uncertainty-averaged decision probability is $$ P(A)=\int P(A\mid\boldsymbol{\theta})q(\boldsymbol{\theta})\,d\boldsymbol{\theta}. $$ It is a probability under the stated parameter-uncertainty distribution and trial model.

The distribution $q(\boldsymbol{\theta})$ needs a defensible interpretation. It may, for example, be a posterior distribution informed by existing data. Alternatively, it may encode explicit uncertainty weights across plausible parameter scenarios. In the latter case, the resulting probability is conditional on those weights; it is not automatically a probability established by frequentist sampling theory. State the source, scale, and dependence structure of parameter uncertainty rather than treating parameter draws as an unexamined computational convenience.

What Is Drawn, and What Is Held Fixed?

A fixed-parameter simulation sets $\boldsymbol{\theta}$ once, then simulates trial participants and outcomes under those values. Repeating the trial under the same parameter values estimates a conditional operating characteristic. A parameter-uncertainty simulation adds an outer step: draw $\boldsymbol{\theta}$ from $q$, then simulate a trial conditional on that draw. The inner trial simulation represents outcome variation given the parameters; the outer draw represents uncertainty about the parameters.

This distinction is not the same as drawing different participant-level characteristics within each simulated trial. Participant variability is part of the conditional trial model: it describes heterogeneity among participants when $\boldsymbol{\theta}$ is fixed. Parameter uncertainty concerns what values the shared model parameters may plausibly take. The two sources can both affect trial outcomes, but they answer different questions and should be represented separately.

Specify the uncertainty.
Define $\boldsymbol{\theta}$, its uncertainty distribution $q(\boldsymbol{\theta})$, and the evidence or scenario rationale supporting that distribution.

Define the trial and its decision event.
Hold the design, analysis rule, and other specified inputs fixed. State precisely what counts as event $A$, such as meeting a prespecified success criterion.

Propagate uncertainty through the trial model.
For each parameter draw, generate outcomes conditional on that draw and determine whether $A$ occurs. Average the conditional probabilities, or the event indicators across draws, according to the simulation design.

Report what the result is conditional on.
Describe the parameter distribution and decision rule alongside the probability. A numerical result without those conditions is difficult to interpret or reproduce.

The calculation follows the law of total probability. For a discrete uncertainty distribution with parameter scenarios $\boldsymbol{\theta}_j$ and weights $w_j$, where $w_j\geq 0$ and $\sum_j w_j=1$, it becomes

P(A)=\sum_j w_jP(A\mid\boldsymbol{\theta}_j).

For a continuous distribution, the corresponding integral averages over its full support. The key operation is to average the conditional probability of the event. In general, that is not equivalent to calculating the probability at the average parameter value.

Worked Decision-Probability Calculations

Worked Example: A Response-Count Success Criterion

Hypothetical data and assumptions. Consider a trial with 10 participants. Conditional on a shared response probability $p$, participant responses are independent Bernoulli outcomes with that probability, so the response count $X$ has a binomial distribution. Trial success is defined as at least 8 responses. Suppose parameter uncertainty is represented by two scenarios: $p=0.80$ with weight $0.50$, and $p=0.40$ with weight $0.50$. These weights are stipulated for illustration, not estimated from a real study.

At fixed $p$, $P(X\geq 8\mid p)$ is the sum of the binomial probabilities for $X=8,9,10$:

P(X\geq 8\mid p)=\binom{10}{8}p^8(1-p)^2+\binom{10}{9}p^9(1-p)+p^{10}.

For $p=0.80$, substitution gives

P(X\geq 8\mid p=0.80) =45(0.80)^8(0.20)^2+10(0.80)^9(0.20)+(0.80)^{10} =0.30199+0.26844+0.10737 =0.67780.

For $p=0.40$, the same calculation gives

P(X\geq 8\mid p=0.40) =45(0.40)^8(0.60)^2+10(0.40)^9(0.60)+(0.40)^{10} =0.01062+0.00157+0.00010 =0.01229.

The uncertainty-averaged probability of trial success is therefore

P(X\geq 8)=0.50(0.67780)+0.50(0.01229)=0.34505.

If instead the simulation fixed $p$ at its uncertainty-weighted mean, $0.50(0.80)+0.50(0.40)=0.60$, the conditional success probability would be $P(X\geq 8\mid p=0.60)=0.16729$. The difference arises because the event probability is nonlinear in $p$. Under the stated scenario weights, the chance of reaching the response-count criterion is about 34.5%, not 16.7%. Neither number is a general property of the trial apart from the stated assumptions and parameter specification.

Worked Example: Probability of Exceeding an Effect Threshold

Hypothetical assumptions. Let $\hat{\delta}$ be a trial estimate of a treatment effect, measured in percentage points. Suppose that conditional on the true effect $\delta$, $\hat{\delta}\sim N(\delta,0.25^2)$, so the standard error is 0.25 percentage points. Define success as $\hat{\delta}>0.20$ percentage points. Represent uncertainty about $\delta$ by two equally weighted scenarios, 0.30 and 0.80 percentage points.

Under a fixed effect $\delta$, the conditional success probability is

P(\hat{\delta}>0.20\mid\delta) =1-\Phi\left(\frac{0.20-\delta}{0.25}\right),

where $\Phi$ is the standard normal cumulative distribution function. At $\delta=0.30$, the standardized threshold is $(0.20-0.30)/0.25=-0.40$, so the probability is $\Phi(0.40)=0.6554$. At $\delta=0.80$, it is $\Phi(2.40)=0.9918$. Averaging across the two scenarios gives

P(\hat{\delta}>0.20) =0.50(0.6554)+0.50(0.9918) =0.8236.

The average effect under these weights is 0.55 percentage points. Fixing $\delta=0.55$ gives $\Phi((0.55-0.20)/0.25)=\Phi(1.40)=0.9192$. The 82.36% uncertainty-averaged probability is lower than the 91.92% plug-in probability, because the low-effect scenario contributes a substantial chance of missing the threshold. The result is a probability of exceeding this particular estimate threshold, not a probability that the treatment effect itself exceeds 0.20.

Variance Decomposition and a Predictive Calculation

The effect of parameter uncertainty on a continuous trial outcome can be made explicit. Let $Y$ be an outcome generated by the trial model. If it has finite variance, the law of total variance gives

\operatorname{Var}(Y) =E_{\boldsymbol{\theta}}\!\left[\operatorname{Var}(Y\mid\boldsymbol{\theta})\right] +\operatorname{Var}_{\boldsymbol{\theta}}\!\left(E[Y\mid\boldsymbol{\theta}]\right).

To justify the result, write $Y-E(Y)=[Y-E(Y\mid\boldsymbol{\theta})]+[E(Y\mid\boldsymbol{\theta})-E(Y)]$, square, and take expectations. The first squared term has expectation $E[\operatorname{Var}(Y\mid\boldsymbol{\theta})]$; the second has expectation $\operatorname{Var}(E[Y\mid\boldsymbol{\theta}])$. The cross-product has expectation zero because $E[Y-E(Y\mid\boldsymbol{\theta})\mid\boldsymbol{\theta}]=0$. Thus the predictive variance includes both conditional trial variation and variation in the conditional mean across parameter values.

Worked Example: Combining Effect Uncertainty and Estimation Error

Hypothetical assumptions. Suppose the treatment effect $\delta$, in percentage points, has uncertainty distribution $N(0.50,0.20^2)$. Conditional on $\delta$, the trial estimate satisfies $\hat{\delta}\mid\delta\sim N(\delta,0.25^2)$. Assume this conditional sampling error is independent of the parameter draw. Define success as $\hat{\delta}>0.20$.

The predictive mean of $\hat{\delta}$ is $0.50$. By the variance decomposition, its variance is the expected conditional variance plus the variance of the conditional mean:

\operatorname{Var}(\hat{\delta}) =0.25^2+0.20^2 =0.0625+0.0400 =0.1025, \qquad SD(\hat{\delta})=\sqrt{0.1025}=0.3202.

Because both distributions are normal and the conditional mean is $\delta$, the marginal distribution is $\hat{\delta}\sim N(0.50,0.1025)$. Consequently,

P(\hat{\delta}>0.20) =1-\Phi\left(\frac{0.20-0.50}{0.3202}\right) =\Phi(0.9369) \approx 0.8255.

Fixing the parameter at its mean instead would give $\Phi((0.50-0.20)/0.25)=\Phi(1.20)\approx0.8849$. Here parameter uncertainty increases the predictive standard deviation and lowers the probability of exceeding the threshold. The effect of uncertainty on a decision probability is not always downward; its direction depends on the event and on how its conditional probability changes over the uncertainty distribution.

Assumptions, Diagnostics, and Alternatives

The calculations are only as credible as their inputs. Before interpreting an uncertainty-averaged result, check that the following conditions are addressed:

The uncertainty distribution is justified. Its location, spread, support, and weights should reflect relevant evidence or be clearly identified as scenario assumptions. A narrow distribution can make uncertainty appear unimportant simply by excluding plausible parameter values.
The conditional trial model is coherent. For each parameter draw, the outcome model and decision rule must be well-defined. For example, a response probability must remain between 0 and 1, and the model must use the stated trial design.
Dependence among parameters is represented when relevant. Drawing parameters independently when their uncertainty is dependent can produce implausible combinations and distort the event probability.
The event matches the intended decision. A probability of crossing a point-estimate threshold is not interchangeable with a probability that an interval excludes a null value, a probability of clinical benefit, or another success criterion.

Useful diagnostics include plotting or tabulating $P(A\mid\boldsymbol{\theta})$ across the uncertainty distribution, checking how much each region of that distribution contributes to $P(A)$, and repeating the calculation under defensible alternative uncertainty distributions. If a small change in a tail, scenario weight, or parameter dependence changes the conclusion, report that sensitivity rather than presenting a single averaged number as robust.

Three approaches answer different questions. A fixed-parameter simulation reports conditional performance at selected values. Uncertainty averaging combines conditional performance according to an explicit distribution. A worst-case or range-based analysis asks how performance behaves over a selected set of values without assigning probabilities to them. The last can be useful when scenario weights are not defensible, but it does not itself produce an uncertainty-averaged probability.

Approach	What it reports	Main limitation
Fixed-parameter	Conditional outcome probability at specified parameter values	Does not represent uncertainty about those values
Uncertainty-averaged	Outcome probability integrated over a stated parameter distribution	Depends on the distribution and its justification
Range or worst-case	Performance across selected values, or at the least favorable value	Does not assign a probability to values unless weights are added

A common mistake is to draw parameter values and participant values in the same undifferentiated step, then describe the resulting spread as “variability.” The correction is to label each source: conditional participant and outcome variation belongs to the trial model, while uncertainty in shared model parameters belongs to $q(\boldsymbol{\theta})$. Another mistake is to use the average parameter value as if it preserved the average event probability. As the examples show, that substitution is generally invalid when $P(A\mid\boldsymbol{\theta})$ is nonlinear.

Finally, a simulation approximates the specified probability with a finite computation. That numerical approximation is distinct from parameter uncertainty: changing the uncertainty distribution changes the target probability, whereas running more simulations improves the approximation to the same target. Reports should keep those interpretations separate.

Key takeaway: To propagate parameter uncertainty, define its distribution, simulate trial outcomes conditional on parameter draws, and average the decision probability over those draws. Report the resulting probability together with the assumptions and decision rule that give it meaning.

Check Your Understanding

Answer each question, then compare with the solutions below.

In one sentence, distinguish a fixed-parameter success probability from an uncertainty-averaged success probability.
A binary outcome has probability $p$, and a trial is declared successful if at least one of two participants responds. What is the success probability at fixed $p$, and why can substituting an average $p$ fail to give the uncertainty-averaged probability?
In the variance decomposition, identify the term for conditional trial variation and the term associated with parameter uncertainty.
Why is it not valid to describe scenario weights as probabilities about parameters unless their interpretation supports that statement?

Solutions. (1) The fixed-parameter probability is $P(A\mid\boldsymbol{\theta})$ at stated values; the uncertainty-averaged probability integrates that conditional probability over $q(\boldsymbol{\theta})$. (2) Conditional independence gives $P(\text{at least one response}\mid p)=1-(1-p)^2$. Since this is nonlinear in $p$, its average over uncertain $p$ is generally not $1-(1-E[p])^2$. (3) $E_{\boldsymbol{\theta}}[\operatorname{Var}(Y\mid\boldsymbol{\theta})]$ is average conditional trial variation; $\operatorname{Var}_{\boldsymbol{\theta}}(E[Y\mid\boldsymbol{\theta}])$ is variation in conditional means across parameter uncertainty. (4) Weights encode probabilities only when they come from an uncertainty distribution with that interpretation; otherwise they define a chosen scenario average, not an objectively established probability about fixed parameters.