Go-No-Go Assurance Under Uncertain Inputs

From Performance at One Input to Assurance Across Inputs

A design can meet a performance target under one assumed input configuration and fail it under another. For example, a design might have adequate power when recruitment is prompt but fall below its target when accrual is slow. A single operating-characteristic calculation answers a conditional question: how does the design perform at these particular inputs? Go-no-go assurance asks a different question: across a specified distribution of plausible inputs, how often does the design meet a prespecified performance target?

This probability depends on the uncertainty distribution or scenario weights used. It is not automatically a probability that the design will succeed in practice, nor a posterior probability that the design is adequate. It is a probability calculated under an explicitly stated representation of uncertainty. The earlier material on scenario weighting and average performance distinguishes a weighted average performance from a probability of crossing a threshold; here the threshold event is the object of interest.

Definition: Let $d$ denote a trial design, $\theta$ the uncertain input parameters, and $g_d(\theta)$ a prespecified performance measure for design $d$ conditional on $\theta$. For a performance cutoff $c$ and an uncertainty distribution $\pi$ for $\theta$, the assurance is $$ A_d = P_{\theta\sim\pi}\{g_d(\theta)\ge c\} =\int I\{g_d(\theta)\ge c\}\,\pi(d\theta), $$ where $I\{\cdot\}$ is 1 when its condition is true and 0 otherwise. A go-no-go rule may compare $A_d$ with a separate minimum assurance level $a_0$.

The two cutoffs serve different purposes. The performance cutoff $c$ says what counts as adequate conditional performance—for example, power of at least 0.80. The assurance threshold $a_0$ says how often that performance must be achieved across uncertain inputs—for example, in at least 0.70 of the specified input distribution. Thus a rule such as “go if $A_d\ge 0.70$” is not the same as requiring average power to be at least 0.70.

Assumptions and Interpretation

The calculation requires a defined design, a scalar performance measure, a cutoff, and a probability distribution for the uncertain inputs. The performance measure must be meaningful at each input value being considered. For instance, $g_d(\theta)$ could be the probability of rejecting $H_0$ under a specified alternative, conditional on nuisance parameters and operational inputs collected in $\theta$. The performance measure and cutoff should be chosen before the assurance calculation is used for a decision.

The distribution $\pi$ may come from elicited plausibility weights, an agreed distribution for uncertain parameters, or a probability distribution representing a stated inferential uncertainty. Its interpretation must be made explicit. If the inputs are represented by a finite scenario set, let scenario $j$ have inputs $\theta_j$ and weight $w_j$, where $w_j\ge 0$ and $\sum_j w_j=1$. Then

A_d=\sum_{j=1}^{J} w_j I\{g_d(\theta_j)\ge c\}.

These weights need not be equal. Equal weights are justified only if the scenarios are intended to have equal probability under the uncertainty representation. If the weights are judgments rather than empirically established probabilities, assurance remains a useful decision summary, but its interpretation is conditional on those judgments.

The performance threshold is inclusive in this definition: a value exactly equal to $c$ counts as meeting the target. If a decision policy instead requires performance strictly above the cutoff, replace $\ge$ with $>$ and use that convention consistently. A discrete scenario grid can make this boundary choice consequential.

Worked Example: Assurance From Weighted Scenarios

Suppose a hypothetical adaptive design has a prespecified target of power at least $c=0.80$. Four input scenarios have conditional powers 0.86, 0.82, 0.74, and 0.91, with respective weights 0.20, 0.30, 0.30, and 0.20. Calculate assurance and distinguish it from weighted average power.

The first, second, and fourth scenarios meet the target; the third does not. Therefore,

$$ A_d=(0.20)(1)+(0.30)(1)+(0.30)(0)+(0.20)(1)=0.70. $$

Under these scenario weights, the design meets the power target in 70% of the represented input distribution. If the minimum acceptable assurance is $a_0=0.65$, the point calculation clears that rule; if $a_0=0.75$, it does not.

The weighted average power is a different quantity:

$$ (0.20)(0.86)+(0.30)(0.82)+(0.30)(0.74)+(0.20)(0.91) =0.172+0.246+0.222+0.182=0.822. $$

Thus average power is 0.822 while assurance is 0.70. The average describes the weighted mean performance, in power units; assurance describes the weight of scenarios that individually clear the 0.80 cutoff.

Estimating Assurance by Simulation

When the input distribution is continuous or has too many scenarios to enumerate, draw $M$ independent input values $\theta_1,\ldots,\theta_M$ from $\pi$. If $g_d(\theta_m)$ can be evaluated for each draw, define $Z_m=I\{g_d(\theta_m)\ge c\}$. Each $Z_m$ is a Bernoulli random variable with success probability $A_d$. The Monte Carlo estimator is the fraction of input draws that meet the target:

\widehat{A}_d=\frac{1}{M}\sum_{m=1}^{M}Z_m.

Because the draws are independent, $E(\widehat{A}_d)=A_d$ and $\operatorname{Var}(\widehat{A}_d)=A_d(1-A_d)/M$. Replacing the unknown $A_d$ by $\widehat{A}_d$ gives the estimated Monte Carlo standard error,

\widehat{SE}(\widehat{A}_d) =\sqrt{\frac{\widehat{A}_d(1-\widehat{A}_d)}{M}}.

This standard error describes the numerical variability from using a finite number of input draws to estimate assurance under the chosen $\pi$. It does not quantify uncertainty about whether $\pi$ itself is a suitable representation of uncertainty. That is a modeling and governance question, not something made small simply by increasing $M$.

Worked Example: Monte Carlo Go-No-Go Assessment

In a hypothetical analysis, $M=1{,}000$ independent input draws are generated from a prespecified uncertainty distribution. The design meets a single performance target on 760 draws. Suppose the minimum acceptable assurance is $a_0=0.70$. Estimate assurance, calculate its Monte Carlo standard error, and assess the result.

The estimated assurance is

\widehat{A}_d=\frac{760}{1{,}000}=0.760.

Its estimated Monte Carlo standard error is

\widehat{SE} =\sqrt{\frac{(0.760)(1-0.760)}{1{,}000}} =\sqrt{\frac{0.1824}{1{,}000}} \approx 0.0135.

An approximate 95% Monte Carlo interval is

0.760\pm 1.96(0.0135) =0.760\pm 0.0265 \approx (0.733,\;0.787).

Both the point estimate and the approximate interval lie above 0.70, so this simulation gives evidence that assurance exceeds the prespecified minimum, subject to the chosen input distribution and performance model. The interval quantifies Monte Carlo error; it does not say that there is a 95% probability the fixed assurance parameter lies in this particular interval. If a formal decision requires a specified confidence-bound rule, that rule should be prespecified rather than selected after seeing the estimate.

Nested Simulation and Classification Near the Cutoff

Often $g_d(\theta)$ is itself a probability that must be estimated by simulating trial outcomes. This creates two layers of computation: outer draws of uncertain inputs and inner trial simulations conditional on each draw. If $R$ simulated trials at $\theta_m$ produce $K_m$ successes, then $\widehat{g}_d(\theta_m)=K_m/R$ estimates conditional performance. A simple nested estimate classifies the input draw as meeting the target when $\widehat{g}_d(\theta_m)\ge c$.

That classification can be wrong because $\widehat{g}_d(\theta_m)$ has simulation error. The concern is greatest when the true $g_d(\theta_m)$ is close to $c$. Consequently, increasing the number of outer draws alone does not necessarily fix an inner-simulation problem: a large outer sample can estimate the wrong classification rule very precisely if the inner estimates are too noisy.

A practical diagnostic is to inspect the uncertainty of conditional performance estimates for draws near the cutoff. Increase inner simulation effort for such draws, or use a sufficiently accurate analytic calculation when available. If $g_d(\theta)$ has no probability mass exactly at $c$, inner estimates become correctly classified with increasing $R$, and the outer estimator converges as $M$ increases. If a non-negligible part of the input distribution lies exactly at the cutoff, the equality convention matters and noisy inner estimates may not reliably classify those inputs; resolve that boundary using more accurate evaluation and the stated rule.

Worked Example: Checking a Noisy Conditional Estimate

For one hypothetical input draw, the target is $c=0.80$. In $R=250$ simulated trials, 202 meet the trial-level success definition, giving estimated conditional performance $202/250=0.808$. Is this enough to confidently classify the input as meeting the target?

Treating the trial outcomes as independent Bernoulli observations conditional on the input, the estimated standard error is

\sqrt{\frac{(0.808)(1-0.808)}{250}} =\sqrt{\frac{0.1551}{250}} \approx 0.0249.

An approximate 95% interval is $0.808\pm1.96(0.0249)$, or approximately $(0.759,\;0.857)$. This interval spans 0.80. The estimate is above the target, but the inner simulation does not sharply resolve whether the true conditional performance clears it. More inner simulations would improve the classification. For comparison, if 3,260 of 4,000 trials meet the success definition, the estimate is $0.815$, with standard error

\sqrt{\frac{(0.815)(0.185)}{4{,}000}}\approx 0.00614.

The corresponding approximate interval is $(0.803,\;0.827)$, which lies above 0.80. The larger inner simulation supports classifying this input as meeting the target, while still representing an estimate rather than an exact conditional probability.

Justification, Diagnostics, and Alternatives

The estimator’s justification follows directly from the definition. For each input draw, $Z_m$ is 1 with probability $A_d$ and 0 otherwise. Therefore $E(Z_m)=A_d$ and $\operatorname{Var}(Z_m)=A_d(1-A_d)$. Independence of the draws gives

E(\widehat{A}_d) =\frac{1}{M}\sum_{m=1}^{M}E(Z_m)=A_d, \qquad \operatorname{Var}(\widehat{A}_d) =\frac{1}{M^2}\sum_{m=1}^{M}\operatorname{Var}(Z_m) =\frac{A_d(1-A_d)}{M}.

The law of large numbers then implies that $\widehat{A}_d$ approaches $A_d$ as $M$ grows. This result relies on draws that represent the stated distribution and on a valid evaluation of whether each input meets the target. If simulation draws are dependent, such as in a correlated sampling scheme, the independent-draw standard error above is not automatically valid.

Before using an assurance estimate for a go-no-go decision, check that the scenario weights sum to one, the input distribution matches the stated decision question, the target and equality convention are prespecified, and the simulation represents the intended trial design. Report both the assurance estimate and its Monte Carlo precision. A small Monte Carlo standard error does not repair an implausible input distribution, omitted input dependence, or a misspecified trial model.

Assurance answers a threshold-crossing question. Weighted average performance answers how large performance is on average, and worst-case operating characteristics answer how the design fares at the least favorable specified input. These summaries can support different decisions; none substitutes automatically for the others. Assurance is especially useful when decision-makers want a probability statement about clearing a target under a stated uncertainty distribution. Its limitations are equally important: different defensible distributions or scenario weights can yield different assurance values, and a scalar assurance result does not explain which inputs caused a design to fall short.

Mistake: “Average power is 0.82, so assurance of achieving 0.80 power is 0.82.” Correction: assurance is the probability or weight of inputs where conditional power is at least 0.80; the weighted mean and threshold-crossing probability are distinct.
Mistake: Treating every listed scenario as equally likely without justification. Correction: use prespecified weights that represent the intended uncertainty distribution, and state what those weights mean.
Mistake: Increasing outer draws while leaving noisy inner estimates near the cutoff unresolved. Correction: check conditional simulation precision near the performance threshold and improve it where classification is uncertain.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A design meets a conditional performance target in scenarios with weights 0.25, 0.15, and 0.20, and fails it in all other scenarios. What is its assurance?
Explain why a weighted mean performance of 0.84 does not, by itself, determine assurance for a cutoff of 0.80.
In 400 independent outer input draws, a design meets the target 300 times. Calculate the estimated assurance and its Monte Carlo standard error.
What is the main concern when an inner-simulation estimate of conditional power is 0.802 for a target of 0.80?

Solutions. (1) Assurance is the total weight of scenarios that meet the target: $0.25+0.15+0.20=0.60$. (2) A mean aggregates the sizes of performance values, whereas assurance counts the probability weight of values at or above 0.80. Different distributions of conditional performance can have the same mean but different proportions above the cutoff. (3) $\widehat{A}_d=300/400=0.750$, and $\widehat{SE}=\sqrt{(0.750)(0.250)/400}\approx0.0217$. (4) Inner simulation error may place the true conditional power on either side of 0.80; inspect its standard error or interval and increase inner simulation precision if needed before treating the input as a definite pass or fail.