Integrating Over Nuisance-Parameter Uncertainty

Why nuisance-parameter uncertainty matters

Assurance averages the probability of trial success over uncertainty in quantities that are not themselves the treatment effect of interest. Even when a design has a clear distribution for plausible treatment effects, its operating characteristics may also depend on the control event rate, outcome variance, or censoring pattern. Treating one of these quantities as known can make the resulting assurance misleadingly precise.

The key calculation is to evaluate conditional Bayesian power at specified values of the treatment effect and nuisance parameters, then average those values over a design distribution. This extends assurance across clinically plausible effect sizes: it does not replace the analysis model or the trial’s success criterion. It specifies how uncertainty about the trial-generating conditions enters the design calculation.

Definition: Let $\theta$ denote the treatment-effect parameter, $\eta$ a nuisance parameter (or vector of nuisance parameters), and $D$ the trial data. Let $S(D)$ indicate whether the prespecified Bayesian success rule is met. Conditional Bayesian power at $(\theta,\eta)$ is $$ C(\theta,\eta)=P\{S(D)=1\mid \theta,\eta,\text{design}\}. $$ If the design distribution for $(\theta,\eta)$ has density $g(\theta,\eta)$, assurance is $$ A=E_g\{C(\theta,\eta)\} =\int C(\theta,\eta)g(\theta,\eta)\,d\theta\,d\eta. $$ For discrete scenarios, the integral is a probability-weighted sum.

Here, $\theta$ describes the effect under which the data are generated, whereas $\eta$ describes other data-generating features that influence the probability of success. Examples of $\eta$ include a control-group event probability, a residual standard deviation, or parameters governing event and censoring times. The analysis still uses its specified likelihood, prior, and decision rule. The design distribution $g$ expresses uncertainty for operating-characteristic calculations; it is not automatically the prior used to analyze trial data.

The distribution may be joint rather than independent. For example, a treatment effect and a control event rate might be related under the clinical scenarios considered. In that case, averaging over a product of separate marginal distributions would discard that relationship and can give a different assurance from averaging over the intended joint distribution.

Why the averaging formula holds

Let $S$ be the event that the trial meets its success rule. Conditional on $(\theta,\eta)$, the probability of that event is $C(\theta,\eta)$. If the design distribution assigns density $g$ to the data-generating parameters, the law of total probability gives the overall probability of success as the average of these conditional probabilities.

\begin{aligned} A &=P(S)\\ &=\int P(S\mid\theta,\eta)g(\theta,\eta)\,d\theta\,d\eta\\ &=\int C(\theta,\eta)g(\theta,\eta)\,d\theta\,d\eta. \end{aligned}

This identity also clarifies the order of the calculation: the conditional power must refer to the same success rule and trial design for every scenario, while the weights must describe the design distribution over scenarios. If nuisance uncertainty is represented by scenarios $\eta_1,\ldots,\eta_K$, with probabilities $w_k$ that sum to 1, then at a fixed effect $\theta$,

A(\theta)=\sum_{k=1}^{K} w_k C(\theta,\eta_k), \qquad \sum_{k=1}^{K}w_k=1.

If both the treatment effect and nuisance parameters vary, use a joint sum or integral over both. One may equivalently average conditional power over nuisance parameters at each effect, then average those results over the effect distribution, provided the relevant conditional distributions and weights are handled consistently.

Worked examples

Worked Example: Uncertain Control Event Rate

Consider a hypothetical equal-allocation trial with 120 participants per group and a binary outcome. The Bayesian success rule is that the posterior probability that the treatment event probability exceeds the control event probability is greater than 0.975. For this operating-characteristic calculation, suppose the treatment-control risk difference is fixed at 0.10, and the control event probability $p_C$ has the following design distribution. The treatment event probability in each scenario is $p_T=p_C+0.10$, so all specified probabilities are valid.

Control event probability $p_C$	Scenario weight	Conditional power
0.10	0.25	0.76
0.25	0.50	0.70
0.40	0.25	0.63

Suppose the conditional powers in the last column have been obtained by evaluating the specified Bayesian analysis rule under the corresponding fixed event probabilities. The design weights sum to $0.25+0.50+0.25=1$. The nuisance-averaged assurance is therefore

A(0.10) =\sum_{k=1}^{3}w_k C(0.10,p_{C,k}) =(0.25)(0.76)+(0.50)(0.70)+(0.25)(0.63) =0.190+0.350+0.1575 =0.6975.

Under this hypothetical design distribution and fixed risk difference, assurance is 0.698, or about 69.8%. This is not conditional power at one “average” control event rate. It is the weighted average of success probabilities at the three specified rates. Changing the scenario weights or the treatment-effect assumption changes the result.

Worked Example: Uncertain Variance for a Normal Endpoint

Suppose a hypothetical trial compares two groups with $n=50$ participants per group. Outcomes are normally distributed with a common standard deviation $\sigma$, and the variance is known within each conditional scenario. The analysis uses a flat prior for the treatment-minus-control mean difference $\delta$, and declares success when the posterior probability that $\delta>0$ exceeds 0.975. Assume the true difference is 5 units in every scenario, with design-distributed standard deviation $\sigma=8,12,$ or $16$ units and respective weights $0.25,0.50,$ and $0.25$.

For a given $\sigma$, the standard error of the observed difference in sample means is $SE=\sigma\sqrt{2/n}$. Under the stated normal model and flat prior, the posterior for $\delta$ is normal with mean equal to the observed difference and standard deviation $SE$. Thus the success rule is equivalent to requiring that the observed difference exceed $1.96SE$. When the true difference is 5, the conditional power is

C(5,\sigma) =P\!\left(\bar{X}_T-\bar{X}_C>1.96SE\mid \delta=5,\sigma\right) =\Phi\!\left(\frac{5}{SE}-1.96\right), \qquad SE=\sigma\sqrt{\frac{2}{50}}.

For $\sigma=8$, $SE=8\sqrt{2/50}=1.600$, giving $C(5,8)=\Phi(5/1.600-1.96)=\Phi(1.165)=0.878$. For $\sigma=12$, $SE=2.400$, giving $C(5,12)=\Phi(5/2.400-1.96)=\Phi(0.1233)=0.549$. For $\sigma=16$, $SE=3.200$, giving $C(5,16)=\Phi(5/3.200-1.96)=\Phi(-0.3975)=0.346$. Therefore,

$$ A(5) =(0.25)(0.878)+(0.50)(0.549)+(0.25)(0.346) =0.2195+0.2745+0.0865 =0.5805. $$

Assurance is approximately 58.1% for this distribution of standard deviations. Larger $\sigma$ means greater sampling uncertainty and, in this example, lower conditional power. The calculation assumes independent normal outcomes, equal group sizes, the stated known-variance analysis within each scenario, and the specified prior and success threshold.

Worked Example: Censoring-Distribution Scenarios

Suppose a hypothetical event-driven trial uses a fixed follow-up plan and a Bayesian posterior success rule for a treatment effect on the event hazard. The treatment effect is fixed for this calculation, but the censoring distribution is uncertain. Investigators specify two plausible censoring scenarios: a lower-censoring scenario with weight 0.60 and a higher-censoring scenario with weight 0.40. Conditional on each scenario, the event-time and censoring-time model is used to calculate the probability of meeting the same posterior success rule; suppose those conditional powers are 0.80 and 0.55, respectively.

The censoring-scenario weights sum to $0.60+0.40=1$, so the assurance averaged over this uncertainty is

$$ A=(0.60)(0.80)+(0.40)(0.55)=0.48+0.22=0.70. $$

The result is 70.0% assurance under the stated distribution over censoring scenarios. This calculation requires that each conditional power incorporate the event accrual, censoring, and analysis specified for its scenario. Averaging only over censoring rates while holding an incompatible event-generation model fixed would not represent the stated design.

Building a defensible nuisance distribution

The arithmetic of averaging is simple; specifying what to average over is the substantive task. A nuisance distribution should represent uncertainty relevant to the proposed design, using clinical knowledge, historical data, external evidence, or explicit scenarios. Its support must be plausible, and it must respect constraints in the data-generating model. For example, event probabilities must lie between 0 and 1, and a treatment event probability constructed by adding a risk difference to a control probability must also remain in that range.

A practical calculation can be organized as follows.

Specify the operating-characteristic target.
State the treatment-effect conditions, nuisance quantities, and Bayesian success rule to be evaluated.

Choose the design distribution.
Define plausible nuisance values and their probabilities, or a continuous distribution. Preserve dependence among nuisance quantities and effects where clinically justified.

Evaluate conditional power.
At each specified parameter combination, calculate the probability that data generated under those conditions meet the same analysis rule.

Average with the specified weights.
Check that discrete weights sum to 1, or that the continuous design density integrates to 1, then report the resulting assurance in context.

For continuous nuisance distributions, the assurance is an integral and may require numerical integration. The integration must cover the intended design distribution rather than an arbitrary range selected for convenience. When effects and nuisance parameters are dependent, the appropriate expression is

A=\int\!\!\int C(\theta,\eta)g(\eta\mid\theta)g(\theta)\,d\eta\,d\theta,

where $g(\eta\mid\theta)$ describes nuisance uncertainty conditional on the effect and $g(\theta)$ is the effect distribution. Replacing $g(\eta\mid\theta)$ with a marginal $g(\eta)$ is justified only when the design distribution treats the quantities as independent.

Diagnostics, alternatives, and limitations

Several checks help reveal when a nuisance-averaged calculation may be unreliable:

Check the conditional-power pattern. Plot or tabulate conditional power across the nuisance range. Abrupt changes may be real, but can also indicate numerical instability, a boundary problem, or inconsistent scenario definitions.
Check the support and weights. Confirm that scenario probabilities are nonnegative and sum to 1, and that all combinations of parameters satisfy the model’s constraints.
Check the analysis remains the same. Conditional calculations should use the same success rule and analysis priors throughout. If the actual planned analysis changes between scenarios, the average no longer describes one fixed design.
Check dependence assumptions. Compare the intended joint distribution with any factorized approximation. Ignoring an important relationship can distort the weighted average.
Check sensitivity to the distribution. Report how assurance changes under reasonable alternative nuisance distributions or scenario weights. A single average can hide low conditional power in an important part of the plausible range.

A nuisance-averaged assurance differs from conditional power at a selected nuisance value, which answers the question “What is the success probability if these nuisance parameters are fixed here?” It also differs from averaging over treatment effects alone, which does not account for nuisance uncertainty unless those quantities have been fixed in the conditional calculation. A worst-case calculation is another alternative: it reports the minimum conditional power over a specified nuisance set rather than a probability-weighted average. The average describes performance under the chosen design distribution; the worst case describes a boundary of performance over the set. Neither is universally preferable.

The result is only as relevant as its assumptions. A poorly calibrated nuisance distribution can produce an assurance that is numerically precise but substantively unconvincing. Scenario averaging can also conceal a weak operating characteristic in a scenario with a small assigned weight. Finally, a nuisance parameter may be estimable from trial data, but that does not eliminate its influence on operating characteristics: the realized data and the analysis rule still depend on the true data-generating conditions.

Common pitfall: Averaging conditional powers evaluated at several nuisance values using equal weights simply because the values were convenient to calculate. Equal weights represent a uniform scenario distribution, not a neutral default. Use weights that encode the intended design uncertainty, and state them explicitly.

Check Your Understanding

Answer each question using the definitions and calculations in this tutorial.

At a fixed treatment effect, conditional powers are 0.60 and 0.90 under two nuisance scenarios. Their design weights are 0.70 and 0.30. What is the nuisance-averaged assurance?
Why is assurance at the mean nuisance value generally not guaranteed to equal assurance averaged over the nuisance distribution?
Suppose the treatment effect and control event probability are associated in the design scenarios. What distribution should be used for averaging?
For the normal-endpoint example, explain how increasing $\sigma$ affects $SE$ and conditional power when the true treatment difference is fixed and positive.
What does a worst-case conditional-power summary answer, and how does it differ from assurance?

Solutions. (1) $A=(0.70)(0.60)+(0.30)(0.90)=0.42+0.27=0.69$, or 69%. (2) Conditional power need not be a linear function of the nuisance parameter, so generally $E\{C(\eta)\}\ne C\{E(\eta)\}$. (3) Use the joint design distribution, or equivalently average using the nuisance distribution conditional on the treatment effect and then average over effects. (4) Since $SE=\sigma\sqrt{2/n}$, increasing $\sigma$ increases $SE$; with a fixed positive effect and the same threshold, the standardized signal decreases and conditional power falls. (5) The worst-case summary is the minimum conditional power over a stated nuisance set. Assurance is the probability-weighted average over a design distribution, so it need not equal that minimum.