End-to-End Assurance Design Workshop

From a decision rule to a design

A sample-size decision is not determined by a target assurance alone. It depends on the treatment effect being planned for, the prior distribution, the rule that will declare success, and any additional operating constraints. An end-to-end design therefore calculates the chance of meeting the Bayesian rule and checks other constraints separately. A design that meets one target but fails another is not a successful design.

The hypothetical example below concerns a two-arm randomized trial with a continuous endpoint measured in points, where higher values favor treatment. It is deliberately simple enough to calculate transparently. The workflow is general: specify the model and success rule, derive the data boundary, evaluate operating characteristics, and select from a prespecified set of candidate sample sizes.

The normal model and known variance make the calculations exact; they are not claims that real trial data will necessarily satisfy these assumptions. Equal allocation gives \(\operatorname{Var}(D\mid\theta)=10^2/n+10^2/n=200/n\). The prior has mean \(2\) points and standard deviation \(4\) points. These choices, including the prior’s positive mean, are consequential: the prior contributes to the posterior decision boundary and the assurance calculation.

Deriving the posterior success boundary

Consider first a rule that declares success when the posterior probability of a beneficial effect exceeds a specified cutoff \(q\): \(P(\theta>0\mid D)\geq q\). For the normal prior and likelihood above, the posterior is normal. Writing \(v=200/n\), its precision is \(1/16+1/v\), and its mean is the precision-weighted average of the prior mean and \(D\).

Let \(z_q\) be the \(q\) quantile of the standard normal distribution. Since the posterior is normal, the rule \(P(\theta>0\mid D)\geq q\) holds exactly when the posterior mean divided by its posterior standard deviation is at least \(z_q\). Solving this inequality for the observed difference gives a data threshold \(c_n(q)\).

The prior-mean contribution \(2/16\) must remain in this expression. It affects the required observed effect and cannot be dropped when substituting numbers. This boundary translates a posterior probability rule into a rule on trial data; it does not itself guarantee a particular frequentist type I error.

Assurance and the separate null-error check

Assurance is the probability of meeting the success rule when the treatment effect is itself drawn from the design prior and the trial data are then generated under that effect. In this model, integrating over \(\theta\sim N(2,16)\) gives the prior-predictive distribution \(D\sim N(2,16+200/n)\). Therefore, for a threshold \(c_n(q)\), assurance has a direct normal-tail calculation.

A hybrid design may also impose a frequentist constraint under the null effect. For the one-sided null \(\theta=0\), the sampling distribution is \(D\mid\theta=0\sim N(0,200/n)\). The null type I error of this decision rule is consequently \(P_0(D\geq c_n(q))=1-\Phi(c_n(q)/\sqrt{200/n})\). This is a separate probability under a fixed null value, not the prior-predictive probability used for assurance.

Checking constraints and selecting a design

The calculations show why the decision rule must be checked against every design requirement. Under the \(0.975\) posterior cutoff, 800 per arm is the first of the nearby candidates to exceed \(0.60\) assurance, but its null type I error is about \(0.0279\), above \(0.025\). Increasing the sample size to 825 or 850 does not solve that problem. The hybrid constraints are not satisfied merely because the assurance target is.

A protocol team could revise the posterior cutoff during design, with scientific and regulatory agreement, and then repeat the full operating-characteristic calculation. The following worked calculation illustrates a prospectively specified \(q=0.99\) rule. Changing the cutoff after looking at trial outcomes would be inappropriate; the rule and constraints need to be set before the trial data are examined.

The design choice is therefore 1100 participants per arm under the \(0.99\) posterior rule, for 2200 participants total, subject to the stated model and candidate grid. This is not a claim that 1100 is the unique or globally smallest feasible sample size under all possible rules or allocation plans. It is the selected candidate after evaluating the specified criteria. If recruitment, cost, or other constraints make this size infeasible, the design inputs or decision rule require an explicit, prospective review and fresh calculations.

Diagnostics, alternatives, and limitations

The closed-form calculations are useful checks on a simulation-based design. They also expose which assumptions drive the answer. In a more complex design, assurance may be estimated by simulation as in the earlier material on simulation estimation of Bayesian power and assurance, with Monte Carlo error reported as in the earlier material on Monte Carlo error for assurance estimates. Common random numbers or importance sampling may improve particular calculations, but they do not repair a misspecified model.

• Check the variance and allocation. Unequal allocation, a different outcome standard deviation, or a variance estimated from data changes the sampling distribution and threshold calculation.
• Check the prior’s role. Here the prior mean is positive, so omitting its contribution changes the boundary. Sensitivity to prior spread or a skeptical prior scenario should be considered when clinically plausible alternatives exist.
• Check the null distribution separately. A posterior probability cutoff does not automatically guarantee a chosen type I error. Calculate the null tail probability at the actual boundary for each candidate.
• Check implementation. Simulation estimates that disagree materially with the analytic result in this simple normal case can signal coding, parameterization, or tail-direction errors.
• Check the clinical decision. This example defines success as evidence that \(\theta>0\). If success should instead require a minimum clinically important benefit, the posterior event and all associated calculations must change.

Assurance differs from power calculated at a single assumed effect: it averages the success probability across a distribution of possible effects, here the design prior. That averaging makes the prior influential and requires sensitivity analysis. Neither assurance nor a null type I error calculation establishes that the model is correct, that the trial will recruit as planned, or that the measured effect answers every clinical question. Attrition, nonadherence, interim monitoring, multiple endpoints, and uncertainty in the outcome variance require additional modeling or design constraints rather than an informal adjustment.