A Regulatory Evidence Package for a Bayesian Trial

Why Assemble an Evidence Package?

A regulatory conclusion should not depend on a single probability quoted without its clinical target, assumptions, or decision context. A useful Bayesian evidence package connects the question the trial was designed to answer with the prior information, the posterior estimate, the prespecified decision rule, and the behavior of that rule across possible repeated trials. These components answer different questions: the posterior summarizes uncertainty about the effect given the observed data and model, while operating characteristics describe how often a decision rule would be triggered under specified true effects.

The worked case below is hypothetical. It uses a simple normal model so that every calculation can be checked directly. Real submissions may require more elaborate outcome, missing-data, safety, and borrowing models, but the same reporting logic applies. The estimand must remain clinically interpretable, and the statistical model and decision rule must be stated precisely enough to reproduce the results.

Start With the Estimand and Model

Suppose higher symptom-score change means greater improvement. The treatment effect \(\delta\) is therefore measured in score points, with positive values favoring treatment. A value of \(\delta=2\) points is stipulated in this hypothetical case as the minimum clinically important benefit. The decision target is not merely whether \(\delta\) is above zero, but whether the evidence supports a benefit exceeding 2 points.

Assume a randomized, parallel-group trial with 20 patients per group. For a deliberately transparent illustration, suppose the outcome is normally distributed within each group with known standard deviation \(\sqrt{20}\) points, and observations are independent across patients. Let \(\hat{\delta}\) be the observed treatment-minus-control difference in mean change. Under this model,

The variance 2 is the sampling variance of the estimated difference, not the variance of an individual outcome. The model assumes the stated outcome scale and week-12 measurement are used consistently, the randomized groups are analyzed as assigned, the normal likelihood and known variance are adequate, and the data are complete for this simplified example. A real analysis must address whether these assumptions are credible, including how missing week-12 outcomes are handled. The known-variance assumption is a simplifying device here, not a general recommendation.

Prior Rationale and Posterior Evidence

Use a normal prior \(\delta\sim N(0,25)\), where the second parameter is the variance and the prior standard deviation is 5 points. Centering at zero expresses no expected direction of benefit before seeing the trial data. A standard deviation of 5 allows substantial effects in either direction: approximately 95% of the prior distribution lies between \(-9.8\) and \(9.8\) points. This is not a “noninformative” prior; it is a stated distribution whose scale should be judged against clinical knowledge and the outcome scale. In a regulatory package, the rationale should explain its source, clinical plausibility, and whether it uses external data. Here it is an illustrative, non-borrowing prior.

For the normal likelihood with known variance and a normal prior, the posterior is normal. Its precision is the sum of prior precision and data precision. With prior mean zero, the posterior variance and mean are as follows.

The posterior standard deviation is \(\sqrt{50/27}=1.36083\) points. Substituting the observed estimate of 4.2 points gives posterior mean \( (25/27)(4.2)=3.88889\) points. The posterior distribution is therefore \(N(3.88889,50/27)\), with the variance—not the standard deviation—as its second parameter.

Justify the Decision Cutoff

The posterior probability rule can be expressed as a cutoff on the observed estimate. This is useful for checking the decision and for evaluating its repeated-trial behavior. Because the posterior is normal, its probability of exceeding 2 is at least 0.90 exactly when its standardized distance above 2 is at least the 90th percentile of the standard normal distribution.

Here \(z_{0.90}=\Phi^{-1}(0.90)\approx1.2815516\). Substituting without rounding intermediate quantities gives

The inequality is inclusive: a value exactly at the cutoff gives posterior probability 0.90 and meets a rule requiring “at least 0.90.” The observed estimate of 4.2 points exceeds the cutoff, consistent with the direct posterior-probability calculation. The derivation relies on this specific prior, likelihood, clinical threshold, and probability criterion; a different choice changes the cutoff.

Operating Characteristics Are a Separate Result

Operating characteristics describe the probability that a decision rule leads to a specified action over hypothetical repetitions of the trial, conditional on a fixed true treatment effect. They are not posterior probabilities. Under the assumed sampling model, \(\hat{\delta}\mid\delta\sim N(\delta,2)\), and the decision is made when \(\hat{\delta}\geq c\), where \(c=4.04349\). Therefore the probability of meeting the rule at a fixed true effect is

This calculation assumes the sampling model and decision rule are fixed and that the true effect is the stated value. It is a conditional repeated-sampling probability, not an estimate of the probability that the true effect takes that value. For example, at the clinical boundary \(\delta=2\), it describes the chance of meeting the rule when the true benefit is exactly 2 points. If “clinically important benefit” means \(\delta>2\), then the rule’s probability of a positive decision at \(\delta=2\) is also a boundary false-positive rate for the composite null \(\delta\leq2\); monotonicity makes it the largest such probability within that null.

Sensitivity, Diagnostics, and Alternatives

A regulatory evidence package should show whether the conclusion depends materially on modeling choices, especially the prior. Repeating the analysis with reasonable alternatives is a direct sensitivity analysis. For an alternative normal prior \(N(m_0,v_0)\), the posterior variance is \((1/v_0+1/2)^{-1}\), and the posterior mean is that variance multiplied by \(m_0/v_0+\hat{\delta}/2\). The likelihood and observed estimate remain fixed in this comparison.

Diagnostics should examine whether the model represents the data and estimand adequately. For a simple normal model, inspect group outcome distributions, influential observations, variance assumptions, and missingness patterns; verify that the analysis includes outcomes in a way consistent with the treatment-policy estimand. For more complex models, posterior predictive checks can identify systematic discrepancies between replicated data and observed data. A check that reveals poor fit is a reason to investigate the model, not a mechanical instruction to accept or reject the trial result.

The main alternative summaries answer related but distinct questions. A frequentist confidence interval describes the long-run coverage behavior of an interval procedure under its assumptions; it is not a posterior probability statement. A Bayesian credible interval gives posterior probability for an interval under the specified model and prior, but alone does not encode the particular clinical threshold or regulatory action. A posterior threshold rule makes that action explicit, while operating-characteristic calculations show its repeated-trial behavior. These approaches can be reported together, provided their interpretations are not conflated.

Important limitations remain. The example treats the sampling variance as known, uses a single continuous outcome, and assumes complete data and a correct normal model. It does not address multiplicity, interim adaptation, missing outcomes, treatment-effect heterogeneity, or joint benefit-risk evidence. Sensitivity to prior scale does not prove that any one prior is correct. The decision threshold and posterior-probability requirement should be clinically justified and specified in advance; operating characteristics should be evaluated for a range of plausible true effects, not selected after seeing the result. A clear package states the estimand, model, prior, posterior summary, decision rule, operating characteristics, and sensitivity results together so readers can see both the evidence and the conditions on which it rests.