Performance Metrics For Bayesian Trial Procedures

Why the Metric Must Match the Question

A Bayesian trial procedure may use a posterior probability threshold to decide whether a treatment is promising, and may also report an effect estimate and a credible interval. These outputs answer different questions. A favorable posterior probability in one trial does not tell us how often the decision rule would incorrectly recommend treatment across repeated trials, whether an estimator systematically overstates the effect, or how often reported intervals cover the true effect.

An operating characteristic describes the behavior of a specified trial procedure over repeated trials under a stated data-generating truth. The procedure includes the design and analysis choices that determine the output: for example, the enrollment plan, endpoint, prior, posterior decision rule, estimator, and interval construction. To evaluate a characteristic, specify the true parameter value or region, the procedure, and the output whose performance matters. The usual repeated-trial probability is not the posterior probability of a parameter given one trial's data.

Definition: Let $\theta$ denote the true parameter governing trial outcomes, and let $D$ denote the data from one trial generated under the specified design at $\theta$. A trial procedure maps $D$ to outputs such as a decision $A(D)$, an estimate $\hat{\psi}(D)$, and an interval $C(D)=[L(D),U(D)]$ for a target $\psi(\theta)$. An operating characteristic is a repeated-trial probability or expectation calculated under the distribution of $D$ at the stated truth $\theta$.

The definitions below presume that the trial design and analysis procedure are fixed, that the target parameter and its units are clear, and that the data-generating distribution represents the setting of interest. If enrollment, missing data, adherence, or other design features are part of the intended setting, they must be reflected in that distribution. A result at one value of $\theta$ is pointwise; it does not automatically hold at other values.

Decision Metrics: False Positives and Power

Suppose $A(D)=1$ means that the procedure declares trial success, such as recommending the treatment under a specified posterior go rule, and $A(D)=0$ means it does not. Let $\Theta_0$ be the null region in which success should not be declared, and let $\Theta_1$ be the alternative region in which success is scientifically or clinically meaningful. These regions must be defined on the parameter scale before evaluating the decision rule. For a superiority question, $\Theta_0$ might represent no benefit or benefit below a clinically meaningful threshold; $\Theta_1$ would then represent benefit above that threshold.

Definition: At a specified truth $\theta\in\Theta_0$, the false-positive probability is $P_\theta(A(D)=1)$. At a specified truth $\theta\in\Theta_1$, power is $P_\theta(A(D)=1)$. For a composite null, a design may also be assessed by the maximum false-positive probability, $\sup_{\theta\in\Theta_0}P_\theta(A(D)=1)$. These are repeated-trial properties of the procedure, not probabilities that the null or alternative is true.

Power depends on the true alternative value: a procedure may have high power for a large treatment effect and lower power for a small but meaningful one. Likewise, a false-positive probability calculated at one null value does not characterize every value in a composite null. A summary averaged across a distribution of possible truths is a different target from a pointwise probability or a worst-case value, so the averaging distribution and purpose must be stated.

Worked Example: False-Positive Probability for a Bayesian Go Rule

Suppose success is declared when the posterior probability that a treatment benefit exceeds zero is greater than 0.95. To assess false positives, a hypothetical calibration exercise generates 600 independent trials under a specified null truth of zero benefit. The rule declares success in 27 trials. Assume each replicate follows the same stated design and analysis procedure.

The false-positive probability at this null truth is the probability of declaring success when the null truth holds. Its estimate from these replicate outcomes is the fraction declared successful:

\widehat{P}_{0}(A(D)=1)=\frac{27}{600}=0.045.

Thus, the estimated false-positive probability is 4.5% at zero benefit under this design and rule. This does not mean that the posterior probability of the null is 4.5%, nor does it establish the false-positive probability at every value in a broader null region. The result is conditional on the specified truth and procedure.

Worked Example: Power at a Meaningful Benefit

Use the same success rule, but now suppose the specified true treatment benefit is 0.10 on the risk-difference scale, and the trial team has defined this value as clinically meaningful. In 500 hypothetical repeated trials under that truth, the rule declares success in 412. Assume the procedure and data-generating conditions are held fixed across replicates.

The estimated power at this effect is:

\widehat{P}_{0.10}(A(D)=1)=\frac{412}{500}=0.824.

The estimated power is 82.4% for a true risk difference of 0.10. It is not the probability that the treatment works after observing a particular trial. Nor does it imply 82.4% power at other effect sizes. To answer a question about the chance of detecting a smaller benefit, the repeated-trial probability must be evaluated at that smaller truth.

Estimation Metrics: Bias

A decision metric concerns whether the procedure declares success. An estimation metric instead concerns the numerical estimate it reports. Let $\hat{\psi}(D)$ estimate the true target $\psi(\theta)$, such as a treatment effect on a specified scale. The bias at $\theta$ is the expected estimate over repeated trials minus the true target. It is measured in the same units as $\psi$, and its sign indicates the direction of average error.

Definition: The bias of $\hat{\psi}$ at truth $\theta$ is $$ \operatorname{Bias}_\theta(\hat{\psi})=E_\theta[\hat{\psi}(D)]-\psi(\theta). $$ A positive value means the estimator is higher than the true target on average; a negative value means it is lower on average.

Bias is not the error in an individual trial estimate. A single estimate can be above or below the truth even when the procedure is unbiased, meaning its expected estimate equals the target. Conversely, a small average bias does not guarantee that individual estimates are close to the truth: variability is a separate feature of performance. The target scale matters as well. Bias on a risk-difference scale is not automatically the same as bias on a log-odds scale.

Worked Example: Interpreting Bias in a Risk-Difference Estimate

Suppose a treatment-effect estimator targets the true risk difference $\psi(\theta)=0.10$. Across 1,000 hypothetical trials generated under the same truth and design, the mean of the reported risk-difference estimates is 0.12. Treating this replicate mean as an estimate of the repeated-trial expectation, the estimated bias is:

\widehat{\operatorname{Bias}}_\theta(\hat{\psi}) =0.12-0.10=0.02.

The estimator overestimates the risk difference by an average of 0.02, or 2 percentage points, in this setting. This finding is specific to the stated truth and trial procedure. It does not imply that every trial estimate is 2 percentage points too high, and it does not describe the spread of estimates around their mean.

Interval Metrics: Coverage and Expected Width

An interval procedure returns endpoints $L(D)$ and $U(D)$ for the target $\psi(\theta)$. Its coverage probability at $\theta$ is the repeated-trial probability that the interval contains the true target. Expected interval width summarizes the average distance between the endpoints. These metrics answer different questions: coverage concerns whether the procedure includes the truth, while width concerns precision on the target scale.

Definition: The coverage probability of $C(D)=[L(D),U(D)]$ at $\theta$ is $$ P_\theta\!\left(\psi(\theta)\in C(D)\right). $$ Its expected width is $$ E_\theta[U(D)-L(D)]. $$ Both are properties of the interval procedure under repeated sampling at the stated truth.

A Bayesian credible interval is constructed from a posterior distribution, but its nominal posterior probability does not by itself guarantee any particular repeated-trial coverage probability. Coverage calibration asks how often intervals produced by the full procedure contain the fixed true value across repeated trials. Expected width, by contrast, gives an average length in the units of the target parameter. A narrow interval can have poor coverage, and a high-coverage interval can be unhelpfully wide. Neither metric alone fully describes interval performance.

Worked Example: Coverage and Width Are Different Targets

A trial procedure reports an interval for treatment effect measured as a risk difference. Across 1,000 hypothetical trials generated at a fixed true risk difference of 0.10, 922 intervals contain 0.10. The sum of the 1,000 interval widths is 160 risk-difference units, where each width is the upper endpoint minus the lower endpoint.

The estimated coverage probability is:

\widehat{\operatorname{Coverage}}_\theta =\frac{922}{1000}=0.922.

The estimated coverage is 92.2% at this truth. The mean width is:

\widehat{E}_\theta[U(D)-L(D)] =\frac{160}{1000}=0.16.

Thus, the average interval width is 0.16 on the risk-difference scale, or 16 percentage points. The coverage result does not establish that the interval has 92.2% posterior probability of containing the fixed truth after a particular trial. The width result also does not say that every interval is 0.16 wide.

A Short Justification: Why These Are Repeated-Trial Targets

For any trial-level output, define an indicator that equals 1 when the event of interest occurs and 0 otherwise. For coverage, let $I(D)=1$ if $\psi(\theta)\in C(D)$, and 0 otherwise. Because the expectation of an indicator equals the probability of its event,

E_\theta[I(D)] =1\cdot P_\theta(I(D)=1)+0\cdot P_\theta(I(D)=0) =P_\theta\!\left(\psi(\theta)\in C(D)\right).

The same argument applies to the success indicator $A(D)$: $E_\theta[A(D)]=P_\theta(A(D)=1)$. This explains why repeated-trial frequencies estimate decision probabilities and coverage, while averages of reported numeric outputs estimate expectations such as mean estimates or interval widths. The truth $\theta$ is held fixed in these definitions; randomness comes from generating new trial data under the specified conditions.

Choosing the Metric and Checking Its Scope

Start from the question, then select the metric whose definition matches it. A question about how often the procedure recommends treatment when no meaningful benefit exists calls for a false-positive probability. A question about how often it recommends treatment when a specified meaningful benefit is true calls for power. A question about systematic over- or underestimation calls for bias. A question about how often reported intervals include the truth calls for coverage. A question about average interval precision calls for expected width.

Performance question	Metric	What must be specified
How often does the rule declare success when success is unwarranted?	False-positive probability	Null region, truth or worst-case target, and success rule
How often does the rule declare success when benefit is meaningful?	Power	Meaningful alternative truth and success rule
Does the estimate systematically overshoot or undershoot?	Bias	Estimator, true target, parameter scale, and truth
How often does the interval contain the true target?	Coverage probability	Interval procedure, target, and truth
How long are intervals on average?	Expected interval width	Interval procedure, parameter scale, and truth

These metrics are complementary, not substitutes. A Bayesian procedure's posterior success criterion defines how the procedure makes a decision; false-positive probability and power describe how that decision behaves over repeated trials. A posterior probability near a threshold is evidence from one observed dataset, whereas an operating characteristic evaluates behavior under a specified fixed truth. Similarly, posterior credible probability and repeated-trial coverage are distinct quantities.

Check that each reported value has an explicit scope. Is it for one truth or a region? Is the null composite, and is the claim pointwise or worst-case? Are results averaged across scenarios, and if so, what weights define that average? Are bias and width reported in interpretable units? A metric without these qualifications can appear more general than the calculation supports.

Common Pitfalls and Limitations

Equating a posterior probability with a false-positive probability. A posterior probability is conditional on the observed data and prior. A false-positive probability is a repeated-trial probability under a specified null truth.
Reporting “power” without an alternative truth. Power varies with the true effect. State the effect size or alternative region to which the reported value applies.
Calling one null-point calculation the composite-null error rate. A value at $\theta=0$ does not establish the maximum over $\Theta_0$. The claim must match the values evaluated.
Treating low bias as complete evidence of good estimation. Bias describes average signed error, not the variability or typical size of errors.
Using interval width as a proxy for coverage. Width measures length, not the probability of including the truth. Report both when both questions matter.

All these metrics depend on the assumed data-generating conditions and the specified procedure. If those conditions do not represent the intended trial setting, the metric may be calculated correctly but answer the wrong performance question. No single metric establishes overall quality: a procedure can have attractive power but unacceptable false-positive behavior, or acceptable coverage but intervals too wide to be useful.

Check Your Understanding

For each question, identify the relevant metric and interpret the supplied calculation. Solutions are included after the questions.

A success rule declares a go decision in 36 of 800 hypothetical trials at a specified truth in the null region. What metric is being estimated, and what is its value?
At a clinically meaningful effect, a rule declares success in 315 of 400 hypothetical trials. Calculate the estimated power.
An estimator targets a true mean treatment difference of 5 mg and has a repeated-trial mean estimate of 4.6 mg. Calculate and interpret its bias.
At a specified truth, 475 of 500 reported intervals contain the true effect. What does this say about coverage, and what does it not say about interval width?

Solutions. (1) This estimates the false-positive probability at the specified null truth: $36/800=0.045$, or 4.5%. (2) Estimated power is $315/400=0.7875$, or 78.75%, at the stated meaningful effect. (3) Bias is $4.6-5=-0.4$ mg, so the estimator is 0.4 mg below the true target on average in this setting. (4) Estimated coverage is $475/500=0.95$, or 95%, at that truth. This calculation gives no average interval width; interval lengths must be evaluated separately.