From a Bayesian Decision to a Trial-Level Property
A Bayesian trial decision is usually made after observing data: for example, proceed if the posterior probability that a treatment effect is beneficial exceeds a specified threshold. That rule answers what to do for the data in hand. It does not, by itself, tell us how often the whole procedure would make that decision if the trial were repeated under a particular state of the world. An operating characteristic answers the latter question.
The distinction matters because a trial procedure is more than its final posterior calculation. Its operating characteristics depend on how participants are assigned and followed, how outcomes arise, what analysis is applied, and how the analysis is converted into a decision. A posterior rule can look identical on paper yet have different repeated-trial behavior when sample size, allocation, missing-data handling, or interim stopping changes.
The scenario \(\eta\) can include the true treatment effect and other features that govern outcomes, such as control response, outcome variability, and the pattern of missingness. It is fixed when defining a particular operating characteristic. The probability is over the random data—and any randomization or algorithmic randomness—generated when the trial procedure is repeated under that scenario. It is not a probability that the scenario itself is true.
A useful way to keep the pieces straight is to describe the procedure as a chain. The design and scenario generate trial data; the specified analysis turns those data into an estimate or posterior distribution; and the decision rule maps the analysis to a decision. Each component must be defined well enough to determine the probability of the event of interest.
What Must Be Specified
For a fixed-sample trial, the specification ordinarily includes the sample size, allocation, outcome model under each scenario, analysis model and prior, and the exact decision threshold. For an adaptive trial, it must also describe what happens at each interim look: when data are reviewed, what stopping or adaptation rules apply, and how the final decision is reached. If the procedure includes a rule for missing outcomes, that rule belongs in the specification too.
The data-generating mechanism used to evaluate the procedure need not match the analysis model. That distinction can be important: a Bayesian analysis may assume a particular model and prior, while the scenario used to assess its repeated-trial behavior may include a different control response or a different outcome distribution. In either case, the assumed scenario must be stated. A probability calculated under an unspecified or implausible scenario has no reliable interpretation.
The analysis prior also has a different role from the scenario. For repeated-trial assessment at a fixed \(\eta\), the prior is part of the analysis rule, but the true parameter is held fixed across repetitions. The prior is not automatically sampled anew for each simulated trial. A design team may use prior information to choose the procedure, but the resulting procedure can still be evaluated at multiple fixed scenarios.
State the data-generating conditions, including the true parameter values and any nuisance features that affect outcomes.
Apply the design under that scenario, including allocation, follow-up, and any interim decisions, to obtain a complete trial data set.
Use the prespecified analysis, then record whether the event of interest—for example, a go decision—occurred.
The operating characteristic is the probability of that recorded event under repeated trials generated by the same procedure and scenario.
Worked Examples: Defining and Calculating Operating Characteristics
Worked Example: A Binary Bayesian Go Rule
Consider a hypothetical single-arm trial with 10 participants and a binary response. Let \(X\) be the number of responses. The analysis uses a \(\operatorname{Beta}(1,1)\) prior for the response probability \(p\), so after observing \(X=x\), the posterior is \(\operatorname{Beta}(x+1,11-x)\). The rule is to declare go if the posterior probability that \(p>0.30\) exceeds 0.90. Find the rule’s go probability when the true response probability is \(p=0.30\) and when it is \(p=0.50\), assuming independent responses with a common probability \(p\).
First translate the posterior threshold into a cutoff on \(x\). For integer beta parameters, the posterior tail probability can be written as \(\Pr(p>0.30\mid X=x)=\Pr\{\operatorname{Binomial}(11,0.30)\leq x\}\). At \(x=4\), this probability is approximately 0.790; at \(x=5\), it is approximately 0.922. Because the cumulative probability increases with \(x\), the go rule is therefore \(X\geq5\).
Under a fixed true value \(p\), the assumed independent Bernoulli outcomes imply \(X\sim\operatorname{Binomial}(10,p)\). At \(p=0.30\), the operating characteristic is \(\Pr(X\geq5)=\sum_{j=5}^{10}{10\choose j}(0.30)^j(0.70)^{10-j}\). The terms sum to approximately \(0.1503\). At \(p=0.50\), the same rule gives \(\Pr(X\geq5)=\sum_{j=5}^{10}{10\choose j}(0.50)^{10}=638/1024\approx0.6230\).
Thus, under the stated outcome assumptions, the procedure declares go in about 15.03% of repeated trials when the true response probability is 0.30, and in about 62.30% when it is 0.50. These are scenario-specific properties of the entire procedure, not posterior probabilities about \(p\) after one trial.
Worked Example: A Normal-Outcome Trial and Its Go Probability
Suppose a hypothetical fixed-sample trial estimates a treatment-control mean difference \(D\), measured in points. Under a fixed true difference \(\delta\), assume \(D\sim N(\delta,0.20^2)\). The analysis uses the prior \(\delta\sim N(0,0.30^2)\), and the decision rule is go if \(\Pr(\delta>0\mid D)>0.95\). Calculate the go probability at \(\delta=0.40\) points and \(\delta=0.60\) points.
The normal likelihood has variance \(0.20^2=0.04\), and the prior has variance \(0.30^2=0.09\). Normal updating gives posterior mean \(kD\), where \(k=0.09/(0.09+0.04)=0.6923\), and posterior variance \(v=(0.09)(0.04)/(0.09+0.04)=0.02769\). Thus the posterior standard deviation is \(\sqrt v\approx0.1664\) points.
The posterior probability of a positive effect exceeds 0.95 exactly when the posterior mean exceeds \(z_{0.95}\sqrt v\), where \(z_{0.95}\approx1.6449\). Since \(k>0\), the data rule is equivalently \(D>c\), with \(c=z_{0.95}\sqrt v/k\approx(1.6449)(0.1664)/0.6923\approx0.3954\) points (approximately 0.39537 using unrounded values).
Under a fixed true \(\delta\), the go probability is therefore \(\Pr_\delta(D>c)=1-\Phi((c-\delta)/0.20)=\Phi((\delta-c)/0.20)\). At \(\delta=0.40\) points, the standardized cutoff is \((0.39537-0.40)/0.20\approx-0.0231\), so the go probability is approximately \(\Phi(0.0231)=0.5092\). At \(\delta=0.60\) points, it is \((0.39537-0.60)/0.20\approx-1.0231\), giving \(\Phi(1.0231)\approx0.8469\).
The calculation uses the sampling distribution for \(D\) under each fixed \(\delta\), while the posterior prior and threshold determine the decision boundary. Neither component alone determines the operating characteristic.
Worked Example: An Interim Conditional Probability Is Not an Operating Characteristic
Consider a hypothetical trial with four independent binary outcomes, each having a fixed true response probability \(p\). Its final rule is go if there are at least three responses. The operating characteristic is \(\Pr\{\operatorname{Binomial}(4,p)\geq3\}=4p^3(1-p)+p^4\). At \(p=0.20\), this equals \(4(0.20)^3(0.80)+(0.20)^4=0.0272\). At \(p=0.80\), it equals \(4(0.80)^3(0.20)+(0.80)^4=0.8192\).
Now suppose the first outcome has been observed and equals one, and consider \(p=0.50\). Conditional on that interim observation, the probability of reaching the final go rule is the probability of at least two responses among the remaining three: \(3/8+1/8=0.5000\). If the first outcome had instead been zero, the conditional probability would be \(1/8=0.1250\), because all three remaining outcomes would need to be responses.
Before any outcome is observed, the repeated-trial go probability at \(p=0.50\) is \(0.50(0.5000)+0.50(0.1250)=0.3125\). The conditional probabilities describe what remains possible given a particular interim result. The operating characteristic averages over all possible trial data under the fixed scenario. In Bayesian decision analysis, predictive success is likewise conditional on current information; it should not be labelled an operating characteristic unless the conditioning and averaging are explicitly part of the question.
Why the Complete Procedure Matters
The normal example also gives a compact derivation of the general idea. For any fixed data rule that declares go when \(D>c\), and a sampling model \(D\sim N(\delta,\sigma_D^2)\), standardizing the event gives \[ P_\delta(\text{go})=P_\delta(D>c) =P\left(\frac{D-\delta}{\sigma_D}>\frac{c-\delta}{\sigma_D}\right) =1-\Phi\left(\frac{c-\delta}{\sigma_D}\right). \] This derivation relies on the stated normal sampling distribution and a fixed cutoff. It does not apply unchanged if the sample size is adapted, the estimator’s distribution is not normal, or the final cutoff depends on interim data. In those cases the probability must reflect the actual procedure.
Several distinct choices can change an operating characteristic: the sample size, allocation ratio, stopping rule, prior used in the analysis, posterior threshold, and treatment of incomplete outcomes. A rule evaluated after excluding participants with missing data is not the same procedure as one using a prespecified imputation model. Similarly, an interim efficacy stop changes the distribution of final trial outcomes and decisions compared with a design that always enrolls the full sample.
An operating characteristic can be a function over scenarios rather than a single number. Reporting it across relevant values of the treatment effect and nuisance parameters shows how the procedure behaves as those conditions vary. A single probability is meaningful only alongside the scenario, event, and procedure to which it refers.
Checks, Alternatives, and Limitations
A calculation is only as credible as its assumptions and implementation. Before interpreting an operating characteristic, check that the event is precisely defined, the fixed scenario is explicit, and the data-generating assumptions represent the situations the trial is intended to address. For a simulation-based calculation, verify that the simulated trial applies the same allocation, interim, analysis, and decision rules as the written design. These checks establish what the probability means; uncertainty in its numerical estimate is a separate issue.
- Incomplete procedure: “Use a Bayesian model and go if the posterior is favorable” is insufficient if “favorable,” the analysis, or the stopping rules are unspecified. Write the full decision rule.
- Unstated truth scenario: A go probability without a specified true effect and relevant nuisance conditions is not a uniquely defined operating characteristic.
- Mixing conditional and repeated-trial probabilities: A probability given interim data describes trials at that interim state. An operating characteristic from trial start averages over the possible data histories.
- Assuming analysis-model correctness automatically: The analysis model and the scenario used to evaluate repeated trials are distinct specifications. State both rather than silently treating them as identical.
- Ignoring trial conduct: Missing outcomes, deviations, or unmodeled stopping can alter the data reaching the analysis and therefore alter the operating characteristic.
Posterior inference answers questions conditional on observed data and the analysis model. Predictive success concerns the chance of meeting a future decision criterion conditional on current information. An operating characteristic instead evaluates the full procedure across repeated data sets under a fixed scenario. These approaches answer different questions; none substitutes for the others. The main limitation is that an operating characteristic is always conditional on its specification. It cannot protect against scenarios or trial behaviors that were omitted.
Check Your Understanding
Answer each question using the distinction between a fixed scenario and repeated trial data.
- A trial uses a posterior go rule, and the reported value is the posterior probability of benefit after the observed data. Is this automatically an operating characteristic? Explain.
- For the four-outcome procedure in the third example, calculate the go probability at \(p=0.50\) from the binomial formula.
- Name three features, besides the posterior threshold, that may need specification to define an operating characteristic for a trial.
- Explain why an interim conditional probability and a trial-start operating characteristic can differ.
Solutions. (1) No. It is a posterior probability conditional on the observed data. An operating characteristic is a repeated-trial probability under a specified scenario. (2) The rule is go for at least three responses, so \(4(0.50)^3(0.50)+(0.50)^4=0.25+0.0625=0.3125\). (3) Examples include the sample size, allocation, outcome-generating assumptions, analysis model and prior, missing-data handling, and interim stopping rules. (4) The interim probability conditions on a particular observed history. The trial-start probability averages over the possible histories under the fixed scenario; in the example, the conditional probabilities 0.5000 and 0.1250 average to 0.3125.