Averaging Operating Characteristics Across Scenarios

Why Average Across Scenarios?

A Bayesian trial decision rule is evaluated under specified data-generating conditions. Those conditions may differ in treatment effect, baseline event risk, accrual, or other features. The operating characteristic for one scenario describes performance conditional on that particular set of conditions. A decision maker may also want a single summary across several scenarios. Such a summary is meaningful only after specifying how much weight each scenario receives.

For example, an average probability of declaring success can summarize expected performance under a stated distribution of plausible scenarios. It does not, by itself, describe the probability of success under every scenario, nor does it guarantee acceptable performance in the least favorable scenario. The weights are part of the estimand: changing them changes the question being answered.

The operating characteristic might be the probability of declaring the treatment successful, the probability of a false-positive decision under a null scenario, or the probability that a credible interval covers its target. The direction of a desirable value depends on the measure: larger success probability may be desirable, while smaller false-positive probability may be desirable. The average must combine the same defined measure across scenarios, not unlike quantities chosen only because they are available.

Expectation and Interpretation

A useful interpretation follows by introducing a scenario label \(S\). Suppose \(P(S=s)=q_s\), and let \(Y\) be an indicator that equals 1 when a specified event occurs, such as a trial declaring success, and 0 otherwise. If \(P(Y=1\mid S=s)=OC_s\), then the law of total expectation gives the weighted average event probability.

This identity explains the interpretation: imagine selecting a scenario according to the stated weights and then conducting a trial under that scenario. The average is the event probability in this two-stage thought experiment. The scenario selection need not actually be random in trial planning. The weights can instead encode a planning distribution, a policy-relevant mix, or an explicitly chosen way to summarize a finite scenario set. In each case, report what the weights represent.

The scenarios should be defined clearly enough that each \(OC_s\) refers to a coherent data-generating condition and the same analysis and decision rule. The weights must be nonnegative and sum to one. If supplied weights do not sum to one but are intended as relative weights, normalize them by dividing each by their total before calculating the average. A weighted sum using unnormalized values is not an average probability.

Weights Define the Target Summary

Weights should be selected to match the purpose of the summary rather than to make the result look favorable. They might represent a defensible distribution over plausible nuisance-parameter values, a planned mix of patient populations or sites, or a policy-defined distribution of conditions. They can also be equal by deliberate convention, in which case the result describes the arithmetic mean across the listed scenarios. Equal weighting does not automatically mean that scenarios are equally plausible.

An average across scenarios answers a different question from a worst-case summary. A worst-case operating characteristic focuses on the most extreme value over the specified set. A weighted average reflects the distribution encoded by \(q_s\), and can be acceptable even when one scenario has poor performance if that scenario receives little weight. For safety-critical or decision-critical conditions, the average should therefore be read alongside scenario-specific results and any required performance constraints.

Estimating the Average by Simulation

In operating-characteristic simulations, \(OC_s\) is usually unknown and is estimated by repeated simulated trials under scenario \(s\). If \(M_s\) independent simulated trials are run and \(R_s\) meet the event definition, then \(\widehat{OC}_s=R_s/M_s\). The weighted estimate is formed using the scenario weights, not by simply pooling all simulated trials when the simulation counts differ.

If the trial outcomes are simulated independently within and across scenarios, each event indicator is Bernoulli with probability \(OC_s\), and the scenario estimates are independent. Conditional on the chosen scenarios and weights, the variance of the weighted estimator is then

Replacing each unknown \(OC_s\) with \(\widehat{OC}_s\) gives an estimated Monte Carlo variance. Its square root is the Monte Carlo standard error, which describes simulation noise in estimating the weighted average. It does not measure uncertainty about whether the scenarios or weights are scientifically appropriate. If simulations are dependent, for example because of a shared random-number or resampling design, the independence-based variance formula may not apply without adjustment.

Diagnostics, Alternatives, and Limitations

Before interpreting an average, verify that all component operating characteristics use the same event definition, decision rule, and relevant analysis conventions. Check that the weights are nonnegative, sum to one, and have a stated rationale. Inspect the scenario-specific values as well as the average. A striking gap between the average and a poor result in a heavily weighted scenario may be important; a poor result in a lightly weighted scenario may still matter if that scenario is scientifically or ethically important.

A practical diagnostic is to recompute the summary under plausible alternative weights. If modest, defensible changes in weights substantially alter the result, the headline average is sensitive to the chosen scenario distribution and should not be presented as a robust single summary. This check does not identify the “correct” weights, but it makes the dependence visible. Another diagnostic is to compare the weighted result with the minimum and maximum scenario-specific values: any weighted average with valid weights must lie between them.

The alternatives answer different questions. Equal weights summarize the listed scenarios symmetrically, regardless of their plausibility. A worst-case summary emphasizes the most adverse scenario in the set. A scenario-specific table keeps the full pattern visible without collapsing it. No one summary replaces the others in every application. In particular, an average can conceal concentrated failures, and a worst-case value depends on which scenarios were included. The scenario set and weights are both substantive choices, not merely computational details.

A common mistake is to average scenario results using the number of simulations run in each scenario as the weights. That calculation gives more influence to scenarios that received more computing effort, even if they were not intended to be more probable or important. For example, the unequal-count simulation above uses weights 0.25 and 0.75 because those were the specified scenario weights, not because 1,000 and 4,000 trials were simulated. The correction is to estimate each \(OC_s\) using its own simulation results and then apply the prespecified \(q_s\). Simulation allocation affects Monte Carlo precision, not the target average.