Start With the Development Decision
A trial simulation can generate many outputs, but an output becomes decision-relevant only when linked to a specific choice. For example, a development team may need to decide whether to advance a regimen to a confirmatory trial. That decision depends not just on an estimated treatment effect, but on what effect is clinically worthwhile, what safety profile is acceptable, and what evidence is required to proceed.
Operating characteristics describe how a prespecified decision procedure behaves under specified true conditions. They are not properties of a trial design in isolation: changing the decision threshold, the definition of clinical value, or the true condition being considered can change the result. A useful definition therefore names both the decision and the state of the world under which it is evaluated.
Before calculating an operating characteristic, specify the estimand or clinical state that matters, the trial outputs used by the rule, the rule itself, and the scenario under which its performance is evaluated. The simulation must represent the trial and analysis relevant to that decision. If any of these definitions changes, the operating characteristic may change even if the treatment and sample size do not.
Success Probability
For a development decision, success should mean that the prespecified rule recommends action under a state in which that action is warranted. Suppose \(\Theta_{\mathrm{worthwhile}}\) is the set of true states in which the regimen meets the development target. Conditional success probability at a particular state is
This is the probability of a go decision under that true state. It is often called power when “go” is exactly rejection of a specified null hypothesis, but the development rule may include more than a hypothesis test. For instance, it may require both a minimum efficacy estimate and an acceptable safety result. In that case, the success probability belongs to the complete rule, not to the efficacy test alone.
“Success” must not be defined merely as a favorable trial result without stating the clinical target. A trial can meet a statistical threshold when the true effect is too small to justify further development. Conversely, a potentially worthwhile effect can fail to meet a stringent decision threshold. Reporting the success probability at one or more relevant true states makes this dependence visible.
Worked Example: Probability of a Go Decision at a Target State
Consider a hypothetical development decision to advance a regimen if the estimated benefit is at least 5 points on a clinical scale and the estimated rate of a specified adverse event is no more than 12%. The rule is “go” only if both conditions are met. At a particular true state judged worthwhile, a trial-outcome distribution assigns probability 0.72 to both conditions being met, 0.08 to efficacy alone meeting its condition, 0.10 to safety alone meeting its condition, and 0.10 to neither meeting its condition.
Define the event. Success is the event that the rule recommends go. Only the outcome category in which both conditions are met triggers go. Therefore,
The success probability at this target state is 72%. The other probabilities sum to \(0.08+0.10+0.10=0.28\), the probability of no-go under this state. The result applies to the stated target state and decision rule; it is not a general probability that the regimen will succeed in development.
False-Positive Risk
A false-positive development decision occurs when the rule recommends go in a true state where the regimen does not meet the stated development target. Let \(\Theta_{\mathrm{not\ worthwhile}}\) be the set of such states. At a specified state \(\theta\) in that set, the conditional false-positive risk is
This probability is often called the false-go probability or false-advancement probability. If the rule is simply a test of a specified null hypothesis and a false go is exactly a rejection when that null is true, the probability is the type I error rate for that null and rule. A broader development decision may instead define an unacceptable region by a clinically meaningful efficacy threshold, a safety constraint, or both. In that case, report false-positive risk for the relevant unacceptable states rather than assuming it is identical to a conventional type I error rate.
The conditional risk is not the probability that the null hypothesis is true after seeing the trial result. It is a long-run probability of the decision rule recommending go, conditional on a particular true state. If a single summary over several unacceptable states is needed, their weights must be stated: different weights produce different averages. Without an explicit weighting basis, reporting risks separately across states is clearer.
Worked Example: False Advancement Under an Unacceptable State
Use the efficacy-and-safety go rule from the previous example. At a hypothetical true state where the benefit is below the minimum worthwhile level, trial-output probabilities are 0.03 for passing both efficacy and safety conditions, 0.17 for passing efficacy only, 0.05 for passing safety only, and 0.75 for passing neither. Go still requires both conditions.
The false-positive event is therefore limited to the “both pass” outcome. Its probability is
The sum \(0.03+0.17+0.05+0.75=1.00\), so the categories account for all possible outcomes. The 17% probability of passing efficacy alone is not a false go under this particular rule, because the safety condition also has to pass. If the decision rule were changed to ignore safety, the false-positive risk would change.
Estimation Accuracy
A go/no-go decision does not fully describe what a trial tells the team. If development choices depend on the size of the estimated treatment effect, the accuracy of that estimate also matters. Let \(\hat{\theta}\) estimate a true quantity \(\theta\). Conditional on \(\theta\), common summaries include bias, mean squared error, root mean squared error, and the probability that the estimate is within a clinically chosen tolerance.
Here \(\delta\) is a prespecified tolerance in the same units as \(\theta\). Bias describes average signed error; RMSE combines variability and bias; and the tolerance probability directly answers how often the estimate is close enough for a stated purpose. No one measure is universally best. A small average bias can coexist with substantial trial-to-trial variability, while a high probability of falling within a broad tolerance can conceal errors that matter near a decision threshold.
Interval performance can also be relevant when the decision uses an interval. For example, coverage is the probability, under a specified true state and repeated application of the analysis, that the reported interval contains \(\theta\). Coverage is not the same as the accuracy of every point estimate, nor does nominal coverage guarantee that a particular interval will be narrow enough to inform the decision.
Worked Example: Accuracy of a Treatment-Effect Estimate
Suppose the true benefit is 10 points. A hypothetical distribution of trial estimates assigns probabilities 0.10, 0.20, 0.40, 0.20, and 0.10 to estimates of 6, 8, 10, 12, and 14 points, respectively. Calculate bias, RMSE, and the probability of estimating within 2 points of the truth.
The expected estimate is
Thus the bias is \(10-10=0\) points. The squared errors at the five possible estimates are 16, 4, 0, 4, and 16 points squared. Hence,
Estimates within 2 points of 10 are 8, 10, and 12, so the probability of meeting that tolerance is \(0.20+0.40+0.20=0.80\), or 80%. The zero bias does not mean every estimate is exact: in this distribution, 20% of estimates are 4 points from the truth.
Decision Loss
Success probability, false-positive risk, and estimation accuracy describe performance, but they do not by themselves say how consequential an error is. A false advancement may consume substantial time and resources; a false stop may discard a worthwhile treatment. Decision loss makes these consequences explicit by assigning a loss to each combination of true state and decision.
For discrete states and trial outcomes, the expected loss follows by summing the loss for each state and outcome, weighted by their joint probabilities. Equivalently, first average over trial outcomes conditional on each state, then average across states using specified state weights. This decomposition shows why a decision rule with a higher success probability need not minimize expected loss: it may also produce more costly false advancement.
The state weights \(P(\theta)\) must have a defensible interpretation, such as an explicitly chosen decision-analysis distribution. If no such weights are appropriate, report expected loss conditionally by state rather than presenting a single overall value. In either case, state the units or scale of loss and explain how its values were chosen.
Worked Example: Expected Loss for a Go/No-Go Rule
Consider two true states: worthwhile and not worthwhile. For a hypothetical decision analysis, assign weights 0.60 and 0.40 to these states. The current rule has a 0.75 probability of go when the treatment is worthwhile and a 0.05 probability of go when it is not worthwhile. Assign loss 20 units to a false no-go, loss 10 units to a false go, and loss 0 to either correct decision.
When the treatment is worthwhile, false no-go occurs with probability \(1-0.75=0.25\). The conditional expected loss is \(0.25(20)=5\) units. When the treatment is not worthwhile, false go occurs with probability 0.05, giving conditional expected loss \(0.05(10)=0.5\) units. Averaging across the stated state weights gives
This value is conditional on the specified state weights, error costs, and decision performance. It is not an intrinsic score for the trial. If the consequences or weights change, the expected loss changes and a different rule could be preferred.
From Trial Outputs to Decision Measures
The mapping is clearest when each operating characteristic is tied to an output of the trial analysis and the exact event or quantity used in its definition. A simulation should apply the same analysis and decision rule that would be used in the development decision.
State which efficacy and safety conditions hold, including whether the treatment is worthwhile under the stated development objective.
These may include an estimated treatment effect, an interval, a test result, or a safety summary. Use only outputs that the actual decision rule uses.
Translate each trial outcome into go or no-go. Write down which output conditions must be met, including any joint efficacy and safety requirements.
For success or false-positive risk, evaluate the probability of go under the relevant true state. For estimation accuracy, summarize the distribution of estimation errors. For loss, weight each decision outcome by its assigned consequence.
The calculation is only as meaningful as its assumptions. The true states must represent the clinical conditions the decision concerns; the modeled trial outputs must be relevant to the analysis; and the decision rule and loss assignments must be explicit. If safety is part of the decision, omitting its joint behavior with efficacy can misrepresent success or false-positive risk. If an estimate is used for a threshold decision, summarizing accuracy only at a distant true value may miss performance near the threshold.
Diagnostics, Alternatives, and Limitations
A first diagnostic is to check that outcome categories are exhaustive and mutually exclusive: their probabilities should sum to 1. Check that the event called “success” matches the stated rule, and that an outcome that fails a required safety gate cannot inadvertently count as a go. For accuracy, verify units and distinguish signed error from absolute or squared error. For expected loss, check that each state-decision pairing has an assigned consequence and that state weights sum to 1 when an overall weighted expectation is reported.
The main alternative to a single aggregate operating characteristic is to report a profile across relevant true states. This makes conditional success and false-positive risks visible without hiding them behind chosen weights. A weighted expected loss answers a different question: it combines decision performance with the relative importance or plausibility assigned to each state. Accuracy summaries are complementary to both approaches because they describe estimation rather than only the action taken or its consequences.
These summaries do not remove uncertainty about whether the assumed trial and analysis reflect future conduct, nor do they establish that the chosen target, risk tolerance, or loss scale is clinically appropriate. A low false-positive risk does not guarantee a high success probability, and a high success probability does not guarantee an accurate effect estimate. Present the measures together when each addresses a distinct part of the decision, and state the conditions to which each applies.
Common mistake: “The false-positive risk is 5%, so there is a 95% probability that a go decision is correct.” This reverses a conditional probability. A 5% false-positive risk means that, under the specified unacceptable true state, the rule recommends go with probability 5%. It does not give the probability that the state is acceptable given a go result; that requires additional information about the distribution of true states and trial outcomes.
Check Your Understanding
Answer each question using the stated decision context. Brief solutions follow.
- A go rule has a 0.68 probability of recommending go when the true effect meets the worthwhile target. What is the success probability at that state?
- At an unacceptable true state, the rule recommends go with probability 0.04. Does this mean there is a 4% probability that the state is unacceptable after observing a go decision?
- An estimator has bias 0 but RMSE 3 mg/L. Can it still have substantial estimation error in individual trials?
- A decision analysis assigns false advancement a loss of 8 units and false stopping a loss of 24 units. Why might a rule with a somewhat higher false-positive risk still have lower expected loss?
Solutions. (1) The conditional success probability is 0.68, or 68%, under the stated worthwhile state and rule. (2) No. The 0.04 is the probability of go conditional on that unacceptable state, not the probability of the state conditional on go. (3) Yes. Zero bias concerns the average signed error; RMSE reflects error magnitude and may be large despite zero average bias. (4) False stopping is assigned three times the loss of false advancement, so a rule that reduces false stops enough may have lower expected loss even if it increases false go decisions. The comparison requires the probabilities of both errors and the stated loss assignments.