Separating False Positives And False Negatives

Why Separate the Two Errors?

A trial decision rule can make two distinct mistakes. It can declare evidence of benefit when the treatment effect is actually null, or it can fail to declare benefit when a clinically relevant effect is present. These errors have different consequences: a false positive may lead to adoption of an ineffective treatment, while a false negative may withhold a useful treatment. A single summary of overall accuracy can conceal this trade-off.

In Bayesian trials, the final decision may be based on a posterior probability or another prespecified rule. Once that rule is fixed, its operating characteristics can be assessed by simulating many trial datasets under scenarios with known true parameter values and applying the rule to every replicate. The resulting error rates describe the long-run behavior of the complete design-and-analysis procedure under those scenarios. They are not posterior probabilities about whether a particular trial conclusion is correct.

Define the Decision and the Truth

Let \(\theta\) denote the data-generating parameter, and let \(D\) be the binary decision from one simulated trial: \(D=1\) means the rule declares evidence of efficacy, and \(D=0\) means it does not. For example, a rule might set \(D=1\) when a prespecified posterior probability exceeds a chosen threshold. The particular rule must be held fixed while its operating characteristics are evaluated.

Definition: At a true scenario \(\theta \in \Theta_0\), where \(\Theta_0\) is the null parameter region, the false-positive probability is \(P_\theta(D=1)\). At a true scenario \(\theta \in \Theta_1\), where \(\Theta_1\) is the alternative region, the false-negative probability is \(P_\theta(D=0)\). At a specified alternative scenario, power is \(P_\theta(D=1)=1-P_\theta(D=0)\).

These probabilities are conditional on the data-generating scenario. For a composite null or alternative, there is not generally one error probability: the probability can vary with \(\theta\). Results should therefore identify the scenario or scenarios used. Calling a result “the false-positive rate” without specifying which null scenarios were evaluated can hide important variation.

The definitions also depend on what counts as a positive decision. In a two-action design, “positive” usually means declaring efficacy. If the protocol has additional actions, such as “inconclusive,” the analyst must specify whether that action is counted as a non-efficacy decision or reported separately. The classification should match the clinical and regulatory question rather than be chosen after seeing the simulation results.

Estimate Each Probability From Its Own Replicates

Suppose \(M_0\) independent simulated trials are generated at a fixed null scenario \(\theta_0\). Let \(I_j=1\) if replicate \(j\) declares efficacy and \(I_j=0\) otherwise. The estimated false-positive probability is the fraction of null replicates that declare efficacy. At an alternative scenario \(\theta_1\), instead count the replicates that do not declare efficacy to estimate the false-negative probability.

\widehat{\mathrm{FP}}(\theta_0) = \frac{1}{M_0}\sum_{j=1}^{M_0} I_j = \frac{K_0}{M_0}, \qquad \widehat{\mathrm{FN}}(\theta_1) = \frac{1}{M_1}\sum_{j=1}^{M_1} J_j = \frac{K_1}{M_1},

Here \(K_0\) is the number of false-positive decisions among the \(M_0\) null replicates, \(J_j=1\) marks a non-efficacy decision in alternative replicate \(j\), and \(K_1\) is the number of such false negatives among \(M_1\) alternative replicates. The estimated power at \(\theta_1\) is \(1-K_1/M_1\). The denominators are different because each probability is estimated under its relevant truth condition.

For independent replicates generated at the same fixed scenario, each error indicator is Bernoulli with the corresponding error probability \(q\). Thus the error count \(K\) has a binomial distribution with size \(M\) and probability \(q\), and \(\widehat q=K/M\) has expectation \(q\) and variance \(q(1-q)/M\). Its Monte Carlo standard error is estimated by substituting \(\widehat q\) for \(q\):

SE(\widehat q) \approx \sqrt{\frac{\widehat q(1-\widehat q)}{M}}.

This standard error measures simulation noise in estimating the operating characteristic, not uncertainty in the treatment effect within a trial. It is largest near \(q=0.5\) and decreases as \(M\) increases. When the estimated error probability is near zero, the plug-in standard error can also be near zero even though the true error probability is not known to be zero. If errors are rare, report the count and denominator and use a suitable binomial interval or an upper bound rather than claiming that the risk vanishes.

Worked Example: False Positives Under a Null Scenario

Consider a hypothetical trial design with a fixed Bayesian decision rule. At a prespecified null scenario, \(M_0=1{,}000\) independent trial datasets are simulated. The rule declares efficacy in \(K_0=34\) replicates. Assume the simulator generates the intended null scenario, the same analysis and decision rule are applied to every replicate, and each replicate is independently generated.

The estimated false-positive probability is \(\widehat{\mathrm{FP}}=34/1{,}000=0.0340\), or 3.40%. Its estimated Monte Carlo standard error is \(\sqrt{0.0340(1-0.0340)/1{,}000}=\sqrt{0.00003284}=0.00573\), or 0.573 percentage points.

The result estimates how often this design declares efficacy at this particular null scenario. It does not say that there is a 3.40% probability the efficacy declaration in any one observed trial is false. That latter question conditions on observed evidence and requires a different probability statement.

Worked Example: False Negatives and Power Under an Alternative

In a separate hypothetical evaluation, the true treatment effect is set to a prespecified beneficial alternative. Among \(M_1=800\) independent simulated trials, \(612\) declare efficacy and \(188\) do not. Suppose the decision rule, trial design, and analysis are unchanged from the specification being evaluated.

The estimated false-negative probability at this alternative is \(188/800=0.235\), or 23.5%. The corresponding estimated power is \(612/800=0.765\), or 76.5%; the two estimates sum to one because the decision has exactly two possible outcomes. The Monte Carlo standard error for the false-negative estimate is \(\sqrt{0.235(1-0.235)/800}=\sqrt{0.0002247}=0.0150\), or 1.50 percentage points.

This is the estimated chance of failing to declare efficacy when the specified alternative is true, under the simulated design and analysis. It is not the chance that the treatment has no effect after a non-significant trial. Power and false-negative probability are scenario-specific complements, not general summaries across all possible treatment effects.

Justification and Practical Interpretation

The estimator follows directly from repeated Bernoulli decisions. At a fixed scenario, let an error indicator \(Z_j\) equal one when replicate \(j\) makes the error of interest. Then \(E(Z_j)=q\), where \(q\) is the true error probability. For independent replicates,

E\left(\frac{1}{M}\sum_{j=1}^{M} Z_j\right) = \frac{1}{M}\sum_{j=1}^{M}E(Z_j) = q, \qquad \mathrm{Var}\left(\frac{1}{M}\sum_{j=1}^{M} Z_j\right) = \frac{1}{M^2}\sum_{j=1}^{M}q(1-q) = \frac{q(1-q)}{M}.

Thus the simulated proportion is unbiased for the scenario-specific error probability, and its Monte Carlo variability decreases as the number of independent replicates increases. This justification relies on applying the same specified procedure to each replicate and on independence across simulated replicates. If the replicates are dependent, or if the decision rule is repeatedly tuned using the same simulation results, this simple variance calculation need not apply.

The two error rates answer different design questions. False-positive probability is relevant to the risk of making an efficacy claim under a null scenario. False-negative probability is relevant to the chance of missing an effect at a particular alternative. Clinical interpretation requires specifying what effect sizes matter and what level of each error risk is acceptable; neither rate alone determines whether the design is appropriate.

Worked Example: Comparing Two Decision Rules

Suppose two hypothetical rules are evaluated separately using \(2{,}000\) null replicates and \(1{,}000\) alternative replicates at a clinically relevant effect. Rule A declares efficacy in \(40\) null replicates and fails to declare efficacy in \(300\) alternative replicates. Rule B has \(100\) false-positive decisions and \(180\) false-negative decisions. Assume each rule is applied consistently within its scenario, and the simulations represent the same null and alternative conditions.

For Rule A, the estimated false-positive probability is \(40/2{,}000=0.0200\), or 2.00%, and the estimated false-negative probability is \(300/1{,}000=0.300\), or 30.0%. Its power at the alternative is 70.0%.

For Rule B, the corresponding estimates are \(100/2{,}000=0.0500\), or 5.00%, and \(180/1{,}000=0.180\), or 18.0%. Its power is 82.0%. Rule B has higher estimated power at this alternative, but also a higher estimated false-positive probability at the null scenario. Which rule is preferable depends on the prespecified constraints and the relative consequences of the errors; neither has uniformly lower error in this comparison.

An overall accuracy percentage would require choosing how often null and alternative conditions occur in the evaluation, and could change with that choice. Reporting both scenario-specific rates makes the trade-off visible without that additional weighting.

Diagnostics, Comparisons, and Common Pitfalls

Before interpreting the estimates, check that the simulated truth matches the label attached to each scenario, the decision rule is identical to the intended trial rule, and the numerator counts the correct decision in the correct denominator. Examine the decision counts as well as the proportions. A reported rate of 0% from 20 replicates is much less informative about a small error probability than 0% from 20,000 replicates. Monte Carlo standard errors or binomial intervals help distinguish a genuine design difference from simulation noise.

When several null or alternative parameter values are relevant, show results by scenario rather than silently pooling them. A null result at one value does not establish the same false-positive probability throughout a composite null, and power at one alternative does not characterize power at every treatment effect. This is particularly important when performance changes near a decision boundary.

False-positive and false-negative probabilities are also not the same as posterior probability thresholds. A posterior probability is calculated within an individual trial under its model and prior. An error probability is estimated across repeated trials generated under a specified truth. Nor should the two rates be replaced by overall accuracy: accuracy combines errors and non-errors according to the mix of scenarios included, while the two conditional rates retain their separate interpretations.

Wrong reasoning: “The false-negative rate is 23.5%, so after this trial fails to declare efficacy, there is a 23.5% probability the treatment works.” Correction: 23.5% is a repeated-trial probability under the specified alternative scenario; it is not a posterior probability conditional on an observed trial result.
Wrong reasoning: “No false positives occurred, so the false-positive probability is zero.” Correction: zero observed events is a simulation result with finite precision. State the count and denominator and quantify or bound the remaining Monte Carlo uncertainty.
Wrong reasoning: “The decision rule with greater power is better.” Correction: compare power alongside false-positive probability and the design requirements; an increase in power may come with an increase in false-positive risk.
Wrong reasoning: “The null scenario passed, so the whole null hypothesis is calibrated.” Correction: a single scenario supports a conclusion only at that scenario unless additional analysis establishes performance across the relevant null region.

Key takeaway: Estimate false positives from replicates generated under null scenarios and false negatives from replicates generated under alternative scenarios. Report each rate with its scenario, count, denominator, and Monte Carlo precision.

Check Your Understanding

Answer each question, then check the solutions below.

At a null scenario, 27 of 900 simulated trials declare efficacy. What is the estimated false-positive probability?
At an alternative scenario, 720 of 1,000 simulated trials declare efficacy. What are the estimated power and false-negative probability?
Why does zero observed false positives in a finite simulation not establish a zero false-positive probability?
Can a false-negative estimate at one alternative be interpreted as the probability that a particular non-efficacy trial nevertheless has a beneficial treatment effect? Explain.

Solutions. (1) \(27/900=0.0300\), or 3.00%. (2) Power is \(720/1{,}000=0.720\), or 72.0%; false-negative probability is \(280/1{,}000=0.280\), or 28.0%. (3) The simulation estimates a probability from a finite number of replicates; an unobserved event may still have positive probability, and Monte Carlo uncertainty remains. (4) No. The estimate is conditional on the specified alternative across repeated simulated trials. It is not a posterior probability conditional on one observed trial.