Meeting Multiple Operating-Characteristic Constraints

Why Multiple Constraints Change Calibration

A decision cutoff is often calibrated against more than one performance target. A trial rule might need to keep the probability of declaring success below a specified limit in every null scenario, while also achieving an adequate success probability under several clinically important alternatives. A cutoff that works for one scenario can fail another. The central question is therefore not whether a cutoff performs well on average, but whether the same cutoff satisfies every prespecified constraint.

This tutorial treats operating characteristics as probabilities under specified trial scenarios. These probabilities may be estimated by simulation, but the constraints themselves concern the underlying scenario-specific probabilities. Monte Carlo uncertainty matters because an estimate that appears to meet a target may be imprecise.

Definition: Let \(S\) be a trial’s decision score, such as a posterior probability, and declare success when \(S\ge c\), where \(c\) is the common decision cutoff. For null scenario \(j\), define the false-positive rate \(q_j(c)=P_j(S\ge c)\). For alternative scenario \(k\), define the success probability \(r_k(c)=P_k(S\ge c)\). Given a false-positive limit \(\alpha\) and a minimum success target \(\beta\), the cutoff is feasible if \(q_j(c)\le\alpha\) for every null scenario and \(r_k(c)\ge\beta\) for every alternative scenario.

Here \(P_j\) and \(P_k\) refer to repeated trials generated under the stated scenario, with the trial design, analysis, prior, and decision rule held fixed. A null scenario may differ from another in nuisance parameters even if both represent no treatment effect. Alternative scenarios specify the treatment effects and other conditions for which success is required. The requirements apply to the listed scenarios; they do not automatically establish performance for unexamined values between them.

For a threshold rule, increasing \(c\) makes success harder to declare. Consequently, each \(q_j(c)\) and \(r_k(c)\) is nonincreasing in \(c\). Each null constraint therefore favors a sufficiently high cutoff, while each alternative constraint favors a sufficiently low cutoff. The calibration problem is to find their overlap.

Combining the Constraints

For each null scenario, identify the cutoffs at which its false-positive rate is at most \(\alpha\). For each alternative, identify cutoffs at which its success probability is at least \(\beta\). The overall feasible set is the intersection of all these sets. When the operating-characteristic curves are monotone, this can be described informally by a lower bound imposed by the most demanding null scenario and an upper bound imposed by the most demanding alternative.

\mathcal{C}_{\mathrm{feasible}} = \left(\bigcap_{j=1}^{J}\{c:q_j(c)\le\alpha\}\right) \cap \left(\bigcap_{k=1}^{K}\{c:r_k(c)\ge\beta\}\right).

If the null constraints require \(c\) to be at least \(L\), and the alternative constraints require it to be at most \(U\), a feasible cutoff exists when the resulting ranges overlap. In a continuous, strictly monotone idealization this is \(L\le U\). Simulated operating characteristics are often discrete or step-shaped, however, so evaluate candidate cutoffs directly rather than assuming a smooth curve or relying on interpolation at a boundary.

Worked Example: Finding the Overlap on a Cutoff Grid

Suppose a hypothetical trial uses a posterior-score cutoff and has two null scenarios and two alternative scenarios. The required false-positive limit is \(\alpha=0.025\), and the minimum success probability is \(\beta=0.80\). The following simulated probabilities are illustrative estimates:

Cutoff \(c\)	Null 1 false-positive rate	Null 2 false-positive rate	Alternative 1 success	Alternative 2 success
0.86	0.020	0.028	0.90	0.85
0.88	0.016	0.024	0.87	0.81
0.90	0.012	0.018	0.84	0.78

At \(c=0.86\), Null 2 fails because \(0.028>0.025\), even though both alternative targets pass. At \(c=0.90\), both null limits pass, but Alternative 2 fails because \(0.78<0.80\). At \(c=0.88\), the two false-positive estimates, \(0.016\) and \(0.024\), are each no greater than \(0.025\), and the two success estimates, \(0.87\) and \(0.81\), are each at least \(0.80\). Thus \(0.88\) meets all four constraints at the point-estimate level.

This grid establishes only that the evaluated cutoff \(0.88\) passes based on those estimates. It does not establish that every cutoff between \(0.86\) and \(0.90\) passes, nor does it account for Monte Carlo uncertainty. Those require additional evaluation.

Monte Carlo Uncertainty Across Several Constraints

In simulation, each estimated operating characteristic is usually a proportion. If \(R\) independent trial replicates are simulated under a scenario and \(X\) of them meet the success rule, then \(\hat{p}=X/R\) estimates the corresponding probability. The Monte Carlo standard error is approximately \(\sqrt{\hat{p}(1-\hat{p})/R}\). The estimate alone does not show how precisely the probability is known.

When several constraints must hold, assessing each with a separate 95% interval does not yield a 95% guarantee that all intervals cover simultaneously. A standard approach is Bonferroni: for \(m\) one-sided bounds, construct each bound with individual coverage \(1-\gamma/m\). The probability that any bound misses is at most the sum of its individual miss probabilities, \(\gamma\), regardless of dependence among the bounds. If approximate Normal bounds are used, the resulting simultaneous coverage is approximate, not guaranteed.

P(\text{at least one bound misses}) \le \sum_{\ell=1}^{m}P(\text{bound }\ell\text{ misses}) \le m\frac{\gamma}{m} =\gamma.

For a conservative assessment, require the one-sided upper bound for every null rate to be at most \(\alpha\), and the one-sided lower bound for every alternative success probability to be at least \(\beta\). With two null and two alternative constraints, there are \(m=4\) bounds. To target 95% simultaneous coverage, set \(\gamma=0.05\); each individual bound then needs coverage \(1-0.05/4=0.9875\), or 98.75%. Exact one-sided binomial bounds at tail probability \(0.0125\) provide the individual coverage needed for the Bonferroni guarantee. Normal-approximation Wald bounds at the same nominal tail level yield only approximately 95% simultaneous coverage, because their individual coverage is not exact.

Worked Example: Assessing Four Constraints With Simulation Bounds

At a candidate cutoff in a hypothetical trial, \(R=10{,}000\) replicates are run in each of four scenarios. The null scenarios produce 72 and 70 false positives; the alternative scenarios produce 8,300 and 8,200 successes. Let \(\alpha=0.01\) and \(\beta=0.80\). First calculate the estimates:

\hat{q}_1=\frac{72}{10{,}000}=0.0072,\quad \hat{q}_2=\frac{70}{10{,}000}=0.0070,\quad \hat{r}_1=0.8300,\quad \hat{r}_2=0.8200.

The point estimates all satisfy their targets. To examine Monte Carlo uncertainty with approximate 95% simultaneous coverage, use one-sided Wald bounds with \(z_{0.9875}\approx2.241\). For Null 1, the estimated standard error is \(\sqrt{0.0072(1-0.0072)/10{,}000}\approx0.0008455\), so the upper bound is \(0.0072+2.241(0.0008455)\approx0.00910\).

For Null 2, the standard error is \(\sqrt{0.0070(1-0.0070)/10{,}000}\approx0.0008337\), giving an upper bound of \(0.0070+2.241(0.0008337)\approx0.00887\).

For Alternative 1, the standard error is \(\sqrt{0.8300(1-0.8300)/10{,}000}\approx0.003756\), and the lower bound is \(0.8300-2.241(0.003756)\approx0.8216\). For Alternative 2, it is \(\sqrt{0.8200(1-0.8200)/10{,}000}\approx0.003842\), giving \(0.8200-2.241(0.003842)\approx0.8114\).

Both approximate null upper bounds are below \(0.01\), and both approximate alternative lower bounds are above \(0.80\). These Wald bounds therefore support the conclusion that the candidate meets all four constraints, with approximately 95% simultaneous coverage under the Normal approximation. They do not guarantee at least 95% simultaneous coverage. If that guarantee is required, calculate exact one-sided binomial bounds at tail probability \(0.0125\) for each scenario and apply Bonferroni. The conclusion should be based on those exact bounds, which may be wider.

When the Constraints Do Not Overlap

Sometimes no cutoff meets all the targets. A null scenario may require a cutoff high enough to control false positives, while a difficult alternative may require a lower cutoff to retain adequate success probability. If the null-imposed lower range lies above the alternative-imposed upper range, selecting a compromise does not make the rule compliant: it simply means at least one target fails.

Worked Example: Recognizing an Infeasible Target Combination

Consider a hypothetical trial with \(\alpha=0.01\) and \(\beta=0.80\). Simulations show that at \(c=0.89\), the most demanding null scenario has estimated false-positive rate \(0.012\), while the weakest alternative has estimated success probability \(0.81\). At \(c=0.91\), those estimates are \(0.008\) and \(0.77\), respectively.

At \(c=0.89\), the alternative target passes, \(0.81\ge0.80\), but the null limit fails, \(0.012>0.01\). At \(c=0.91\), the null limit passes, \(0.008\le0.01\), but the alternative target fails, \(0.77<0.80\). Monotonicity explains the conflict: raising the cutoff can reduce the false-positive rate, but cannot increase the alternative success probability. The data therefore indicate a tension between the requirements; they do not justify declaring either cutoff feasible. Further evaluation should check the full curves and Monte Carlo uncertainty before concluding that the constraint sets are disjoint.

If the conflict is confirmed, possible responses include revisiting whether the target values and scenarios reflect the design objective, or evaluating a changed trial design or decision rule. Each change creates a new calibration problem: all constraints must be reassessed under the revised design. Quietly dropping a difficult scenario or relaxing a target after seeing the results changes the stated objective and should not be presented as having met the original requirements.

Diagnostics, Alternatives, and Limitations

A reliable assessment separates three questions: whether the point estimates meet the targets, whether Monte Carlo uncertainty is small enough to support that conclusion, and whether the chosen scenarios adequately represent the intended operating conditions. A pass on the first question is not automatically a pass on the other two.

Inspect each constraint separately. Reporting only the worst null rate and best alternative success probability can conceal which scenario fails. Record the scenario, estimate, target, and uncertainty bound for every requirement.
Check cutoff monotonicity. For a fixed score and rule \(S\ge c\), simulated estimates should generally decrease as \(c\) increases. Small reversals can occur from Monte Carlo noise; large or systematic reversals may indicate inconsistent simulation inputs, coding errors, or a changed decision rule.
Use suitable binomial bounds. Wald bounds can behave poorly near zero or one, or with few replicates. Exact binomial bounds avoid that particular approximation, though they can be conservative. Bonferroni can also be conservative when many constraints are assessed.
Do not confuse simulation precision with scenario coverage. More replicates reduce Monte Carlo error for the scenarios simulated. They do not establish performance at unexamined parameter values or remedy a misspecified trial model.

A grid search is transparent and useful for identifying feasible candidate cutoffs, but a coarse grid may miss a narrow feasible range. A finer grid or direct evaluation around the boundary can help, with uncertainty assessed at the selected cutoff. Pointwise intervals at every cutoff do not automatically provide simultaneous coverage over the entire search; selecting a cutoff after looking at noisy estimates can make simple pointwise claims optimistic. A pre-specified candidate set and appropriate simultaneous assessment clarify what has been established.

Another alternative is to optimize a single summary, such as average success probability or a weighted utility, subject to selected constraints. Such optimization can be useful when trade-offs are explicitly part of the objective. It is not equivalent to requiring every scenario-specific target: a favorable average can hide a poor result in one clinically important scenario. Multiple-constraint calibration is most informative when the scenarios and limits are chosen before examining the calibration results.

Key takeaway: A cutoff is feasible when every underlying null rate meets its false-positive limit and every underlying alternative success probability meets its target. To establish this from simulation with the intended confidence level, require the corresponding one-sided bounds to meet the targets. If the required cutoff ranges do not overlap, there is no single cutoff that meets the specified operating-characteristic constraints.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

With a common cutoff, two null rates are \(0.018\) and \(0.023\), and two alternative success probabilities are \(0.84\) and \(0.79\). For \(\alpha=0.025\) and \(\beta=0.80\), do the point estimates meet all constraints?
Why does a cutoff that controls the false-positive rate in one null scenario not necessarily control it in every null scenario?
For four one-sided bounds and a Bonferroni target of 95% simultaneous coverage, what individual coverage level is required?
At a cutoff, an alternative estimate is \(0.82\), but its lower uncertainty bound is \(0.78\) for a target of \(0.80\). What can be concluded?
As \(c\) increases, how should false-positive and success probabilities change for a rule that declares success when \(S\ge c\)?

Solutions. (1) No. The second alternative estimate fails because \(0.79<0.80\), although both null estimates pass. (2) Null scenarios can have different distributions of the decision score, so their probabilities \(P_j(S\ge c)\) need not be equal; each must be checked. (3) Each bound needs individual coverage \(1-0.05/4=0.9875\), or 98.75%. (4) The point estimate passes, but the uncertainty bound does not support concluding that the target is met at the stated bound level. (5) Both probabilities are nonincreasing, since increasing the cutoff can only remove trials from the event \(S\ge c\).