Searching For A Calibrated Decision Cutoff

Searching for a Cutoff That Meets a Constraint

A decision rule based on a posterior probability often declares a treatment successful when the posterior probability of benefit exceeds a chosen cutoff. That cutoff is a design choice: a lower cutoff tends to declare success more often, while a higher one tends to be more stringent. Calibration asks which cutoff makes a specified operating characteristic satisfy a target, such as keeping the false-positive rate at or below 0.05 under a null scenario.

The earlier material on calibrating a posterior probability cutoff establishes the goal of choosing a threshold against a simulated operating-characteristic criterion. Here the focus is the search itself: how to locate a cutoff efficiently, how to preserve feasibility during the search, and how to diagnose whether the assumed ordering of cutoffs is valid.

Definition: For simulated trial replicate \(i\), let \(q_i\) be the posterior probability used by the decision rule and let \(c\in[0,1]\) be its cutoff. The replicate-level decision is \(D_i(c)=I(q_i\ge c)\), where \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. In a specified generating scenario, the operating characteristic is \(R(c)=P(D(c)=1)\). From \(N\) replicates, its estimate is \(\hat R(c)=\frac{1}{N}\sum_{i=1}^{N}D_i(c)\). For a false-positive-rate constraint under a null scenario, the calibration goal is to find a cutoff with \(\hat R(c)\le r_0\), where \(r_0\) is the chosen target.

This definition assumes that the simulated scenario is held fixed while cutoffs are compared, the analysis producing each \(q_i\) is unchanged, and the decision is made only by comparing \(q_i\) with \(c\). In particular, the same replicate-level posterior probability should be used to evaluate every candidate cutoff. These conditions make the decisions nested: raising the cutoff cannot turn a non-success into a success.

Why the False-Positive Rate Is Monotone

For \(c_1<c_2\), any replicate with \(q_i\ge c_2\) also has \(q_i\ge c_1\). Therefore, for each replicate, \(D_i(c_2)\le D_i(c_1)\). Averaging these inequalities over the replicates gives

\hat R(c_2)=\frac{1}{N}\sum_{i=1}^{N}D_i(c_2) \le \frac{1}{N}\sum_{i=1}^{N}D_i(c_1)=\hat R(c_1).

Taking expectations under the fixed generating scenario gives the same ordering for the true operating characteristic: \(R(c_2)\le R(c_1)\). This is a non-increasing relationship, not necessarily a strictly decreasing one. Between posterior-probability values observed in the replicates, the estimated rate may stay constant; at an observed value it can drop. Thus the estimated operating characteristic is generally a step function, and interpolation between cutoffs is not justified by this ordering alone.

Key result: Under the fixed-rule assumptions above, the set of cutoffs satisfying \(\hat R(c)\le r_0\) is an upper set: if a cutoff is feasible, every larger cutoff is also feasible. This result supports grid search and bisection. It does not imply that an estimated feasible cutoff necessarily meets the target for the true operating characteristic.

Grid Search

A grid search evaluates a prespecified set of candidate cutoffs, then selects a feasible one according to the design objective. If the objective is the least stringent cutoff on the grid that meets an upper bound, choose the smallest grid value with \(\hat R(c)\le r_0\). The result is the smallest feasible tested value, not necessarily the exact smallest feasible value over all real cutoffs.

Worked Example: Finding the First Feasible Grid Value

Suppose ten hypothetical trial replicates are generated under a null scenario. Their posterior probabilities are 0.96, 0.91, 0.87, 0.83, 0.80, 0.76, 0.72, 0.68, 0.54, and 0.31. The false-positive-rate target is \(r_0=0.30\), and the candidate grid is 0.70, 0.80, 0.85, and 0.90. For each candidate, count the scores at least as large as the cutoff.

At \(c=0.70\), seven scores meet the decision rule, so \(\hat R(0.70)=7/10=0.70\). At \(c=0.80\), five scores meet it, giving \(\hat R(0.80)=5/10=0.50\). At \(c=0.85\), three scores meet it, giving \(\hat R(0.85)=3/10=0.30\). At \(c=0.90\), two scores meet it, giving \(\hat R(0.90)=2/10=0.20\).

The first candidate on the grid to satisfy the target is 0.85, since \(0.30\le0.30\), whereas the preceding candidate has rate \(0.50>0.30\). The calibrated grid value is therefore 0.85. This small example illustrates the search, but ten replicates would generally provide little precision for estimating a clinical-trial operating characteristic.

Grid search is straightforward and transparent. Its resolution is limited by the spacing of the candidate values: a coarse grid can skip over useful cutoffs, while a fine grid requires more evaluations. If all cutoffs are evaluated on the same replicate-level posterior probabilities, the resulting estimated rates must be non-increasing. A rate that rises at a higher cutoff is a diagnostic warning, not a feature to smooth away.

Bisection With a Feasible Endpoint

When the rate is non-increasing, bisection can narrow the search interval without evaluating every point on a fine grid. Maintain a lower endpoint \(c_L\) known to be infeasible and an upper endpoint \(c_U\) known to be feasible. At each iteration, evaluate the midpoint. If it meets the constraint, replace \(c_U\) with the midpoint; otherwise replace \(c_L\). The upper endpoint remains feasible throughout.

Bracket the target.
Choose \(c_L<c_U\) with \(\hat R(c_L)>r_0\) and \(\hat R(c_U)\le r_0\). If no feasible upper endpoint is found, expand or reconsider the search range rather than applying bisection without a bracket.

Evaluate the midpoint.
Set \(c_M=(c_L+c_U)/2\) and calculate \(\hat R(c_M)\) using the same simulated replicates.

Keep the feasible side.
If \(\hat R(c_M)\le r_0\), set \(c_U=c_M\). Otherwise set \(c_L=c_M\). Monotonicity preserves the infeasible-lower, feasible-upper bracket.

Stop and report carefully.
Stop when \(c_U-c_L\) is no greater than the chosen search tolerance. Report the feasible upper endpoint, along with its estimated rate and the target. Do not report the infeasible lower endpoint as a solution.

Worked Example: Preserving Feasibility When Rounding

Suppose 1,000 hypothetical null replicates are used with a target false-positive rate of 0.200. A bisection search has narrowed the bracket so that the lower endpoint \(c_L=0.7734\) has 201 successes and the upper endpoint \(c_U=0.7734375\) has 200. Thus

\hat R(0.7734)=\frac{201}{1000}=0.201, \qquad \hat R(0.7734375)=\frac{200}{1000}=0.200.

The upper endpoint meets the target; the lower endpoint does not. The difference of one success is consistent with a replicate having \(q_i=0.77342\): it is counted at 0.7734 because \(0.77342\ge0.7734\), but not at 0.7734375 because \(0.77342<0.7734375\).

Therefore, reporting 0.7734 as the feasible cutoff would be incorrect for these replicates. Reporting 0.7734375 preserves the demonstrated feasibility. If the operational cutoff must be reported to hundredths, rounding upward to 0.78 cannot increase the empirical rate, by monotonicity; its rate is at most 0.200 on this same replicate set. In any other rounding scheme, recalculate the rate at the rounded cutoff before claiming it meets the criterion.

The bisection bracket bounds the location of the transition in the estimated step function to the specified tolerance; it does not guarantee that the selected cutoff is uniquely optimal. The estimated rate can be constant across part or all of the bracket. Also, the true operating characteristic is not known exactly: a cutoff meeting the target in a finite simulation can have Monte Carlo uncertainty. The search procedure addresses locating a cutoff for the estimated criterion; uncertainty in that estimate is a separate issue.

Monotonicity Checks and Failure Modes

A practical monotonicity check evaluates several ordered cutoffs using the same replicate-level scores and confirms that the counts of decisions do not increase as the cutoff rises. This check tests both the assumed structure and the implementation. If the rule is exactly \(I(q_i\ge c)\), a non-monotone sequence of counts cannot arise from Monte Carlo randomness in those fixed scores.

Worked Example: Diagnosing a Monotonicity Check

In 100 hypothetical null replicates, the same posterior scores produce 31 decisions at cutoff 0.70, 18 at 0.80, and 7 at 0.90. The estimated rates are \(31/100=0.31\), \(18/100=0.18\), and \(7/100=0.07\). The sequence is non-increasing, as required.

Now suppose a program reports 0.31, 0.18, and 0.23 for those cutoffs. If it used the same 100 scores and the same decision rule, 23 decisions at 0.90 cannot exceed the 18 decisions at 0.80: every score at least 0.90 is also at least 0.80. The program must be checked for such problems as applying different comparison signs, misaligning replicate results, or changing another part of the decision rule between evaluations.

If separate simulations were run at each cutoff, the estimated rates need not be ordered because each estimate has simulation variability. That pattern alone does not prove the true operating characteristic is non-monotone. For a direct structural check, retain one set of replicate-level scores and reevaluate the decisions at each cutoff. If the actual decision rule includes additional cutoff-dependent conditions, establish whether nesting still holds before using bisection.

Grid search and bisection answer slightly different practical needs. A grid search displays the rate at every tested candidate and makes irregularities easy to inspect; it is often preferable when the candidate cutoffs are few or operationally prespecified. Bisection is more efficient when the cutoff is continuous, a monotone bracket is available, and a narrow search interval is desired. Neither method creates information absent from the simulations, and neither should conceal a failed monotonicity check.

Common Pitfalls and Limitations

Rounding down a feasible endpoint. A tiny decrease in cutoff can include additional scores and violate the target. Report the feasible endpoint or reevaluate the criterion at the operationally rounded value.
Treating a coarse grid result as an exact optimum. The smallest feasible tested cutoff is only the smallest on that grid. Refine the grid or use bisection if a more precise boundary is needed.
Using bisection without a valid bracket. The method requires one endpoint above the target and another at or below it, as well as monotonicity over the search range.
Expecting strict decreases. Rates may remain unchanged across many cutoffs because no replicate scores lie in that interval. A plateau is compatible with a non-increasing step function.
Confusing estimated feasibility with assured true performance. The empirical rate is based on a finite number of replicates. Meeting the target in that simulation does not by itself establish that the true rate is below the target.

The central safeguard is to define the scenario, analysis, decision rule, and target before searching. Then use paired replicate-level results to verify the ordering, retain an explicitly feasible endpoint, and state the cutoff precision honestly. If monotonicity fails under a rule that should be nested, investigate the implementation before interpreting a calibrated value.

Check Your Understanding

Answer each question, then compare with the solutions below.

For fixed replicate scores and the rule \(D_i(c)=I(q_i\ge c)\), why must the estimated operating characteristic be non-increasing in \(c\)?
A grid has estimated rates 0.12, 0.08, and 0.05 at cutoffs 0.70, 0.80, and 0.90. For a target of 0.08, what is the smallest feasible tested cutoff?
A bisection search has an infeasible lower endpoint and a feasible upper endpoint. The midpoint is feasible. Which endpoint should be updated, and why?
At a reported cutoff \(c\), the estimated rate is exactly the target. The cutoff is rounded downward for presentation. What must be done before claiming the rounded cutoff is feasible?
Separate simulations at increasing cutoffs yield slightly non-monotone estimated rates. Does this alone refute monotonicity of the true operating characteristic?

Solutions. (1) If \(c_1<c_2\), every score satisfying \(q_i\ge c_2\) also satisfies \(q_i\ge c_1\); hence \(D_i(c_2)\le D_i(c_1)\) for every replicate, and averaging preserves the inequality. (2) The smallest feasible tested cutoff is 0.80, since its rate equals the 0.08 target and the rate at 0.70 exceeds it. (3) Replace the upper endpoint with the midpoint. The midpoint meets the constraint, so it is a feasible upper endpoint and the bracket remains valid. (4) Recompute the estimated rate at the rounded-down cutoff using the replicate scores. A lower cutoff may include more successes, so the original rate does not establish feasibility after rounding. (5) No. Separate simulations introduce Monte Carlo variability; evaluate the same replicate-level scores at each cutoff to check the nested decision rule directly.