Threshold Selection Without Test-Set Bias

Why the Threshold Is Part of the Prediction Procedure

A classifier may produce a score for each subject, but a decision often requires assigning a class. The threshold converts the score into that decision. Because different thresholds produce different classifications, choosing a threshold after inspecting test-set outcomes amounts to adapting the procedure to the test set. The resulting test performance no longer provides an unbiased evaluation of that full procedure.

The threshold is therefore a decision rule that must be chosen using training data or a separate validation set, according to a rule specified before the held-out test outcomes are examined. The model, score direction, threshold-selection rule, and threshold value should be fixed before evaluating the test set. This builds on the earlier material on protecting a held-out test set from reuse: threshold selection is another form of model selection, even if the model’s coefficients never change.

Definition: Let $S$ be a model score for a subject, with larger scores indicating stronger evidence for the positive class. A threshold $t$ classifies a subject as positive when $S \ge t$, and negative otherwise. Threshold selection is the choice of $t$ using data and a prespecified decision rule. The selected threshold, together with the fitted model and any score-processing steps, forms part of the prediction procedure.

The score need not be a probability. If it is a probability, the probability refers to the positive outcome in a defined target population. A numerical threshold has meaning only in relation to the score’s scale and direction. For a score where smaller values indicate greater evidence for the positive class, reverse the inequality or transform the score before applying the rule. The fitted model and the threshold must also be paired: changing the model can change the score scale, so a threshold selected for one fitted model may not work for another.

Choosing a Threshold From Decision Costs

A cost-based threshold starts with the consequences of the two types of classification error. Let $C_{\mathrm{FP}}$ be the cost of a false positive and $C_{\mathrm{FN}}$ the cost of a false negative. Assume correct classifications have zero cost, the costs are fixed for the decision being made, and the model score $p$ is a credible estimate of the probability of the positive outcome for the subject. For a subject with estimated probability $p$, predicting positive has expected cost $C_{\mathrm{FP}}(1-p)$; predicting negative has expected cost $C_{\mathrm{FN}}p$.

Choose positive when its expected cost is no greater than the cost of choosing negative. Comparing these quantities gives the cost-based threshold:

C_{\mathrm{FP}}(1-p) \le C_{\mathrm{FN}}p \quad\Longleftrightarrow\quad p \ge \frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}.

This result assumes that the two stated error costs capture the relevant consequences and that the probability scale is suitable for the decision. If costs vary across subjects, actions have costs beyond these two errors, or the score is not a probability, this simple threshold need not be appropriate. In particular, a cost-based rule does not by itself make an uncalibrated score a probability.

Worked Example: Applying a Prespecified Cost Threshold

Suppose a hypothetical screening decision assigns a cost of \$200 to a false positive and \$800 to a false negative. The model outputs probabilities for the positive outcome. Under the assumptions above, the threshold is

t = \frac{200}{200+800}=0.20.

Thus, classify a subject as positive if the predicted probability is at least 0.20. Now apply this already chosen threshold to eight hypothetical held-out subjects. Their scores and outcomes are: positive outcomes with scores 0.91, 0.72, 0.27, and 0.11; negative outcomes with scores 0.38, 0.22, 0.18, and 0.06. The held-out outcomes are used here only to evaluate the fixed rule.

At $t=0.20$, scores 0.91, 0.72, 0.27, 0.38, and 0.22 are classified positive. Therefore, there are three true positives, two false positives, one false negative, and two true negatives. The total cost is

2(\$200)+1(\$800)=\$1{,}200, \qquad \frac{\$1{,}200}{8}=\$150\text{ per held-out subject}.

The observed average cost is \$150 per subject in this hypothetical test set. It describes the fixed rule’s results on these subjects; it is not a reason to revise the threshold using their outcomes.

Choosing a Threshold to Meet an Operating Constraint

Some applications specify a requirement such as sensitivity of at least 0.90, or specificity of at least 0.80, rather than a complete cost for each error. The threshold can then be selected on training or validation data to meet that constraint while optimizing a stated secondary objective. For example, choose the lowest threshold that achieves a prespecified minimum specificity if the aim is to maximize sensitivity subject to that constraint. Here, sensitivity is the fraction of positive-outcome subjects classified positive; specificity is the fraction of negative-outcome subjects classified negative.

On a finite validation sample, thresholds can be evaluated at the observed score values. The inequality convention matters: with the rule $S \ge t$, a subject whose score equals the threshold is classified positive. If several thresholds achieve the same objective, specify a tie rule in advance, such as choosing the higher threshold to reduce positive classifications. An empirical constraint is a sample result, not a guarantee that the same constraint will hold in a new population.

Worked Example: Meeting a Minimum Specificity

Consider ten hypothetical validation subjects: five with positive outcomes and scores 0.95, 0.82, 0.68, 0.55, and 0.31; and five with negative outcomes and scores 0.88, 0.61, 0.44, 0.25, and 0.10. The prespecified goal is to maximize validation sensitivity subject to specificity of at least 0.80. With five negative subjects, this requires no more than one false positive.

At $t=0.68$, the positive classifications are the scores 0.95, 0.88, 0.82, and 0.68. This gives three true positives and one false positive, so sensitivity is $3/5=0.60$ and specificity is $4/5=0.80$. Lowering the threshold to 0.61 classifies both 0.88 and 0.61 among the negative subjects as positive, giving two false positives and specificity $3/5=0.60$; the constraint fails. Raising the threshold to 0.82 gives two true positives and one false positive, so sensitivity falls to $2/5=0.40$ while specificity remains 0.80. Thus, among thresholds meeting the constraint, $t=0.68$ has the greatest observed sensitivity.

Apply $t=0.68$, without changing it, to eight hypothetical held-out subjects: positive outcomes with scores 0.93, 0.73, 0.66, and 0.40; negative outcomes with scores 0.85, 0.62, 0.20, and 0.08. The rule predicts positive for 0.93, 0.85, and 0.73. It yields two true positives, one false positive, two false negatives, and three true negatives. Held-out sensitivity is $2/4=0.50$, and specificity is $3/4=0.75$. The lower observed specificity than the validation target does not show that the threshold was selected incorrectly: the target was an empirical validation constraint, and performance varies across samples.

Selection, Evaluation, and a Short Justification

Threshold selection may use the training data, but using the same data both to fit a flexible model and to choose a threshold can make the selection overly adapted to those observations. A separate validation set, or an appropriately nested resampling procedure, provides a clearer separation: fit the model using the designated training data, choose the threshold using data reserved for that choice, then evaluate the locked procedure on subjects not used for either task. If a final model is refitted, its score scale may change; the threshold cannot automatically be assumed to transfer. The fitting and selection procedure must be defined so that the evaluated model-threshold pair is the one intended for deployment.

The reason test-set threshold selection creates optimism can be stated directly. Suppose there are several candidate thresholds and the estimated test performance at candidate $j$ is $\widehat{M}_j=M_j+\varepsilon_j$, where $M_j$ is its true performance and $\varepsilon_j$ is sampling error. Selecting the candidate with the best observed test performance also selects partly for favorable error. For a metric where larger is better, the maximum of the noisy estimates tends to exceed the maximum of their true values:

E\!\left[\max_j \widehat{M}_j\right] \ge \max_j E[\widehat{M}_j] = \max_j M_j.

The inequality follows because $\max_j \widehat{M}_j$ is at least $\widehat{M}_k$ for every fixed $k$; taking expectations and then choosing the $k$ with the largest $M_k$ gives the result. The magnitude of the optimism depends on the data and candidates; it is not a correction formula. The practical implication is to select the threshold without the held-out outcomes, then evaluate it once.

Worked Example: Empirical Cost Selection on Validation Data

A hypothetical validation set has three positive outcomes with scores 0.80, 0.60, and 0.30, and three negative outcomes with scores 0.70, 0.40, and 0.10. Let the prespecified costs be $C_{\mathrm{FP}}=1$ unit and $C_{\mathrm{FN}}=3$ units. Compare thresholds $0.80, 0.60, 0.40,$ and $0.30$, using $S \ge t$.

At $t=0.80$, there is one true positive and two false negatives, with no false positives. Total cost is $0(1)+2(3)=6$ units. At $t=0.60$, there are two true positives, one false negative, and one false positive, for cost $1(1)+1(3)=4$ units. At $t=0.40$, there are two true positives, one false negative, and two false positives, for cost $2(1)+1(3)=5$ units. At $t=0.30$, all three positives are detected and two negatives are false positives, for cost $2(1)+0(3)=2$ units. Among these prespecified candidates, choose $t=0.30$.

For a separate hypothetical held-out set with positive-outcome scores 0.75 and 0.25 and negative-outcome scores 0.55 and 0.12, the fixed threshold predicts positive for 0.75 and 0.55. That gives one false positive and one false negative, for cost $1(1)+1(3)=4$ units, or one unit per subject. This held-out result evaluates the chosen threshold; it does not replace the validation choice.

Diagnostics, Alternatives, and Limitations

Before interpreting the result, check that the score direction and equality rule are correct, that the threshold was selected using only permitted data, and that the selected operating point actually meets its stated validation objective. Inspect the number of positive and negative subjects contributing to the constraint: with very few negatives, one additional false positive can materially change specificity. Also check whether the threshold is stable across reasonable validation resamples or folds. Large variation can signal that the data do not support a precise operating point. Any such resampling must keep the relevant fitting and threshold-selection steps inside the training portion of each evaluation split.

Threshold rules answer different questions. A fixed threshold of 0.50 is a convenient convention for probability scores, but it does not generally minimize error costs or satisfy a clinical operating constraint. Maximizing a summary such as Youden’s index selects a balance between sensitivity and specificity, not necessarily the best decision under unequal costs. Maximizing F1 emphasizes precision and sensitivity while omitting true negatives; it is not a general substitute for specifying consequences. A threshold-free summary such as area under the ROC curve describes ranking over thresholds, but does not identify the threshold for a particular action. Choose among these approaches based on the intended use, not whichever gives the most favorable test result.

A threshold chosen to satisfy a validation constraint may fail to satisfy it in deployment because the validation sample is finite and the deployment population may differ. Decision costs, outcome prevalence, and the consequences of an incorrect classification are context-dependent. If those inputs change, the appropriate threshold may change as well. The threshold must also remain tied to the model and score scale for which it was selected. These considerations make a clear record essential: the selection data, rule, costs or constraint, score direction, tie convention, selected value, and held-out evaluation should all be documented.

Wrong reasoning: “The test-set sensitivity is highest at 0.42, so 0.42 is the best threshold and the test sensitivity at 0.42 is an unbiased estimate.”
Correction: If test outcomes were used to compare thresholds, they were used for selection. Choose the threshold using training or validation data under a prespecified rule, then evaluate that fixed threshold on held-out subjects.

Key takeaway: A threshold is part of the prediction procedure. Select it using training or validation data to address prespecified costs or operating constraints, lock it before accessing held-out outcomes, and evaluate the fixed model-threshold pair on those subjects.

Check Your Understanding

Answer each question, then check the brief solutions below.

If false-positive cost is 3 units and false-negative cost is 7 units, what probability threshold follows from the two-cost rule?
Why is choosing the threshold that maximizes test-set sensitivity and reporting that sensitivity on the same test set biased?
A validation threshold meets a specificity requirement, but held-out specificity is lower. Does this alone establish that the validation procedure was invalid?
Why might a threshold selected for one fitted model not be suitable after refitting that model?
What does maximizing a threshold-free ranking metric fail to specify for a decision that requires a positive or negative classification?

Solutions. (1) $3/(3+7)=0.30$. (2) The test outcomes helped select the threshold, so the reported performance reflects selection as well as evaluation and tends to be optimistic. (3) No. The validation constraint is an empirical result, and held-out performance varies; the threshold can still have been selected correctly without test-set reuse. (4) Refitting can change the score scale or subject rankings, so the original threshold may no longer represent the same operating point. (5) It summarizes ranking across thresholds but does not choose the cutoff that meets a specified cost or operating constraint.