Selecting Classification Thresholds Without Leakage

Why Threshold Selection Needs Its Own Validation

A binary classifier often produces a score for each sample rather than a final positive-or-negative decision. The decision requires a threshold: samples scoring at or above it are called positive, and those below it are called negative. In genomic applications, the threshold can substantially change how many affected samples are detected and how many unaffected samples are incorrectly flagged.

A threshold is part of the prediction procedure. If it is chosen to make sensitivity or specificity look best on the same outcomes later used to report those measures, the reported performance is optimistic: those outcomes have influenced the procedure being evaluated. The remedy is to choose the threshold using training data or inner-validation predictions, then assess it on untouched outer-fold predictions. The outer fold must not influence the threshold, the model, or any other analysis choice.

These definitions assume that both classes are represented in the evaluation set; otherwise the corresponding denominator is zero and that measure is undefined. A score need not be a probability for a threshold rule to be used. If the score is a probability, \(t=0.5\) is not automatically the right threshold: the appropriate choice depends on the intended use and the relative consequences of false negatives and false positives.

Choose the Threshold Inside the Training Process

In nested cross-validation, an outer fold is set aside for evaluation. Threshold selection belongs entirely inside the outer training data. One option is to generate inner-validation predictions within that training set, select a threshold from those predictions using a declared rule, fit the model on the full outer training set, and apply the selected threshold to scores for the untouched outer fold. The outer-fold labels are revealed only for evaluation.

A threshold-selection rule should be stated before examining outer-fold results. Common choices include maximizing Youden’s index, \(J=\text{sensitivity}+\text{specificity}-1\), on inner-validation predictions, or choosing the threshold that achieves a prespecified minimum specificity while maximizing sensitivity. The first treats the two rates symmetrically; the second reflects an operational constraint. Neither is universally best. If the consequences of errors differ, the rule should reflect the application rather than being selected after seeing outer results.

Thresholds chosen from fitted training predictions can be used, but those predictions may be unusually favorable because the model was fitted to the same samples. Inner-validation predictions offer a more defensible basis for selection. Whatever selection method is used, the outer fold remains reserved for evaluation.

Evaluate the Selected Rule on Untouched Outer Predictions

After selecting a threshold using only outer-training data, apply it to the scores for the corresponding outer fold. Compare the resulting predictions with that fold’s labels to calculate sensitivity and specificity. Do not revise the threshold after seeing these results. In cross-validation, repeat the full selection-and-evaluation process separately for each outer fold.

A fold-specific threshold is usually appropriate because each outer training set can select its own threshold using its own inner predictions. If performance is summarized across outer folds, state how the fold results were combined. For example, pooling the outer-fold confusion counts gives pooled sensitivity \(=\sum TP/(\sum TP+\sum FN)\) and pooled specificity \(=\sum TN/(\sum TN+\sum FP)\). This describes the predictions at the thresholds selected within the respective folds; it does not mean one universal threshold was selected for all samples.

Threshold Selection Under an Operational Constraint

Sometimes the objective is not to balance sensitivity and specificity. A laboratory might require estimated specificity of at least 0.80 on inner-validation predictions, then seek the greatest sensitivity among thresholds meeting that constraint. The constraint and selection rule must be fixed using training data, not adjusted in response to outer-fold results.

Why Reusing Evaluation Outcomes Is Optimistic

The source of optimism can be seen by considering several candidate thresholds. Each threshold produces a noisy estimate of performance. Choosing the threshold with the largest observed estimate tends to select one that performed well partly by chance. If \(\widehat{M}_j\) is an estimate of a performance measure for candidate \(j\), then

The expectation inequality follows because the maximum is at least each candidate’s estimate on every data set; taking expectations preserves that inequality. Under comparable unbiased candidate estimates, the selected maximum therefore tends to exceed the best candidate’s expected performance. The same logic applies if a threshold is chosen to maximize sensitivity subject to an observed specificity requirement. Evaluating the selected threshold on an independent outer fold avoids using the same random outcomes for both selection and assessment.

This justification does not imply that every selected threshold is severely optimistic, nor does it quantify the size of the optimism. The amount depends on such factors as the number of candidate thresholds, the variability of the estimates, and their dependence. The practical safeguard is procedural: keep outer outcomes out of threshold selection.

Diagnostics, Alternatives, and Limitations

Thresholds can be unstable when small changes in the inner data produce large changes in the selected value. Check the sensitivity and specificity across the candidate thresholds, the number of samples in each class, and whether the chosen value lies in a broad region of similar performance or at an isolated peak. If small changes alter the choice substantially, report that instability and avoid presenting a single threshold as universally established.

A prespecified threshold is an alternative when it has a defensible basis independent of the evaluation data. It removes data-driven threshold selection from the validation procedure, though the resulting sensitivity and specificity still need evaluation. A single validation split can be used to select a threshold, but the choice may be sensitive to that split. Nested cross-validation repeats selection within each outer training set and gives an evaluation that accounts for the selection process; it does not make the estimate exact or eliminate sampling uncertainty.

A common mistake is to inspect outer-fold sensitivity and specificity at several thresholds, select the most attractive one, and then report those same outer-fold results as unbiased evaluation. The correction is to treat that inspection as additional tuning: the outer outcomes are no longer untouched. Select a threshold using training or inner-validation predictions, or obtain new independent evaluation data for an assessment after the choice. Also distinguish a threshold from a model parameter: the threshold converts scores to class labels, but selecting it is still a data-dependent decision and must be included in the validation design.