Scores, Thresholds, and Binary Decisions

From Scores to Decisions

A binary classifier can produce a numerical score for each case, but a score is not itself a class label. A score may estimate the probability of an event, or it may simply rank cases from lower to higher risk. To turn scores into positive or negative decisions, an analyst must specify a threshold and a rule for cases that land exactly on it.

For example, a hypothetical insurer might use a model to estimate the probability that a submitted claim requires a particular review. The model’s estimated probability is a score. If the review team treats probabilities of at least 0.40 as a positive decision for review, each score is converted to a label by comparing it with 0.40. The threshold determines the decision rule; it does not change the scores themselves.

Definition: A score is a numerical output assigned to a case by a model. A threshold is a specified cut-off on that score scale. A thresholded binary decision assigns one of two labels according to a rule such as “positive if the score is at least the threshold; negative otherwise.” The positive label must represent the event or class of interest.

When the score $s$ is an estimated probability, it is usually between 0 and 1. A common rule is to predict positive when $s \geq t$, where $t$ is the chosen probability threshold. Under that rule, a score exactly equal to the threshold produces a positive decision. An alternative rule could use $s > t$, but the boundary convention must be stated; otherwise, cases exactly at the threshold have an ambiguous classification.

A score need not be a probability. A model might output an index on a scale from 0 to 100, a log-odds value, or another risk score. In those cases, apply a threshold on the score’s own scale and verify which direction indicates greater support for the positive class. A threshold of 0.5 has no automatic meaning for an arbitrary score.

Worked Example: Applying a Probability Threshold

A hypothetical model estimates the probability that each of six policy applications will require a manual review. The review class is designated positive. Apply the rule “positive if the score is at least 0.50; negative otherwise.” The model scores are 0.82, 0.50, 0.49, 0.07, 0.68, and 0.50, in that order.

Method. Compare each score with $t=0.50$, using the stated $\geq$ rule. This example tests the boundary convention: a score equal to 0.50 is positive, not negative.

For the first application, $0.82 \geq 0.50$, so the decision is positive. For the second, $0.50 \geq 0.50$, so it is positive. For the third, $0.49 < 0.50$, so it is negative. For the fourth, $0.07 < 0.50$, so it is negative. For the fifth, $0.68 \geq 0.50$, so it is positive. For the sixth, $0.50 \geq 0.50$, so it is positive.

The decisions, in the original order, are therefore positive, positive, negative, negative, positive, positive. The calculation uses the supplied scores and threshold only; it does not establish whether the model is well calibrated or whether this threshold is appropriate for the review process.

Common mistake avoided. Treating “at least 0.50” as “greater than 0.50” would incorrectly label both applications scored exactly 0.50 as negative. The inequality in the decision rule controls those cases.

Thresholds Depend on the Score Scale

Before making a decision, identify what a score represents, what direction indicates stronger evidence for the positive class, and which threshold is specified. A probability threshold is meaningful only when the score is being interpreted as a probability. Even then, a threshold of 0.50 is a convention, not a universal requirement.

Name the positive class.
State which event or category counts as positive. The label describes the decision target, not whether a case is desirable or undesirable.

Identify the score scale and direction.
Determine whether the score is a probability or another numerical quantity, and whether higher or lower values support a positive decision.

Apply the threshold rule.
Compare each score with the stated threshold using the stated inequality, including its treatment of equality.

Report decisions as labels.
Keep the original score distinct from the resulting positive or negative decision. Do not describe a thresholded label as though it were a probability.

Worked Example: Thresholding a Non-Probability Risk Score

A hypothetical claims triage model returns a risk index from 0 to 100. Larger values indicate greater support for the positive class, defined as “send for specialist review.” The policy is to send a case for review when its index is at least 72. Apply the rule to scores 88, 72, 71.9, 65, and 104.

Method. Use the score-scale rule “positive if score $\geq 72$.” The fact that the score range resembles a percentage does not make these values estimated probabilities.

For 88, $88 \geq 72$, so the decision is positive. For 72, $72 \geq 72$, so it is positive, including the boundary. For 71.9, $71.9 < 72$, so it is negative. For 65, $65 < 72$, so it is negative. For 104, $104 \geq 72$, so it is positive under the stated rule.

The resulting decisions are positive, positive, negative, negative, and positive, respectively. The value 104 is outside the stated 0-to-100 description, so in a real workflow it should be checked as a possible data or scale error. Given the problem’s instruction to apply the rule to the supplied score, however, it meets the threshold.

Common mistake avoided. Reading an index of 72 as a 72% event probability would add an interpretation not supported by the score definition. Thresholding requires comparison on the correct scale; it does not by itself supply a probability interpretation.

Why a Cost-Based Threshold Can Differ From 0.50

If a score is a calibrated probability, a decision threshold can be derived from the relative costs of the two possible errors. This provides a reason that a rational decision rule may use a threshold below or above 0.50. The derivation assumes two actions, two possible true states, known error costs, and no additional costs that differ between the actions. It is not a general rule for arbitrary, uncalibrated scores.

Result: Suppose $p=P(\text{positive event}\mid\text{available information})$, the cost of a false positive is $C_{FP}>0$, and the cost of a false negative is $C_{FN}>0$. If correct decisions have zero cost and the objective is to minimize expected cost for an individual case, predict positive when $$ p \geq \frac{C_{FP}}{C_{FP}+C_{FN}}. $$ At equality, the two actions have the same expected cost; assigning equality to positive is one explicit tie rule.

To justify the result, compare the expected costs of the two possible decisions. If the decision is positive, a cost $C_{FP}$ is incurred only when the event is actually negative, which has probability $1-p$. Its expected cost is therefore $(1-p)C_{FP}$. If the decision is negative, a cost $C_{FN}$ is incurred only when the event is actually positive, which has probability $p$. Its expected cost is $pC_{FN}$. Choose positive when its expected cost is no greater than the negative decision’s expected cost:

(1-p)C_{FP} \leq pC_{FN} \quad\Longleftrightarrow\quad C_{FP} \leq p(C_{FP}+C_{FN}) \quad\Longleftrightarrow\quad p \geq \frac{C_{FP}}{C_{FP}+C_{FN}}.

The rearrangement uses $C_{FP}+C_{FN}>0$, so division by that quantity does not reverse the inequality. If the cost assumptions change, the threshold may change too. If the score is not a calibrated probability, substituting it for $p$ does not justify this expected-cost calculation.

Worked Example: Deriving a Threshold From Error Costs

A hypothetical benefit administrator uses a calibrated probability $p$ that a case contains a specified payment error. A positive decision triggers an investigation. Assume each false positive costs $120 in unnecessary investigation, each false negative costs $480 in missed error, and correct decisions have zero cost. For cases scored 0.18, 0.20, and 0.35, derive the cost-minimizing threshold and make each decision. Use positive at or above the threshold.

Method. Compare the expected cost of investigating with the expected cost of not investigating. The assumptions permit using the calibrated probability in the expected-cost formula.

The threshold is

t=\frac{C_{FP}}{C_{FP}+C_{FN}} =\frac{120}{120+480} =\frac{120}{600} =0.20.

For $p=0.18$, the expected cost of a positive decision is $(1-0.18)(120)=0.82(120)=\$98.40$. The expected cost of a negative decision is $0.18(480)=\$86.40$. Since $86.40 is lower, choose negative.

For $p=0.20$, the positive decision costs $(1-0.20)(120)=0.80(120)=\$96.00$ in expectation. The negative decision costs $0.20(480)=\$96.00$. The costs are equal; the stated tie rule assigns positive.

For $p=0.35$, the positive decision costs $(1-0.35)(120)=0.65(120)=\$78.00$. The negative decision costs $0.35(480)=\$168.00$. Since $78.00 is lower, choose positive.

Thus, the decisions are negative, positive, and positive. At the boundary, both actions have the same expected cost, which checks the threshold calculation.

Common mistake avoided. Using 0.50 without considering the unequal error costs would choose negative at $p=0.35$, even though the expected cost of a positive decision is lower. The cost-based threshold is 0.20 under the assumptions given.

Interpreting and Reporting the Decision Rule

A threshold does not make a model’s scores more or less accurate; it converts them into decisions. The same scores can produce different labels under different thresholds, while the scores themselves remain unchanged. For a clear analysis, report the positive class, what the score represents, the threshold, the inequality used, and how equality is handled.

Threshold choice should reflect the decision objective and the consequences of errors, not an assumption that 0.50 is always correct. If a threshold is selected by examining validation results, keep that selection separate from evaluation on a protected final test set, as in the earlier material on validation design. A score-to-label rule is also distinct from the later analysis of how resulting labels are summarized.

Key takeaway: A score is model output; a binary label is a decision made from that output. State the score scale, positive class, threshold, and boundary rule. Use a probability-based cost threshold only when the score is a suitable probability and the decision-cost assumptions apply.

This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus: a sound interpretation begins by separating what the model outputs from what the decision rule does with it.

Check Your Understanding

For each problem, apply the stated score scale and threshold rule. Treat a score equal to the threshold as positive whenever the rule says “at least.”

A probability model gives scores 0.39, 0.40, and 0.61. The positive class is “refer for review,” with positive defined as score at least 0.40. State the three decisions in order. (Answer: positive, positive, positive.)
A hypothetical inspection index uses higher values for greater concern. The rule is positive if the index is at least 6.5. Classify scores 6.4, 6.5, and 8.1. (Answer: negative, positive, positive.)
A calibrated probability is used with false-positive cost $90 and false-negative cost $120. Find the cost-based threshold, then classify probabilities 0.40 and 0.50 using positive at or above the threshold. (Answer: threshold $90/(90+120)=3/7\approx0.4286$; negative, positive.)
A calibrated event probability is $p=0.25$. A positive decision has false-positive cost $30, a negative decision has false-negative cost $150, and correct decisions have zero cost. Find the threshold and compare the expected costs of both decisions at this $p$. (Answer: threshold $30/(30+150)=1/6\approx0.1667$; positive cost is $0.75(30)=\$22.50$, negative cost is $0.25(150)=\$37.50$, so choose positive.)
A model outputs a risk index from 0 to 10, with larger values indicating stronger support for the positive class. A policy says positive only when the index is strictly greater than 7. Classify scores 7.0 and 7.1, and explain why treating the index as a probability would be unjustified. (Answer: negative, positive; the index is not defined as a probability.)

What You'll Learn