Separating Logistic Probability Estimates From Class Decisions

From a Fitted Probability to a Class Decision

A logistic regression model estimates an event probability for each set of predictor values. A classification rule then converts that probability into a label such as “event” or “non-event.” These are different outputs: the model supplies the probability, while a chosen threshold determines the class label. Changing the threshold changes the decisions, but it does not by itself refit the model or change its fitted probabilities.

This distinction matters when a decision has consequences. A hypothetical insurer might use a fitted probability of a specified claim outcome to flag applications for review. The probability is a model estimate; whether the application is flagged depends on the threshold selected for that operational decision. A different threshold can flag more or fewer applications even when the fitted model is exactly the same.

Definition: For a binary outcome \(Y\), let \(Y=1\) denote the event and \(Y=0\) denote the non-event. A fitted logistic model gives an estimated event probability \(\hat p_i=P(Y_i=1\mid x_i)\) for observation \(i\). Given a threshold \(c\), where \(0<c<1\), classify observation \(i\) as an event if \(\hat p_i\ge c\), and as a non-event if \(\hat p_i<c\). The fitted probability is not the class label.

The convention in this tutorial is to classify a case as an event when its fitted probability is at least the threshold. Thus, if \(\hat p_i=c\), it is classified as an event. A problem that specifies a different convention should be followed as stated.

Confusion-Matrix Counts and Performance Measures

To assess classifications, compare them with observed outcomes. For a set of cases whose outcomes are known, each case falls into one of four categories. “Positive” means classified as the event, not that the outcome is inherently beneficial.

	Observed event \(Y=1\)	Observed non-event \(Y=0\)
Classified event	True positive (TP)	False positive (FP)
Classified non-event	False negative (FN)	True negative (TN)

Definition: Sensitivity is the proportion of observed events classified as events: \(\mathrm{Sensitivity}=\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})\). Specificity is the proportion of observed non-events classified as non-events: \(\mathrm{Specificity}=\mathrm{TN}/(\mathrm{TN}+\mathrm{FP})\). Both denominators are determined by the observed outcomes, not by the number classified into each group.

Sensitivity answers, “Among the actual events, what fraction did the rule identify?” Specificity answers, “Among the actual non-events, what fraction did the rule correctly leave unflagged?” If a denominator is zero, the corresponding measure is undefined; for example, sensitivity cannot be calculated from a sample containing no observed events.

These measures are calculated for a particular threshold and a particular set of observed outcomes. They are not properties of the fitted probabilities alone. Accuracy, \((\mathrm{TP}+\mathrm{TN})/n\), is another possible summary, but it can conceal poor performance on a less common class. The threshold and the measure should therefore be interpreted in the context of the decision.

Worked Example: Comparing Two Thresholds for the Same Model

A hypothetical model estimates the probability of a specified claim event for eight policies. The outcomes have since been observed. For this example, take the displayed probabilities as the fitted probabilities from one fixed logistic model.

Observed outcome	Fitted probabilities
Event	0.82, 0.61, 0.44, 0.27
Non-event	0.73, 0.48, 0.32, 0.08

Method. Apply each threshold to every fitted probability, then count the four outcomes in the confusion matrix. The same probabilities and observed labels are used at both thresholds.

At \(c=0.50\). Among the four events, 0.82 and 0.61 are at least 0.50, so TP \(=2\) and FN \(=4-2=2\). Among the four non-events, 0.73 and 0.48 are at least 0.50, so FP \(=2\) and TN \(=4-2=2\). Therefore sensitivity \(=2/(2+2)=0.5000\), and specificity \(=2/(2+2)=0.5000\).

At \(c=0.30\). Among the events, 0.82, 0.61, and 0.44 are at least 0.30, giving TP \(=3\) and FN \(=1\). Among the non-events, 0.73, 0.48, and 0.32 are at least 0.30, giving FP \(=3\) and TN \(=1\). Thus sensitivity \(=3/4=0.7500\), and specificity \(=1/4=0.2500\).

Check. At each threshold, TP \(+\) FN \(=4\) observed events and FP \(+\) TN \(=4\) observed non-events. Lowering the threshold increased sensitivity here, while reducing specificity. No fitted probability changed. The common mistake avoided is to treat a threshold change as though it were a new logistic model fit.

Why the Threshold Changes Decisions, Not Probabilities

In logistic regression, the fitted probability is obtained by applying the inverse-logit function to a linear predictor. For predictors \(x\), write the fitted linear predictor as \(\hat\eta(x)\). Then \(\hat p(x)=\exp(\hat\eta(x))/(1+\exp(\hat\eta(x)))\). Once the fitted coefficients and predictor values are fixed, \(\hat\eta(x)\) and \(\hat p(x)\) are fixed.

The rule \(\hat p(x)\ge c\) can also be expressed as a cutoff on the linear predictor. This makes explicit why changing \(c\) changes the classification boundary while leaving the fitted model untouched.

\begin{aligned} \hat p(x)\ge c &\iff \frac{\exp(\hat\eta(x))}{1+\exp(\hat\eta(x))}\ge c\\ &\iff \exp(\hat\eta(x))\ge c\{1+\exp(\hat\eta(x))\}\\ &\iff (1-c)\exp(\hat\eta(x))\ge c\\ &\iff \hat\eta(x)\ge \log\!\left(\frac{c}{1-c}\right). \end{aligned}

The second equivalence follows by multiplying by \(1+\exp(\hat\eta(x))\), which is positive. The third follows by subtracting \(c\exp(\hat\eta(x))\) from both sides. Since \(1-c>0\), division by \(1-c\) preserves the inequality; taking logarithms also preserves it because the logarithm is increasing on positive values. Thus the probability threshold corresponds to a cutoff on the fitted linear predictor. Raising \(c\) raises that cutoff and cannot add cases to the event-class group.

Key result: With a fixed fitted logistic model, increasing the threshold can only keep the same classifications or change some event classifications to non-event classifications. Consequently, on a fixed evaluation set, TP cannot increase and FP cannot increase; FN cannot decrease and TN cannot decrease. Sensitivity cannot increase, and specificity cannot decrease. These are weak inequalities: a change in threshold may have no effect if no fitted probabilities lie between the two thresholds.

Worked Example: Applying the Equality Convention

A hypothetical review rule uses a fixed logistic model. Five policies have observed outcomes and fitted event probabilities as follows: three are events with probabilities 0.91, 0.65, and 0.58; two are non-events with probabilities 0.65 and 0.22. Find the confusion-matrix counts, sensitivity, and specificity at threshold \(c=0.65\).

Method. Classify a case as an event when its probability is at least 0.65. In particular, both probabilities equal to 0.65 meet the rule.

Among the three observed events, 0.91 and 0.65 meet the threshold. Therefore TP \(=2\) and FN \(=3-2=1\). Among the two observed non-events, 0.65 meets the threshold, while 0.22 does not. Therefore FP \(=1\) and TN \(=2-1=1\).

Sensitivity \(=2/(2+1)=2/3=0.6667\). Specificity \(=1/(1+1)=1/2=0.5000\). The counts check: TP \(+\) FN \(=3\) events and FP \(+\) TN \(=2\) non-events.

The common mistake avoided is treating “at least 0.65” as “greater than 0.65.” Under the stated convention, equality counts as an event classification.

Selecting a Threshold for a Decision

There is no universally best threshold. A lower threshold generally catches more observed events but also classifies more non-events as events. A higher threshold generally reduces false positives but misses more events. Whether that trade-off is desirable depends on the decision being made, the consequences of each error, and any required performance target.

For example, a decision maker might assign a cost to a false negative and a different cost to a false positive. A simple comparison can then be made across candidate thresholds by multiplying each error count by its specified cost. This is a decision rule applied after fitting; it is not a change to the logistic regression coefficients. The costs in such an analysis must be provided or justified, rather than inferred from the fitted probabilities.

Worked Example: Choosing Between Thresholds Under Error Costs

A hypothetical monitoring program uses a fixed model to classify eight accounts for further review. A false negative costs 5 units, while a false positive costs 1 unit. The observed events have fitted probabilities 0.95, 0.72, 0.41, and 0.18; the observed non-events have probabilities 0.83, 0.55, 0.38, and 0.09. Compare thresholds \(c=0.70\) and \(c=0.40\), choosing the one with lower total error cost.

At \(c=0.70\). Of the events, 0.95 and 0.72 meet the threshold, so TP \(=2\) and FN \(=2\). Of the non-events, only 0.83 meets it, so FP \(=1\) and TN \(=3\). Sensitivity \(=2/4=0.5000\), and specificity \(=3/4=0.7500\). The total error cost is \(5(2)+1(1)=11\) units.

At \(c=0.40\). Of the events, 0.95, 0.72, and 0.41 meet the threshold, so TP \(=3\) and FN \(=1\). Of the non-events, 0.83, 0.55, and 0.38 meet it, so FP \(=3\) and TN \(=1\). Sensitivity \(=3/4=0.7500\), and specificity \(=1/4=0.2500\). The total error cost is \(5(1)+1(3)=8\) units.

The lower threshold has the smaller stated error cost, 8 rather than 11 units, so it is preferred among these two candidates under the given costs. It has higher sensitivity and lower specificity. This conclusion is conditional on the supplied costs and candidate thresholds; it does not establish that the fitted probabilities or model are better at \(c=0.40\).

Common Exam Trap: Reversing the Counts

For a threshold \(c=0.40\), suppose the observed events have fitted probabilities 0.90, 0.35, and 0.20, while the observed non-events have probabilities 0.70, 0.30, and 0.10. The correct classifications are based on whether each value is at least 0.40.

Incorrect calculation: Counting two observed events as meeting the threshold and two observed non-events as meeting it would give TP \(=2\), FN \(=1\), FP \(=2\), and TN \(=1\). This is wrong: among the event probabilities, only 0.90 is at least 0.40; among the non-event probabilities, only 0.70 is at least 0.40.

Correction: TP \(=1\), FN \(=2\), FP \(=1\), and TN \(=2\). Therefore sensitivity \(=1/(1+2)=0.3333\), and specificity \(=2/(2+1)=0.6667\). Check that the two row totals by observed outcome are each 3. A reliable approach is to classify the event and non-event groups separately before calculating either measure.

Key takeaway: First identify the fitted probability, then apply the stated threshold, then compare each classification with the observed outcome. Sensitivity uses the number of observed events as its denominator; specificity uses the number of observed non-events. Changing the threshold changes classifications and their performance measures, not the fitted probabilities from a fixed model.

Check Your Understanding

For each problem, use the convention that a fitted probability equal to the threshold is classified as an event. Report sensitivity and specificity to four decimal places.

A validation set contains three observed events with fitted probabilities 0.76, 0.49, and 0.31, and three observed non-events with probabilities 0.68, 0.42, and 0.15. At threshold 0.50, find TP, FN, FP, TN, sensitivity, and specificity. (Answer: TP \(=1\), FN \(=2\), FP \(=1\), TN \(=2\); sensitivity \(=0.3333\), specificity \(=0.6667\).)
Four events have probabilities 0.80, 0.40, 0.39, and 0.10; four non-events have probabilities 0.70, 0.40, 0.20, and 0.05. At threshold 0.40, find the counts and both measures. (Answer: TP \(=2\), FN \(=2\), FP \(=2\), TN \(=2\); sensitivity \(=0.5000\), specificity \(=0.5000\).)
Three events have probabilities 0.91, 0.66, and 0.25; three non-events have probabilities 0.72, 0.52, and 0.12. At threshold 0.70, calculate sensitivity and specificity. (Answer: TP \(=1\), FN \(=2\), FP \(=1\), TN \(=2\); sensitivity \(=0.3333\), specificity \(=0.6667\).)
Three events have probabilities 0.88, 0.55, and 0.22; three non-events have probabilities 0.77, 0.44, and 0.18. Compare sensitivity and specificity at thresholds 0.60 and 0.30. (Answer: At 0.60, sensitivity \(=0.3333\) and specificity \(=0.6667\); at 0.30, sensitivity \(=0.6667\) and specificity \(=0.3333\).)
A program assigns cost 4 units to each false negative and 1 unit to each false positive. Four events have probabilities 0.95, 0.62, 0.36, and 0.14; four non-events have probabilities 0.81, 0.57, 0.29, and 0.06. Compare thresholds 0.60 and 0.30 using total error cost. (Answer: At 0.60, TP \(=2\), FN \(=2\), FP \(=1\), TN \(=3\), and cost \(=4(2)+1(1)=9\); at 0.30, TP \(=3\), FN \(=1\), FP \(=3\), TN \(=1\), and cost \(=4(1)+1(3)=7\). Choose 0.30 under the stated costs.)

What You'll Learn