From Regression Estimates to Classification Rules
For a binary response, estimating the probability of class 1 at each feature value and assigning a class are related but distinct tasks. The probability estimate contains information about how likely class 1 is; a classifier turns that estimate into a discrete decision. The standard plug-in construction makes this conversion by comparing the estimated probability with one half.
Let \(X\) be a feature vector and \(Y\in\{0,1\}\) a response. The binary regression function is \(\eta(x)=P(Y=1\mid X=x)\), understood as a measurable version of the conditional probability. Suppose a training sample \(S_n\) is used to produce an estimate \(\hat\eta_{S_n}(x)\). The subscript emphasizes that the estimated function, and hence the classifier, can vary from one training sample to another.
The threshold has a direct decision-theoretic interpretation. If the conditional probability of class 1 is \(t\), predicting 1 has conditional error probability \(1-t\), while predicting 0 has conditional error probability \(t\). The smaller error is obtained by predicting 1 when \(t\geq 1/2\). A plug-in classifier uses the same decision rule after replacing the unknown \(\eta(x)\) with an estimate.
For the probability interpretation to apply literally, it is natural to take \(\hat\eta_{S_n}(x)\) in \([0,1]\). The threshold formula itself is still a mathematical rule for a real-valued estimate. In either case, the output is a classifier: it takes values in \(\{0,1\}\), not in the interval of probabilities.
Two Different Kinds of Error
A regression estimate can be assessed by how close its values are to \(\eta\). For example, its integrated squared error, conditional on the training sample, is
where \(P_X\) is the feature distribution. This quantity measures probability-estimation error across the feature space. It is a random quantity before the training sample is observed.
Classification risk instead measures the probability of predicting the wrong label for a new observation. Conditional on \(S_n\), define
where the test pair \((X,Y)\) is drawn from the population independently of the training sample. One may also average this risk over the randomness in \(S_n\), obtaining the expected risk \(E_{S_n}[R(\hat h_{S_n}\mid S_n)]\). Neither quantity is the same as integrated regression error: regression error assesses the estimated probabilities, whereas classification risk assesses the labels produced after thresholding.
Worked Example: Regression Error and Risk on Three Feature Values
Suppose \(X\) takes values \(a,b,c\) with probabilities \(0.5,0.3,0.2\). The true regression function and an estimated function are
Thresholding the estimate gives \(\hat h(a)=1\), \(\hat h(b)=1\), and \(\hat h(c)=0\). Its integrated squared regression error is
At \(a\), predicting 1 has conditional error \(1-0.8=0.2\); at \(b\), predicting 1 has error \(1-0.4=0.6\); at \(c\), predicting 0 has error \(0.1\). Therefore the classification risk is
The Bayes rule thresholds the true regression function, so its predictions are \(1,0,0\). Its risk is \(0.5(0.2)+0.3(0.4)+0.2(0.1)=0.24\). Thus this estimate has squared regression error \(0.019\) and classification risk \(0.30\); these figures describe different features of the procedure.
What the Thresholding Step Guarantees
For a fixed training sample, the thresholding step is optimal for the conditional probabilities represented by the estimate. This is a statement about the estimated probabilities, not a guarantee that the resulting rule is optimal for the true population.
Proof. Fix \(x\) and write \(t=\hat\eta_{S_n}(x)\). If a classifier predicts 0 at \(x\), its integrand is \(t\). If it predicts 1, its integrand is \(1-t\). When \(t\geq 1/2\), \(1-t\leq t\), so prediction 1 minimizes the integrand. When \(t<1/2\), \(t<1-t\), so prediction 0 minimizes it. The plug-in rule makes exactly these pointwise minimizing choices. Integrating the pointwise inequality over \(P_X\) proves that its estimated risk is no larger than that of any measurable classifier. At \(t=1/2\), the two integrands are equal. \(\square\)
The proposition explains why thresholding is the natural classification step once a probability estimate has been chosen. It does not say that the estimated risk equals true risk: the integrand uses \(\hat\eta_{S_n}\), not \(\eta\). Estimation quality determines how well those probabilities represent the population.
How Decisions Determine Excess Risk
The Bayes classifier is \(h^*(x)=\mathbf{1}\{\eta(x)\geq 1/2\}\), using the same convention at a tie. The exact excess-risk identity for binary classification expresses the difference between a classifier's risk and Bayes risk in terms of its decisions:
Derivation. Given \(X=x\), predicting 0 has error probability \(\eta(x)\), and predicting 1 has error probability \(1-\eta(x)\). The Bayes rule chooses the smaller of these two quantities. If \(h(x)=h^*(x)\), their conditional risks agree, so the excess is zero. If they differ, the excess is the difference between the larger and smaller quantities:
Taking expectation over \(X\) yields the identity. At \(\eta(x)=1/2\), the two predictions have equal conditional risk, and the displayed weight is zero, so the tie convention does not affect the result. \(\square\)
For a plug-in classifier, this identity says that regression estimation matters for classification through the decisions made after thresholding. Differences between \(\hat\eta_{S_n}(x)\) and \(\eta(x)\) that leave the prediction unchanged do not change that point's contribution to excess risk. Conversely, an estimate on the wrong side of one half can change the prediction. The size of the resulting risk contribution depends on the true \(\eta(x)\), not just on the size of the estimation error.
Worked Example: Different Regression Errors, Identical Classification Risk
Let \(X\) have two equally likely values, \(u\) and \(v\), with \(\eta(u)=0.51\) and \(\eta(v)=0.90\). Consider two estimates:
Both estimates exceed one half at both feature values, so both classifiers always predict 1. Their classification risks are therefore equal:
Their integrated squared regression errors, however, differ. For estimate \(A\), the deviations are \(0.5001-0.51=-0.0099\) and \(0.99-0.90=0.09\), giving
For estimate \(B\), the deviations are \(0.90-0.51=0.39\) and \(0.51-0.90=-0.39\), so
The regression errors are quite different, but the thresholded decisions—and therefore the classification risks—are identical. In this example both classifiers are also Bayes rules, since both true regression values exceed one half.
Worked Example: A Small Probability Error That Changes a Decision
Let \(P(X=a)=0.99\) and \(P(X=b)=0.01\), with \(\eta(a)=0.90\) and \(\eta(b)=0.51\). Suppose an estimate equals \(0.90\) at \(a\) and \(0.49\) at \(b\). It predicts 1 at \(a\), but 0 at \(b\), whereas the Bayes classifier predicts 1 at both values.
The integrated squared regression error is
The plug-in classifier has risk \(0.99(0.10)+0.01(0.51)=0.1041\). The Bayes risk is \(0.99(0.10)+0.01(0.49)=0.1039\). The excess risk is \(0.0002\): at \(b\), the wrong decision increases conditional error by \(0.51-0.49=0.02\), and that feature value occurs with probability \(0.01\). This verifies that a small integrated regression error can coexist with a threshold-crossing decision; the risk impact also depends on how often that feature value occurs and on its true conditional probabilities.
Training-Sample Variability and a Common Pitfall
Because \(\hat\eta_{S_n}\) depends on the training sample, the plug-in classifier is itself random before training. For a fixed observed sample, one can examine its conditional risk and regression error. To describe performance across possible training samples, average those quantities over the distribution of \(S_n\). Keeping these two sources of randomness distinct avoids confusing uncertainty in the fitted rule with the randomness of a new test observation.
- Regression estimation error concerns the numerical differences between \(\hat\eta_{S_n}(x)\) and \(\eta(x)\).
- Classification risk concerns the probability that the thresholded prediction differs from \(Y\).
- Excess classification risk compares that risk with the risk of \(h^*\); the exact identity weights incorrect decisions by \(|2\eta(X)-1|\).
Common mistake: It is tempting to argue that a classifier with smaller integrated squared regression error must have smaller classification risk. That conclusion does not follow just from the definitions. Thresholding discards the magnitude of estimates once their side of one half is known, while classification risk depends on the labels and the feature distribution. The second worked example shows that substantially different regression errors can yield identical classifiers. A comparison of classification risks requires information about the decisions made and the population, not only a comparison of regression-error summaries.
Check Your Understanding
Use the definitions and derivations above to answer the following questions.
- If \(\hat\eta_{S_n}(x)=0.5\), what class does the stated tie convention assign?
- For a fixed training sample, what distribution is used to integrate the squared regression error?
- Explain why minimizing risk computed from \(\hat\eta_{S_n}\) does not by itself establish that the resulting classifier minimizes true population risk.
- In the second worked example, why do the two estimates have the same classification risk despite their different integrated squared errors?
- In the third worked example, identify the feature value at which the plug-in decision differs from the Bayes decision and calculate its contribution to excess risk.