Clipping Regression Estimates Before Classification

Why Clip a Regression Estimate?

In binary classification, a regression function often represents the conditional probability of the positive class: \(\eta(x)=P(Y=1\mid X=x)\), where \(Y\in\{0,1\}\). Its values therefore lie between zero and one. An estimate \(\hat\eta(x)\), however, need not respect those limits. For example, a fitted linear regression can produce a negative value or a value greater than one.

Clipping replaces any such value by the nearest endpoint of the probability interval. Two useful questions then arise. Does this repair worsen the estimate’s error relative to the true conditional probability? And does it change the plug-in classifier that thresholds the estimate? The pointwise answer to the first question is no, and for thresholds strictly between zero and one the answer to the second is also no.

The map changes only estimates outside the probability interval. It leaves values already in the interval untouched. In this tutorial, the target \(\eta(x)\) is assumed to be in \([0,1]\) at every point under consideration, as is the case for a version of a binary conditional probability. The claims below are pointwise: they apply separately at each such \(x\), and do not require a particular estimation method.

Clipping Cannot Increase Pointwise Absolute Error

Proof. Consider the three cases in the definition of \(C\). If \(0\leq u\leq 1\), then \(C(u)=u\), so the two errors are equal. If \(u<0\), then \(C(u)=0\), and \(v\geq 0\) gives

If \(u>1\), then \(C(u)=1\). Since \(v\leq 1\),

All possible values of \(u\) are covered, proving the claim. \(\square\)

Applying the result with \(v=\eta(x)\) and \(u=\hat\eta(x)\) gives the regression-estimation statement directly.

Consequences for Integrated Error

The pointwise inequality also gives an integrated guarantee without any independence assumption or model for how the estimate was obtained. Let \(P_X\) be the distribution of the feature vector \(X\). For any measurable real-valued function \(r\), compare its absolute regression error with that of its clipped version.

Proof. For every \(x\), the pointwise result gives \(|C(r(x))-\eta(x)|\leq |r(x)-\eta(x)|\). Both sides are nonnegative measurable functions: \(C\) is continuous, and \(r\) and \(\eta\) are measurable. Integrating a pointwise inequality between nonnegative measurable functions with respect to the same measure preserves the inequality. The integrals may be finite or infinite; if the right-hand integral is finite, so is the left-hand integral. \(\square\)

The same argument applies to a fitted estimate that depends on training data: for each fixed realization of the data, apply the pointwise inequality to the resulting function and then integrate over \(X\). If an expectation over the training data is also taken, the inequality remains valid because the integrands are nonnegative. This does not require clipping to make a poor estimate good; it says only that clipping cannot make its absolute regression error larger.

Why the Standard Threshold Rule Is Preserved

For a real-valued score function \(r\) and a threshold \(\tau\), define the threshold classifier by \(h_{r,\tau}(x)=\mathbf{1}\{r(x)\geq\tau\}\), where \(\mathbf{1}\) is one when its condition holds and zero otherwise. The standard binary plug-in rule thresholds a conditional probability estimate at one-half. More generally, the same clipping argument applies to any threshold strictly between zero and one.

Proof. If \(u<0\), then \(C(u)=0<\tau\), and also \(u<\tau\), so both indicators are zero. If \(0\leq u\leq 1\), then \(C(u)=u\), so the two indicators compare the same value with \(\tau\). If \(u>1\), then \(C(u)=1>\tau\), and also \(u>\tau\), so both indicators are one. These cases exhaust the real line. Substituting \(u=r(x)\) proves equality of the classifiers pointwise. \(\square\)

Because one-half is strictly between zero and one, clipping an estimated regression function does not alter its standard plug-in classifier:

The equality includes a threshold tie: if \(\hat\eta(x)=1/2\), clipping leaves it equal to \(1/2\), and both rules classify it as positive under the stated \(\geq\) convention. For any interior threshold, clipping therefore preserves every classification decision, not just the overall classification risk.

What Clipping Does—and Does Not—Guarantee

Clipping has two distinct consequences that should not be conflated. It weakly reduces pointwise absolute regression error, and it preserves the classifier for an interior threshold. Since the classifier is unchanged, any classification quantity determined solely by that classifier is unchanged as well. In particular, clipping by itself cannot correct a classification decision that was already on the wrong side of the threshold.

This distinction matters in the three-point example. At \(b\), the true probability is \(0.6\), so the one-half Bayes decision is positive, but the estimate \(0.4\) gives a negative decision. Clipping leaves \(0.4\) unchanged, so that disagreement remains. At \(a\) and \(c\), clipping improves the probability estimates without changing their decisions. The improvement in regression error is real, but it is not a promise of improved classification performance.

A common mistaken argument is: “Clipping changes the score, so it might move a point across one-half and improve or worsen the classifier.” For this particular operation and an interior threshold, that reasoning overlooks where changed scores can lie. A score below zero is below every interior threshold before and after clipping; a score above one is above every interior threshold before and after clipping. Scores between zero and one do not change. Thus no score crosses the threshold.

Another pitfall is to conclude that clipping must strictly reduce error. The theorem gives a weak inequality. For an in-range estimate, the error is exactly unchanged; an out-of-range estimate can also have equal error in boundary cases. The reliable conclusion is “cannot increase,” not “always decreases.”