Pointwise Bias and Variance Near the Decision Boundary

Why Pointwise Bias and Variance Matter

A plug-in classifier estimates the conditional probability of class 1 and assigns the class according to whether that estimate exceeds one half. The effect of estimation error depends on where the observation lies. When the true conditional probability is close to one half, even a small perturbation can change the predicted class, but the cost of that mistake is small. Farther from one half, a mistake is less likely to occur, yet it has greater conditional cost.

This tutorial studies that tradeoff at a fixed feature value $x$. The estimator varies with the training sample, so its bias and variance at $x$ determine aspects of the distribution of the resulting classification decision. They do not, by themselves, determine its error probability: that also depends on the shape of the estimator's sampling distribution.

Definition: Let $\eta(x)=P(Y=1\mid X=x)$, and let $\hat{\eta}(x)$ be a regression estimate based on a random training sample. The pointwise bias and variance are $$ b(x)=E[\hat{\eta}(x)]-\eta(x), \qquad v(x)=\operatorname{Var}(\hat{\eta}(x)). $$ The plug-in rule predicts class 1 when $\hat{\eta}(x)\geq 1/2$, and class 0 otherwise. Expectations and probabilities involving $\hat{\eta}(x)$ below are over the randomness in the training sample, with $x$ held fixed.

Write $a(x)=|\eta(x)-1/2|$, the pointwise distance from the Bayes threshold. For $\eta(x)>1/2$, the Bayes classifier predicts 1; for $\eta(x)<1/2$, it predicts 0. At equality, either class has the same conditional zero-one risk. The pointwise excess risk of a classifier is its conditional risk minus the conditional risk of the Bayes choice.

The Exact Pointwise Regret Identity

For a fixed prediction, predicting the less probable class costs more than predicting the more probable class by exactly the difference between their conditional probabilities. This gives the pointwise identity below, and averaging it over the training sample gives the expected regret of a random plug-in rule.

Theorem: If $\eta(x)\neq 1/2$, the expected pointwise excess risk of the plug-in rule is $$ \mathcal{R}(x) = 2\left|\eta(x)-\frac12\right| P\!\left(\text{the plug-in rule predicts the class opposite to the Bayes class at }x\right). $$ If $\eta(x)=1/2$, then $\mathcal{R}(x)=0$, regardless of the plug-in prediction.

Proof. Fix $x$ and abbreviate $\eta=\eta(x)$. Predicting 1 has conditional error probability $1-\eta$, while predicting 0 has conditional error probability $\eta$. If $\eta>1/2$, the Bayes prediction is 1, and choosing 0 instead increases conditional risk by $$ \eta-(1-\eta)=2\eta-1. $$ If $\eta<1/2$, the Bayes prediction is 0, and choosing 1 instead increases conditional risk by $$ (1-\eta)-\eta=1-2\eta. $$ In either case the increase is $2|\eta-1/2|$. The increase is zero when the plug-in rule agrees with the Bayes prediction, so its expected increase is this cost multiplied by the probability of disagreement. At $\eta=1/2$, the two predictions have equal conditional risk. $\square$

This identity separates two quantities that are sometimes conflated: the probability of a wrong decision and the cost of that decision. The first can be large near the boundary; the second is then small. Far from the boundary, the cost is larger, but the estimate generally has to move farther to cross the threshold.

How Bias and Variance Enter

The pointwise bias moves the center of the estimator's distribution relative to the true $\eta(x)$. Bias directed toward one half tends to make a boundary crossing more likely; bias directed away tends to make it less likely. Variance describes spread around that center. To turn these summaries into an exact probability, a distributional assumption is needed.

Result: Suppose, at a fixed $x$, that $\hat{\eta}(x)$ has a normal sampling distribution with mean $\eta(x)+b(x)$ and variance $v(x)>0$. Then, for $\eta(x)\neq 1/2$, the probability of a wrong plug-in decision is $$ \Phi\!\left( -\frac{a(x)+\operatorname{sign}(\eta(x)-1/2)b(x)}{\sqrt{v(x)}} \right), $$ where $\Phi$ is the standard normal cumulative distribution function. Consequently, $$ \mathcal{R}(x) = 2a(x)\, \Phi\!\left( -\frac{a(x)+\operatorname{sign}(\eta(x)-1/2)b(x)}{\sqrt{v(x)}} \right). $$

Derivation. Put $s=\operatorname{sign}(\eta(x)-1/2)$, so $s$ is either 1 or $-1$, and write $\hat{\eta}(x)=\eta(x)+b(x)+Z$, where $Z\sim N(0,v(x))$. The signed distance of the estimate from the threshold is $$ s\left(\hat{\eta}(x)-\frac12\right) = a(x)+s b(x)+sZ. $$ A wrong decision occurs when this signed distance is negative. Since $sZ\sim N(0,v(x))$, standardizing that normal variable gives the displayed probability. A normal distribution with positive variance assigns probability zero to an exact tie at the threshold, so the choice of tie convention does not affect this calculation. Multiplying by the pointwise cost from the theorem proves the regret formula. $\square$

The numerator $a(x)+s b(x)$ is the signed distance of the estimator's mean from the threshold, measured in the Bayes direction. Bias toward the threshold reduces this distance; bias away increases it. If the numerator is negative, the estimator's mean is already on the wrong side, and increasing variance need not reduce the error probability. Thus “more variance means more classification error” is not a general statement without conditions on the distribution and mean.

Worked Example: A Noisy Estimate Close to the Boundary

At a fixed $x$, suppose the true conditional probability is $\eta(x)=0.52$, and the estimate is unbiased and normally distributed with standard deviation $0.05$. Compute the probability of a wrong decision and the expected pointwise regret.

Here $a(x)=0.52-0.50=0.02$, $b(x)=0$, and $\sqrt{v(x)}=0.05$. The standardized distance to the threshold is $0.02/0.05=0.4$, so $$ P(\text{wrong decision at }x)=\Phi(-0.4)\approx 0.344578. $$ The conditional cost of a wrong decision is $2a(x)=0.04$. Therefore $$ \mathcal{R}(x)=0.04(0.344578)\approx 0.0137831. $$ The estimate crosses the threshold fairly often, but each crossing has a small conditional cost because the true probability is close to one half.

Worked Example: Bias Toward the Threshold

Suppose $\eta(x)=0.55$, and $\hat{\eta}(x)$ is normally distributed with standard deviation $0.02$. Compare an unbiased estimator with one having bias $b(x)=-0.03$.

For the unbiased estimator, the mean is $0.55$, the distance to the threshold is $0.05$, and the standardized distance is $0.05/0.02=2.5$. Thus $$ P(\text{wrong})=\Phi(-2.5)\approx 0.00620967, \qquad \mathcal{R}(x)=0.10(0.00620967)\approx 0.000620967. $$ With bias $-0.03$, the mean is $0.55-0.03=0.52$. The distance of that mean from the threshold is only $0.02$, giving $$ P(\text{wrong})=\Phi(-1)\approx 0.158655, \qquad \mathcal{R}(x)=0.10(0.158655)\approx 0.0158655. $$ The pointwise variance is unchanged, but bias toward the threshold substantially increases both the chance of a classification error and expected regret. This comparison is specific to the stated normal distributions; bias and variance alone do not determine these probabilities in general.

A Variance Bound When Bias Is Smaller Than the Margin

Even without a normality assumption, variance gives a useful upper bound when the pointwise bias is not large enough to move the estimator's mean across the threshold. The bound makes explicit why estimates farther from the boundary are more protected against random fluctuations, provided the bias remains controlled.

Theorem: Fix $x$ with $a(x)>0$. Suppose $\hat{\eta}(x)$ has finite variance $v(x)$ and $|b(x)|<a(x)$. Then $$ P(\text{wrong decision at }x) \leq \min\!\left\{1,\frac{v(x)}{(a(x)-|b(x)|)^2}\right\}, $$ and hence $$ \mathcal{R}(x) \leq 2a(x)\min\!\left\{1,\frac{v(x)}{(a(x)-|b(x)|)^2}\right\}. $$

Proof. Let $s=\operatorname{sign}(\eta(x)-1/2)$, $a=a(x)$, $b=b(x)$, and $Z=\hat{\eta}(x)-E[\hat{\eta}(x)]$. Then $E[Z]=0$ and $\operatorname{Var}(Z)=v(x)$. A wrong decision requires the estimate to reach or cross the threshold in the direction opposite to the Bayes class. In signed coordinates, this implies $$ s\left(\hat{\eta}(x)-\frac12\right)\leq 0, \qquad a+s b+sZ\leq 0. $$ Therefore $sZ\leq -(a+s b)$. Since $a+s b\geq a-|b|>0$, this event implies $$ |Z|\geq a-|b|. $$ Chebyshev's inequality gives $$ P(\text{wrong decision}) \leq P\!\left(|Z|\geq a-|b|\right) \leq \frac{v(x)}{(a-|b|)^2}. $$ A probability is also at most 1, which gives the minimum in the claim. Multiplication by $2a(x)$, using the exact pointwise regret identity, proves the regret bound. $\square$

This bound is conservative: Chebyshev's inequality uses only the variance and does not use the detailed tail shape. Its condition $|b(x)|<a(x)$ is important. If the bias is at least as large as the margin, the estimator's mean could lie on the wrong side, and this argument cannot guarantee that errors require a large noise fluctuation.

Worked Example: A Distant Boundary Crossing

Suppose the true probability is $\eta(x)=0.80$, while the estimate has distribution $N(0.78,0.10^2)$. Calculate the bias, the error probability, and the expected pointwise regret.

The pointwise margin is $a(x)=0.80-0.50=0.30$. The estimator's mean is $0.78$, so its bias is $b(x)=0.78-0.80=-0.02$, directed toward the threshold. Its standard deviation is $0.10$. A wrong decision occurs when $\hat{\eta}(x)<0.50$, giving $$ P(\text{wrong}) = \Phi\!\left(\frac{0.50-0.78}{0.10}\right) = \Phi(-2.8) \approx 0.00255513. $$ The cost of a wrong decision is $2a(x)=0.60$. Thus $$ \mathcal{R}(x) = 0.60(0.00255513) \approx 0.00153308 \approx 0.001533. $$ The estimate is much more likely to cross the threshold in the near-boundary example than here, but the cost per crossing is larger here. The small error probability keeps the resulting pointwise regret small.

Interpreting Boundary and Nonboundary Errors

The exact identity clarifies what “errors near the boundary matter less” does and does not mean. It means that the conditional regret for each wrong decision is small when $a(x)$ is small. It does not mean errors near the boundary are rare; in fact, a small shift in the estimate may reverse the decision there. Conversely, a point well away from the boundary has a larger cost per mistake, but a threshold crossing usually requires a larger estimation error.

Pointwise calculations also should not be confused with population regret. To obtain population excess risk, pointwise regret must be averaged over the distribution of $X$. The overall contribution of a region depends on how much probability mass $X$ places there, as well as on the pointwise error probabilities and costs. A point with a high error probability alone does not establish that a large part of the population risk comes from its neighborhood.

Common Pitfall

Wrong reasoning: “The variance is large, so the classification error probability must be large.” Variance does not determine a tail probability by itself, and a biased estimator may have its mean on either side of the threshold.
Correction: State the distributional assumption when computing an exact probability, as in the normal examples. Without that assumption, use a valid bound such as Chebyshev's inequality, checking that the bias is smaller than the margin when applying the bound above.
Wrong reasoning: “A wrong decision near one half has negligible impact because its cost is small.” Small pointwise cost does not imply zero population contribution; the region's probability mass and the frequency of errors also matter.

Key takeaway: At a fixed feature value, expected plug-in regret is exactly the margin cost $2|\eta(x)-1/2|$ multiplied by the probability of crossing the decision threshold in the wrong direction. Bias shifts that crossing probability, variance controls fluctuation, and the margin controls the cost of the resulting mistake.

Check Your Understanding

Answer each question using the pointwise perspective developed here.

Why can a plug-in classifier have a high probability of error at a point but a small expected pointwise regret?
Under a normal sampling model, what happens to the standardized distance to the threshold when bias points toward one half?
Why do pointwise bias and variance alone not determine the exact error probability?
State the condition needed to apply the variance bound, and explain what role it plays in the proof.
What additional information is needed to move from pointwise regret to population excess risk?