The Classification Margin Variable

Why Probability Near One Half Matters

In binary classification, the conditional class probability determines which label is preferable at a feature value. When that probability is near one half, the two labels have nearly equal conditional error. An error there costs little in classification risk. When the probability is far from one half, choosing the less probable label costs more.

The classification margin variable records this distance from a tie. Its distribution describes how much feature probability lies at different levels of conditional ambiguity. It does not, by itself, describe which points a fitted classifier will misclassify; rather, it tells us how costly errors are according to where they occur.

Definition and Basic Interpretation

Let $X$ be a feature vector and let $Y\in\{-1,+1\}$ be the class label. Write $\eta(X)=P(Y=+1\mid X)$ for a version of the conditional class probability. Define a Bayes classifier $g^*$ by choosing $+1$ when $\eta(X)\geq 1/2$ and $-1$ otherwise. At a tie, either label has the same conditional risk; the stated convention fixes one choice.

Definition: The classification margin variable is $M=|\eta(X)-1/2|$. It takes values in $[0,1/2]$. Its distribution function is $G(t)=P(M\leq t)$, for $t\geq 0$. Thus $G(t)$ is the probability that the conditional class probability lies within distance $t$ of one half.

The margin is a random variable because it depends on the random feature vector $X$. It is not the signed score $Yf(X)$ sometimes called a margin in other settings. Here it is defined specifically from the conditional class probability, and it is nonnegative. A large value means the conditional preference for one label is strong; a value near zero means the labels are nearly tied.

The distribution function $G$ summarizes the amount of probability near a tie. In particular, $G(0)=P(M=0)$ is the probability of an exact tie, while $G(t)-G(0)$ is the probability of a non-tie whose margin is at most $t$. These quantities matter because a wrong decision at a small positive margin has small excess risk. At an exact tie, either decision has zero excess risk.

Worked Example: A Three-Point Feature Distribution

Suppose $X$ takes values $A,B,C$ with probabilities $0.2,0.5,0.3$, respectively, and the conditional probabilities of label $+1$ are $\eta(A)=0.5$, $\eta(B)=0.2$, and $\eta(C)=0.9$. Find the margin distribution and the Bayes risk.

At $A$, $M=|0.5-0.5|=0$. At $B$, $M=|0.2-0.5|=0.3$. At $C$, $M=|0.9-0.5|=0.4$. Therefore the margin has probability $0.2$ at 0, $0.5$ at 0.3, and $0.3$ at 0.4. Equivalently, $G(t)=0$ for $t<0$, $G(t)=0.2$ for $0\leq t<0.3$, $G(t)=0.7$ for $0.3\leq t<0.4$, and $G(t)=1$ for $t\geq0.4$.

The Bayes classifier chooses $+1$ at $A$ under the tie convention, $-1$ at $B$, and $+1$ at $C$. Its conditional error probabilities are $0.5,0.2,0.1$. Thus its overall risk is $0.2(0.5)+0.5(0.2)+0.3(0.1)=0.10+0.10+0.03=0.23$.

Margin and Excess Classification Risk

For a measurable classifier $g\colon X\mapsto\{-1,+1\}$, define its classification risk by $R(g)=P(g(X)\ne Y)$. The excess risk $R(g)-R(g^*)$ compares it with the Bayes classifier. The following identity makes precise how the margin weights classification mistakes.

Theorem: For any measurable classifier $g$, with $M=|\eta(X)-1/2|$ and the Bayes classifier defined above, $$ R(g)-R(g^*)=2E\!\left[M\,\mathbf{1}\{g(X)\ne g^*(X)\}\right]. $$

Proof. Conditional on $X$, predicting $+1$ has error probability $1-\eta(X)$, while predicting $-1$ has error probability $\eta(X)$. The Bayes decision selects the smaller of these two error probabilities, with either choice allowed when they are equal. If $g(X)=g^*(X)$, the conditional excess risk is zero. If the two decisions differ and $\eta(X)>1/2$, the Bayes decision is $+1$ and the conditional excess is $\eta(X)-(1-\eta(X))=2\eta(X)-1=2M$. If they differ and $\eta(X)<1/2$, the Bayes decision is $-1$ and the conditional excess is $(1-\eta(X))-\eta(X)=1-2\eta(X)=2M$. At a tie, the conditional excess is zero, which also equals $2M$. Therefore the conditional excess is $2M\mathbf{1}\{g(X)\ne g^*(X)\}$ in every case. Taking expectations gives the identity. $\square$

The factor of two follows from defining $M$ as the distance of $\eta(X)$ from one half. For example, if $\eta(X)=0.8$, then $M=0.3$, and choosing the wrong label increases conditional error by $0.6$. Defining a different variable, such as $|2\eta(X)-1|$, would change the factor in the identity.

Worked Example: Errors on a Small-Margin Interval

Let $X$ be uniform on $[0,1]$, and suppose $\eta(x)=(1+x)/2$. Find the distribution of $M$, then calculate the excess risk of a classifier that predicts $-1$ for $x\leq a$ and $+1$ for $x>a$, where $0\leq a\leq1$.

Here $\eta(x)\geq1/2$, so $M=|\eta(X)-1/2|=X/2$. For $0\leq t\leq1/2$, $G(t)=P(X/2\leq t)=P(X\leq2t)=2t$; also $G(t)=1$ for $t\geq1/2$. There is no atom at zero.

The Bayes classifier predicts $+1$ except at the tie $x=0$. The stated classifier disagrees with it on $0<X\leq a$, up to endpoints of probability zero. Using the excess-risk identity, $$ R(g)-R(g^*)=E[X\mathbf{1}\{X\leq a\}] =\int_0^a x\,dx=\frac{a^2}{2}. $$ For $a=0.2$, the disagreement probability is $0.2$, but the excess risk is $0.2^2/2=0.02$. The relatively small cost arises because these disagreements occur where the conditional probability is close to one half.

What the Small-Margin Distribution Can Bound

Let $A=\{g(X)\ne g^*(X)\}$ be the disagreement event. Splitting this event at a threshold $t$ separates errors on points close to a tie from errors farther away. This gives a useful bound without assuming any particular form for $G$.

Result: For every $t\geq0$, $$ R(g)-R(g^*)= 2E[M\mathbf{1}\{A,\ M\leq t\}] + 2E[M\mathbf{1}\{A,\ M>t\}], $$ and the first term satisfies $$ 0\leq 2E[M\mathbf{1}\{A,\ M\leq t\}]\leq 2tG(t). $$ If all disagreements occur where $M\geq t$, then $R(g)-R(g^*)\geq2tP(A)$.

Proof. The events $\{A,M\leq t\}$ and $\{A,M>t\}$ are disjoint and their union is $A$. Splitting the expectation in the excess-risk identity over these two events proves the equality. On $\{M\leq t\}$, $0\leq M\leq t$, so $0\leq2M\mathbf{1}\{A,M\leq t\}\leq2t\mathbf{1}\{M\leq t\}$. Taking expectations gives the bound $2tP(M\leq t)=2tG(t)$. If $A$ implies $M\geq t$, then $2M\mathbf{1}_A\geq2t\mathbf{1}_A$; taking expectations yields the final inequality. $\square$

The bound says that even if a classifier disagrees with the Bayes rule throughout the entire region $M\leq t$, the excess risk from that region cannot exceed $2tG(t)$. A large $G(t)$ means there is substantial probability near a tie, where errors can be inexpensive. A small $G(t)$ means little probability lies in that particular near-tie region. The distribution therefore helps describe which kinds of disagreements can have small cost.

Worked Example: A Margin Distribution With a Gap Near Zero

Let $X$ be uniform on $[-1,1]$. Suppose $\eta(x)=0.3$ for $x<0$ and $\eta(x)=0.7$ for $x\geq0$. Find the margin distribution and calculate the excess risk if a classifier disagrees with the Bayes rule on a set of feature probability $0.25$.

For either half of the feature space, $M=|0.3-0.5|=|0.7-0.5|=0.2$. Hence $P(M=0.2)=1$: $G(t)=0$ for $0\leq t<0.2$, and $G(t)=1$ for $t\geq0.2$. There is no feature probability at margins arbitrarily close to zero.

Every disagreement has conditional excess $2M=0.4$. Therefore, if the disagreement set has probability $0.25$, the overall excess risk is $0.4(0.25)=0.10$. The Bayes risk in this model is $0.3$, since the conditional error is $0.3$ at every feature value. The classifier’s risk is consequently $0.3+0.10=0.40$.

What the Margin Distribution Does Not Tell Us

The distribution $G$ is a summary of conditional ambiguity under the feature distribution. It is not a complete description of a learning problem. In particular, it does not specify which feature values a fitted classifier gets wrong, how difficult it is to estimate $\eta$, or how a hypothesis class restricts possible decision boundaries. Two problems can have the same margin distribution and still differ in those respects.

The excess-risk identity depends on the joint placement of disagreements and margins: an error on a small-margin point contributes less than an error on a large-margin point. Knowing only the marginal distribution of $M$ does not tell us whether a particular classifier’s errors fall in one region or the other. To use $G$ in a learning analysis, it must be combined with information about the classifier or the learning procedure.

Common Pitfalls

Using the wrong scale. With $M=|\eta-1/2|$, a wrong decision has conditional excess $2M$, not $M$. Check the definition before applying a formula.
Treating a tie as a costly mistake. When $\eta=1/2$, both labels have the same conditional error. A disagreement with the chosen tie convention has zero excess risk.
Reading $G(t)$ as a misclassification probability. It is the probability of a small margin, not the probability that a fitted classifier is wrong. The two events are different.
Assuming a margin distribution alone determines learning difficulty. It describes how conditional error cost varies over feature probability, but not estimation difficulty or where a learner’s errors occur.

Key takeaway: The margin variable $M=|\eta(X)-1/2|$ measures conditional ambiguity, and its distribution $G(t)=P(M\leq t)$ tracks probability near a tie. Excess risk is exactly twice the margin averaged over classifier disagreements, so the location of errors matters as much as their number.

Check Your Understanding

Work through these questions using the definition and risk identity.

If $\eta(X)=0.65$ at a feature value, what is the margin there, and what is the conditional excess risk of choosing the wrong label?
What does $G(0)$ measure? Why does disagreement at an exact tie contribute no excess risk?
In the uniform-feature example with $\eta(x)=(1+x)/2$, what is $G(0.1)$?
Why does the bound $2tG(t)$ apply to the excess risk from disagreements restricted to $M\leq t$?
Can two classification problems have the same distribution of $M$ but different learning difficulty? Explain what information the margin distribution leaves out.