Margin Distributions Around the Bayes Threshold

Why Describe Margins Near the Threshold?

In binary classification, the Bayes threshold separates feature values for which class 1 is more likely from those for which class 0 is more likely. The regression function may be far from the threshold for most feature values, or a large amount of feature probability may lie very close to it. These situations have different implications for how sensitive classification is to small changes in the regression function. A convenient way to describe that concentration is to study the distribution of the distance from the threshold.

Let $X$ be a feature vector and let $Y\in\{0,1\}$ be its class label. Write $P_X$ for the marginal distribution of $X$, and let $\eta(x)=P(Y=1\mid X=x)$ be a measurable version of the binary regression function. The Bayes threshold is $1/2$. This tutorial focuses on the distribution of the margin around that threshold, not on the construction of a classifier.

Definition: The absolute Bayes margin is the random variable $$ T=|2\eta(X)-1|. $$ It takes values in $[0,1]$. Its distribution function is $G(t)=P(T\le t)$. Thus $G(t)$ is the probability that a randomly drawn feature vector has margin at most $t$ from the Bayes threshold.

The factor of 2 makes the margin equal to 0 at the threshold and equal to 1 when $\eta(X)$ is 0 or 1. Equivalently, $T=2|\eta(X)-1/2|$. Since conditional probabilities are defined up to $P_X$-null sets, changing the chosen version of $\eta$ on a null set does not change the distribution of $T$.

The Margin Condition

The margin condition bounds the lower tail of $T$: it limits how much feature probability can be concentrated arbitrarily close to the threshold. A larger exponent means the bound requires that this probability decrease more quickly as the margin level shrinks.

Definition: The distribution of $X$ and the regression function satisfy a margin condition with exponent $\alpha$ if there are constants $\alpha\ge 0$ and $C\ge 1$ such that $$ P\bigl(|2\eta(X)-1|\le t\bigr)\le C t^\alpha \qquad\text{for every }0<t\le 1. $$ Equivalently, $G(t)\le Ct^\alpha$ on this range. A version stated only for $0<t\le t_0$, for some $t_0>0$, is called a local margin condition; the range away from zero can be included by enlarging the constant.

The exponent and constant are part of the condition: an exponent alone does not determine a bound. The case $\alpha=0$ imposes no restriction, since $G(t)\le1$ already. A positive exponent is informative about how the distribution behaves near zero. The condition is an upper bound, not a claim that $G(t)$ equals a power of $t$, and it does not require $T$ to have a density.

Worked Example: A Point Mass at the Bayes Threshold

Suppose $X$ takes values $a$ and $b$, with $P(X=a)=1/4$ and $P(X=b)=3/4$. Let $\eta(a)=1/2$ and $\eta(b)=0.8$. Find the margin distribution and decide which positive margin exponents are possible.

At $X=a$, $T=|2(1/2)-1|=0$. At $X=b$, $T=|2(0.8)-1|=0.6$. Therefore $T$ has probability $1/4$ at zero and probability $3/4$ at $0.6$, so $$ G(t)= \begin{cases} 0, & t<0,\\ 1/4, & 0\le t<0.6,\\ 1, & 0.6\le t. \end{cases} $$ For any $\alpha>0$, the margin condition would require $1/4\le Ct^\alpha$ for every sufficiently small positive $t$. But $Ct^\alpha$ tends to zero as $t$ decreases to zero, so this inequality cannot hold for any finite $C$. For $\alpha=0$, $G(t)\le1$ holds. The positive probability exactly on the threshold is what prevents a positive exponent.

Worked Example: Uniform Margins from a Uniform Feature

Let $X$ be uniform on $[0,1]$, and set $\eta(x)=x$. Compute $G(t)$ and find the largest possible margin exponent.

Here $T=|2X-1|$. For $0\le t\le1$, the event $T\le t$ is equivalent to $$ \frac{1-t}{2}\le X\le\frac{1+t}{2}. $$ The interval has length $(1+t)/2-(1-t)/2=t$. Since $X$ is uniform, $G(t)=t$ for $0\le t\le1$. Thus the margin condition holds with $\alpha=1$ and $C=1$.

It also holds for every $0\le\alpha\le1$, because $t\le t^\alpha$ when $0<t\le1$ and $\alpha\le1$. It cannot hold for $\alpha>1$: it would require $t\le Ct^\alpha$, or $t^{1-\alpha}\le C$, for every $0<t\le1$. The left side is unbounded as $t$ decreases to zero. The largest possible exponent is therefore 1.

Worked Example: A Quadratically Small Lower Tail

Let $X$ have density $f_X(x)=2x$ on $[0,1]$, and set $\eta(x)=(1+x)/2$. Find the distribution of the margin and its largest exponent.

The density integrates to one, since $\int_0^1 2x\,dx=1$. Also $0\le\eta(x)\le1$, and $$ T=|2\eta(X)-1|=|1+X-1|=X. $$ For $0\le t\le1$, it follows that $$ G(t)=P(X\le t)=\int_0^t 2x\,dx=t^2. $$ The condition holds with $\alpha=2$ and $C=1$. It holds for every $0\le\alpha\le2$, since $t^2\le t^\alpha$ on $(0,1]$ in that range. If $\alpha>2$, it would require $t^2\le Ct^\alpha$, so $t^{2-\alpha}\le C$; this fails as $t$ decreases to zero. The largest exponent is 2.

Consequences and Exponent Conventions

Two basic facts help interpret the definition. First, a positive exponent rules out an atom at the threshold. Second, if a bound holds for one exponent, it holds for any smaller nonnegative exponent on the same range. These statements concern the distribution of the margin alone; they do not require a smooth regression function.

Theorem: If the margin condition holds with $\alpha>0$, then $P(T=0)=0$. If it holds with exponent $\alpha$, then it also holds with any exponent $\beta$ satisfying $0\le\beta\le\alpha$, using the same constant $C$.

Proof. The events $\{T\le 1/k\}$, for positive integers $k$, decrease to $\{T=0\}$, because $T\ge0$. Continuity of probability for decreasing events gives $$ P(T=0)=\lim_{k\to\infty}P(T\le1/k) \le\lim_{k\to\infty}C(1/k)^\alpha=0. $$ For the second claim, if $0<t\le1$ and $\beta\le\alpha$, then $t^\alpha\le t^\beta$. Hence $$ G(t)\le Ct^\alpha\le Ct^\beta. $$ Both claims follow. $\square$

The largest exponent for which a distribution satisfies a margin condition is sometimes called its margin exponent. This refers to the strongest polynomial upper bound available, not necessarily to an exact power law. For example, a distribution might satisfy a bound with exponent $\alpha$ even if its lower tail is smaller than $t^\alpha$.

Some sources measure distance from the threshold using $S=|\eta(X)-1/2|$, rather than $T=|2\eta(X)-1|$. Since $T=2S$, the same exponent is used, but the constant changes. In particular, $$ P(S\le t)=P(T\le 2t)\le C(2t)^\alpha $$ whenever the bound for $T$ applies at $2t$. Conversely, a bound $P(S\le t)\le C_0t^\alpha$ implies $P(T\le t)\le C_0(t/2)^\alpha$. When a convention uses a different scaling, compare constants and ranges as well as exponents.

Why the Distribution Near Zero Matters

The margin distribution summarizes how much probability lies close to the decision threshold, without requiring the reader to inspect the full shape of $\eta(x)$. A margin condition is useful because observations with small margins are precisely those whose conditional class probabilities are close to a tie. If $G(t)$ is small for small $t$, relatively little feature probability lies in that near-threshold region; if $G(t)$ stays bounded away from zero as $t$ shrinks, there is an atom on the boundary.

The condition alone does not specify where in the feature space these margins occur, nor does it assert that the regression function is smooth. It is a statement about the joint effect of the marginal feature distribution and the values of $\eta$. Two different feature distributions and regression functions can have the same margin distribution. Later analyses can combine this lower-tail description with other assumptions, but the margin condition itself is only a bound on $P(T\le t)$.

Common Pitfalls

Excluding equality by accident. Using $P(T<t)$ instead of $P(T\le t)$ can hide an atom at zero. For positive exponents, the condition as defined here includes the threshold and therefore rules out such an atom.
Comparing exponents without checking the scale. Replacing $|2\eta(X)-1|$ with $|\eta(X)-1/2|$ changes the argument of the distribution function by a factor of 2. The exponent is unchanged, but constants and ranges must be adjusted.
Reading an upper bound as an equality. The statement $G(t)\le Ct^\alpha$ does not say that $G(t)=Ct^\alpha$, or that the margin has a density.
Treating every exponent as equally informative. A condition with exponent zero is automatic. Moreover, a bound at exponent $\alpha$ also gives bounds at smaller exponents, so the strongest available exponent is the most informative one.

Key takeaway: The margin variable $T=|2\eta(X)-1|$ measures distance in conditional class probability from the Bayes threshold, and $G(t)=P(T\le t)$ records how much feature probability is near that threshold. A margin condition bounds this lower tail by a power of $t$.

Check Your Understanding

Use the definition and examples above to answer the following questions.

If $T$ has probability $0.1$ at zero, can a margin condition hold with any positive exponent? Explain.
If $G(t)\le 5t^{3/2}$ for $0<t\le1$, what other nonnegative exponents does this same bound imply?
For the uniform-feature example with $\eta(x)=x$, why does an exponent greater than 1 fail?
Suppose a source defines its margin as $|\eta(X)-1/2|$. How is that variable related to $T$, and what happens to the exponent when the scale changes?
Does a margin condition require the distribution of $T$ to have a density? State what the condition actually bounds.