Estimating Margin Exponents in Design Examples

Why the Design Distribution Matters

In binary classification, small margins occur where the conditional probability of class 1 is close to one half. How much probability the design distribution places near those locations determines how often such small margins arise. Consequently, the order at which the conditional probability crosses one half is only part of the answer: the local behavior of the covariate density matters as well.

This tutorial calculates the resulting small-margin exponent in explicit design examples. The central calculation is an integration: translate a small probability gap into a neighborhood of the crossing, then measure the design probability of that neighborhood.

Margin and Local Boundary Behavior

Let $X$ be a covariate and $Y\in\{0,1\}$ a binary response. Write $\eta(x)=P(Y=1\mid X=x)$. The quantity $|2\eta(X)-1|$ is the conditional class-probability gap on the doubled scale. It is zero where the two classes have equal conditional probability.

Definition: The margin random variable is $M=|2\eta(X)-1|$. A distribution has small-margin exponent $\alpha>0$ with leading constant $K>0$ if $$ P(M\le t)\sim Kt^\alpha \qquad\text{as }t\downarrow 0, $$ where $\sim$ means that the ratio of the two sides tends to 1. In particular, this asymptotic implies a margin bound $P(M\le t)\le Ct^\alpha$ for all sufficiently small $t$, for some finite $C$.

The factor 2 in this definition is a convention. Using $|\eta(X)-1/2|$ instead changes constants in the calculations below, but not the exponent. In the examples, the equality $\eta(x)=1/2$ holds only at finitely many points, each of design probability zero, so $P(M=0)=0$.

Consider first a crossing at the endpoint $x=0$ of a one-dimensional design space. Suppose the design has density $f_X(x)\sim c x^a$, and the margin gap $g(x)=|2\eta(x)-1|$ satisfies $g(x)\sim d x^\kappa$ as $x\downarrow0$, with $c,d,\kappa>0$ and $a>-1$. The restriction $a>-1$ ensures that the density is integrable near zero. The parameter $\kappa$ describes the order of the crossing; $a$ describes how the design density behaves there.

Theorem: Suppose $X$ has a density on $[0,1]$ with $f_X(x)\sim c x^a$ as $x\downarrow0$, where $c>0$ and $a>-1$. Suppose $g(x)\sim d x^\kappa$ there, where $d,\kappa>0$, and that $g$ is bounded away from zero outside every sufficiently small neighborhood of 0. Then $$ P(g(X)\le t)\sim \frac{c}{a+1}d^{-(a+1)/\kappa}t^{(a+1)/\kappa}. $$ Thus the small-margin exponent is $\alpha=(a+1)/\kappa$.

Proof. First, integrate the density near zero. Since $f_X(x)/(cx^a)\to1$, for every $\varepsilon\in(0,1)$, sufficiently small $x$ satisfies $(1-\varepsilon)cx^a\le f_X(x)\le(1+\varepsilon)cx^a$. Integrating these inequalities from 0 to $r$, and then letting $\varepsilon\downarrow0$, gives $$ P(X\le r)=\int_0^r f_X(x)\,dx\sim \frac{c}{a+1}r^{a+1}. $$

Likewise, for every $\varepsilon\in(0,1)$, sufficiently small $x$ satisfies $d(1-\varepsilon)x^\kappa\le g(x)\le d(1+\varepsilon)x^\kappa$. Because $g$ is bounded away from zero outside a neighborhood of 0, the event $g(X)\le t$, for small enough $t$, must lie within that neighborhood. There, the inequalities give the event inclusions $$ \left\{X\le \left(\frac{t}{d(1+\varepsilon)}\right)^{1/\kappa}\right\} \subseteq \{g(X)\le t\} \subseteq \left\{X\le \left(\frac{t}{d(1-\varepsilon)}\right)^{1/\kappa}\right\}. $$

Apply the density-integral asymptotic to the two outer probabilities. Divide by $t^{(a+1)/\kappa}$, take lower and upper limits as $t\downarrow0$, and then let $\varepsilon\downarrow0$. Both bounds converge to $\frac{c}{a+1}d^{-(a+1)/\kappa}$, proving the stated asymptotic and exponent. $\square$

Worked Examples

Worked Example: A Density That Vanishes at the Crossing

Let $X$ have density $f_X(x)=3x^2$ on $[0,1]$, and let $\eta(x)=1/2+x/4$. Calculate the small-margin exponent and the exact probability for small $t$.

The density is normalized because $\int_0^1 3x^2\,dx=1$. The regression function lies between $1/2$ and $3/4$, and the margin is $M=|2\eta(X)-1|=X/2$. Thus $M\le t$ exactly when $X\le2t$. For $0<t\le1/2$, $$ P(M\le t)=\int_0^{2t}3x^2\,dx=(2t)^3=8t^3. $$

Here $a=2$, because the density is proportional to $x^2$, and $\kappa=1$, because the gap is proportional to $x$. The theorem gives $\alpha=(2+1)/1=3$, agreeing with the exact calculation. The design puts relatively little probability near the crossing, which makes very small margins comparatively rare.

Worked Example: An Integrable Density That Increases Near the Crossing

Let $X$ have density $f_X(x)=\tfrac12 x^{-1/2}$ on $[0,1]$, and let $\eta(x)=1/2+x^2/4$. Find the small-margin exponent.

The density is integrable and normalized: $\int_0^1 \tfrac12 x^{-1/2}\,dx=1$. The margin is $M=X^2/2$, so $M\le t$ is equivalent to $X\le\sqrt{2t}$. For $0<t\le1/2$, this threshold is at most 1, and $$ P(M\le t)=\int_0^{\sqrt{2t}}\frac12 x^{-1/2}\,dx =\left.\sqrt{x}\right|_0^{\sqrt{2t}} =(2t)^{1/4}. $$

The density power is $a=-1/2$, which is allowed because $a>-1$, and the crossing order is $\kappa=2$. Therefore $\alpha=(a+1)/\kappa=(1/2)/2=1/4$. The density increases without bound near zero, but its integral there is finite; it places more design probability near the crossing and produces a smaller exponent.

Worked Example: An Interior Crossing with a Vanishing Density

Let $X$ have density $f_X(x)=\tfrac34|x|^{1/2}$ on $[-1,1]$, and set $\eta(x)=1/2+(1/5)\operatorname{sign}(x)|x|^2$, with $\operatorname{sign}(0)=0$. Calculate the small-margin exponent.

The density integrates to one, since $\int_{-1}^1|x|^{1/2}\,dx=4/3$. Also, $\eta(x)$ ranges from $3/10$ to $7/10$, so it is a valid conditional probability. The interior crossing is at zero, and $M=|2\eta(X)-1|=(2/5)|X|^2$. Thus, for $0<t\le2/5$, the event $M\le t$ is $|X|\le\sqrt{5t/2}$. Writing $r=\sqrt{5t/2}$, we obtain $$ P(M\le t)=\int_{-r}^{r}\frac34|x|^{1/2}\,dx =\frac32\int_0^r x^{1/2}\,dx =r^{3/2} =\left(\frac{5t}{2}\right)^{3/4}. $$

The density behaves like $|x|^{1/2}$ on both sides of the interior crossing, while the gap behaves like $|x|^2$. The probability of a radius-$r$ neighborhood is proportional to $r^{3/2}$, so substituting $r$ proportional to $t^{1/2}$ gives exponent $3/4$. This example has the same local power calculation as a one-sided boundary, but includes both sides of the crossing; the two sides affect the constant, not the exponent.

When Several Regions Can Produce Small Margins

A design can have several locations where $\eta(x)=1/2$. Each such region contributes probability to $P(M\le t)$. The total small-margin behavior is determined by the contribution that decreases most slowly as $t$ approaches zero—that is, by the smallest exponent among the regions.

Result: Suppose there are finitely many disjoint neighborhoods of margin-zero locations. Suppose the probability contributed by neighborhood $j$ is asymptotic to $K_jt^{\alpha_j}$, where $K_j>0$ and $\alpha_j>0$, and suppose the margin is bounded away from zero outside these neighborhoods. Then $$ P(M\le t)\sim \left(\sum_{j:\,\alpha_j=\alpha_*}K_j\right)t^{\alpha_*}, \qquad \alpha_*=\min_j\alpha_j. $$

Proof. For small enough $t$, no point outside the stated neighborhoods can satisfy $M\le t$. The neighborhoods are disjoint, so their event probabilities add: $P(M\le t)=\sum_j P_j(M\le t)$, where $P_j$ denotes the probability contribution from neighborhood $j$. Divide by $t^{\alpha_*}$. If $\alpha_j=\alpha_*$, the $j$th term tends to $K_j$. If $\alpha_j>\alpha_*$, its ratio is asymptotic to $K_jt^{\alpha_j-\alpha_*}$, which tends to zero. Summing the finitely many limits proves the result. $\square$

Reading and Checking an Exponent

The power $\alpha$ can be read as the slope of the small-margin probability on a log-log scale: if $P(M\le t)\sim Kt^\alpha$, then $\log P(M\le t)$ is approximately $\log K+\alpha\log t$ for small $t$. In a specified model, however, the reliable route is to derive the event $M\le t$ and integrate the design density, rather than infer a slope from intuition alone.

A frequent mistake is to report the density power $a$ as the margin exponent. A density proportional to $x^a$ assigns probability proportional to $r^{a+1}$, not $r^a$, to $[0,r]$: integration adds one to the power. A second step then converts the neighborhood radius $r$ to the margin threshold $t$; if the gap is proportional to $x^\kappa$, then $r$ is proportional to $t^{1/\kappa}$. Together these give $(a+1)/\kappa$.

For an interior crossing, integrate on both sides when both are present. This can double a leading constant, but it does not double the exponent. Likewise, replacing $|2\eta-1|$ by $|\eta-1/2|$ changes the threshold by a constant factor, leaving the exponent unchanged. These checks help distinguish a genuine change in the small-margin rate from a change in units.

These calculations matter because margin assumptions summarize how much design probability lies near ambiguous classification decisions. A model with the same crossing order can have a different exponent under a different design density; conversely, a density concentrated near the crossing can offset a relatively gradual crossing. The exponent is therefore a property of the conditional probabilities and the design distribution together, not of the regression function alone.

Check Your Understanding

For each question, identify the neighborhood probability before simplifying its power of $t$.

If $f_X(x)\sim c x^3$ near a one-sided crossing and the gap is proportional to $x^2$, what is the small-margin exponent?
Why does a density proportional to $x^{-2/3}$ near zero define an integrable local design density, and what is the probability order of $X\le r$?
For an interior crossing with density proportional to $|x|^a$ and gap proportional to $|x|^\kappa$, what exponent follows when both sides have the same local behavior?
Two disjoint margin-zero regions contribute orders $t^{1/2}$ and $t^{3/4}$. Which one determines the overall exponent, and why?
Does multiplying the margin gap by a positive constant change its exponent? Explain using the threshold neighborhood.