Tangential Crossings and Flat Regression Functions

Why Flatness Changes the Margin Distribution

For binary classification, the Bayes decision changes at the threshold where the regression function equals one half. How quickly that function moves away from one half determines how much probability can accumulate near the decision boundary. A regular crossing has a nonzero first derivative in the direction normal to the boundary. A tangential contact or a flatter crossing moves away more slowly, so a wider region of feature space has a small margin.

This tutorial quantifies that effect. The key calculation is that if the regression function departs from one half at order \(k\) in the normal direction, then the probability of a margin at most \(t\) is typically of order \(t^{1/k}\). This conclusion requires both a local expansion near the boundary and control of near-threshold points elsewhere.

Definition: Let \(Y\in\{0,1\}\), let \(X\) have distribution with density \(q\), and define the binary regression function \(\eta(x)=P(Y=1\mid X=x)\). The margin at \(x\) is \(M(x)=|2\eta(x)-1|\), and the random margin is \(M=M(X)\). Thus \(M\le t\) means that the conditional class probability is close to one half, on the scale \(2\eta-1\).

Write \(S=\{x:\eta(x)=1/2\}\) for the threshold set in the support of \(X\). In a regular geometric setting, \(S\) is a smooth hypersurface. Let \(u\) denote signed distance from \(S\), with \(u=0\) on \(S\), and let \(s\in S\) identify the nearest boundary point. The normal coordinate \(u\) is the relevant direction for measuring how quickly \(\eta\) leaves the threshold.

Order of Contact: Crossing or Tangency

Suppose that, uniformly along \(S\), the regression function has the expansion \(\eta(s+u n(s))-1/2=a(s)u^k+r(s,u)\), where \(n(s)\) is a unit normal, \(k\) is a positive integer, and the remainder is negligible compared with \(|u|^k\). Assume \(a(s)\) is bounded away from zero. The margin then has leading behavior \(2|a(s)||u|^k\).

When \(k\) is odd, the leading term changes sign as \(u\) changes sign: this is a crossing of the Bayes threshold. When \(k\) is even, it has the same sign on both sides: this is a tangential contact, with the threshold approached and then left on the same side. In either case, the small-margin probability depends on \(|u|^k\), so the order \(k\) determines its exponent.

Worked Example: A Quadratic Tangency

Let \(X\) be uniform on \([-1,1]\) and set \(\eta(x)=1/2+x^2/8\). This is a valid regression function: it ranges from \(1/2\) to \(5/8\). The threshold set is \(S=\{0\}\), and the function touches one half without crossing it.

The margin is \(M(x)=|2\eta(x)-1|=x^2/4\). Thus \(M\le t\) exactly when \(|x|\le 2\sqrt{t}\). For \(0\le t\le 1/4\), this interval lies inside \([-1,1]\). Since the density of \(X\) is \(1/2\),

P(M\le t)=\frac{1}{2}(4\sqrt{t})=2t^{1/2}.

The exponent is \(1/2\), the reciprocal of the order \(k=2\). The calculation also verifies the constant, not just the order.

Worked Example: A Fourth-Order Flat Contact

Keep \(X\) uniform on \([-1,1]\), but take \(\eta(x)=1/2+x^4/8\). The regression function ranges from \(1/2\) to \(5/8\), so it is valid, and again the threshold set is \(\{0\}\). Now \(M(x)=x^4/4\), so \(M\le t\) is equivalent to \(|x|\le (4t)^{1/4}\).

For \(0\le t\le 1/4\), the interval is contained in the support. Its length is \(2(4t)^{1/4}\), and the density is \(1/2\). Therefore

P(M\le t)=\frac{1}{2}\,2(4t)^{1/4}=4^{1/4}t^{1/4}.

The fourth-order contact produces exponent \(1/4\), so small-margin probability decreases more slowly than in the quadratic example. At the same small value of \(t\), more probability lies near the threshold.

Worked Example: A Cubic Crossing

Let \(X\) again be uniform on \([-1,1]\), and define \(\eta(x)=1/2+x^3/4\). This ranges from \(1/4\) to \(3/4\), so it is a valid regression function. Its sign relative to one half changes at zero, making this an order-three crossing rather than a tangency.

Here \(M(x)=|x|^3/2\), and \(M\le t\) is equivalent to \(|x|\le (2t)^{1/3}\). For \(0\le t\le 1/2\), the interval is inside the support, and hence

P(M\le t)=\frac{1}{2}\,2(2t)^{1/3}=(2t)^{1/3}.

The exponent is \(1/3\). The parity of the order determines whether the regression function crosses the threshold, while the magnitude of the order determines the small-margin exponent.

A Boundary-Level Asymptotic

The examples give an exact one-dimensional calculation. The same exponent appears around a smooth boundary in any dimension. The following result states conditions that ensure that the local expansion accounts for all sufficiently small margins.

Theorem: Let \(K\subset\mathbb{R}^d\) be compact, and suppose \(X\) has density \(q\) on \(K\). Assume \(\eta\) is continuous on \(K\), its threshold set \(S=\{x\in K:\eta(x)=1/2\}\) is a compact smooth hypersurface lying in the interior of \(K\), and \(q\) is continuous and strictly positive on a neighborhood of \(S\). For some integer \(k\ge1\), suppose there is a continuous function \(a\) on \(S\), with \(0<\inf_{s\in S}|a(s)|\le\sup_{s\in S}|a(s)|<\infty\), such that \(\eta(s+u n(s))-1/2=a(s)u^k+r(s,u)\), with \(\sup_{s\in S}|r(s,u)|/|u|^k\to0\) as \(u\to0\). Then, as \(t\downarrow0\),

P(M\le t) = t^{1/k} \left[ 2\int_S q(s)\{2|a(s)|\}^{-1/k}\,d\mathcal{H}^{d-1}(s) +o(1) \right].

Here \(n(s)\) is a unit normal, chosen consistently in a tubular neighborhood, and \(\mathcal{H}^{d-1}\) denotes surface measure. The expression in brackets has a positive finite leading constant under the stated assumptions. In particular, \(P(M\le t)\) is asymptotic to that constant times \(t^{1/k}\).

Proof. Choose a sufficiently small tubular neighborhood \(T_\delta=\{s+u n(s):s\in S,\ |u|<\delta\}\), so each point there has unique normal coordinates. The expansion and the lower bound on \(|a|\) imply that, after reducing \(\delta\) if needed, there are constants \(c,C>0\) such that

c|u|^k\le M(s+u n(s))\le C|u|^k \qquad(s\in S,\ |u|<\delta).

Continuity of \(\eta\) on compact \(K\), together with the assumption that its entire threshold set in \(K\) is \(S\), gives a positive gap outside any fixed neighborhood of \(S\). In particular, \(\inf_{x\in K\setminus T_\delta}M(x)>0\). Consequently, for all sufficiently small \(t\), the event \(M\le t\) is contained in \(T_\delta\). This step prevents other parts of the support from contributing additional near-threshold probability.

In tubular coordinates, Lebesgue volume has element \(J(s,u)\,du\,d\mathcal{H}^{d-1}(s)\), where \(J(s,u)\to1\) uniformly in \(s\) as \(u\to0\). Set \(u=t^{1/k}v\). For each fixed \(s,v\),

\frac{M(s+t^{1/k}v n(s))}{t} \longrightarrow 2|a(s)||v|^k.

The event indicator therefore converges to the indicator of \(|v|<(2|a(s)|)^{-1/k}\), except at the two endpoints in \(v\), which form a set of measure zero. The inequalities \(c|u|^k\le M\le C|u|^k\) imply that any contributing \(v\) lies in the fixed bounded interval \(|v|\le c^{-1/k}\). On this interval, \(q(s+t^{1/k}v n(s))J(s,t^{1/k}v)\) is uniformly bounded, because \(q\) is continuous near the compact set \(S\). Thus the dominated convergence theorem applies to the integral over \(S\) and this bounded interval. The limiting interval in \(v\) has length \(2(2|a(s)|)^{-1/k}\). Combining its length with the limiting density \(q(s)\) gives the displayed constant. The gap outside \(T_\delta\) means this integral accounts for the whole event, proving the result. \(\square\)

Reading the Exponent and Its Limitations

The theorem says that the reciprocal of the normal order of contact controls the small-margin exponent. A linear crossing has \(k=1\) and probability of order \(t\). A quadratic contact has order \(t^{1/2}\); an order-four contact has order \(t^{1/4}\). For small \(t\), a smaller exponent means a larger value of \(t^{1/k}\), and therefore more probability very near the Bayes threshold.

The constant also has a geometric interpretation. Each boundary point contributes according to the density \(q(s)\) there and the local coefficient \(|a(s)|\): a larger coefficient means the regression function leaves one half more quickly, narrowing the near-threshold strip. The factor of two counts the two normal directions. This is why the margin behavior is not determined by the order \(k\) alone.

Important condition: The local expansion near \(S\) does not by itself determine \(P(M\le t)\). The theorem also excludes small margins away from \(S\). Continuity of \(\eta\) on compact \(K\), with no threshold points outside \(S\), guarantees the required positive gap. Alternatively, one can assume directly that the small-margin event outside a fixed tubular neighborhood is empty for all sufficiently small \(t\).

Common Mistake

A tempting argument is: “The expansion near the boundary gives probability of order \(t^{1/k}\), so that must be the total small-margin probability.” This only computes the contribution from a neighborhood of the stated boundary. If \(\eta\) is merely measurable elsewhere, there may be points far from that boundary where \(\eta\) gets arbitrarily close to one half. Those regions can contribute a larger probability and change the exponent or invalidate the proposed asymptotic. The correction is to verify a positive margin gap off the neighborhood, or to impose a direct condition ruling out remote near-threshold points. Continuity on compact support, when the zero set is exactly \(S\), is one standard way to obtain that gap.

A second interpretive error is to call every even-order contact a crossing. For an even \(k\) and nonzero leading coefficient, the leading term has the same sign on both sides of \(S\); the threshold is touched, not crossed. This distinction matters for the shape of the Bayes decision regions, although the absolute-margin calculation uses the same order \(t^{1/k}\).

Check Your Understanding

Use the definition \(M=|2\eta(X)-1|\) and distinguish the local calculation from conditions needed to make it a global probability statement.

If \(\eta-1/2\) is proportional to \(u^5\) near a smooth boundary, what is the predicted small-margin exponent, and does the leading term cross the threshold?
For \(\eta(x)=1/2+x^6/8\) with \(X\) uniform on \([-1,1]\), derive \(P(M\le t)\) for sufficiently small \(t\).
Why does an even-order leading term with nonzero coefficient describe tangency rather than crossing?
Which hypotheses in the theorem ensure that near-threshold points away from \(S\) do not change the asymptotic?
In the boundary-level constant, what roles do \(q(s)\) and \(|a(s)|\) play?