Why Plot a Margin Curve?
A classifier’s decision boundary separates covariate values assigned to different classes. In binary classification, a fitted probability near the decision threshold signals a score close to that boundary. A margin curve summarizes how much of the covariate distribution lies within a chosen distance of the threshold, repeating the calculation over a range of distances. Rather than reporting only one number, it shows how quickly the near-boundary population grows as the definition of “near” is widened.
The curve is a diagnostic of boundary difficulty: a large share of covariates with small margins means many observations lie near the classification threshold. Such observations are particularly sensitive to changes in estimated probabilities. The curve is not, by itself, a measure of predictive accuracy or probability calibration; those require different comparisons.
Definition and Scale
Let \(Y\in\{0,1\}\), let \(X\) denote the covariates, and write \(\eta(x)=P(Y=1\mid X=x)\). The Bayes threshold under zero-one loss is \(1/2\). We use the normalized margin \(M(x)=|2\eta(x)-1|\), which ranges from 0 to 1: zero means the conditional probability is exactly at the threshold, and values near 1 mean it is far from the threshold. Equivalently, the unnormalized distance is \(|\eta(x)-1/2|=M(x)/2\). Stating the scale matters because the same threshold number means different things under these two conventions.
For a fitted probability function \(\hat\eta\), define its estimated normalized margin by \(\hat M(x)=|2\hat\eta(x)-1|\). Given validation covariates \(X_1,\ldots,X_m\), the empirical estimated margin curve is
At each threshold \(t\), this is the fraction of validation covariates whose fitted probabilities fall within \(t/2\) of \(1/2\). For example, \(t=0.2\) means \(\hat\eta(X_i)\in[0.4,0.6]\). Evaluating the formula on a grid of thresholds and plotting the resulting fractions produces a nondecreasing step curve.
Worked Examples
Worked Example: A Discrete Covariate Distribution
Suppose \(X\) takes three values \(a,b,c\), with probabilities \(0.2,0.5,0.3\), respectively. The true conditional probabilities are \(\eta(a)=0.5\), \(\eta(b)=0.6\), and \(\eta(c)=0.9\). Find the population margin curve.
The normalized margins are \(M(a)=|2(0.5)-1|=0\), \(M(b)=|2(0.6)-1|=0.2\), and \(M(c)=|2(0.9)-1|=0.8\). The corresponding probability masses are \(0.2,0.5,0.3\).
For \(0\le t<0.2\), only \(a\) has margin at most \(t\), so \(C(t)=0.2\). For \(0.2\le t<0.8\), both \(a\) and \(b\) qualify, giving \(C(t)=0.2+0.5=0.7\). For \(0.8\le t\le1\), all three qualify, giving \(C(t)=1\). In particular, \(C(0)=0.2\): one fifth of the covariate distribution lies exactly on the threshold.
Worked Example: A Smooth Boundary
Let \(X\) be uniform on \([-1,1]\), and suppose \(\eta(x)=(1+x)/2\). Then \(M(x)=|2\eta(x)-1|=|x|\). For \(0\le t\le1\), the event \(M(X)\le t\) is \(-t\le X\le t\). A uniform variable on an interval of length 2 falls in this subinterval of length \(2t\) with probability \(2t/2=t\). Thus \(C(t)=t\).
The curve starts at zero and rises linearly to one. For instance, \(C(0.1)=0.1\): ten percent of the covariate distribution is within normalized margin \(0.1\) of the boundary, equivalently within \(0.05\) of \(1/2\) in conditional-probability units. The gradual increase reflects covariate mass spread across distances from the boundary rather than concentrated at a few discrete margins.
Worked Example: Calculating an Empirical Curve
A fitted model is evaluated at ten validation covariates. Its predicted probabilities are \(0.51, 0.54, 0.62, 0.75, 0.91, 0.49, 0.45, 0.30, 0.08, 0.52\). Compute the empirical estimated curve at selected thresholds.
The normalized fitted margins, in the same order, are \(0.02,0.08,0.24,0.50,0.82,0.02,0.10,0.40,0.84,0.04\). At \(t=0.05\), three margins qualify (\(0.02,0.02,0.04\)), so \(\hat C_{10}(0.05)=3/10=0.3\). At \(t=0.10\), five qualify: those three, plus \(0.08\) and \(0.10\). Hence \(\hat C_{10}(0.10)=5/10=0.5\).
At \(t=0.25\), the margin \(0.24\) also qualifies, giving \(6/10=0.6\). At \(t=0.50\), the margins \(0.40\) and \(0.50\) qualify as well, giving \(8/10=0.8\). The two remaining margins, \(0.82\) and \(0.84\), enter by \(t=0.85\), where the curve reaches 1. The equality at \(t=0.10\) and \(t=0.50\) is included by the definition.
Basic Properties and an Area Summary
A margin curve is a cumulative distribution function evaluated on the margin variable. This gives it structural properties that help with both interpretation and checking calculations.
Proof. If \(s\le t\), then \(\{M(X)\le s\}\subseteq\{M(X)\le t\}\), so \(C(s)\le C(t)\). To show right-continuity, take any sequence \(t_k\downarrow t\). The events \(\{M(X)\le t_k\}\) decrease to \(\{M(X)\le t\}\): a value no greater than every \(t_k\) is no greater than their limit \(t\). Continuity of probability for decreasing events therefore gives \(C(t_k)\to C(t)\). Since \(M(X)\in[0,1]\), \(P(M(X)\le1)=1\), so \(C(1)=1\).
For the area identity, write \(C(t)=E[\mathbf{1}\{M(X)\le t\}]\). The integrand is nonnegative and measurable, so Tonelli’s theorem permits exchanging expectation and integration:
For a fixed value \(M(X)=m\), the indicator is one for \(t\in[m,1]\), an interval of length \(1-m\); its value at a single endpoint does not affect the integral. This proves the identity. A larger area corresponds to a smaller average normalized margin, and thus, in this specific sense, more probability mass close to the threshold. The area compresses the curve to one summary, while the full curve shows at which thresholds the mass accumulates. \(\square\)
Worked Example: Comparing Curve Areas
In the discrete example, the margins \(0,0.2,0.8\) have probabilities \(0.2,0.5,0.3\). Therefore \(E[M]=0(0.2)+0.2(0.5)+0.8(0.3)=0.34\), and the area under the curve is \(1-0.34=0.66\). The piecewise curve gives the same result: its area is \(0.2(0.2)+0.6(0.7)+0.2(1)=0.66\).
For the uniform-boundary example, \(C(t)=t\), so its area is \(\int_0^1t\,dt=1/2\). The corresponding margin \(M=|X|\) has expectation \(1/2\), again giving \(1-E[M]=1/2\). These summaries are comparable because both use the same normalized margin scale; their curves still reveal more detail than the areas alone.
What the Curve Does—and Does Not—Diagnose
A high curve at small thresholds indicates that a substantial part of the covariate distribution receives probabilities close to the classification threshold. That can signal a difficult boundary: modest changes in fitted probabilities may change predicted classes for many covariates. A curve that stays low until larger thresholds indicates less covariate mass close to the threshold. It does not guarantee that the fitted labels are correct or that the probabilities are well estimated.
The population curve uses the true \(\eta\), which is generally unknown. The empirical curve instead uses \(\hat\eta\), so it describes the margins assigned by a particular fitted model on the assessment covariates. If those covariates are an independent validation sample and the fitted function is held fixed, the empirical fraction estimates the validation-distribution probability of a small fitted margin. Its interpretation remains conditional on the model and on the sample used for assessment. A curve calculated on the training data can be misleading if the fitting procedure has produced overly confident or otherwise distorted predictions.
In particular, the curve should not be read as a calibration plot. Calibration asks whether predicted probabilities agree with observed event frequencies; the margin curve only counts how many scores lie near the threshold. Nor is it a class-imbalance summary: imbalance concerns the marginal frequency of the labels, while the margin curve concerns conditional probabilities across covariates. Those quantities may be related in a particular problem, but neither determines the other.
For a useful diagnostic, report the margin convention and threshold scale, show several thresholds rather than one arbitrarily selected value, and say whether the curve uses true or fitted probabilities and which covariates were used. Comparing curves across models is most interpretable when they use the same validation covariates and the same margin definition. A visibly higher fitted curve then means more validation covariates receive scores near the threshold; it does not establish that the model has worse accuracy.
Common Mistake
Wrong reasoning: “The curve is high near zero, so the classifier must have low accuracy.” The curve contains no observed labels and does not count incorrect predictions. It measures how concentrated the probability scores are near the decision threshold.
Correction: Use the curve to describe fitted or true margin mass, and use labeled assessment data to evaluate classification accuracy. A high near-zero curve flags potential sensitivity around the boundary, not a guaranteed prediction error. Conversely, a low curve can result from scores pushed far from the threshold, even if those probabilities are poorly calibrated.
Check Your Understanding
Use the normalized margin \(M(x)=|2\eta(x)-1|\) unless a question specifies otherwise.
- Why does \(C(0)\) measure probability mass exactly on the Bayes threshold?
- If a fitted probability is \(0.42\), what is its normalized margin, and which thresholds include it?
- Explain why a population margin curve must be nondecreasing and right-continuous.
- What does the area under a population margin curve equal, and what does a larger area indicate?
- Give one reason a fitted margin curve could look reassuring even when the fitted probabilities are not reliable.