Why Boundary Estimation Has a Quadratic Risk Cost
A fitted linear discriminant rule changes its classification only where its estimated boundary differs from the Bayes boundary. If parameter estimates are close to their population values, the changed region is a thin strip around the true boundary. Its thickness is first order in the estimation error, while the loss per point in that strip is also first order in distance from the boundary. Their product gives a second-order excess-risk term.
The coefficient of that term is not determined by parameter variance alone. It also depends on how much probability mass is weighted at the boundary, how sharply the true score changes in the normal direction, and how sensitive the score is to each parameter. These geometric and distributional contributions are made explicit below. We distinguish carefully between a local expansion in probability and an approximation to expected risk: the latter needs additional control of the remainder.
For Gaussian classes with a common covariance matrix, the log weighted-density ratio is affine in \(x\), so this is the LDA setting. The derivation below is stated for a smooth regular boundary, which also makes clear which features of the argument are geometric rather than specific to Gaussian densities.
The Local Boundary Expansion
Write \(p\) for the dimension of \(x\), and \(m\) for the dimension of \(\theta\). Assume \(g\) is continuously differentiable in \(x\) and \(\theta\) in a neighborhood of \(B\), and that \(\nabla_x g(s;\theta)\neq 0\) at every boundary point \(s\). Define \(\kappa(s)=\|\nabla_xg(s;\theta)\|\), \(a(s)=\nabla_\theta g(s;\theta)\), and \(q(s)=w_0(s)=w_1(s)\) for \(s\in B\). The equality defining \(q\) follows from \(g(s;\theta)=0\). Let \(dS(s)\) denote surface measure on \(B\).
For a small parameter perturbation \(\delta\), the first-order change in the score at a boundary point is \(a(s)^{\mathsf T}\delta\). The normal unit vector pointing toward increasing score is \(n(s)=\nabla_xg(s;\theta)/\kappa(s)\). A point a signed distance \(t\) along that normal has true score \(g(s+t n(s);\theta)=\kappa(s)t+o(|t|)\). Thus the perturbed boundary is displaced, to first order, by \(-a(s)^{\mathsf T}\delta/\kappa(s)\).
Proof. At a point \(x\) where the estimated and Bayes rules disagree, the excess misclassification contribution per unit volume is \(|w_1(x)-w_0(x)|\); where they agree it is zero. Hence excess risk is the integral of this absolute difference over the disagreement region. Use normal coordinates \(x=s+t n(s)\) near the boundary. The true score there satisfies \(g(x;\theta)=\kappa(s)t+o(|t|)\).
The estimated score is \(g(x;\theta+\delta)=g(x;\theta)+a(s)^{\mathsf T}\delta+o(\|\delta\|+|t|)\), uniformly in the local neighborhood. Its zero therefore occurs at signed distance \(h_\delta(s)=-a(s)^{\mathsf T}\delta/\kappa(s)+o(\|\delta\|)\). The true and estimated decisions disagree between \(t=0\) and \(t=h_\delta(s)\), up to a remainder of smaller order.
It is important to expand the weighted-density difference correctly. Since \(w_1(x)=w_0(x)e^{g(x;\theta)}\), \[ |w_1(x)-w_0(x)|=w_0(x)|e^{g(x;\theta)}-1|. \] As \(x\to s\), continuity gives \(w_0(x)=q(s)+o(1)\), and \(e^z-1=z+o(|z|)\) as \(z\to0\). Consequently, \(|w_1(x)-w_0(x)|=q(s)\kappa(s)|t|+o(|t|)\). This is a local approximation, not an exact equality away from the boundary.
The normal-coordinate volume element is \((1+o(1))\,dt\,dS(s)\) as \(t\to0\). Integrating the leading contribution across the disagreement strip gives \[ \int_{\min(0,h_\delta)}^{\max(0,h_\delta)} q(s)\kappa(s)|t|\,dt = \frac{q(s)\kappa(s)}{2}h_\delta(s)^2. \] Substituting the first-order displacement yields \(\frac{q(s)}{2\kappa(s)}(a(s)^{\mathsf T}\delta)^2+o(\|\delta\|^2)\). Integration over \(B\), together with the stated uniform and tail conditions, gives the result. \(\square\)
From Parameter Uncertainty to Expected Excess Risk
For a random fitted parameter, put \(\Delta_n=\hat\theta-\theta\), and define the quadratic term \[ Q(\Delta_n)=\frac{1}{2}\int_B \frac{q(s)}{\kappa(s)} \bigl(a(s)^{\mathsf T}\Delta_n\bigr)^2\,dS(s). \] If \(nE[\Delta_n\Delta_n^{\mathsf T}]\to V\), then the expected quadratic term has a simple limit. But this second-moment condition alone does not make the expected risk equal to the expected quadratic term.
Derivation. For each \(s\), the expectation of the quadratic form is \(E[(a(s)^{\mathsf T}\Delta_n)^2] =a(s)^{\mathsf T}E[\Delta_n\Delta_n^{\mathsf T}]a(s)\). The boundary-integrability assumption permits integration of this expression, and the second-moment limit gives the displayed leading term. The expectation-level remainder condition then transfers the expansion from \(Q\) to the actual risk.
A common way to establish the second-moment condition is to show that \(\sqrt n\Delta_n\) has a limiting distribution with covariance \(V\), has negligible asymptotic bias, and has sufficiently controlled moments (for example, uniform integrability of its squared components). Those conditions address the quadratic term. The separate expectation-level remainder condition is still needed unless it follows from stronger bounds on the risk expansion.
Worked Examples
Worked Example: An Estimated Intercept in a Gaussian LDA Rule
Let \(X\in\mathbb R^2\), with equal class priors and \(X\mid Y=1\sim N((d/2,0),I_2)\), \(X\mid Y=0\sim N((-d/2,0),I_2)\), where \(d>0\). The true log weighted-density ratio is \(g(x)=dx_1\): the true boundary is \(x_1=0\). Suppose the fitted score is \(g(x)+\delta\), an intercept perturbation.
At \(s=(0,x_2)\), \(\kappa(s)=\|\nabla g\|=d\), and \(a(s)=1\). The two weighted densities agree on the boundary, with \(q(s)=\frac12\phi(d/2)\phi(x_2)\), where \(\phi\) is the standard normal density. Therefore \[ Q(\delta)=\frac{\delta^2}{2d} \int_{-\infty}^{\infty}\frac12\phi(d/2)\phi(x_2)\,dx_2 =\frac{\phi(d/2)}{4d}\delta^2. \] If \(E[\delta_n^2]=\tau^2/n+o(n^{-1})\) and the expected remainder is \(o(n^{-1})\), then \(E[\mathcal E]=\tau^2\phi(d/2)/(4dn)+o(n^{-1})\). The boundary density supplies the factor \(\phi(d/2)\), while the score gradient supplies the factor \(1/d\).
Worked Example: Uncertainty in Intercept and Boundary Slope
Use the same Gaussian model, but let the fitted score be \(dx_1+\delta_0+\delta_1x_2\). The parameter perturbation is \(\Delta=(\delta_0,\delta_1)^{\mathsf T}\). Along the true boundary, \(a(s)=(1,x_2)^{\mathsf T}\). Suppose \(nE[\Delta\Delta^{\mathsf T}]\to V=\begin{pmatrix}4&1\\1&2\end{pmatrix}\). This matrix is positive definite: its leading diagonal entry is \(4>0\), and its determinant is \(8-1=7>0\).
The integrand's parameter-uncertainty factor is \[ a(s)^{\mathsf T}Va(s) =4+2x_2+2x_2^2. \] Using \(\int\phi(x_2)\,dx_2=1\), \(\int x_2\phi(x_2)\,dx_2=0\), and \(\int x_2^2\phi(x_2)\,dx_2=1\), its integral against \(\phi(x_2)\) is \(4+0+2=6\). Thus \[ E[\mathcal E] = \frac{1}{2d}\cdot\frac12\phi(d/2)\cdot\frac{6}{n} +o(n^{-1}) = \frac{3\phi(d/2)}{2dn}+o(n^{-1}), \] provided the expected remainder condition holds. Slope uncertainty matters because its effect depends on position \(x_2\) along the boundary; its contribution is included by integrating the squared sensitivity over that boundary.
Worked Example: Why a Variance Calculation May Fail for Expected Risk
In the same Gaussian model, consider the estimated score \(dx_1+Z_nx_2\). Let \(Z_n=c\) and \(Z_n=-c\) each with probability \(1/(nc^2)\), and let \(Z_n=0\) otherwise, for fixed \(c>0\) and sufficiently large \(n\). Then \(E[Z_n]=0\) and \(E[Z_n^2]=c^2\{2/(nc^2)\}=2/n\), so \(Z_n\to_p0\).
Conditional on \(Z_n=z\), the score for class 1 has mean \(d^2/2\) and variance \(d^2+z^2\); for class 0 it has mean \(-d^2/2\) and the same variance. The conditional misclassification probability is therefore \(\Phi\{-d^2/(2\sqrt{d^2+z^2})\}\). At \(z=0\), the Bayes error is \(\Phi(-d/2)\). The actual conditional excess risk is \[ R(c)=\Phi\left(-\frac{d^2}{2\sqrt{d^2+c^2}}\right)-\Phi(-d/2), \] the same for \(z=c\) and \(z=-c\). Hence \(E[\mathcal E]=2R(c)/(nc^2)\).
The local quadratic formula, using \(a(s)=x_2\) and \(E[Z_n^2]=2/n\), would instead predict \(\phi(d/2)/(2dn)\). These coefficients need not agree: as \(c\to\infty\), \(R(c)\) approaches the finite value \(1/2-\Phi(-d/2)\), so \(2R(c)/c^2\to0\), whereas \(\phi(d/2)/(2d)>0\). Thus convergence in probability and the correct variance do not suffice. Rare, large perturbations can affect expected risk at order \(n^{-1}\) even though they disappear in probability.
Common Mistakes and Interpretation
- Using an exact boundary approximation away from the boundary. The identity is \(w_1(x)-w_0(x)=w_0(x)\{e^{g(x;\theta)}-1\}\). Near \(s\in B\), \(w_0(x)=q(s)+o(1)\), so the magnitude is \(q(s)|g(x;\theta)|+o(|g(x;\theta)|)\). Replacing \(w_0(x)\) by \(q(s)\) as an exact equality is incorrect.
- Keeping only parameter variance and ignoring geometry. The coefficient includes \(q(s)\), the normal score gradient \(\kappa(s)\), and the score sensitivity \(a(s)\). Omitting any of these generally changes the answer.
- Inferring an expected-risk expansion from convergence in probability. A remainder that is small in probability can still have a non-negligible expectation. Establish an expectation-level remainder bound, or impose sufficient moment and uniform-integrability conditions that imply one.
- Using covariance when bias contributes at the same order. The quadratic term uses \(E[\Delta_n\Delta_n^{\mathsf T}]\), not just \(\operatorname{Cov}(\Delta_n)\). A bias of order \(n^{-1/2}\) contributes to this second moment and must be included.
The result explains why estimated LDA boundaries can have small excess risk even when their parameters are uncertain: only errors that move a high-density portion of the boundary in a sensitive direction contribute substantially. It also clarifies the limit of this approximation. It is a local statement about small boundary perturbations; turning it into an expected-risk statement requires control of exceptional estimation errors as well as a covariance calculation.
Check Your Understanding
Check both the boundary calculation and the conditions behind its expected-risk interpretation.
- Why is the weighted-density difference near a boundary point \(s\) approximately \(q(s)|g(x;\theta)|\), rather than exactly equal to it?
- In the local theorem, what roles are played by \(q(s)\), \(\kappa(s)\), and \(a(s)\)?
- For an intercept perturbation in the two-dimensional Gaussian example, explain why the leading excess risk is proportional to \(\delta^2\), not \(|\delta|\).
- Which second-moment matrix appears in the expected-risk formula, and how can a non-negligible bias affect it?
- Why do convergence in probability and a variance of order \(1/n\) fail, by themselves, to guarantee the expected-risk approximation?