Why Parameter Growth Can Have a Directional Effect
In an interpolating classifier, the training labels may already be correct while the numerical margins continue to change. In a positively homogeneous network, multiplying every parameter by the same positive constant multiplies each output by a power of that constant. Thus parameter norms can grow without changing the classifier’s predictions, while the direction of the parameter vector continues to encode how the classifier separates the data.
This creates a possible implicit bias: gradient-based optimization may favor certain parameter directions, and some such directions satisfy a margin-maximization condition. The qualification matters. Positive homogeneity alone does not prove that optimization converges, that its direction converges, or that any limiting direction is a global maximum-margin solution. We will make a conditional stationarity result precise and identify the assumptions it needs.
Homogeneity and Parameter-Space Margin
For a nonzero parameter vector, write \(\theta=ru\), where \(r=\|\theta\|_2>0\) and \(\|u\|_2=1\). Homogeneity gives \(m_i(ru)=r^L m_i(u)\). Consequently, scaling a parameter vector by a positive constant does not change its margin signs, but it scales their magnitudes. The direction \(u\) determines the relative geometry of the margins on the unit sphere.
A natural parameter-space margin objective is \( \max_{\|u\|_2=1}\min_i m_i(u) \). It asks for a unit-norm parameter direction whose smallest training margin is as large as possible. This is an optimization problem in the parameters; it need not be the same as maximizing margin under a norm on the network’s output functions.
Worked Example: A Two-Factor Homogeneous Classifier
Take one training observation \(x=1,y=1\), and define \(f_{a,b}(x)=abx\). The parameter vector is \((a,b)\), and scaling both coordinates by \(c>0\) gives \(f_{ca,cb}(x)=c^2 f_{a,b}(x)\). This model is homogeneous of degree \(2\). Its training margin is \(m(a,b)=ab\).
On the unit circle \(a^2+b^2=1\), the inequality \((a-b)^2\geq 0\) gives \(2ab\leq a^2+b^2=1\). Hence the largest possible margin is \(1/2\), attained at \(a=b=1/\sqrt{2}\). The classifier’s best unit-norm direction is therefore balanced between its two factors.
Now use exponential loss \(H(a,b)=e^{-ab}\) and gradient flow, meaning \(\dot a=-\partial H/\partial a\) and \(\dot b=-\partial H/\partial b\). Direct differentiation gives \(\dot a=b e^{-ab}\) and \(\dot b=a e^{-ab}\). Therefore \(\frac{d}{dt}(a^2-b^2)=2a\dot a-2b\dot b=0\): the difference of squared factors is invariant. Starting with \(a,b>0\), both factors increase, and \(q=ab\) satisfies \(\dot q=(a^2+b^2)e^{-q}>0\). If \(q\) were bounded, then \(a^2+b^2\geq 2ab=2q\) alone would not give a positive lower bound when \(q\) is near zero; instead, since \(a,b\) are increasing and initially positive, \(a^2+b^2\geq a(0)^2+b(0)^2>0\). A bounded \(q\) would then keep \(e^{-q}\) bounded below and force \(\dot q\) bounded below by a positive constant, a contradiction. Thus \(q\to\infty\). The invariant \(a^2-b^2\) then implies \((a-b)^2=a^2+b^2-2ab\) remains bounded while the norm diverges, so the normalized direction approaches \((1,1)/\sqrt{2}\), the maximum-margin direction.
A Radial Identity for Exponential-Loss Gradient Flow
For a finite sample, consider the empirical exponential loss \(H(\theta)=\frac{1}{n}\sum_{i=1}^n e^{-m_i(\theta)}\). Assume each margin is differentiable and homogeneous of degree \(L\), and consider the gradient-flow equation \(\dot\theta=-\nabla H(\theta)\). The next identity describes the radial component of this dynamics. It does not by itself establish convergence to a particular direction.
Proof. Differentiating \(m_i(c\theta)=c^L m_i(\theta)\) with respect to \(c\) at \(c=1\) gives Euler’s identity \(\theta^\mathsf{T}\nabla m_i(\theta)=L m_i(\theta)\). The gradient-flow equation is \(\dot\theta=\frac{1}{n}\sum_i e^{-m_i(\theta)}\nabla m_i(\theta)\). Taking its inner product with \(\theta\) and applying Euler’s identity yields the claimed derivative of the squared norm. Separately, the chain rule gives \(\frac{d}{dt}H(\theta(t))=\nabla H(\theta(t))^\mathsf{T}\dot\theta(t)=-\|\nabla H(\theta(t))\|_2^2\). Both identities follow directly under the stated differentiability assumptions. \(\square\)
If \(H(\theta(t))\to 0\), then each individual exponential term tends to zero, since the terms are nonnegative and the sample is finite. Thus every margin tends to \(+\infty\). The radial identity then has a positive right-hand side at sufficiently late times. But neither loss decrease nor norm growth determines which unit direction is selected.
Worked Example: Degree Three and the Unit-Sphere Margin
Consider \(f_{a,b,c}(x)=abcx\) and the single observation \(x=y=1\). Scaling all three parameters by \(s>0\) multiplies the output by \(s^3\), so the model is degree \(3\). Its margin is \(abc\). Under the constraint \(a^2+b^2+c^2=1\), a positive product is largest when the three squared coordinates are equal: by the arithmetic-geometric mean inequality, \(a^2b^2c^2\leq ((a^2+b^2+c^2)/3)^3=1/27\). Taking nonnegative cube roots shows \(abc\leq 1/(3\sqrt{3})\), with equality at \(a=b=c=1/\sqrt{3}\). This verifies the maximum explicitly.
The example illustrates why the degree matters: scaling the parameter norm by \(s\) scales the margin by \(s^3\), not by \(s\). The unit-sphere optimization removes that overall scale and isolates a directional question. It does not show that gradient flow in every degree-three model reaches this direction.
When a Limiting Direction Is Margin-Stationary
A useful result is conditional. It makes explicit the additional asymptotic information needed to infer a margin condition from gradient-based optimization. Write a trajectory as \(\theta(t)=r(t)u(t)\), with \(r(t)=\|\theta(t)\|_2\) and \(\|u(t)\|_2=1\). Define the normalized exponential weights \(\alpha_i(t)=e^{-m_i(\theta(t))}/\sum_j e^{-m_j(\theta(t))}\). Assume the margins are continuously differentiable near the limiting direction.
Proof. By homogeneity, \(m_i(\theta(t))=r(t)^L m_i(u(t))\). Let \(m_*=\min_j m_j(u_*)\). If \(m_i(u_*)>m_*\), choose an index \(j\) attaining \(m_*\). Convergence of the directions implies that for some \(\delta>0\), eventually \(m_i(u(t))-m_j(u(t))>\delta\). Therefore \(\alpha_i(t)\leq e^{-r(t)^L\delta}\), so \(\alpha_i^*=0\). The limiting weights are nonnegative and sum to one because each finite-time vector of weights has those properties. They can thus have positive weight only on indices attaining the minimum margin at \(u_*\).
The assumed angular condition, together with continuity of the gradients and convergence of the weights, gives \((I-u_*u_*^\mathsf{T})g=0\), where \(g=\sum_i\alpha_i^*\nabla m_i(u_*)\). Since \(\|u_*\|_2=1\), this means \(g=\lambda u_*\) for some scalar \(\lambda\). Euler’s identity gives \(u_*^\mathsf{T}g=\sum_i\alpha_i^* Lm_i(u_*)=Lm_*\), because the weights are supported on minimum-margin indices. Hence \(\lambda=Lm_*\), proving the displayed condition.
To see its optimization meaning, introduce a scalar \(s\) and write the unit-sphere problem as maximizing \(s\) subject to \(s\leq m_i(u)\) for every \(i\) and \(\|u\|_2=1\). The limiting weights are nonnegative multipliers for the active margin constraints, sum to one by stationarity in \(s\), and satisfy the required stationarity in \(u\) together with a multiplier for the sphere constraint. Thus they give KKT stationarity. KKT stationarity is a local first-order condition, not a proof of global optimality. \(\square\)
The angular condition is an extra hypothesis, not an automatic consequence of the norm diverging or the loss vanishing. For ReLU networks, differentiability can also fail at activation boundaries; applications must specify how the optimization handles those points and verify appropriate conditions. Results for particular network architectures may establish stronger conclusions, but those conclusions rely on their own assumptions.
Worked Example: A Dead ReLU Direction
Let \(f_w(x)=\operatorname{ReLU}(wx)=\max(0,wx)\), with one observation \(x=y=1\). The model is positively homogeneous of degree \(1\) in \(w\). The parameter \(w=1\) gives margin \(1\), so the sample is separable. At \(w=-1\), however, the output and margin are both zero, and the exponential loss is \(H(w)=e^{-\operatorname{ReLU}(w)}=1\).
For \(w<0\), \(\operatorname{ReLU}(w)=0\) and its derivative is zero. Consequently the derivative of \(H\) is zero there, and gradient flow initialized at \(w=-1\) remains at \(w=-1\). Its loss does not vanish, its norm does not diverge, and it does not approach a maximum-margin direction. This does not contradict the theorem: the trajectory fails its hypotheses. It demonstrates that positive homogeneity and the existence of a separator alone do not guarantee successful optimization.
What the Implicit-Bias Claim Does—and Does Not—Say
The useful interpretation is conditional: when optimization drives the loss down, the parameter norm grows, and the normalized dynamics settle into a direction with the required angular balance, the limiting direction has a margin-stationarity certificate. The certificate identifies a balance among gradients of the observations with the smallest margins. It can explain why parameter directions selected by optimization resemble solutions to explicit margin problems.
A common mistake is to reason, “The model is homogeneous and the training loss tends to zero; therefore gradient descent must find the global maximum-margin direction.” This conclusion does not follow. Vanishing loss gives growing training margins, but not convergence of the normalized parameters. Even if a direction converges and satisfies the theorem’s stationarity condition, it might be a nonglobal stationary point. Architecture, initialization, activation patterns, and the optimization trajectory can matter.
Check Your Understanding
Use the definitions and conditional result above to answer these questions.
- For a degree-\(L\) homogeneous model, how does scaling the parameter vector by \(c>0\) change each training margin?
- In the two-factor example, why does the inequality \(2ab\leq a^2+b^2\) identify a maximum margin on the unit circle?
- What does the radial identity imply about the norm once all margins are positive, and what does it fail to establish about the direction?
- Why must any limiting exponential weights in the theorem be supported on minimum-margin observations?
- Explain why KKT stationarity does not by itself establish that a direction globally maximizes the minimum margin.
- Which hypothesis of the conditional theorem fails for gradient flow initialized at \(w=-1\) in the dead-ReLU example?