Why Activation Smoothness Matters
A network’s activation affects the shapes its units can produce and the constructions available for approximating a target. ReLU units have a corner, so one-dimensional ReLU networks naturally represent piecewise-linear functions. Sigmoid and other smooth activations have derivatives of every order, which can be used to build smooth features by combining units with nearby parameters. These differences matter when the target itself is smooth, but they do not by themselves determine an approximation rate.
To make the comparison precise, consider approximation on a compact interval \(I=[a,b]\), measured in the uniform norm \(\|g\|_{\infty,I}=\sup_{x\in I}|g(x)|\). A one-hidden-layer network with activation \(\sigma\) has the form \(g(x)=\beta_0+\sum_{j=1}^{m}\beta_j\sigma(w_jx+c_j)\). The number of units, parameter sizes, and error criterion all affect approximation performance. The discussion below separates what follows from the shape of an activation from what requires additional assumptions.
ReLU, \(\rho(t)=\max\{t,0\}\), is continuous but not differentiable at zero. The logistic sigmoid, \(\sigma(t)=1/(1+e^{-t})\), is infinitely differentiable and nonpolynomial. The hyperbolic tangent is another smooth nonpolynomial activation. The main contrast is not that one class can approximate every smooth target and the other cannot: ReLU approximation results already give useful rates for regular targets. Rather, each activation offers different convenient representations, and a comparison is meaningful only after specifying the target class, norm, architecture, and parameter constraints.
ReLU and Smooth Targets: Linear Interpolation
For a twice-differentiable function on an interval, linear interpolation is a direct ReLU construction. Let \(a=x_0<x_1<\cdots<x_m=b\) be equally spaced knots with spacing \(h=(b-a)/m\), and let \(I_m f\) be the function that agrees with \(f\) at each knot and is linear between successive knots.
To prove the error bound, fix one subinterval \([x_i,x_{i+1}]\), and write \(L\) for the line joining \((x_i,f(x_i))\) and \((x_{i+1},f(x_{i+1}))\). The interpolation remainder formula gives, for each \(x\) in this subinterval, \(f(x)-L(x)=f''(\xi)(x-x_i)(x-x_{i+1})/2\) for some \(\xi\in(x_i,x_{i+1})\) (with the endpoint errors equal to zero). Therefore \[ |f(x)-L(x)|\le \frac{M}{2}(x-x_i)(x_{i+1}-x)\le \frac{Mh^2}{8}, \] because the product is at most \(h^2/4\). Taking the maximum over all subintervals proves the claimed uniform bound.
For the representation, let \(s_i=(f(x_{i+1})-f(x_i))/h\) be the slope on interval \(i\). Then, on \([a,b]\), \(I_mf(x)=f(a)+s_0(x-a)+\sum_{i=1}^{m-1}(s_i-s_{i-1})\rho(x-x_i)\). Before \(x_1\), the hinge terms vanish and the slope is \(s_0\). Each time \(x\) passes a knot \(x_i\), the slope changes by \(s_i-s_{i-1}\), so on every interval the formula has exactly the slope and value of the interpolant. This proves exact representability. If an implementation requires an affine term to be built from ReLU units, use \(x=\rho(x)-\rho(-x)\). \(\square\)
Worked Example: ReLU Approximation of a Quadratic
Approximate \(f(x)=x^2\) on \([0,1]\) using \(m=4\) equal intervals. The knots are \(0,1/4,1/2,3/4,1\), so \(h=1/4\), and \(f''(x)=2\), giving \(M=2\).
The theorem gives \(\|f-I_4f\|_{\infty,[0,1]}\le 2(1/4)^2/8=1/64\). This bound is attained at the midpoint of each interval: on \([0,1/4]\), for instance, the interpolant is \(I_4f(x)=x/4\). The error is \(x^2-x/4\), whose absolute value is largest at \(x=1/8\), where it equals \(1/64\). Thus the actual uniform error is exactly \(1/64\).
The slopes on the four intervals are \(1/4,3/4,5/4,7/4\). The slope changes at the three interior knots are each \(1/2\). Hence \(I_4f(x)=\frac14x+\frac12\rho(x-\frac14)+\frac12\rho(x-\frac12)+\frac12\rho(x-\frac34)\) on \([0,1]\), a ReLU representation with three hinge units and an affine term.
Smooth Activations and Finite-Difference Features
A smooth activation can produce polynomial-shaped features in a limit by combining its values at nearby inputs. This is a construction, not a claim that a finite network automatically achieves a particular target error with controlled weights.
For the proof, take \(\varepsilon>0\) small enough that every \(c+j\varepsilon x\), for \(0\le j\le q\) and \(|x|\le R\), lies in the neighborhood where the derivatives exist. Taylor’s theorem gives \(\sigma(c+u)=\sum_{r=0}^{q}\sigma^{(r)}(c)u^r/r!+r_{q+1}(u)\), where \(|r_{q+1}(u)|\le C|u|^{q+1}\) for these \(u\), with \(C\) finite because \(\sigma^{(q+1)}\) is bounded on a compact neighborhood.
In the weighted sum, the terms of degree \(r<q\) cancel, while \(\sum_{j=0}^{q}(-1)^{q-j}{q\choose j}j^q=q!\). These identities follow because the \(q\)-th forward difference annihilates polynomials of degree below \(q\) and gives \(q!\) for the monomial \(j^q\). The degree-\(q\) term in the sum is therefore \(\sigma^{(q)}(c)\varepsilon^q x^q\). After division, it equals \(x^q\). The remaining terms have absolute value at most \(C\varepsilon^{q+1}R^{q+1}\sum_{j=0}^{q}{q\choose j}j^{q+1}\). Dividing by \(|\sigma^{(q)}(c)|\varepsilon^q\) gives a bound proportional to \(\varepsilon\), uniformly for \(|x|\le R\). This tends to zero and proves the result. \(\square\)
If \(\sigma\) is smooth and nonpolynomial on the real line, then for every \(q\ge1\) there is some \(c\) with \(\sigma^{(q)}(c)\ne0\). Otherwise \(\sigma^{(q)}\) would vanish everywhere, forcing \(\sigma\) to be a polynomial of degree at most \(q-1\). This explains why smooth nonpolynomial activations offer finite-difference constructions for features of arbitrarily high degree. It does not, by itself, bound the number of units or the sizes of the coefficients needed at a particular finite accuracy.
Worked Example: A Sigmoid Construction Approaching \(x^2\)
For the logistic sigmoid, \(\sigma'(t)=\sigma(t)(1-\sigma(t))\) and \(\sigma''(t)=\sigma(t)(1-\sigma(t))(1-2\sigma(t))\). At \(c=1\), \(\sigma(1)\approx0.7311\), so \(\sigma''(1)\approx-0.0909\), which is nonzero. The theorem applies with \(q=2\).
For \(\varepsilon=0.1\), the three-unit combination is \(G_{0.1}(x)=\frac{\sigma(1+0.1x)-2\sigma(1)+\sigma(1-0.1x)}{0.01\,\sigma''(1)}\). Its numerator is a centered second difference. Taylor expansion about 1 shows that its leading term is \(0.01\,\sigma''(1)x^2\); the constant and linear terms cancel. The remaining error after division is bounded, uniformly on \(|x|\le1\), by \(\frac{0.1^2}{12|\sigma''(1)|}\sup_{t\in[0.9,1.1]}|\sigma^{(4)}(t)|\). This bound follows by Taylor expansion through the cubic term, with fourth-order remainders for the two shifted values. It is finite, and the theorem ensures the error tends to zero as \(\varepsilon\) decreases. The factor \(1/\varepsilon^2\) also shows why small approximation error need not come with small coefficients.
Worked Example: Smooth Output Does Not Ensure Stable Derivatives
Consider \(g_k(x)=\sigma(kx)\) on \([-1,1]\), where \(k>0\). Each \(g_k\) is infinitely differentiable, and \(g_k'(x)=k\sigma'(kx)\). At zero, \(\sigma'(0)=1/4\), so \(g_k'(0)=k/4\). For \(k=20\), the derivative at zero is \(5\), even though the output always lies between 0 and 1.
As \(k\) increases, the transition becomes steeper. Thus smoothness of each network output does not imply a uniform bound on its derivatives across networks. If the approximation goal includes derivative accuracy, or requires controlled derivatives, output smoothness alone is not enough: one must also control parameter magnitudes or establish derivative-error bounds directly.
Comparing the Constructions
| Activation | Useful structure | What the structure does not guarantee |
|---|---|---|
| ReLU | Exact piecewise-linear interpolation; a \(C^2\) target has interpolation error at most \(Mh^2/8\). | A smooth target is not represented exactly by a finite piecewise-linear interpolant in general. |
| Sigmoid | Smooth units; finite differences can approach powers when the required derivative is nonzero. | The finite-difference coefficients can grow as the perturbation shrinks; smoothness alone gives no efficient finite-size rate. |
| Other smooth nonpolynomial activations | When the relevant derivatives are nonzero, the same local construction applies; smooth nonpolynomial functions have nonzero derivatives of every order somewhere. | The location and size of those derivatives, parameter constraints, and architecture still affect quantitative performance. |
For continuous targets on compact domains, standard universal-approximation results give density for ReLU and for common nonpolynomial activations such as sigmoid. Density says that any positive error tolerance can be met by some network; it does not specify the required width, depth, parameter magnitudes, or rate as the tolerance shrinks. The earlier material on approximation rates and analytic targets supplies more quantitative settings under additional assumptions. Here, the finite-difference result identifies one mechanism by which activation derivatives can help, while the interpolation result provides a direct rate for a particular ReLU construction.
Common Pitfall
A common mistake is to reason, “The target is smooth and the activation has many derivatives, so the smooth activation must approximate it faster.” That conclusion does not follow. The finite-difference construction divides by \(\varepsilon^q\), so its coefficients become large as \(\varepsilon\) decreases; and the theorem gives convergence for a particular feature, not a complete network-size bound for an arbitrary target. Conversely, ReLU’s lack of differentiability does not prevent accurate approximation of smooth targets: the interpolation error above decreases quadratically with knot spacing when the target has a bounded second derivative.
Check Your Understanding
Answer using the assumptions and constructions stated above.
- For a \(C^2\) target with \(\sup|f''|\le 3\) on an interval, what uniform error bound does equal-spacing ReLU interpolation give when the spacing is \(h=0.2\)?
- In the finite-difference theorem, why is the condition \(\sigma^{(q)}(c)\ne0\) needed?
- What happens to the finite-difference construction’s coefficient scale as \(\varepsilon\) decreases for a feature of degree \(q\)?
- Why does the smoothness of \(\sigma(kx)\) for every finite \(k\) not imply a uniform bound on its derivative at zero?
- What additional information would be needed to turn a density statement into a quantitative approximation-rate claim?