Deep Network Approximation Of Analytic Functions

Why Analyticity Changes the Approximation Rate

A function that is analytic near an interval has more structure than one that is merely continuous or finitely differentiable. The key consequence for approximation is that its polynomial approximants can converge geometrically: increasing the degree by a fixed amount can reduce the error by a fixed multiplicative factor. A deep network can represent these polynomials, so the analytic bound becomes a network approximation bound once the activation and representation cost are specified.

The complex domain matters. Smoothness on the real interval alone does not give the geometric bound below. We require an analytic extension to a complex neighborhood whose size is measured by a Bernstein ellipse.

From Analyticity to a Geometric Error Bound

The following result turns a bound on the complex ellipse into a uniform approximation bound on the real interval \([-1,1]\). The degree-\(m\) Chebyshev truncation is \(p_m(x)=\sum_{k=0}^{m}a_kT_k(x)\), where \(a_k\) are the Chebyshev expansion coefficients of \(f\).

Proof. Set \(g(w)=f((w+w^{-1})/2)\). The map \(w\mapsto(w+w^{-1})/2\) sends the circle \(|w|=r\) to an ellipse with semiaxes \((r+r^{-1})/2\) and \((r-r^{-1})/2\). Thus \(g\) is analytic in an annulus containing \(\rho^{-1}\le |w|\le\rho\). Its Laurent expansion has coefficients \(c_k\), and Cauchy's integral formula on \(|w|=\rho\) gives, for \(k\ge1\),

On the unit circle, \(w=e^{it}\) and \((w+w^{-1})/2=\cos t\). The symmetry \(g(w)=g(w^{-1})\) makes the Laurent coefficients at \(k\) and \(-k\) equal. Consequently the Chebyshev coefficient for \(T_k\), \(k\ge1\), is \(a_k=2c_k\), so \(|a_k|\le2M_\rho\rho^{-k}\). Since \(|T_k(x)|\le1\) for \(x\in[-1,1]\), the tail is bounded by

The series converges uniformly on \([-1,1]\), as the displayed geometric majorant is summable; hence the tail bound holds uniformly. \(\square\)

Turning the Polynomial Bound into a Network Bound

An approximation rate in polynomial degree is not automatically a rate in network size. To make that connection precise, consider networks with the quadratic activation \(\sigma(t)=t^2\), affine operations, and skip connections. Count depth as the number of successive activation stages and size as the number of activation units. The following construction is specific to this activation; it is not a claim about every network architecture.

Derivation. First, multiplication can be performed exactly using two square activations:

Thus a constant number of square units and affine combinations implements one multiplication. Identity transmission through activation stages is also exact, since \(t=((t+1)^2-(t-1)^2)/4\). Constants and affine additions are handled by affine maps. Write the polynomial in monomial form as \(q(x)=q_0+x(q_1+x(q_2+\cdots+xq_m))\). Horner evaluation uses \(m\) multiplication steps and affine additions. Replacing each multiplication by the identity above requires only a fixed number of units and stages; carrying \(x\) and the current value through the stages has constant overhead per step. The total depth and size are therefore both bounded by a constant times \(m\), and the output equals \(q(x)\) exactly. \(\square\)

Combining this lemma with the theorem yields a network guarantee: for any \(\varepsilon>0\), choose \(m\) so that \(2M_\rho\rho^{-m}/(\rho-1)\le\varepsilon\), then represent \(p_m\) by the stated network. In particular, it suffices to take

when the right-hand side is positive, rounding up to an integer. The resulting size and depth are \(O(\log(1/\varepsilon))\) with \(\rho\) and \(M_\rho\) fixed. This statement counts units, not the magnitude or finite-precision encoding of weights; it also depends on the specified quadratic activation.

Products of Analytic Factors

Products often arise when a target has several analytic components. Approximate each factor on the same interval and multiply the approximating polynomials. The error is controlled by the factor errors and their uniform bounds; the degree of the product is at most the sum of the factor degrees.

Proof. Subtract the two products by replacing factors one at a time:

The theorem gives \(|f_j-p_j|\le\varepsilon_j\) on the interval. Hence \(|p_i|\le|f_i|+\varepsilon_i\le M_i+\varepsilon_i\), while \(|f_i|\le M_i\). Apply these bounds term by term and use the triangle inequality to obtain the stated result. The product polynomial has degree at most \(\sum_jm_j\); the network lemma represents it with size and depth \(O(\sum_jm_j)\). \(\square\)

What the Rate Does—and Does Not—Say

The geometric rate comes from analytic continuation: a larger admissible ellipse (larger \(\rho\)) gives faster decay \(\rho^{-m}\), though the corresponding bound \(M_\rho\) may also increase. One chooses an ellipse on which both analyticity and a useful magnitude bound hold. The network conclusion then follows through an explicit representation of the approximating polynomial.

A common mistake is to infer exponential accuracy in the number of units for any deep network merely because the target is analytic. The reasoning omits two necessary links: an analytic-to-polynomial approximation bound and a cost bound for representing the polynomial in the chosen architecture. Here the first link is the Chebyshev theorem, and the second relies on quadratic activation and exact multiplication. Change the activation, constrain the weights, or change how network size is counted, and the representation claim must be reconsidered. Analyticity alone is not an architecture-independent network theorem.

A second pitfall is to use real-variable smoothness as if it supplied the same ellipse bound. The proof uses a complex contour and a finite supremum \(M_\rho\); it requires an analytic extension around the entire ellipse. Singularities close to the interval restrict how large \(\rho\) can be and therefore limit the available geometric rate.