Choosing Network Architecture For An Approximation Target

Architecture Is a Response to the Target

A network architecture should be chosen to match the function being approximated, not simply to maximize depth or width. A smooth target, a composition of simpler functions, a sum of separate coordinate effects, and a rapidly oscillating target can require very different resources. In each case, the useful question is: what approximation guarantee does a candidate architecture provide, and what does that guarantee cost?

The discussion below treats approximation on a specified domain, measured in uniform error. This makes the design calculations explicit; other norms require their own approximation certificates. The earlier material on smooth activations, ReLU approximation, compositional classes, sparse representations, and oscillatory functions supplies the relevant approximation tools. Here the emphasis is on using such tools to choose an architecture for a target and tolerance.

A useful design starts from a proven bound of the form “this architecture has error at most \(B\).” The tolerance \(\varepsilon\) then determines a sufficient width, depth, or number of active parameters by solving \(B\le\varepsilon\). The bound is a guarantee, not necessarily the smallest achievable error: the target may have extra structure that the generic certificate does not use.

Smooth Targets: Use a Certificate to Set the Width

For a smooth one-dimensional target, a simple piecewise-linear approximation already illustrates how a quantitative error bound determines network width. Let \(f\) be twice continuously differentiable on \([0,1]\), and write \(M=\sup_{x\in[0,1]}|f''(x)|\). Divide the interval into \(N\) equal subintervals and linearly interpolate \(f\) at the grid points. The resulting spline can be represented by a ReLU network with one hidden layer and a number of units proportional to \(N\).

To prove the bound, consider one subinterval \([a,b]\), of length \(h=1/N\), and a point \(x\in(a,b)\). The interpolation remainder formula gives \(f(x)-I_N f(x)=f''(\xi)(x-a)(x-b)/2\) for some \(\xi\in(a,b)\). Since \((x-a)(b-x)\le h^2/4\), it follows that \[ |f(x)-I_N f(x)|\le \frac{M}{2}\frac{h^2}{4}=\frac{M}{8N^2}. \] At the grid points the error is zero. Taking the supremum over every subinterval proves the result.

This interpolant is a continuous piecewise-linear function with knots \(t_j=j/N\). Any such spline can be written as an affine function plus a sum of hinge functions \(c_j(x-t_j)_+\), where \((u)_+=\max(u,0)\). Because \((x-t_j)_+\) is a ReLU unit, the spline is represented by a one-hidden-layer ReLU network. With an affine skip connection, it needs at most \(N-1\) knot units; without one, an additional unit can represent the affine term on \([0,1]\), giving a width of at most \(N\). Thus the error bound translates directly into a width choice.

The example also shows why the target’s regularity matters. A bound based on a second derivative becomes more useful when that derivative is controlled. If the target is less smooth or has sharp changes, a smoothness-based certificate may be weak; if it is smoother, an architecture and approximation result that exploit higher-order smoothness may be preferable. The activation should be selected together with the available theorem: ReLU is natural for piecewise-linear constructions, while smooth activations can be advantageous when the target and the desired approximation scheme call for smooth approximants.

Compositions: Allocate Error Through the Layers

When a target is a composition, approximating each constituent to the same tolerance is not automatically the right design. Error in an inner function is transformed by the outer function. A Lipschitz constant gives a direct way to account for that effect.

For each \(x\in D\), add and subtract \(g(\hat h(x))\), then apply the triangle inequality: \[ |g(h(x))-\hat g(\hat h(x))| \le |g(h(x))-g(\hat h(x))|+|g(\hat h(x))-\hat g(\hat h(x))| \le L\delta_h+\delta_g. \] The assumed set contains both arguments needed for the Lipschitz and outer approximation bounds. Taking the supremum proves the claim.

This is the basic depth design principle for compositions: preserve the stages of the target when those stages are simpler than the fully expanded function. The depth is chosen to reflect the composition, while widths within stages are set by their local approximation requirements. Error budgets need not be equal across stages; if one stage has a large amplification factor or is more expensive to approximate, it may deserve a different allocation.

Separable Targets: Build Parallel Branches

A target that is a sum of functions of separate coordinates has no interaction terms between those coordinates. A fully connected architecture can represent it, but may spend parameters on interactions that the target does not contain. Parallel subnetworks followed by a sum preserve the known structure. For \(f(x)=\sum_{j=1}^r a_j u_j(x_j)\), if branch \(j\) approximates \(u_j\) with uniform error \(\delta_j\), then \[ |f(x)-\hat f(x)|\le\sum_{j=1}^r|a_j|\delta_j. \] This follows by writing the difference as \(\sum_j a_j(u_j(x_j)-\hat u_j(x_j))\) and applying the triangle inequality. It gives an explicit error budget for the branches.

For more terms or unequal branch costs, the same inequality suggests unequal error allocations: choose \(\delta_j\) so that the weighted sum \(\sum_j|a_j|\delta_j\) is within tolerance while expensive branches are not over-approximated. The design benefit comes from the function’s structure, not merely from choosing a narrow network.

Oscillatory Targets: Depth Can Replace Exponential Width

Rapid oscillation is a setting in which depth can make a dramatic difference. The following exact example is the tent map \(T:[0,1]\to[0,1]\), defined by \(T(x)=2x\) for \(0\le x\le1/2\) and \(T(x)=2(1-x)\) for \(1/2\le x\le1\). It has the ReLU representation \[ T(x)=2(x)_+-4(x-\tfrac12)_++2(x-1)_+,\qquad x\in[0,1]. \] Its iterates \(T^m\) have \(2^m\) linear intervals, with slope magnitude \(2^m\) on each interval. Each new iteration doubles the number of intervals.

A one-hidden-layer ReLU network with \(W\) units is an affine function plus \(W\) hinge functions. Each hinge can add at most one breakpoint, so the output has at most \(W+1\) linear intervals. Since \(T^m\) changes slope at each of its \(2^m-1\) interior grid points, exact representation by such a network requires \(W\ge2^m-1\). In contrast, composing \(T\) with itself \(m\) times gives a deep representation with width 3 at each stage: at each stage the three units compute \((z)_+\), \((z-\tfrac12)_+\), and \((z-1)_+\) from the preceding scalar output \(z\); their affine combination is \(T(z)\). Thus depth \(m\), width 3, and a number of nonzero parameters proportional to \(m\) suffice.

A Practical Architecture-Selection Sequence

The examples suggest a disciplined sequence for turning a target and tolerance into a design. The approximation theorem or construction comes first; parameter counting comes after the error bound has been checked.

Common Pitfall: Choosing by Size Alone

A common mistake is to assume that a larger network, or a deeper network, must be better for every target. That reasoning confuses representational capacity with an approximation guarantee. A wide shallow network can be effective for a spline-like target, while a compositional or oscillatory target may admit a much more economical deep representation. Conversely, depth alone does not establish a useful approximation rate: the activation, connectivity, target assumptions, and error norm all matter.

A second mistake is to turn a sufficient upper bound into a claim of necessity. The quadratic example shows that 16 intervals certify error below \(0.001\); it does not show that fewer intervals cannot work. A lower bound requires a separate argument, such as the breakpoint count for exact representation of \(T^6\). Keep these statements distinct when comparing architectures.