Network Architecture And Approximation Budgets

Why Architecture Is an Approximation Budget

An approximation statement about a neural network is meaningful only after the allowed networks have been specified. The same target function may be compared with networks having one or many hidden layers, narrow or wide layers, different activations, or different limits on the number of parameters. These choices determine the approximation class over which the best error is measured.

This tutorial makes those constraints explicit. It focuses on architecture and parameter accounting, not on a claim that one architecture approximates a particular function well. In particular, increasing a budget enlarges the available class, but by itself does not guarantee that the best error decreases strictly.

Feedforward Networks and Their Architecture

Consider a network with input \(x\in\mathbb{R}^{d_0}\), \(L\) hidden layers, and output in \(\mathbb{R}^{d_{L+1}}\). The hidden widths are \(d_1,\ldots,d_L\). Fix an activation function \(\sigma:\mathbb{R}\to\mathbb{R}\), applied separately to each coordinate in each hidden layer. The output layer is affine in the final hidden representation.

Definition: A feedforward network with hidden widths \(d_1,\ldots,d_L\) is specified by matrices \(A_l\in\mathbb{R}^{d_l\times d_{l-1}}\) and bias vectors \(b_l\in\mathbb{R}^{d_l}\), for \(l=1,\ldots,L+1\). Its hidden representations and output are

h_0=x,\qquad h_l=\sigma(A_lh_{l-1}+b_l)\quad (1\le l\le L),\qquad N(x)=A_{L+1}h_L+b_{L+1}.

Here \(d_0\) is the input dimension and \(d_{L+1}\) is the output dimension. The depth convention in this tutorial is that \(L\) counts hidden layers; the final affine output map is not counted as a hidden layer. The width of hidden layer \(l\) is \(d_l\).

For scalar regression, \(d_{L+1}=1\). If \(L=0\), there are no hidden representations: the network is simply \(N(x)=A_1x+b_1\), an affine function. Stating this convention matters because some accounts count all affine maps when reporting depth, while others count only hidden layers.

The activation is also part of the specification. For example, the ReLU activation is \(\sigma(t)=\max\{t,0\}\). Choosing a different fixed activation changes the function class, even when all widths and parameter slots are unchanged. Unless explicitly stated otherwise, the activation is fixed rather than learned.

Counting Parameters Correctly

An affine map from \(d_{l-1}\) inputs to \(d_l\) outputs has \(d_ld_{l-1}\) weight entries and \(d_l\) bias entries. Thus it has \(d_l(d_{l-1}+1)\) parameter slots. This count includes every location available for a weight or bias, even if a particular fitted value at that location is zero.

Definition: For fixed widths \(d_0,\ldots,d_{L+1}\), the number of parameter slots, including all biases, is

P_{\mathrm{slots}}=\sum_{l=1}^{L+1}d_l(d_{l-1}+1).

This formula also covers \(L=0\): with \(d_1\) output coordinates, the affine network has \(d_1(d_0+1)\) slots. In the scalar-output case this is \(d_0+1\).

Worked Example: Counting a Two-Hidden-Layer Network

Take input dimension \(d_0=2\), hidden widths \(d_1=3\) and \(d_2=2\), and scalar output \(d_3=1\). Count all weights and biases by affine map.

The first map sends \(\mathbb{R}^2\) to \(\mathbb{R}^3\), so it has \(3\cdot2=6\) weights and 3 biases, or \(3(2+1)=9\) slots.

The second map sends \(\mathbb{R}^3\) to \(\mathbb{R}^2\), so it has \(2\cdot3=6\) weights and 2 biases, or \(2(3+1)=8\) slots.

The output map sends \(\mathbb{R}^2\) to \(\mathbb{R}\), so it has 2 weights and 1 bias, or \(1(2+1)=3\) slots. The total is therefore \(9+8+3=20\) slots. Directly from the formula, \(3(2+1)+2(3+1)+1(2+1)=9+8+3=20\).

A slot count is not the same as a count of nonzero parameter values. For a fixed network, its number of nonzero parameters counts only entries of the matrices and bias vectors whose values are not zero. That number can be smaller than \(P_{\mathrm{slots}}\). Some approximation results impose a budget on nonzero parameters, while others constrain the architecture's total slots; these are different restrictions and should not be conflated.

Worked Example: Slots and Nonzero Parameters in a ReLU Network

Let the input be scalar, use ReLU in both hidden layers, and take widths \(d_1=2\), \(d_2=1\), with scalar output. Define \(h_1=(\max\{x,0\},\max\{-x,0\})\), \(h_2=\max\{h_{1,1}+h_{1,2}-1,0\}\), and \(N(x)=2h_2\).

The first layer has two weights and two biases, hence \(2(1+1)=4\) slots. Its two weights are \(1\) and \(-1\), while both biases are zero, so it has 2 nonzero entries.

The second layer has two weights and one bias, hence \(1(2+1)=3\) slots. Its weights are \(1,1\) and its bias is \(-1\), giving 3 nonzero entries. The output layer has one weight and one bias, hence \(1(1+1)=2\) slots; its weight is \(2\) and its bias is zero, giving 1 nonzero entry. Thus there are \(4+3+2=9\) slots but \(2+3+1=6\) nonzero parameters.

Since \(\max\{x,0\}+\max\{-x,0\}=|x|\), this network computes \(N(x)=2\max\{|x|-1,0\}\). For instance, \(N(0)=0\), \(N(1)=0\), \(N(2)=2\), and \(N(-2)=2\). The calculation illustrates why a statement such as “the network has six parameters” is ambiguous unless it says whether it means slots or nonzero values.

Approximation Classes Under Budgets

Fix an input domain \(\Omega\), an activation \(\sigma\), and a norm or error measure appropriate to the approximation problem. For a fixed architecture, let \(\mathcal{N}(d_1,\ldots,d_L;\sigma)\) denote the set of all functions computed by networks with those widths and arbitrary real weights and biases. The architecture specifies where parameters may occur; the values of those parameters vary across the class.

Definition: For a target function \(f\) and a function norm \(\|\cdot\|\), the best approximation error over a network class \(\mathcal{F}\) is

E(f;\mathcal{F})=\inf_{g\in\mathcal{F}}\|f-g\|.

A depth, width, or parameter budget defines \(\mathcal{F}\) by restricting which architectures or parameter values are allowed. The infimum need not be attained.

For example, a class with at most \(L_{\max}\) hidden layers and width at most \(W\) consists of networks with any number \(k\) of hidden layers satisfying \(0\le k\le L_{\max}\), each width at most \(W\), and the chosen activation. One may further restrict the total number of slots to at most \(P\), or instead restrict the number of nonzero parameter values to at most \(S\). The former constrains architecture size; the latter constrains parameter sparsity within an architecture.

These budgets answer different questions. Depth limits the number of successive nonlinear transformations. Width limits the number of coordinates available at each hidden layer. A slot budget bounds the total number of locations for weights and biases. An activation choice determines the nonlinear operation at each hidden coordinate. None of these descriptors alone specifies an approximation error: the target, domain, norm, and allowed parameter values also matter.

Worked Example: Comparing Two Explicit Budgets

For scalar input and scalar output, consider a one-hidden-layer network of width 4 and a two-hidden-layer network with widths 2 and 2. Use the same fixed activation in both.

For width 4, the input-to-hidden map has \(4(1+1)=8\) slots, and the output map has \(1(4+1)=5\). The total is \(8+5=13\).

For widths \((2,2)\), the first map has \(2(1+1)=4\) slots, the second has \(2(2+1)=6\), and the output map has \(1(2+1)=3\). The total is \(4+6+3=13\).

Thus these architectures have the same slot count but different depths and widths. A budget of 13 slots does not determine a unique architecture or imply that one of these two classes contains the other. Their function classes depend on the activation and on how the available parameters are arranged across layers.

Two Consequences of Class Inclusion

The basic conclusions about budgets follow from set inclusion, not from a special property of neural networks. The first result records the effect of relaxing a constraint.

Proposition: If \(\mathcal{F}_1\subseteq\mathcal{F}_2\), then for every target \(f\) and norm \(\|\cdot\|\), \(E(f;\mathcal{F}_2)\le E(f;\mathcal{F}_1)\).

Proof. Every \(g\in\mathcal{F}_1\) also belongs to \(\mathcal{F}_2\). Hence the set of errors \(\{\|f-g\|:g\in\mathcal{F}_2\}\) contains the set \(\{\|f-g\|:g\in\mathcal{F}_1\}\). The infimum over the larger set cannot exceed the infimum over the smaller set. Therefore \(E(f;\mathcal{F}_2)\le E(f;\mathcal{F}_1)\). \(\square\)

For the network classes just defined, increasing a depth cap, width cap, or slot cap while keeping the other specifications fixed yields a class containing the original class: every network previously allowed is still allowed. The proposition therefore shows that the best approximation error cannot increase. It does not show strict improvement; the larger class may add only functions that do not improve the infimum for the particular target and norm.

Proposition: If \(\mathcal{F}_{L,W,P}\) consists of networks with at most \(L\) hidden layers, each of width at most \(W\), and at most \(P\) parameter slots, then \(\mathcal{F}_{L,W,P}\subseteq\mathcal{F}_{L',W',P'}\) whenever \(L\le L'\), \(W\le W'\), and \(P\le P'\), with the input dimension, output dimension, and activation held fixed.

Proof. Take any network in \(\mathcal{F}_{L,W,P}\). It has some number \(k\le L\) of hidden layers, each of width at most \(W\), and at most \(P\) slots. Since \(L\le L'\), the same \(k\) satisfies \(k\le L'\). Since \(W\le W'\), every hidden layer still satisfies the width constraint \(W'\). Its architecture and parameter values have not changed, so its slot count remains at most \(P\), and hence at most \(P'\). It satisfies all three constraints defining \(\mathcal{F}_{L',W',P'}\), and therefore belongs to that class. Since the network was arbitrary, the set inclusion follows. \(\square\)

Common Mistakes and Practical Interpretation

Reversing the bias count. A map from \(d_{l-1}\) coordinates to \(d_l\) coordinates has \(d_l\) biases, one per output coordinate. Its count is \(d_l(d_{l-1}+1)\), not \((d_l+1)d_{l-1}\).
Counting only weights. Unless biases are explicitly excluded, they are parameters and must be included in the slot count.
Using a hidden-layer formula for an affine model. With no hidden layer and scalar output, there are \(d_0\) weights and one bias, or \(d_0+1\) slots.
Treating equal parameter counts as equal architectures. Different depth and width configurations can have the same total number of slots while defining different function classes.
Assuming a larger budget must improve the achieved error. It guarantees only that the best error cannot worsen when the class contains the smaller one. Strict improvement requires additional facts about the target and the classes.

Key takeaway: State the activation, hidden-layer convention, widths, output dimension, and parameter-counting rule before interpreting an approximation claim. Architecture budgets define the candidate function class; approximation error is then measured by optimizing over that class.

Check Your Understanding

Work through these questions by writing the layer-by-layer parameter counts explicitly.

A scalar-output network has input dimension 3 and hidden widths 4 and 2. How many slots does each affine map have, and what is the total?
For a scalar-input, scalar-output affine network with no hidden layer, how many weights and biases are there?
Explain the difference between the number of parameter slots and the number of nonzero parameters.
Two architecture classes satisfy \(\mathcal{F}_1\subseteq\mathcal{F}_2\). What inequality must hold between their best approximation errors, and why?
Can two networks with equal slot counts have different numbers of hidden layers? Give an explicit pair of scalar-input, scalar-output architectures with equal counts.