Balancing Neural Width, Depth, and Estimation Error

Why Width and Depth Must Be Balanced

A neural network can become more expressive by increasing its width, its depth, or both. Greater expressiveness can reduce approximation error, but a larger or more complex class can also make estimation from a finite sample harder. The useful architecture is therefore not necessarily the largest one that can be trained: it is a choice that makes the two errors small together.

The balance depends on two ingredients. The first is an approximation profile: a bound describing how the best risk in a width-\(W\), depth-\(D\) class improves as \(W\) and \(D\) increase. The second is an estimation bound for fitting within that class. This tutorial derives the architecture and risk rate that follow from a particular, transparent pair of bounds. The rate is conditional on those bounds; it is not a universal rate for all neural networks or data distributions.

Setup and Assumptions

Let \(R(f)\) denote the population risk of predictor \(f\), and let \(R^*=\inf_f R(f)\) be the target risk over the prediction problem under consideration. For each pair of positive integers \(W,D\), let \(\mathcal F_{W,D}\) be a neural-network class with width \(W\) and depth \(D\), using a fixed convention for counting layers. Write \(f^*_{W,D}\) for a risk minimizer in this class, when one exists. If the infimum is not attained, the same statements can be made with an arbitrarily accurate approximant.

We also need an estimation bound. Norm-based generalization results control the gap between the population risk and the empirical risk, or the excess risk of a fitted predictor, using a norm-dependent complexity measure. The specific norm and its proof depend on the network class and loss. For this rate calculation, assume those results give the following bound for a fitted predictor \(\hat f_{W,D}\):

simultaneously over the candidate architectures being considered, on an event of probability at least \(1-\eta\). Here \(A_3>0\), \(n\) is sample size, and \(\delta_n\geq 0\) allows for optimization error or another stated fitting tolerance. This form corresponds, for example, to a norm-based complexity envelope that grows at most proportionally to \(\sqrt{WD}\), with other constants absorbed into \(A_3\). The bound is an assumption about the considered classes and fitting procedure, not a consequence of width and depth alone.

Adding the approximation and estimation bounds gives the architecture-specific risk guarantee

The first two terms reward increasing width and depth. The third penalizes their joint growth. The task is to choose \(W\) and \(D\) so that none of these three terms dominates unnecessarily.

Worked Examples of the Balance

General Balancing Result

The preceding calculations extend to any positive approximation exponents. The result below gives both a rate-achieving scaling and a matching lower bound for the three-term proxy. That lower bound concerns the proxy itself; it does not claim a minimax lower bound for the underlying statistical problem.

Proof. For the upper bound, first take the real-valued scales \(W_0=n^{r/p}\) and \(D_0=n^{r/q}\). Their approximation terms satisfy $$ W_0^{-p}=n^{-r},\qquad D_0^{-q}=n^{-r}. $$ The estimation term has order $$ \sqrt{\frac{W_0D_0}{n}} =n^{(r/p+r/q-1)/2}. $$ Since \(r(2+1/p+1/q)=1\), we have \((r/p+r/q-1)/2=-r\). Thus all three terms are constant multiples of \(n^{-r}\). Rounding \(W_0,D_0\) up to integers preserves this order: for \(x\geq1\), \(\lceil x\rceil\leq2x\), so rounding changes each term by at most a constant factor.

For the lower bound, write \(x=A_1W^{-p}\), \(y=A_2D^{-q}\), and \(z=A_3\sqrt{WD/n}\). Define weights $$ u=\frac{r}{p},\qquad v=\frac{r}{q},\qquad w=2r. $$ They are positive and sum to one, because \(r(1/p+1/q+2)=1\). The weighted arithmetic-geometric mean inequality gives \(\Phi_n(W,D)=x+y+z\geq x^u y^v z^w\). In this product, the powers of \(W\) cancel: they are \(-pu+w/2=-r+r=0\). The powers of \(D\) cancel in the same way. The remaining power of \(n\) is \(-w/2=-r\). Hence $$ \Phi_n(W,D)\geq A_1^u A_2^v A_3^w n^{-r} $$ for every \(W,D\geq1\). Taking the infimum proves the lower bound and completes the proof. \(\square\)

If the fitting bound includes \(\delta_n\), the resulting risk guarantee is of order \(n^{-r}+\delta_n\). In particular, the optimization error must be no larger than the target rate to retain an \(n^{-r}\) guarantee. If a norm-based bound instead has an additional logarithmic factor, that factor must remain in the balance; it cannot be discarded when stating a finite-sample guarantee.

How to Use the Rate Responsibly

The calculation is useful because it turns two qualitative claims—“larger networks approximate better” and “more complex classes are harder to estimate”—into a concrete sample-size-dependent architecture. Its conclusion is only as strong as its inputs. A valid use should check that the approximation profile applies at the selected width and depth, that the norm-based estimation bound applies to the fitted class, and that the guarantee holds for the architecture-selection procedure being used. A bound proved separately for one fixed architecture does not automatically hold simultaneously over many architectures.

A common mistake is to set \(W=D\) before examining the approximation profile. That can be reasonable when the two exponents and constants are comparable, but it is not generally justified. In the asymmetric example, depth has a fourth-power approximation term while width has only a first-power term; the balanced choices consequently have different powers of \(n\). The correction is to balance all three terms using the given exponents, rather than to assume a symmetric architecture.

Another pitfall is to treat the norm-based estimation term as a universal parameter-count penalty. It is not. The norm constraints, data and loss assumptions, constants, and any logarithmic or confidence terms matter. If the available bound has a different dependence on \(W\) and \(D\), use that dependence in place of \(\sqrt{WD/n}\) and redo the balance. The main lesson is the method: state the approximation and estimation bounds precisely, then choose width and depth so their contributions are of comparable order.