Universal Approximation By One-Hidden-Layer Networks

Why a Universal Approximation Result Matters

A flexible model class is useful only if it can represent functions of interest, at least to a chosen accuracy. For one-hidden-layer networks, the basic question is whether a finite sum of activated affine functions can approximate an arbitrary continuous target on a compact input domain. A positive answer is powerful, but its precise meaning matters: it asserts that some finite network works for each target and tolerance, not that the network represents the target exactly or that its required width is small.

We study uniform approximation: the error is controlled at every point of the domain. The central result below applies to any compact $K\subset\mathbb{R}^d$ and a continuous sigmoidal activation. Its proof uses a duality argument: if the network functions were not dense, a nonzero signed measure would annihilate every hidden-unit function. The activation assumption rules out such a measure.

Network Class and Activation Assumption

Definition: Fix a positive integer $d$, a compact set $K\subset\mathbb{R}^d$, and an activation function $\sigma:\mathbb{R}\to\mathbb{R}$. A one-hidden-layer network with finite width $m$ has the form $$ g(x)=a_0+\sum_{j=1}^{m}a_j\sigma(w_j^\top x+b_j), \qquad x\in K, $$ where $a_0,a_j,b_j\in\mathbb{R}$ and $w_j\in\mathbb{R}^d$. The network class allows every finite width $m$ and every choice of these parameters. A continuous sigmoidal activation is a continuous function satisfying $\lim_{z\to-\infty}\sigma(z)=0$ and $\lim_{z\to+\infty}\sigma(z)=1$.

The logistic function $\sigma(z)=1/(1+e^{-z})$ is a continuous sigmoidal activation. These assumptions ensure that $\sigma$ is bounded and that, as its input is rescaled, it approaches a step function away from the step's threshold. That limiting behavior is the key to the proof.

Write $C(K)$ for the real-valued continuous functions on $K$, equipped with the uniform norm $\|f\|_\infty=\sup_{x\in K}|f(x)|$. Saying that the network class is dense in $C(K)$ means that every $f\in C(K)$ can be approximated to any strictly positive uniform error by a network with some finite width.

Worked Example: An Exactly Represented Constant

Let $K=[0,1]$, use the logistic activation, and take the target $f(x)=2$. Choose output bias $a_0=2$ and width $m=1$ with output coefficient $a_1=0$, $w_1=1$, and $b_1=0$. Then $$ g(x)=2+0\cdot\sigma(x)=2 $$ for every $x\in[0,1]$. Thus $\|f-g\|_\infty=0$. This example is exact because the network includes an output bias; density is a broader claim about targets that need not have such a simple representation.

Worked Example: A Linear Target With an Explicit Error Bound

Take $K=[0,1]$, $f(x)=x$, and the logistic activation. Since $\sigma(0)=1/2$, consider the width-one network $$ g(x)=40\sigma(0.1x)-20. $$ To bound its error, note that $\sigma'(z)=\sigma(z)(1-\sigma(z))$, so $0\leq\sigma'(z)\leq1/4$, and $\sigma''(z)=\sigma'(z)(1-2\sigma(z))$, so $|\sigma''(z)|\leq1/4$. Taylor's theorem around zero therefore gives $$ \left|\sigma(z)-\frac12-\frac{z}{4}\right|\leq\frac{z^2}{8}. $$ Put $z=0.1x$. The linear term in $g(x)$ is $40(0.1x/4)=x$, and the constant terms cancel. Hence $$ |g(x)-x|\leq40\frac{(0.1x)^2}{8}=0.05x^2\leq0.05 $$ for every $x\in[0,1]$. Therefore $\|g-f\|_\infty\leq0.05$. This calculation gives a concrete network and a verified error bound, rather than only an existence claim.

A Measure-Theoretic Lemma

The proof of density uses a fact about how sigmoidal ridge functions test measures. A finite signed Borel measure $\mu$ on compact $K$ is a difference of finite positive measures; its total variation $|\mu|$ is finite. The pushforward of $\mu$ under a map $T$ is the signed measure assigning $A$ the value $\mu(T^{-1}(A))$.

Lemma: Let $\sigma$ be continuous and sigmoidal, and let $\mu$ be a finite signed Borel measure on compact $K\subset\mathbb{R}^d$. If $$ \int_K \sigma(w^\top x+b)\,d\mu(x)=0 $$ for every $w\in\mathbb{R}^d$ and $b\in\mathbb{R}$, then $\mu=0$.

Proof. Fix $u\in\mathbb{R}^d$, and let $\nu$ be the pushforward of $\mu$ under $x\mapsto u^\top x$. This measure is supported on a compact interval, since $K$ is compact. For any real $c$ and $t>0$, apply the assumption with $w=tu$ and $b=-tc$. It gives $$ \int_{\mathbb{R}}\sigma(t(s-c))\,d\nu(s)=0. $$ The finite measure $|\nu|$ has at most countably many atoms. Thus, whenever $|\nu|(\{c\})=0$, the integrand converges $|\nu|$-almost everywhere, as $t\to\infty$, to the indicator of $(c,\infty)$. It is bounded uniformly in $t$, because $\sigma$ is bounded. Dominated convergence, applied to the positive and negative parts of $\nu$, yields $\nu((c,\infty))=0$.

The values of $c$ with $|\nu|(\{c\})=0$ are dense. For any two such values $a<b$, $$ \nu((a,b])=\nu((a,\infty))-\nu((b,\infty))=0. $$ These intervals, together with the fact that $\nu$ has compact support, determine $\nu$; equivalently, intervals with such endpoints generate the Borel sets and the finite signed measure is zero on them. Hence $\nu=0$. This holds for every $u$.

It follows that the Fourier transform of $\mu$ vanishes: for every $\xi\in\mathbb{R}^d$, $$ \int_K e^{i\xi^\top x}\,d\mu(x)=0, $$ because this integral is the Fourier transform of the pushforward under $x\mapsto \xi^\top x$, which is the zero measure. To see that this implies $\mu=0$, let $\phi_\varepsilon$ be the centered Gaussian density on $\mathbb{R}^d$ with covariance matrix $\varepsilon I_d$, where $\varepsilon>0$. Its Fourier representation is an integrable Gaussian-weighted integral. Fubini's theorem applies because $\mu$ is finite and the Gaussian weight is integrable. The vanishing Fourier transform therefore gives $$ \int_K \phi_\varepsilon(y-x)\,d\mu(x)=0 $$ for each $y\in\mathbb{R}^d$. For any continuous compactly supported $q:\mathbb{R}^d\to\mathbb{R}$, another application of Fubini gives $$ \int_K (q*\phi_\varepsilon)(x)\,d\mu(x)=0. $$ As $\varepsilon\downarrow0$, the Gaussian densities form an approximate identity, so $q*\phi_\varepsilon\to q$ uniformly. Finiteness of $|\mu|$ then implies $\int_K q\,d\mu=0$. Every continuous function on compact $K$ has a continuous extension to $\mathbb{R}^d$; multiplying that extension by a continuous compactly supported function equal to one on $K$ gives a compactly supported extension. Thus $\mu$ integrates every continuous function on $K$ to zero, which implies $\mu=0$ by uniqueness in the representation of continuous linear functionals on $C(K)$. $\square$

Universal Approximation Theorem

Theorem: Let $K\subset\mathbb{R}^d$ be compact, and let $\sigma:\mathbb{R}\to\mathbb{R}$ be continuous and sigmoidal. For every $f\in C(K)$ and every $\epsilon>0$, there is a finite-width one-hidden-layer network $g$ of the form defined above such that $$ \sup_{x\in K}|f(x)-g(x)|<\epsilon. $$

Proof. The network class is a linear subspace of $C(K)$: sums of two finite networks are finite networks, and scalar multiples remain in the class. Suppose its uniform closure were a proper closed subspace of $C(K)$. The Hahn–Banach theorem would then give a nonzero continuous linear functional on $C(K)$ that is zero on the network class. By the Riesz representation theorem for compact spaces, this functional has the form $h\mapsto\int_K h\,d\mu$ for a nonzero finite signed regular Borel measure $\mu$.

Every single hidden-unit function $x\mapsto\sigma(w^\top x+b)$ is in the network class (take output bias zero and one hidden unit with output coefficient one). Therefore the annihilating functional satisfies $$ \int_K \sigma(w^\top x+b)\,d\mu(x)=0 $$ for all $w,b$. The lemma implies $\mu=0$, contradicting that the functional is nonzero. The closure cannot be proper, so it is all of $C(K)$. This is precisely the stated uniform approximation property. $\square$

Applying the Density Claim

Worked Example: Approximating a Quadratic Target

Let $K=[0,1]$, $f(x)=x^2$, and $\epsilon=0.01$. The target is continuous on the compact interval. For the logistic activation, the theorem therefore guarantees a finite-width network $g$ satisfying $$ \sup_{x\in[0,1]}|x^2-g(x)|<0.01. $$ For example, this entails $|g(0.4)-0.16|<0.01$ and $|g(1)-1|<0.01$, as well as the same error bound at every other point in the interval. The theorem does not specify the width or parameters of this network; it establishes their existence for this target and tolerance.

Worked Example: A Two-Variable Target on a Square

Let $K=[0,1]^2$ and $f(x_1,x_2)=x_1x_2$. This function is continuous on $K$. For points $x,y\in K$, $$ |x_1x_2-y_1y_2| \leq |x_1||x_2-y_2|+|y_2||x_1-y_1| \leq |x_2-y_2|+|x_1-y_1|, $$ which also verifies continuity directly. Given tolerance $0.02$, the theorem supplies a finite network with inputs in $\mathbb{R}^2$ whose uniform error on the entire square is less than $0.02$. In particular, at $(0.5,0.5)$ its output is within $0.02$ of $0.25$, and the same uniform guarantee holds at the square's boundary and interior. The hidden units may use different affine combinations of the two coordinates.

These examples illustrate both the scope and the limits of the theorem. It covers continuous targets on compact domains and provides a uniform guarantee, including for multivariable functions. It does not give a width bound, a parameter-construction algorithm, or an approximation rate as a function of width.

Interpretation and a Common Pitfall

Universal approximation is a statement about the union of the network classes over all finite widths: each target and tolerance has some finite network. It does not say that one fixed width approximates every continuous function to arbitrary accuracy. Nor does density imply that the target is exactly a finite network, that parameters can be found easily, or that approximation is statistically reliable with finite data. Those are separate questions from the function-approximation result proved here.

Pitfall: reading density as exact representation. The claim $\|f-g\|_\infty<\epsilon$ for every $\epsilon>0$ allows a different finite network $g$ for each tolerance. It does not establish that one finite network has zero error.
Pitfall: replacing uniform error with pointwise error. Pointwise convergence at each input does not, by itself, give a single error bound that holds across the domain. The theorem explicitly controls the supremum over all of $K$.
Pitfall: dropping the activation assumptions. The proof uses a continuous bounded activation with distinct limits at positive and negative infinity. The result proved here is for continuous sigmoidal activations, not an assertion for every possible activation.

Key takeaway: For a continuous sigmoidal activation, finite one-hidden-layer networks are dense in $C(K)$ under the uniform norm for every compact $K\subset\mathbb{R}^d$. This establishes expressive sufficiency, not exact representation, efficient approximation, or a quantitative rate.

Check Your Understanding

Use the definitions and proof to answer the following questions.

What does uniform density guarantee for a fixed continuous target and a chosen positive tolerance?
Why does the proof consider a signed measure that annihilates every hidden-unit function?
In the measure lemma, where is dominated convergence used, and why is its use valid?
Does the theorem give a bound on the width needed to approximate $x_1x_2$ within a specified tolerance? Explain.
Why does approximation to every positive tolerance not imply exact representation by one finite network?