Piecewise-Smooth Targets And ReLU Approximation

Why Interfaces Need Separate Treatment

A function can be very regular inside each of several regions and still be difficult to approximate uniformly across their boundaries. For instance, a target may be constant on either side of a line but jump across it. The smooth-region approximation problem and the representation of the line are distinct tasks: an accurate approximation on each side does not, by itself, specify how a network should switch between the two.

ReLU networks can represent clipped linear transitions around interfaces. Away from those transition zones, they can select the appropriate regional network. This gives a useful error decomposition: control approximation error within each region, then bound the contribution from neighborhoods of the interfaces. The latter matters particularly for integral losses, because a thin neighborhood can have small volume even when the target jumps.

Definition: Let \(D=[-R,R]^d\), where \(R>0\). A finite collection of distinct hyperplanes \(H_\ell=\{x:a_\ell^\mathsf{T}x=b_\ell\}\), \(1\leq \ell\leq M\), partitions the complement of the hyperplanes into open cells. Here \(\|a_\ell\|_2=1\). A bounded function \(f:D\to\mathbb R\) is piecewise smooth with respect to this arrangement if its restriction to each open cell is the restriction of a smooth function on that cell. Values on the hyperplanes may be specified separately. In the approximation result below, the needed regional property is that each cell admits a network approximation with a stated uniform error; smoothness is one setting in which such regional approximations are obtained.

ReLU Gates for Regions

Write \(\sigma(t)=\max\{t,0\}\). The clipped ramp

\operatorname{clip}(t,0,1)=\sigma(t)-\sigma(t-1)

is exactly representable by a ReLU network. For a hyperplane \(H_\ell\), a signed distance is \(a_\ell^\mathsf{T}x-b_\ell\), since the normal vector has unit length. Given a width \(\delta>0\), define the gate for the side with sign \(s\in\{-1,1\}\) by

q_{\ell,s}(x)=\operatorname{clip}\!\left(\frac{s(a_\ell^\mathsf{T}x-b_\ell)}{\delta},0,1\right).

This gate is zero on the wrong side of the hyperplane and one at signed distance at least \(\delta\) on the designated side. Between those locations it changes linearly. Each cell corresponds to a feasible pattern of signs across the \(M\) hyperplanes. Multiplying the corresponding side-gates makes a cell gate: it is one where all its signed distances are at least \(\delta\), and zero as soon as any one of its constraints is violated.

Products are not generally exact ReLU operations. We use the standard multiplication approximation fact: on any fixed bounded square, a ReLU network can approximate \((u,v)\mapsto uv\) uniformly to any prescribed positive accuracy. Repeated application gives an arbitrarily accurate network for any fixed finite product on a bounded input set. This fact allows the ideal products in the construction below to be implemented with a controlled uniform error.

A Regional Approximation Bound

Suppose the arrangement has \(J\) open cells \(C_1,\ldots,C_J\). For each cell, let \(g_j\) be a ReLU network that approximates \(f\) there. To keep the eventual error bound finite even where a regional network is used outside its own cell, clip each \(g_j\) to the interval \([-B-\varepsilon,B+\varepsilon]\). If \(|f|\leq B\) and the original regional error is at most \(\varepsilon\), this clipping does not increase that error on the cell.

Theorem: Let \(D=[-R,R]^d\), and suppose \(|f(x)|\leq B\) on \(D\). Let \(M\) unit-normal hyperplanes induce \(J\) open cells, and suppose for each cell there is a bounded ReLU network \(g_j\) with \(|g_j(x)|\leq B+\varepsilon\) on \(D\) and \[ |g_j(x)-f(x)|\leq\varepsilon\qquad(x\in C_j). \] For any \(\delta>0\) and \(\eta>0\), a ReLU network \(\widehat f\) can be constructed so that, with \[ A_\delta=\{x\in D:\ |a_\ell^\mathsf{T}x-b_\ell|<\delta\text{ for at least one }\ell\}, \] \[ \|\widehat f-f\|_{L^p(D)}^p \leq |D|(\varepsilon+\eta)^p+ |A_\delta|\bigl(J(B+\varepsilon)+B+\eta\bigr)^p,\qquad 1\leq p<\infty. \] Moreover, \[ |A_\delta|\leq \min\!\left\{|D|,\;2M\delta\sqrt d\,(2R)^{d-1}\right\}. \] Here \(L^p(D)\) uses Lebesgue measure, and values assigned to \(f\) on the hyperplanes do not affect the bound.

To prove the result, let the signs of cell \(C_j\) be \(s_{j\ell}\in\{-1,1\}\), and form its ideal gate \[ G_j(x)=\prod_{\ell=1}^M q_{\ell,s_{j\ell}}(x). \] Consider the ideal glued function \(F(x)=\sum_{j=1}^J G_j(x)g_j(x)\). Every gate lies in \([0,1]\). Therefore \(|F(x)|\leq J(B+\varepsilon)\) on \(D\). The function \(F\) is an intermediate expression; multiplication networks will implement it to uniform error at most \(\eta\).

Identify the good set.
If \(x\notin A_\delta\), it is at distance at least \(\delta\) from every hyperplane. Its own cell gate is one. Every other cell has at least one opposite sign, whose gate is zero at \(x\). Thus exactly one gate is one and all others are zero.

Control error near interfaces.
On \(A_\delta\), the global bound on \(F\), the bound on \(f\), and the implementation error give \(|\widehat f-f|\leq J(B+\varepsilon)+B+\eta\).

Integrate the two bounds.
The good set has measure at most \(|D|\), and the exceptional set has measure \(|A_\delta|\). Integrating the \(p\)th powers of the two pointwise bounds gives the stated \(L^p\) inequality.

It remains to verify the measure estimate and that the ideal expression can be implemented. For each hyperplane choose a coordinate \(k\) with \(|a_{\ell k}|\geq1/\sqrt d\); such a coordinate exists because \(\sum_k a_{\ell k}^2=1\). Holding the other coordinates fixed, the condition \(|a_\ell^\mathsf{T}x-b_\ell|<\delta\) restricts \(x_k\) to an interval of length at most \(2\delta/|a_{\ell k}|\leq2\delta\sqrt d\). Integrating over the other \(d-1\) coordinates in the box shows that this one strip has volume at most \(2\delta\sqrt d(2R)^{d-1}\). The union bound over \(M\) strips gives the second term in the minimum; the whole set also has volume at most \(|D|\).

Finally, each \(q_{\ell,s}\) is a ReLU network and has values in \([0,1]\); each \(g_j\) is bounded by assumption. Approximate the finitely many products defining the \(G_j\), and then the products \(G_jg_j\), by multiplication networks on bounded input ranges. Because there are finitely many terms, choosing the accuracy of each product sufficiently small makes the total uniform error at most \(\eta\). This proves both claims of the theorem. \(\square\)

Worked Constructions

Worked Example: A Jump at One Interface

Take \(D=[-1,1]\), interface \(x=0\), and \(f(x)=-1\) for \(x\leq0\), \(f(x)=1\) for \(x>0\). The two regional approximants are the constants \(g_-(x)=-1\) on the left cell and \(g_+(x)=1\) on the right cell. Their errors are zero on their respective open cells; no claim is made that either regional network approximates \(f\) on the closed cell containing the shared boundary.

For \(0<\delta\leq1\), let \(v(x)=\operatorname{clip}(x/\delta,0,1)\). The glued network is \(F(x)=(1-v(x))g_-(x)+v(x)g_+(x)=2v(x)-1\). For \(x\leq0\), \(F(x)=-1\); for \(x\geq\delta\), \(F(x)=1\). On \(0<x<\delta\), the pointwise error is \(2(1-x/\delta)\). Thus the error is supported on a set of length \(\delta\), and \[ \int_{-1}^{1}|F(x)-f(x)|^2\,dx =4\int_0^\delta(1-x/\delta)^2\,dx =4\delta\int_0^1(1-u)^2\,du =\frac{4\delta}{3}. \] The full interface strip \((-\delta,\delta)\) has length \(2\delta\), but the actual error support for this gate and boundary convention is only its positive half. The calculation distinguishes the strip used in a general bound from the exact support in this example.

Worked Example: Two Crossing Interfaces in a Square

Let \(D=[-1,1]^2\) and take the interfaces \(x_1=0\) and \(x_2=0\). For \(\delta=0.05\), the vertical strip \(|x_1|<0.05\) has width \(0.1\) and area \(0.1\cdot2=0.2\). The horizontal strip also has area \(0.2\). Their intersection is the central square of side \(0.1\), with area \(0.01\). Inclusion-exclusion therefore gives \[ |A_\delta|=0.2+0.2-0.01=0.39,\qquad |D\setminus A_\delta|=4-0.39=3.61. \] The union-bound estimate is \(0.2+0.2=0.4\), a valid upper bound but not the exact area.

For example, assign the constant \(c_{s_1,s_2}\in[-1,1]\) to each open quadrant, with arbitrary values on the axes. The four regional networks can be those four constants, so their regional errors are zero. Products of the two relevant side-gates form each quadrant gate. Outside the two strips exactly one gate is one and the other three are zero, so the glued expression equals the appropriate quadrant constant there. Any nonzero discrepancy is confined to the computed exceptional set of area \(0.39\); its size within that set depends on the four assigned constants and the gate transitions.

Worked Example: Smooth Approximation on One Side of a Jump

On \(D=[-1,1]\), define \(f(x)=x^2\) for \(-1<x<0\) and \(f(x)=2+x\) for \(0<x<1\), with any bounded values at \(-1\), \(0\), and \(1\). Use the grid \(-1,-0.5,0\) to approximate \(x^2\) on the left. The linear interpolant through \((-1,1)\), \((-0.5,0.25)\), and \((0,0)\) is \[ g_-(x)= \begin{cases} -1.5x-0.5,&-1\leq x\leq-0.5,\\ -0.5x,&-0.5\leq x\leq0. \end{cases} \] It is exactly representable by a ReLU network: its slope changes from \(-1.5\) to \(-0.5\) at \(-0.5\), a slope change of \(1\), so one hinge term \(\sigma(x+0.5)\) suffices in addition to the first affine segment. On the right use \(g_+(x)=2+x\), which is exact.

On an interval of length \(h=0.5\), the linear interpolant to \(x^2\) has error \(e(x)=(x-a)(b-x)\), where \(a,b\) are the interval endpoints. This follows because the interpolant minus \(x^2\) is a quadratic with roots at \(a,b\) and leading coefficient \(-1\). Its maximum is at the midpoint, where \(e=h^2/4=0.25/4=1/16\). Thus the left regional error is at most \(1/16\); the right error is zero. A gate of width \(\delta\) around zero selects these networks away from the interface strip. The theorem then separates the \(1/16\) regional error from the additional contribution confined to that strip.

Choosing an Error Criterion

The result explains why a small interface neighborhood can be acceptable for \(L^p\) approximation while still preventing a small global supremum error. If there are \(M\) interfaces and the region networks have error \(\varepsilon\), the strip-volume bound is proportional to \(M\delta\), capped by the domain volume. Reducing \(\delta\) shrinks this exceptional region. In practice, however, narrower ramps have steeper slopes, since each gate scales a signed distance by \(1/\delta\). Interface geometry and network parameter constraints therefore matter when converting the existence bound into a size or stability guarantee.

This separation is also useful in approximation arguments: one can first obtain regional networks from smooth-function approximation results, then add gates, and finally choose \(\delta\) according to the loss being controlled. The theorem does not assert a particular network-size rate for the regional approximants; that rate depends on the smoothness assumptions and the approximation result used to construct them.

Common Mistake: Treating Cell Boundaries as Ordinary Interior Points

A jump target need not have a single continuous trace on an interface. It is therefore incorrect to define regional errors as suprema over closed cells and then assert that distinct regional constants have zero error on both closed cells. At a shared boundary, the target has one assigned value, and at least one regional constant may disagree with it. The theorem instead assumes error on open cells and isolates interface neighborhoods; hyperplanes have Lebesgue measure zero, so their assigned values do not affect finite-\(p\) integrals. For uniform approximation, by contrast, boundary values must be handled explicitly, and a discontinuous target generally cannot be uniformly approximated arbitrarily well by continuous ReLU networks.

A second related pitfall is to add the areas of strips and call the sum their exact union area. Strips can overlap. The sum is a safe union-bound upper bound, but an exact geometric calculation must subtract overlaps, as in the crossing-strip example.

Key takeaway: For targets that are smooth within finitely many regions, ReLU approximation has two distinct components: regional approximation error and the error introduced near interfaces. ReLU gates select regional networks away from those interfaces, while strip-volume bounds quantify the exceptional part for integral losses.

Check Your Understanding

Use the definitions and construction above to answer the following.

Why does a regional error assumption on open cells avoid the boundary contradiction that can arise for a jump target?
For a single unit-normal hyperplane in \([-R,R]^d\), explain how slicing in a coordinate with \(|a_k|\geq1/\sqrt d\) gives a strip-volume bound.
In the square example, what is the exact area of the union of the two strips, and what is the union-bound upper bound?
For the one-dimensional jump example, calculate the squared \(L^2\) error when \(\delta=0.3\).
Why does an \(L^p\) bound based on a small interface neighborhood not imply a small uniform error across a jump?