One-Dimensional ReLU Splines And Exact Representation

From Slope Changes to ReLU Units

The ReLU activation is \(\sigma(u)=\max\{u,0\}\). As a function of its input, it is zero to the left of zero and linear to the right. Shifting its input gives a hinge: \(\sigma(x-\tau)\) is zero for \(x\leq \tau\) and has slope one for \(x>\tau\). Multiplying a hinge changes the size and direction of its slope change.

This makes one-dimensional piecewise-linear functions especially transparent. Between successive knots, such a function has a constant slope. At each knot, its slope changes by a definite amount. A shifted ReLU contributes exactly one such change, so the function can be built by adding one hinge for each knot, together with an affine baseline.

Definition: A function \(f:\mathbb{R}\to\mathbb{R}\) is continuous piecewise linear with finitely many knots if there are finitely many ordered points \(\tau_1<\cdots<\tau_K\) such that \(f\) is affine on each interval cut out by these points and is continuous on \(\mathbb{R}\). The \(\tau_j\) are knots; a knot at which the slopes on the two sides differ is a genuine slope-change knot.

For a function with knots \(\tau_1,\ldots,\tau_K\), let \(s_0\) be its slope to the left of \(\tau_1\), and \(s_j\) its slope between \(\tau_j\) and \(\tau_{j+1}\), with \(s_K\) the slope to the right of \(\tau_K\). The change at knot \(\tau_j\) is \(s_j-s_{j-1}\). The exact representation is

f(x)=a+s_0x+\sum_{j=1}^{K}(s_j-s_{j-1})(x-\tau_j)_+, \qquad (u)_+=\max\{u,0\},

where \(a+s_0x\) is the affine expression for \(f\) to the left of the first knot. A zero slope change contributes no hinge and can be omitted. This formula is an identity, not an approximation: each hinge adjusts the slope after its knot while leaving the function unchanged before that knot.

Worked Example: A Single Hinge

Consider \(f(x)=3+2x-4(x-1)_+\). Its knot is \(\tau_1=1\). To the left, the hinge is zero, so \(f(x)=3+2x\) and \(s_0=2\). To the right, \((x-1)_+=x-1\), giving

f(x)=3+2x-4(x-1)=7-2x,\qquad x\geq 1.

Thus \(s_1=-2\), and the slope change is \(s_1-s_0=-2-2=-4\), exactly the coefficient of the hinge. At the knot, the left expression gives \(3+2(1)=5\); the right expression gives \(7-2(1)=5\). The function is continuous, with slope \(2\) before the knot and slope \(-2\) after it.

Exact Representation Theorem

Theorem: Let \(f:\mathbb{R}\to\mathbb{R}\) be continuous and affine on the intervals determined by finitely many ordered knots \(\tau_1<\cdots<\tau_K\). Let \(a+s_0x\) be its affine expression on the leftmost interval, and let \(s_j\) be its slope on the interval immediately to the right of \(\tau_j\). Then \(f(x)=a+s_0x+\sum_{j=1}^K(s_j-s_{j-1})(x-\tau_j)_+\) for every \(x\in\mathbb{R}\). A knot with zero slope change can be removed from the sum.

The formula follows by checking the slope on each interval and using continuity at the knots. In particular, this argument requires no differentiability at a knot: the slopes are the constant slopes on the intervals to its left and right.

Start to the left of the knots.
For \(x<\tau_1\), every \((x-\tau_j)_+\) is zero, so the right-hand side equals \(a+s_0x=f(x)\).

Check the slope on each interval.
On \((\tau_\ell,\tau_{\ell+1})\), exactly the hinges with \(j\leq\ell\) are active. The right-hand side has slope \(s_0+\sum_{j=1}^{\ell}(s_j-s_{j-1})=s_\ell\), by telescoping.

Use continuity to match values.
Each hinge is continuous, so the proposed expression is continuous at every knot. It agrees with \(f\) on the leftmost interval and has the same slope on each successive interval; continuity therefore makes the values agree at each successive interval as well.

These steps prove the identity on every interval, including the two unbounded tails. If there are no knots, the assertion reduces to the affine formula \(f(x)=a+s_0x\). A knot whose adjacent slopes are equal has coefficient zero and does not affect the representation. \(\square\)

Writing the Formula as a ReLU Network

A one-hidden-layer ReLU network with scalar input and output has the form

g(x)=\beta_0+\sum_{r=1}^{m}\alpha_r\sigma(w_rx+b_r),

where \(m\) is the number of hidden units and the displayed parameters are real. The hinge \((x-\tau_j)_+\) is the ReLU unit \(\sigma(x-\tau_j)\). The affine term can also be represented using ReLU units, because \(x=\sigma(x)-\sigma(-x)\). Consequently, the theorem gives the network representation

f(x)=a+s_0\sigma(x)-s_0\sigma(-x) +\sum_{j=1}^{K}(s_j-s_{j-1})\sigma(x-\tau_j).

Result: Every continuous piecewise-linear function on \(\mathbb{R}\) with finitely many knots has an exact one-hidden-layer ReLU representation. The formula above uses at most \(K+2\) hidden units: two for the affine term and one for each knot. If the network permits a direct affine term outside the hidden layer, the \(K\) hinge units suffice.

The count is an upper bound, not a claim that every representation needs that many units. For instance, the affine part needs no hidden units if an output-layer affine term is allowed. Also, knots with no slope change can be left out. The meaningful count for the hinge portion is the number of genuine slope changes.

Worked Example: A Bounded Ramp

Construct a ramp that is zero up to \(x=-1\), rises linearly to height \(2\) at \(x=2\), and stays at height \(2\) thereafter. The interval from \(-1\) to \(2\) has length \(3\), so the desired slope there is \(2/3\). The slope changes from \(0\) to \(2/3\) at \(-1\), then from \(2/3\) to \(0\) at \(2\). Thus

r(x)=\frac{2}{3}(x+1)_+-\frac{2}{3}(x-2)_+.

For \(x\leq-1\), both hinges are zero, so \(r(x)=0\). For \(-1<x<2\), only the first hinge is active, so \(r(x)=\frac{2}{3}(x+1)\), which rises from \(0\) toward \(\frac{2}{3}(3)=2\). For \(x\geq2\), both hinges are active, and \(r(x)=\frac{2}{3}(x+1)-\frac{2}{3}(x-2)=\frac{2}{3}(3)=2\). The negative coefficient at \(2\) cancels the rising slope, producing the flat plateau.

Exact Interpolation at Knots

The same construction gives a direct formula for linear interpolation. Take ordered input locations \(x_0<x_1<\cdots<x_m\) and assigned values \(y_0,\ldots,y_m\). There is a unique continuous function that passes through the specified points and is linear on each interval between consecutive locations. Extend it linearly beyond the two endpoint locations. Its interval slopes are

s_j=\frac{y_{j+1}-y_j}{x_{j+1}-x_j}, \qquad j=0,\ldots,m-1.

The leftmost affine expression is \(y_0+s_0(x-x_0)\). At each interior location \(x_j\), for \(1\leq j\leq m-1\), the slope changes by \(s_j-s_{j-1}\). The interpolant is therefore

I(x)=y_0+s_0(x-x_0)+ \sum_{j=1}^{m-1}(s_j-s_{j-1})(x-x_j)_+.

Worked Example: Interpolating Four Points

Interpolate the points \((0,1)\), \((1,3)\), \((3,2)\), and \((4,4)\), with linear extensions outside \([0,4]\). The three interval slopes are

s_0=\frac{3-1}{1-0}=2,\qquad s_1=\frac{2-3}{3-1}=-\frac12,\qquad s_2=\frac{4-2}{4-3}=2.

The slope changes at the interior knots are \(s_1-s_0=-\frac12-2=-\frac52\) at \(x=1\), and \(s_2-s_1=2-(-\frac12)=\frac52\) at \(x=3\). The leftmost affine expression is \(1+2x\), so the interpolant is

I(x)=1+2x-\frac52(x-1)_++\frac52(x-3)_+.

Check each interval. For \(x\leq1\), \(I(x)=1+2x\), which gives \(I(0)=1\) and \(I(1)=3\). For \(1\leq x\leq3\), \(I(x)=1+2x-\frac52(x-1)=\frac72-\frac12x\), giving \(I(1)=3\) and \(I(3)=2\). For \(x\geq3\), \(I(x)=\frac72-\frac12x+\frac52(x-3)=2x-4\), giving \(I(3)=2\) and \(I(4)=4\). The adjacent formulas agree at both knots, as required.

Why the Representation Is Useful

The slope-change formula identifies the role of each network unit: its location is a knot, and its output coefficient is the change in slope there. This gives a precise description of what one-dimensional ReLU networks can express with finitely many units. Conversely, each unit \(\sigma(wx+b)\) with \(w\neq0\) is continuous and piecewise linear, with its only possible slope change at \(x=-b/w\); a unit with \(w=0\) is constant. A finite sum of such units is therefore also continuous and piecewise linear, with finitely many possible knots. In one dimension, finite ReLU networks and finite continuous piecewise-linear functions are two descriptions of the same basic structure.

Exact representation of this class should not be confused with approximation of a general target function. A function that is not piecewise linear with finitely many knots will not, in general, equal one of these finite formulas. Questions about how well networks can approximate smoother or broader function classes concern approximation error and rates, rather than the exact identity derived here.

Common Mistake

Using the slope after a knot as the hinge coefficient. A hinge changes the slope by its coefficient; it does not set the new slope to that coefficient. If the slope changes from \(2\) to \(-2\), the hinge coefficient is \(-2-2=-4\), not \(-2\).
Forgetting continuity when matching interval formulas. Matching slopes alone does not fix vertical offsets. In the theorem, the affine baseline fixes the value on the first interval, and continuity of the hinges propagates that value across each knot.
Calling interpolation an approximation when the target is the interpolant itself. The ReLU formula passes exactly through the specified points and is exactly linear between them. Whether it approximates another underlying function well is a separate question.

Key takeaway: A one-dimensional continuous piecewise-linear function is determined by its leftmost affine expression, its knot locations, and its slope changes. Each slope change becomes the coefficient of one shifted ReLU hinge, yielding an exact representation.

Check Your Understanding

Use the hinge formula and verify values or slopes directly where appropriate.

A function has leftmost expression \(1-x\), a knot at \(x=2\), and slope \(3\) to the right of that knot. What is its hinge coefficient and its ReLU formula?
Write a ReLU formula for a ramp that is zero for \(x\leq0\), rises with slope \(1/2\) on \((0,4)\), and is constant thereafter. What is its plateau height?
For a piecewise-linear function whose successive slopes are \(1\), \(4\), and \(2\), what are the hinge coefficients at its two knots?
Why does checking the slope on each interval, by itself, not establish equality of two piecewise-linear functions?
Can a finite one-hidden-layer ReLU network represent exactly a continuous function with infinitely many genuine slope-change knots? Explain using the structure of its units.