Approximating Nonlinear Relationships With Step Functions

From a Curved Pattern to a Step Function

A response may change gradually as a predictor increases. For example, expected claim cost might rise faster at older building ages than at younger ages. A regression tree does not fit a continuously bending curve directly: it divides predictor values into regions and assigns one constant prediction to each region. The resulting function is a step function. Adding regions can let the fitted levels follow the broad shape of a curved pattern more closely.

This is an approximation, not a claim that the underlying relationship is truly constant within each interval. The fitted prediction stays fixed until a threshold is crossed, then changes to the level for the next region. Consequently, even a smooth response pattern can produce a prediction with abrupt jumps.

Definition: For training observations $(x_i,y_i)$, a piecewise-constant regression-tree model partitions the predictor values into disjoint regions $R_1,\ldots,R_M$. It predicts a constant $c_m$ whenever $x\in R_m$. With squared-error fitting, the fitted level in region $R_m$ is the mean response of the training observations in that region: $$ \hat{f}(x)=\hat{c}_m=\frac{1}{n_m}\sum_{i:x_i\in R_m}y_i,\qquad x\in R_m, $$ where $n_m$ is the number of observations in $R_m$. The training sum of squared errors is $\sum_{m=1}^{M}\sum_{i:x_i\in R_m}(y_i-\hat{c}_m)^2$.

For this tutorial, consider a single numeric predictor and a continuous response fitted by squared error. The regions are intervals created by threshold splits. For example, a threshold $t$ can separate $x\le t$ from $x>t$. The calculations of region means and training error require no distributional assumption. To interpret performance on future observations, however, the training data should reasonably represent the population of interest; a low training error alone does not establish that future predictions will be accurate.

Why the Region Prediction Is a Mean

Within one fixed region, every observation receives the same prediction $c$. The best such constant is the value that minimizes the region’s sum of squared errors. The short derivation below shows why that value is the sample mean.

Write the region’s error.
For responses $y_1,\ldots,y_n$ in the region, the squared error for a proposed constant $c$ is $Q(c)=\sum_{i=1}^{n}(y_i-c)^2$.

Differentiate with respect to the constant.
$Q'(c)=-2\sum_{i=1}^{n}(y_i-c)$. Setting this derivative equal to zero gives $nc=\sum_{i=1}^{n}y_i$.

Check that the stationary point minimizes error.
$Q''(c)=2n>0$, so $Q(c)$ is strictly convex and the stationary point is its unique minimum. Thus $c=\bar{y}$, the region’s sample mean.

The mean is therefore not an arbitrary choice of representative level: it is the constant prediction that gives the smallest squared error for the observations in that region. This result concerns squared-error fitting; a different loss function can lead to a different optimal constant.

Worked Example: Three Regions Following a Curved Pattern

A hypothetical portfolio study records a continuous risk score $x$ and a response $y$, measured in thousands of dollars. The seven observations are $(0,0),(1,1),(2,4),(3,9),(4,16),(5,25),(6,36)$. Use the fixed regions $x\le1$, $1<x\le4$, and $x>4$. Find the fitted level in each region and the total training squared error. Compare it with the one-region fit.

Method. Compute the response mean within each region, then add the squared deviations from those means. No stochastic distribution is needed for these training-data calculations.

First region: its responses are $0,1$, so its fitted level is $(0+1)/2=0.5$. Its squared error is $(0-0.5)^2+(1-0.5)^2=0.25+0.25=0.5$.

Second region: its responses are $4,9,16$, so its fitted level is $(4+9+16)/3=29/3$. Its squared error is \[ (4-29/3)^2+(9-29/3)^2+(16-29/3)^2 =289/9+4/9+361/9=654/9=218/3. \]

Third region: its responses are $25,36$, so its fitted level is $(25+36)/2=30.5$. Its squared error is $(25-30.5)^2+(36-30.5)^2=30.25+30.25=60.5$.

The three-region total is $0.5+218/3+60.5=401/3$, or $133.6667$ when rounded to four decimal places. For one region, the mean of all seven responses is $91/7=13$. The one-region squared error is $\sum_i(y_i-13)^2=1092$. Thus, these three intervals fit the observed curvature more closely than a single constant does. The calculation is checked by noting that each region’s squared error is nonnegative and their sum is $133.6667$.

Common mistake to avoid: do not use the overall mean $13$ as the prediction in each of the three regions. Each region has its own mean, found from only the responses assigned to that region.

What Adding Regions Changes

A threshold split replaces one region and its single fitted level with smaller regions that can have different fitted levels. If the split is added while retaining the original partition’s other regions, training squared error cannot increase. The old constant remains an available candidate for both new regions; optimizing each new region separately can only match or improve on that candidate’s error. In symbols, the set of predictions allowed by the finer partition includes the predictions allowed by the coarser partition.

This guarantee is about the training observations and the squared-error objective. It does not say that the finer step function is closer to the unknown response pattern everywhere, or that it will predict new observations better. A new level is estimated from fewer observations in its region, and its value may reflect chance variation as well as the underlying pattern. Choosing or limiting tree size is a separate question.

Feature	Fewer regions	More regions
Prediction shape	Fewer, wider constant intervals	More, narrower constant intervals
Training squared error	Cannot be lower than that of a nested finer partition	Cannot be higher than that of its nested coarser partition
Approximation of a curved pattern	May miss changes in the response level	Can track changes more closely, but remains discontinuous
Interpretation	Each fitted level summarizes a broader range	Each fitted level summarizes a smaller range

Worked Example: Tracking a Quadratic Pattern With More Regions

A hypothetical model-development exercise uses the responses $y=x^2$ at predictor values $x=0,1,\ldots,7$. Compare the training squared error for (a) one region, (b) two regions split at $x=3.5$, and (c) four regions split into the pairs $\{0,1\},\{2,3\},\{4,5\},\{6,7\}$. The two-region partition is nested in the four-region partition.

One region: the responses are $0,1,4,9,16,25,36,49$. Their sum is $140$, so the fitted level is $140/8=17.5$. The sum of squared responses is $4676$. Therefore the squared error is \[ \sum_i(y_i-17.5)^2=\sum_i y_i^2-8(17.5)^2=4676-2450=2226. \]

Two regions: the left responses are $0,1,4,9$, with mean $14/4=3.5$. Their squared error is $12.25+6.25+0.25+30.25=49$. The right responses are $16,25,36,49$, with mean $126/4=31.5$. Their squared error is $240.25+42.25+20.25+306.25=609$. The total is $49+609=658$.

Four regions: the four pairwise means are $(0+1)/2=0.5$, $(4+9)/2=6.5$, $(16+25)/2=20.5$, and $(36+49)/2=42.5$. The corresponding errors are $0.5$, $12.5$, $40.5$, and $84.5$. For instance, the last pair contributes $(36-42.5)^2+(49-42.5)^2=42.25+42.25=84.5$. The total is $0.5+12.5+40.5+84.5=138$.

The training errors are $2226$, $658$, and $138$, respectively. They decrease as the nested partitions add regions, as expected. The four-region levels rise from $0.5$ to $42.5$, allowing the step function to reflect the increasing response. The final check is that each level is the mean of its pair, and the four pair errors sum to $138$.

Common mistake to avoid: a lower training error is not proof of better predictions on new cases. The comparison measures fit to these eight specified responses only.

Step Functions Need Not Be Smooth or Monotone

A tree’s thresholds determine where its predictions can change; the response means determine the levels. Those levels are calculated separately, so the fitted function is not automatically smooth, and it is not automatically increasing or decreasing. Even when predictor values are ordered, the response averages in successive regions can move up and down.

Worked Example: Ordered Regions With Nonmonotone Levels

For a hypothetical risk score, suppose the observations at $x=1,\ldots,6$ have responses $1,8,2,3,9,4$. Divide the predictor into three consecutive pairs: $x=1,2$, $x=3,4$, and $x=5,6$. Find the fitted levels, each region’s squared error, and the total.

First pair: the mean is $(1+8)/2=4.5$, and the squared error is $(1-4.5)^2+(8-4.5)^2=12.25+12.25=24.5$.

Second pair: the mean is $(2+3)/2=2.5$, and the squared error is $(2-2.5)^2+(3-2.5)^2=0.25+0.25=0.5$.

Third pair: the mean is $(9+4)/2=6.5$, and the squared error is $(9-6.5)^2+(4-6.5)^2=6.25+6.25=12.5$.

The fitted levels are $4.5,2.5,6.5$, which are not monotone: they first decrease and then increase. The total squared error is $24.5+0.5+12.5=37.5$. As a check, summing the six squared deviations directly gives $12.25+12.25+0.25+0.25+6.25+6.25=37.5$.

Common mistake to avoid: do not add a region’s squared deviations twice or replace them with its sum of squared responses. The region error is specifically the sum of $(y_i-\hat{c}_m)^2$, and the total here is $37.5$.

The pattern in this example is useful to keep in mind when interpreting a fitted tree. An increasing average response across the full predictor range does not force every sequence of local region means to increase. A monotonicity constraint would have to be imposed separately; it is not a consequence of fitting ordinary regression-tree means.

Common Exam Traps

Using a single mean for several regions. Calculate a separate sample mean from the observations in each terminal region.
Confusing number of observations with number of regions. A region’s mean divides its response sum by the count of observations assigned to that region.
Assuming each extra split lowers training error strictly. It can leave error unchanged, for example when the best fitted levels on both sides are equal. The guarantee is nonincrease, not strict decrease.
Calling a step function a smooth curve. It is constant inside each region and can jump at region boundaries.
Inferring future accuracy from training error alone. More regions cannot worsen the minimized training error for nested partitions, but this comparison alone says nothing conclusive about performance on new data.

Key takeaway: A regression tree approximates a nonlinear response pattern by assigning a sample-mean prediction to each threshold-defined region. Adding nested regions gives the model more distinct levels and cannot increase training squared error, but the resulting step function may still jump, fluctuate, and fail to improve predictions beyond the training data.

Check Your Understanding

For each problem, use squared-error fitting and calculate a separate mean for each stated region.

Responses at $x=1,2,3,4,5$ are $2,4,8,10,16$. Use regions $x\le2$ and $x>2$. Find both fitted levels and the total squared error. (Answer: levels $3$ and $34/3$; total squared error $110/3$, or $36.6667$ to four decimal places.)
Responses at $x=0,1,2,3,4,5$ are $0,1,4,9,16,25$. Use three regions containing the pairs $\{0,1\},\{2,3\},\{4,5\}$. Find the levels and total squared error. (Answer: levels $0.5,6.5,20.5$; total squared error $53.5$.)
Responses at $x=1,\ldots,6$ are $3,5,4,12,11,13$. Split after $x=3$. Find the two fitted levels and total squared error. (Answer: levels $4$ and $12$; total squared error $4$.)
At $x=1,2,3,4,5$, responses are $2,3,4,10,11$. Compare the training squared errors for a split after $x=2$ and a split after $x=3$. Which split has lower error? (Answer: errors $34$ and $2.5$, respectively; the split after $x=3$ has lower error.)
For $y=x^2$ at $x=0,\ldots,7$, compare the one-region error with the four-region error using the consecutive pairs $\{0,1\},\{2,3\},\{4,5\},\{6,7\}$. State the reduction in training error. (Answer: one-region error $2226$, four-region error $138$, reduction $2088$.)

What You'll Learn