Worked Example of a Tree Approximating Curvature

Why a Tree Can Approximate Curvature

A regression tree does not fit one smooth curve across the predictor range. Instead, it divides that range into regions and predicts a constant response within each region. With enough regions, the resulting step function can follow a curved pattern approximately. The practical question is where to split: a useful threshold makes the predictions within its child regions closer to the observed responses.

The calculations below use hypothetical training data with one numerical predictor $x$ and a quantitative response $y$. At each terminal node, the prediction is the mean response among the observations in that node. A candidate split assigns observations with $x$ at or below a threshold to the left child and those above it to the right child.

Definition: For a node containing $n$ responses $y_1,\ldots,y_n$, its squared-error total is $$ \operatorname{SSE}=\sum_{i=1}^{n}(y_i-\bar y)^2, $$ where $\bar y$ is the node mean. For a proposed split, the total child error is the left-child SSE plus the right-child SSE. The split improvement is the parent SSE minus this total child error. A larger improvement means a greater reduction in training squared error.

For a numerical predictor, candidate thresholds need only be considered between consecutive distinct observed predictor values. The observations on each side of a threshold determine the two child means and their errors. A split is useful in the training data when its improvement is positive; among the candidate thresholds under consideration, the largest improvement gives the greatest immediate reduction in squared error.

Why the Split-Improvement Formula Holds

A convenient calculation avoids recomputing every squared deviation directly. For responses with count $n$, sum $T$, and sum of squares $Q$, the SSE around their mean is $Q-T^2/n$. This follows by expanding the definition and using $\bar y=T/n$:

\sum_{i=1}^{n}(y_i-\bar y)^2 =\sum_{i=1}^{n}y_i^2-2\bar y\sum_{i=1}^{n}y_i+n\bar y^2 =Q-\frac{T^2}{n}.

There is also a useful direct expression for the improvement from splitting a parent into left and right children. Write their counts and means as $n_L,n_R,\bar y_L,\bar y_R$, with $n=n_L+n_R$. The parent mean is $\bar y=(n_L\bar y_L+n_R\bar y_R)/n$. The parent SSE equals the two child SSEs plus the squared deviations of the child means from the parent mean, weighted by their counts. Therefore,

\operatorname{SSE}_{\text{parent}} -\operatorname{SSE}_{\text{left}} -\operatorname{SSE}_{\text{right}} =n_L(\bar y_L-\bar y)^2+n_R(\bar y_R-\bar y)^2 =\frac{n_Ln_R}{n}(\bar y_L-\bar y_R)^2.

The final equality follows by substituting the weighted parent mean into both squared deviations and simplifying. This identity shows why a split with separated child means can have a large improvement, while also accounting for how many observations are in each child. It also proves that the minimized child SSE cannot exceed the parent SSE: the improvement is nonnegative.

Worked Example: Comparing Every Split for a Curved Pattern

A hypothetical portfolio analyst records a predictor index $x$ and response $y$, a scaled measure of expected claim cost, for four segments. The data are $x=1,2,3,4$ and $y=1,4,9,16$, following a curved pattern. Calculate the parent error, assess every possible one-split tree, and select the best split.

Method. Use $Q-T^2/n$ for each node’s SSE, then subtract the combined child error from the parent error. This calculation avoids the common mistake of using the parent’s sum of squares for a child that contains only some of the observations.

The parent response sum is $T=1+4+9+16=30$, and its sum of squares is $Q=1+16+81+256=354$. Thus the parent mean is $30/4=7.5$, and the parent SSE is

354-\frac{30^2}{4}=354-225=129.

Split after $x=1$. The left child contains $1$, so its SSE is $0$. The right child contains $4,9,16$, with sum $29$ and sum of squares $353$. Its SSE is $353-29^2/3=353-841/3=218/3$. The combined child error is $218/3$, so the improvement is $129-218/3=169/3$, approximately $56.333$.

Split after $x=2$. The left child contains $1,4$. Its sum is $5$, its sum of squares is $17$, and its SSE is $17-25/2=9/2=4.5$. The right child contains $9,16$. Its sum is $25$, its sum of squares is $337$, and its SSE is $337-625/2=49/2=24.5$. The combined child error is $4.5+24.5=29$, so the improvement is $129-29=100$.

Split after $x=3$. The left child contains $1,4,9$, with sum $14$ and sum of squares $98$. Its SSE is

98-\frac{14^2}{3}=98-\frac{196}{3}=\frac{98}{3}\approx32.667.

The right child contains only $16$, so its SSE is $0$. The combined child error is $98/3$, and the improvement is $129-98/3=289/3$, approximately $96.333$.

The best of the three candidate splits is after $x=2$, with improvement $100$. It gives the stepwise predictions $2.5$ for $x=1,2$ and $12.5$ for $x=3,4$. As a check, the child errors add to $29$, and $129-29=100$, as required.

Growing a Shallow Tree and Reading Its Predictions

A tree can make another split within one of the children, subject to the chosen size or depth limit. The next example continues from the best first split above. This illustrates two distinct tasks: calculating which available split reduces error most, and then translating the resulting terminal nodes into predictions over the predictor range.

Worked Example: Adding One Split to a Two-Leaf Tree

Use the same hypothetical four observations, but now allow at most three terminal nodes. The first split is after $x=2$, producing nodes $\{1,4\}$ and $\{9,16\}$. Choose which child to split, calculate the resulting training error, and state the predictions.

Method. Compare the improvement available from splitting each current node. A split of a two-observation node into two single-observation nodes gives each resulting node SSE $0$.

The left node $\{1,4\}$ currently has SSE $4.5$, so splitting it into two singletons reduces its error by $4.5$. The right node $\{9,16\}$ currently has SSE $24.5$, so splitting it into two singletons reduces its error by $24.5$. With a limit of three leaves, only one of these two splits can be added. The larger available reduction is $24.5$, so split the right node between $x=3$ and $x=4$, at threshold $3.5$.

The three terminal-node predictions are:

For $x\leq2.5$, the node contains responses $1,4$, so its prediction is $(1+4)/2=2.5$.
For $2.5<x\leq3.5$, the node contains response $9$, so its prediction is $9$.
For $x>3.5$, the node contains response $16$, so its prediction is $16$.

The remaining training error is the SSE in the unsplit left node, $4.5$, plus zero in each singleton node, for a total of $4.5$. Relative to the parent SSE of $129$, the overall improvement is $129-4.5=124.5$. This agrees with the first split’s improvement of $100$, plus the additional improvement of $24.5$. The check confirms that the second split reduces error by exactly the amount calculated for its node.

The added split makes the observed predictions closer to the four responses, but it leaves two terminal nodes with one observation each. The predictions are therefore exact for those training observations, not evidence that the underlying relationship is known exactly between or beyond them.

A Different Curved Pattern Can Favor a Different Threshold

A tree’s split is selected by the observed response values and the resulting errors, not by a requirement that the threshold be at the center of the predictor range. Curvature can be approximated unevenly: if one part of the range differs substantially from the rest, isolating that part may be the best immediate split.

Worked Example: A Curved Pattern With an Uneven Best Split

In a hypothetical study, a numerical exposure index $x=1,2,3,4,5$ is paired with scaled response values $y=20,13,8,5,4$. The responses decrease and flatten across the range. Compare all one-split thresholds, select the best, and interpret its predictions.

Method. Compute each candidate’s combined child SSE using $Q-T^2/n$, then compare the improvements against the same parent SSE.

For the parent, $T=20+13+8+5+4=50$ and $Q=400+169+64+25+16=674$. Therefore,

\operatorname{SSE}_{\text{parent}}=674-\frac{50^2}{5}=674-500=174.

Split after $x=1$. The singleton left child has SSE $0$. The right child has values $13,8,5,4$, with sum $30$ and sum of squares $234$. Its SSE is $234-900/4=9$. The combined error is $9$, giving improvement $174-9=165$.

Split after $x=2$. The left values $20,13$ have SSE $569-1089/2=24.5$. The right values $8,5,4$ have sum $17$, sum of squares $105$, and SSE $105-289/3=26/3$. The combined error is $24.5+26/3=199/6$, approximately $33.167$. The improvement is $174-199/6=845/6$, approximately $140.833$.

Split after $x=3$. The left values $20,13,8$ have sum $41$, sum of squares $633$, and SSE $633-1681/3=218/3$. The right values $5,4$ have SSE $0.5$. The combined error is $218/3+0.5=439/6$, approximately $73.167$, and the improvement is $174-439/6=605/6$, approximately $100.833$.

Split after $x=4$. The left values $20,13,8,5$ have sum $46$, sum of squares $658$, and SSE $658-2116/4=129$. The singleton right child has SSE $0$. The improvement is $174-129=45$.

The greatest improvement is $165$, so the selected split isolates $x=1$. Its predictions are $20$ at $x=1$ and $(13+8+5+4)/4=7.5$ for $x=2,3,4,5$. The resulting child error is $0+9=9$, and $174-9=165$, which checks the improvement calculation. Although the original responses follow a changing pattern, this one-split tree represents them with only two constant predictions.

Common Mistake: Reusing a Parent’s Sum of Squares

A child’s SSE must be calculated from the responses actually in that child. In the first example, the parent’s sum of squares is $354$, but the left child after the split at $x=3$ contains only $1,4,9$. Its sum of squares is $98$, not $354$.

The incorrect calculation would use $354-196/3$ for that left-child SSE, producing an error larger than the parent SSE and a negative split improvement. The correction is $98-196/3=98/3$, approximately $32.667$; the right child’s error is $0$, so the actual improvement is $129-98/3=289/3$, approximately $96.333$. A negative improvement is also a warning to check the arithmetic: because each child can use the parent mean if needed, minimizing error separately within the children cannot increase the minimized SSE.

Key takeaway: To assess a regression-tree split, calculate each child’s error from its own responses, add the child errors, and subtract that total from the parent error. The selected split makes the largest immediate reduction among the candidate thresholds; its terminal-node means create a stepwise approximation, not a smooth curve.

Check Your Understanding

For each problem, calculate the requested error or improvement before checking the final answer.

A node has responses $2,6,10$. Find its SSE. Then split off the response $2$ as the left child and find the improvement. (Answer: Parent SSE $32$; split improvement $24$.)
Four ordered predictor values have responses $3,7,11,19$. Find the parent SSE and the improvement from splitting after the second observation. (Answer: Parent SSE $140$; improvement $100$.)
A proposed split has two observations in the left child and three in the right child. Their means are $4$ and $10$, respectively. Find the split improvement using the child-count formula. (Answer: $43.2$.)
For ordered responses $2,5,8,14,17$, split after the third observation. Calculate the two child SSEs, total child error, and improvement. (Answer: Left SSE $18$, right SSE $4.5$, total child error $22.5$, improvement $79.5$.)
A parent node has SSE $90$. One candidate split has child SSEs $12$ and $33$; another has child SSEs $20$ and $18$. Which split is better, and by how much more does it reduce error? (Answer: The second split; its improvement is $52$, which is $3$ greater than the first split’s improvement of $49$.)

What You'll Learn