Selecting Regression Splits By Squared Error

Why Squared Error Selects a Regression Split

A regression tree predicts a numerical response by dividing observations into nodes and assigning a prediction within each node. For the squared-error criterion, that prediction is the mean response in the node. A proposed split is useful when the two resulting groups have less total within-node variation than the unsplit group.

This tutorial focuses on evaluating candidate splits for a numeric predictor. The candidate cut points are treated as given; the task is to calculate and compare their squared-error improvements. The key comparison is between the parent node’s sum of squared errors and the combined sums of squared errors of its two child nodes.

Definition: For a node containing $n$ observations with responses $y_1,\ldots,y_n$, its within-node sum of squared errors (SSE) is $$ \operatorname{SSE}=\sum_{i=1}^{n}(y_i-\bar{y})^2, \qquad \bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i. $$ For a proposed split, the child SSE is the sum of the SSEs of the left and right child nodes, each calculated using that child’s own response mean. The reduction in SSE is parent SSE minus child SSE.

The SSE measures the total squared distance between observed responses and the prediction made for their node. It is a total, not an average: a node with more observations can contribute more SSE than a smaller node. When the parent node and candidate split are fixed, selecting the split with the lowest child SSE is equivalent to selecting the split with the greatest reduction in SSE.

Calculating a Node’s SSE

A direct calculation finds the node mean, subtracts it from each response, squares the differences, and adds them. A useful alternative follows by expanding the squares:

\begin{aligned} \operatorname{SSE} &=\sum_{i=1}^{n}(y_i-\bar{y})^2\\ &=\sum_{i=1}^{n}y_i^2-2\bar{y}\sum_{i=1}^{n}y_i+n\bar{y}^2\\ &=\sum_{i=1}^{n}y_i^2-\frac{\left(\sum_{i=1}^{n}y_i\right)^2}{n}. \end{aligned}

The last equality holds because $\sum y_i=n\bar{y}$. Thus the middle terms combine as $-2n\bar{y}^2+n\bar{y}^2=-n\bar{y}^2$, and $n\bar{y}^2=(\sum y_i)^2/n$. This identity is a convenient arithmetic check; it gives the same SSE as the direct deviations. It requires $n>0$, since a node with no observations has no mean and cannot be scored this way.

For a split, apply the calculation separately to the two children. If their response means are $\bar{y}_L$ and $\bar{y}_R$, then $\operatorname{SSE}_{\text{children}}=\operatorname{SSE}_L+\operatorname{SSE}_R$. Do not center both children on the parent mean: each leaf makes its own prediction.

Calculate the parent SSE.
Use all responses in the current node and its response mean.

Partition the observations.
Use the proposed cut point to assign each observation to the left or right child.

Calculate both child SSEs.
Find each child’s mean and sum its squared deviations from that mean.

Compare the result.
Add child SSEs and subtract that total from the parent SSE. Among the candidates being considered, a larger reduction is better under this criterion.

Worked Split Comparisons

Worked Example: Comparing Two Cut Points

A hypothetical insurer uses a claim-processing score $x$ to predict the handling cost $y$, measured in hundreds of dollars, for six claims. In increasing order of $x$, the responses are $2,4,4,8,10,12$. Compare the candidate cuts after the second and after the third observation. Use the response mean as the prediction in each node.

Method. Calculate the parent SSE once, then calculate each pair of child SSEs. The parent responses sum to $40$, and their squared responses sum to $344$. Therefore,

\operatorname{SSE}_{\text{parent}} =344-\frac{40^2}{6} =344-\frac{800}{3} =\frac{232}{3}.

Cut after the second observation. The left child has responses $2,4$, mean $3$, and SSE $(2-3)^2+(4-3)^2=2$. The right child has responses $4,8,10,12$, mean $34/4=8.5$, and SSE $(4-8.5)^2+(8-8.5)^2+(10-8.5)^2+(12-8.5)^2=20.25+0.25+2.25+12.25=35$. The child SSE is $2+35=37$, so the reduction is $\frac{232}{3}-37=\frac{121}{3}$.

Cut after the third observation. The left child has responses $2,4,4$, mean $10/3$, and SSE $36-\frac{10^2}{3}=\frac{8}{3}$, since its squared responses sum to $36$. The right child has responses $8,10,12$, mean $10$, and SSE $(8-10)^2+(10-10)^2+(12-10)^2=8$. Thus child SSE is $\frac{8}{3}+8=\frac{32}{3}$, and the reduction is $\frac{232}{3}-\frac{32}{3}=\frac{200}{3}$.

Since $\frac{200}{3}>\frac{121}{3}$, the cut after the third observation has the greater squared-error reduction and the lower child SSE. The parent SSE is the same for both candidates, so either comparison gives the same choice. The calculation avoids choosing by visual impression or by comparing only one child.

Worked Example: Using the Computational Formula

A hypothetical maintenance team orders five equipment units by a sensor reading $x$. Their next-month repair costs $y$, in thousands of dollars, are $3,3,7,7,8$. Compare a cut after the second unit with a cut after the third. Do not round intermediate values.

The parent responses sum to $28$, and the squared responses sum to $180$. The parent SSE is $180-\frac{28^2}{5}=180-\frac{784}{5}=\frac{116}{5}$.

Cut after the second unit. The left responses are $3,3$, so their mean is $3$ and their SSE is $0$. The right responses are $7,7,8$, with sum $22$ and squared sum $162$. Their SSE is $162-\frac{22^2}{3}=162-\frac{484}{3}=\frac{2}{3}$. The child total is $\frac{2}{3}$, so the reduction is $\frac{116}{5}-\frac{2}{3}=\frac{338}{15}$.

Cut after the third unit. The left responses are $3,3,7$, with sum $13$ and squared sum $67$. Their SSE is $67-\frac{13^2}{3}=67-\frac{169}{3}=\frac{32}{3}$. The right responses are $7,8$, with mean $7.5$ and SSE $(7-7.5)^2+(8-7.5)^2=0.5=\frac{1}{2}$. The child total is $\frac{32}{3}+\frac{1}{2}=\frac{67}{6}$, so the reduction is $\frac{116}{5}-\frac{67}{6}=\frac{361}{30}$.

The first cut is preferred: its reduction $\frac{338}{15}=\frac{676}{30}$ exceeds $\frac{361}{30}$. As a check, the child SSE for the first cut, $\frac{2}{3}$, is lower than the second cut’s $\frac{67}{6}$. The two comparisons agree.

Worked Example: Checking Each Child Against Its Own Mean

A hypothetical energy provider orders five sites by a numeric demand indicator. Their observed peak loads $y$, in megawatts, are $1,2,3,4,10$. Compare the candidate cuts after the third and fourth sites.

The parent sum is $20$, and the squared sum is $130$, so the parent SSE is $130-\frac{20^2}{5}=50$.

Cut after the third site. The left child $1,2,3$ has mean $2$, giving SSE $1+0+1=2$. The right child $4,10$ has mean $7$, giving SSE $9+9=18$. The child total is $20$, and the reduction is $50-20=30$.

Cut after the fourth site. The left child $1,2,3,4$ has mean $2.5$, giving SSE $2.25+0.25+0.25+2.25=5$. The right child contains only $10$, so its mean is $10$ and its SSE is $0$. The child total is $5$, and the reduction is $50-5=45$.

The cut after the fourth site is better by the squared-error criterion because its child SSE is smaller and its reduction is larger. The arithmetic is consistent: for either cut, parent SSE minus child SSE equals the stated reduction. This criterion alone measures fit to these observations; it does not establish that a split will predict well on new observations.

Interpreting the Criterion and Avoiding Traps

The squared-error rule compares total error, so child-node sizes matter through the number and responses of observations in each child. It does not take an unweighted average of the two child errors. Nor does a smaller child SSE mean that every individual prediction is closer; it means that the sum of squared deviations from the respective child means is smaller.

A frequent exam error is to compute the two child SSEs correctly but average them before comparing candidates. For the second worked example, that incorrect calculation would give $(0+\frac{2}{3})/2=\frac{1}{3}$ for the first cut and $(\frac{32}{3}+\frac{1}{2})/2=\frac{67}{12}$ for the second. It happens to preserve the ranking there because both candidates have two children. But it is not the tree’s total squared error, and the unweighted average can mislead when comparing structures with different numbers of children or otherwise changes the requested quantity. The correction is to use the sum of child SSEs, as specified by the criterion.

Another trap is to subtract child SSE from parent SSE in the wrong order. A useful check is that splitting and then allowing each child its own mean cannot increase the minimum SSE: the parent mean remains an available common prediction, and each child can choose its own mean instead. Therefore the reduction should be nonnegative for a valid split evaluated this way. A negative result signals an arithmetic or definition error.

Key takeaway: For each candidate split, calculate each child’s SSE around its own response mean, add the child SSEs, and compare that total with the parent SSE. With the parent fixed, the best candidate under this criterion has the largest reduction in SSE.

Check Your Understanding

For each question, assume that the prediction in a node is its response mean and that the listed cut separates the responses in their stated order.

For responses $2,4,6,8$, calculate the parent SSE and the child SSE for a cut after the second response. Find the reduction. (Answer: parent SSE $20$; child SSE $4$; reduction $16$.)
For responses $1,1,5,7$, calculate the reduction from a cut after the second response. (Answer: parent SSE $27$; child SSE $2$; reduction $25$.)
For responses $2,4,5,9,10$, compare cuts after the second and third responses. Which cut has the greater reduction? (Answer: after the third response; reductions are $30$ and $245/6$, respectively.)
For responses $0,2,4,4,8,12$, compare cuts after the third and fourth responses. Show which candidate has the smaller child SSE. (Answer: after the fourth response; child SSEs are $40$ after the third and $19$ after the fourth.)
A node has responses $2,2,6,10$. A proposed cut places $2,2,6$ in the left child and $10$ in the right child. Calculate the parent SSE, child SSE, and reduction, and state whether the split improves squared error. (Answer: parent SSE $40$; child SSE $8$; reduction $32$, so it improves squared error.)

What You'll Learn