Evaluating a First Split
A regression tree begins with all training observations in one parent node. Its first decision is which predictor and cut to use to divide that node into two children. The question is not which split gives the most balanced groups or the most different-looking predictor values. Under squared-error fitting, the best first split is the one that leaves the smallest combined within-child sum of squared errors.
This tutorial works through small datasets so that each candidate can be checked directly. The response values are treated as fixed observed data; no probability distribution, interest rate, or mortality assumption is needed. Unless stated otherwise, each observation has equal weight, the response is continuous, and the split creates two nonempty children.
For a numeric predictor, sort the observations by its value and consider valid cut points between adjacent distinct values. A cut can be represented by a threshold between those values; the midpoint is a convenient choice. For a categorical predictor, a candidate split groups its levels into two nonempty sets. In either case, the response values determine the score. The predictor values determine membership in the children, not the child predictions.
Why Reduction in Error Gives the Same Ranking
Every candidate first split starts from the same parent node, so its parent SSE is fixed. Subtracting the children’s combined SSE from that fixed quantity shows that minimizing the split score is equivalent to maximizing the reduction. More specifically, let \(n_L\) and \(n_R\) be the child sizes, \(n=n_L+n_R\), and let \(\bar{y}_L\), \(\bar{y}_R\), and \(\bar{y}\) be their means and the parent mean. Then
To see why, take an observation in \(L\) and write \(y_i-\bar{y}=(y_i-\bar{y}_L)+(\bar{y}_L-\bar{y})\). Squaring and summing over \(L\) gives its within-child SSE, plus \(n_L(\bar{y}_L-\bar{y})^2\), plus a cross term. That cross term is zero because \(\sum_{i\in L}(y_i-\bar{y}_L)=0\). The same argument applies to \(R\). Thus the parent SSE is the sum of the two child SSEs and the two between-mean terms. Since
the between-mean terms add to \(\frac{n_Ln_R}{n}(\bar{y}_L-\bar{y}_R)^2\), as stated. The identity is a useful arithmetic check: a split cannot have a negative reduction in SSE. It assumes the unweighted squared-error criterion used here; other criteria or observation weights require their corresponding formulas.
Worked Comparisons
The procedure is deliberately local: compare the valid candidates for the current parent node, select one first split, and calculate the means in its children. This calculation alone does not decide how a tree should be grown beyond that split.
Worked Example: Comparing Two Predictors on Six Observations
A hypothetical service team records response time \(y\), in hours, for six cases. It also records a numeric workload score \(x\) and a two-level routing category \(z\). Find the best first split among every cut of \(x\) and the split by \(z\), then report the child means.
| Case | \(x\) | \(z\) | \(y\) (hours) |
|---|---|---|---|
| 1 | 1 | A | 2 |
| 2 | 2 | A | 4 |
| 3 | 3 | B | 3 |
| 4 | 4 | B | 8 |
| 5 | 5 | A | 7 |
| 6 | 6 | B | 9 |
Method and parent score. For any group \(G\), calculate \(\operatorname{SSE}(G)=\sum y_i^2-(\sum y_i)^2/n_G\). The parent has \(\sum y_i=33\) and \(\sum y_i^2=223\), so its mean is \(33/6=5.5\) hours and its SSE is \(223-33^2/6=41.5\) hours squared.
Numeric predictor candidates. The distinct \(x\)-values are 1 through 6, so the five thresholds are 1.5, 2.5, 3.5, 4.5, and 5.5.
At \(x=1.5\), the left child is \(\{2\}\) and has SSE 0. The right child has responses \(4,3,8,7,9\), with sum 31 and sum of squares 219. Its SSE is \(219-31^2/5=26.8\). The total is 26.8.
At \(x=2.5\), the left responses \(2,4\) have mean 3 and SSE \((2-3)^2+(4-3)^2=2\). The right responses \(3,8,7,9\) have sum 27 and sum of squares 203, so their SSE is \(203-27^2/4=20.75\). The total is 22.75.
At \(x=3.5\), the left responses \(2,4,3\) have sum 9 and sum of squares 29, giving SSE \(29-9^2/3=2\) and mean 3. The right responses \(8,7,9\) have sum 24 and sum of squares 194, giving SSE \(194-24^2/3=2\) and mean 8. The total is 4.
At \(x=4.5\), the left responses \(2,4,3,8\) have sum 17 and sum of squares 93, giving SSE \(93-17^2/4=20.75\). The right responses \(7,9\) have SSE 2. The total is 22.75.
At \(x=5.5\), the left responses \(2,4,3,8,7\) have sum 24 and sum of squares 142, giving SSE \(142-24^2/5=26.8\). The right child has SSE 0, so the total is 26.8.
Categorical predictor candidate. Category A has responses \(2,4,7\): its sum is 13, its sum of squares is 69, and its SSE is \(69-13^2/3=38/3\). Category B has responses \(3,8,9\): its sum is 20, its sum of squares is 154, and its SSE is \(154-20^2/3=62/3\). The combined score is \(100/3\), or approximately 33.3333.
Selection and check. The smallest score is 4, for the threshold \(x=3.5\). Its child means are 3 hours on the left and 8 hours on the right. The reduction is \(41.5-4=37.5\). The identity also gives \((3\cdot3/6)(8-3)^2=1.5\cdot25=37.5\), confirming the calculation. The common mistake avoided here is selecting a predictor by visual separation or group balance instead of comparing its total child SSE.
Worked Example: Checking a Close Numeric Cut Against a Category Split
A hypothetical equipment-monitoring study records a continuous response \(y\), in units of a measurement index, and two predictors for five devices. Find the best first split and its child means.
| Device | Numeric setting \(p\) | Class \(c\) | \(y\) |
|---|---|---|---|
| 1 | 2 | A | 4 |
| 2 | 4 | A | 4 |
| 3 | 6 | A | 10 |
| 4 | 8 | B | 12 |
| 5 | 10 | B | 15 |
Parent and numeric cuts. The response sum is 45, the mean is 9, and the sum of squared responses is \(16+16+100+144+225=501\). Thus the parent SSE is \(501-45^2/5=96\).
After \(p=2\), the right child responses \(4,10,12,15\) have mean \(41/4=10.25\). Their SSE is \((4-10.25)^2+(10-10.25)^2+(12-10.25)^2+(15-10.25)^2=64.75\); the singleton left child contributes 0.
After \(p=4\), the left child \(4,4\) has mean 4 and SSE 0. The right child \(10,12,15\) has mean \(37/3\) and SSE \(10^2+12^2+15^2-37^2/3=469-1369/3=38/3\). The total is \(38/3\), approximately 12.6667.
After \(p=6\), the left child \(4,4,10\) has mean 6 and SSE \(16+16+16=48\)—but this would incorrectly measure deviations from the mean. Rechecking those deviations gives \((4-6)^2+(4-6)^2+(10-6)^2=4+4+16=24\). The right child \(12,15\) has mean 13.5 and SSE \(2.25+2.25=4.5\). The total is 28.5.
After \(p=8\), the left child \(4,4,10,12\) has mean 7.5 and SSE \(12.25+12.25+6.25+20.25=51\); the right singleton contributes 0. The total is 51.
Category candidate and selection. Class A has responses \(4,4,10\), mean 6, and SSE 24. Class B has responses \(12,15\), mean 13.5, and SSE 4.5. Its total is 28.5. Therefore the best candidate is the numeric cut between 4 and 6, conventionally represented by \(p=5\). The child means are 4 and \(37/3\), approximately 12.3333. The split reduction is \(96-38/3=250/3\), approximately 83.3333. The weighted-mean identity gives \((2\cdot3/5)(37/3-4)^2=250/3\), a check on the result. The common mistake avoided is using a response group’s sum of squares as its SSE without subtracting the squared sum divided by its size.
Worked Example: Reporting the Winning Split and Verifying Its Score
A hypothetical calibration exercise records five response values \(y\) for ordered settings \(x=1,2,3,4,5\). The responses are \(3,3,5,9,10\). A second predictor \(h\) divides the first, third, and fourth observations into group L and the second and fifth into group H. Compare all cuts of \(x\) with the split by \(h\), then report the best split.
Parent score. The response mean is \((3+3+5+9+10)/5=6\). The parent SSE is \((3-6)^2+(3-6)^2+(5-6)^2+(9-6)^2+(10-6)^2=44\).
Numeric cuts. After \(x=1\), the singleton has SSE 0 and the responses \(3,5,9,10\) have mean 6.75 and SSE 41.75, so the total is 41.75. After \(x=2\), the responses \(3,3\) have SSE 0; the responses \(5,9,10\) have mean 8 and SSE \(9+1+4=14\), giving total 14. After \(x=3\), the left responses \(3,3,5\) have mean \(11/3\) and SSE \(3^2+3^2+5^2-11^2/3=8/3\). The right responses \(9,10\) have mean 9.5 and SSE 0.5. The total is \(8/3+1/2=19/6\), approximately 3.1667. After \(x=4\), the left responses \(3,3,5,9\) have mean 5 and SSE \(4+4+0+16=24\); the right singleton contributes 0.
Category comparison and result. Group L has responses \(3,5,9\), mean \(17/3\), and SSE \(115-17^2/3=8/3\). Group H has responses \(3,10\), mean 6.5, and SSE \(49/2=24.5\). The category split score is \(8/3+49/2=163/6\), approximately 27.1667. Thus the best first split is after \(x=3\), with child means \(11/3\) and \(19/2\), approximately 3.6667 and 9.5. Its reduction is \(44-19/6=245/6\), approximately 40.8333. Here the error to avoid is treating a low score from one child as enough: the decision uses the combined SSE of both children.
Practical Checks Before Choosing
A careful first-split calculation has three separate pieces: generate the valid candidate partitions, compute each partition’s combined child SSE, and report the child means for the selected partition. Keeping those steps distinct helps prevent common arithmetic and interpretation errors.
- Use total error, not a simple average of the two node scores. The criterion is the sum of the child SSEs. Each observation contributes its squared deviation from its own child mean.
- Recalculate each child mean from its own responses. The parent mean is not generally the prediction in either child.
- Do not compare raw squared errors across different response units. The SSE has squared response units, and its value depends on the response scale. Within one dataset, all candidates use the same response scale.
- Check that the children cover the parent exactly. Every observation belongs to one child, neither child is empty, and the two child sizes sum to the parent size.
- Do not infer a unique choice from a tie. If two candidate splits have the same score, both minimize the stated criterion; a tie-breaking rule would be needed to select one.
A first split is a greedy choice: it is selected from the candidates at the current node using the current data. The calculations establish which candidate best reduces squared error at that step; they do not by themselves establish that it is best to keep growing the tree.
Check Your Understanding
For each problem, use unweighted squared-error fitting and compare the requested candidate splits.
- A numeric predictor orders four responses as \(1,1,5,5\). At the cut between the second and third observations, calculate the two child means and the combined child SSE. (Answer: Means 1 and 5; combined SSE 0.)
- Four observations have responses \(2,4,6,8\). A candidate split places the first two in one child and the last two in the other. Find both child means, the child SSEs, and the total split score. (Answer: Means 3 and 7; child SSEs 2 and 2; total 4.)
- Five responses in numeric-predictor order are \(1,2,7,8,9\). Compare the cuts after observations 2 and 3. Which is better, and what is its score? (Answer: The cut after observation 2 is better; its score is \(0.5+2=2.5\), compared with \(127/6\), approximately 21.1667, after observation 3.)
- The parent responses are \(0,2,4,10\), and a candidate cut places the first two in one child and the last two in the other. Find the parent SSE, split score, and reduction in SSE. (Answer: Parent SSE 56; split score 20; reduction 36.)
- Candidate A has child SSEs 3 and 11; candidate B has child SSEs 5 and 7. Which candidate is preferred under squared-error splitting? Is the choice unique? (Answer: Both have total SSE 14, so they tie and neither is uniquely preferred by this criterion.)