From a Dataset to a Small Regression Tree
A regression tree turns predictor values into a set of simple decision rules. Each split divides the observations into two groups, and each terminal node predicts a constant response. Building a small tree requires more than finding a threshold that looks plausible: compare the total squared error in both child nodes, respect the stopping constraints, and follow the resulting rules consistently when predicting.
The method here uses the squared-error criterion for a quantitative response. It is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. The splitting calculations do not require a particular probability distribution for the response. They describe how the tree fits the training observations; they do not, by themselves, establish that the fitted predictions are accurate for new data.
For a numerical predictor \(x\), a candidate threshold \(t\) sends observations with \(x\leq t\) to one child and observations with \(x>t\) to the other. In a basic regression tree, each child predicts its own response mean. A useful split lowers the total SSE, not necessarily the SSE of each child considered separately.
Why Squared-Error Reduction Has a Simple Form
Let the parent node contain \(n\) observations. A proposed split creates a left child with \(n_L\) observations and mean \(\bar{y}_L\), and a right child with \(n_R\) observations and mean \(\bar{y}_R\), where \(n=n_L+n_R\). The parent mean is the weighted mean of the child means:
Within each child, decompose each response’s deviation from the parent mean into its deviation from the child mean and the difference between the child and parent means. The cross-product terms sum to zero because the deviations from a child’s mean sum to zero. Thus:
For the last equality, substitute \(\bar{y}_L-\bar{y}=\frac{n_R}{n}(\bar{y}_L-\bar{y}_R)\) and \(\bar{y}_R-\bar{y}=-\frac{n_L}{n}(\bar{y}_L-\bar{y}_R)\) into the preceding line. The resulting coefficient is \(n_Ln_R^2/n^2+n_Rn_L^2/n^2=n_Ln_R/n\). Therefore, for a fixed parent node, choosing the split with the largest reduction is equivalent to choosing the split with the smallest child SSE.
Worked Example: Grow a Tree Under a Depth Limit
Worked Example: Grow a Tree Under a Depth Limit
A hypothetical insurer records a numerical predictor \(x\), a binary segment label \(z\), and a quantitative response \(y\) for eight policy groups. The response values are training data, not reported company experience. Use squared-error splitting, require at least two observations in each child, and stop at depth two. Find the successive splits, terminal-node rules, and predictions for new groups with \((x,z)=(3,A),(6,B),(8,A)\).
| Observation | \(x\) | \(z\) | \(y\) |
|---|---|---|---|
| 1 | 1 | A | 2 |
| 2 | 2 | B | 2 |
| 3 | 3 | A | 4 |
| 4 | 4 | B | 4 |
| 5 | 5 | A | 10 |
| 6 | 6 | B | 10 |
| 7 | 7 | A | 12 |
| 8 | 8 | B | 12 |
Step 1: Calculate the root SSE. The response mean is \((2+2+4+4+10+10+12+12)/8=56/8=7\). The root SSE is \(2(2-7)^2+2(4-7)^2+2(10-7)^2+2(12-7)^2=50+18+18+50=136\).
Step 2: Compare root splits. The candidate numerical thresholds are between successive observed \(x\) values; writing \(x\leq k\) specifies the corresponding partition. The child SSE totals are:
| Root split | Child SSE total |
|---|---|
| \(x\leq 1\) | \(752/7\), approximately 107.43 |
| \(x\leq 2\) | \(208/3\), approximately 69.33 |
| \(x\leq 3\) | \(688/15\), approximately 45.87 |
| \(x\leq 4\) | 16 |
| \(x\leq 5\) | \(688/15\), approximately 45.87 |
| \(x\leq 6\) | \(208/3\), approximately 69.33 |
| \(x\leq 7\) | \(752/7\), approximately 107.43 |
For example, at \(x\leq 2\), the left child has responses \(2,2\) and SSE 0. The right child has responses \(4,4,10,10,12,12\), mean \(52/6=26/3\), and SSE \(520-(52^2/6)=208/3\). This gives a total of \(208/3\), not the left child’s SSE alone.
At \(x\leq 4\), the left responses \(2,2,4,4\) have mean 3 and SSE \(2(2-3)^2+2(4-3)^2=4\). The right responses \(10,10,12,12\) have mean 11 and SSE \(2(10-11)^2+2(12-11)^2=4\). Their total SSE is \(4+4=8\). This is not 16, so the candidate table must be checked against the child calculations: the correct child SSE total at \(x\leq4\) is 8.
Consequently, compare the corrected \(x\leq4\) total of 8 with the other candidate totals; it is the smallest. The binary split on \(z\) also does not improve the root SSE: each segment has responses \(2,4,10,12\), mean 7, and SSE 68, giving a total of 136. The root split is therefore \(x\leq4\), with reduction \(136-8=128\).
Step 3: Split each child if eligible. In the left child, the responses are \(2,2,4,4\), with mean 3 and SSE 4. Splitting at \(x\leq2\) creates two children with responses \(2,2\) and \(4,4\); each has SSE 0 and size two. This split is eligible, reduces SSE by 4, and reaches depth two. In the right child, responses \(10,10,12,12\) have mean 11 and SSE 4. Splitting at \(x\leq6\) creates two size-two children with SSE 0 each, reducing SSE by 4. The maximum depth constraint prevents any further splitting.
Step 4: State the terminal rules and predictions. The four terminal-node rules and their predictions are:
- If \(x\leq4\) and \(x\leq2\), predict 2.
- If \(x\leq4\) and \(x>2\), predict 4.
- If \(x>4\) and \(x\leq6\), predict 10.
- If \(x>4\) and \(x>6\), predict 12.
For \((3,A)\), \(3\leq4\) and \(3>2\), so the prediction is 4. For \((6,B)\), \(6>4\) and \(6\leq6\), so the prediction is 10. For \((8,A)\), \(8>4\) and \(8>6\), so the prediction is 12. The label \(z\) does not appear in any selected split and therefore does not affect these predictions.
Check. The final training SSE is zero because each terminal node contains two identical response values. The total reduction is \(128+4+4=136\), which equals the root SSE. This verifies both the split calculations and the reductions added across the two child nodes.
Method and mistake avoided. This is greedy squared-error splitting: at each node, compare eligible candidate splits using both child SSEs. Do not infer that a predictor matters just because it is in the dataset, or add reductions for more child splits than the tree actually contains.
Worked Example: Select a Second Split and Predict
Worked Example: Select a Second Split and Predict
A hypothetical maintenance study records a predictor \(x=1,2,3,4,5,6\) and response values \(y=1,1,5,5,8,8\), in that order. Build a tree to depth two with a minimum of two observations per child, then predict for \(x=4.5\) and \(x=6\).
Root calculation. The response mean is \(28/6=14/3\). The root SSE is the sum of squares minus the squared sum divided by the sample size:
For \(x\leq2\), the child responses are \(1,1\) and \(5,5,8,8\). The first child SSE is 0; the second child mean is \(26/4=6.5\) and its SSE is \(2(5-6.5)^2+2(8-6.5)^2=9\). The total is 9. For \(x\leq4\), the child responses are \(1,1,5,5\) and \(8,8\); their SSEs are 16 and 0, respectively, for a total of 16. For \(x\leq3\), the groups are \(1,1,5\) and \(5,8,8\); their means are \(7/3\) and 7, and their SSEs are \(8/3\) and 18, for a total of \(62/3\). For \(x\leq5\), the first group \(1,1,5,5,8\) has mean 4 and SSE 36, while the other child has SSE 0. Thus \(x\leq2\), with child SSE 9, is the best root split.
The right child has responses \(5,5,8,8\), mean 6.5, and SSE 9. Splitting it at \(x\leq4\) gives one child with responses \(5,5\) and one with \(8,8\), so the child SSE total is 0. Both children have size two, and the split is allowed at depth two. The terminal rules are \(x\leq2\), predicting 1; \(2<x\leq4\), predicting 5; and \(x>4\), predicting 8.
Since \(4.5>4\), the prediction for \(x=4.5\) is 8. Since \(6>4\), the prediction for \(x=6\) is also 8. The root reduction is \(112/3-9=85/3\); the second reduction is 9. The final SSE is zero, and the total reduction \(85/3+9=112/3\) agrees with the root SSE.
Method and mistake avoided. First choose the root using total child SSE, then repeat the comparison inside the eligible child. Do not predict by interpolating between neighboring response values: a regression tree assigns the mean of the terminal node reached by the observation.
Worked Example: Enforce a Minimum Leaf Size
Worked Example: Enforce a Minimum Leaf Size
A hypothetical equipment review has five observations with \(x=1,2,3,4,5\) and responses \(y=2,2,2,10,10\). A split must leave at least three observations in each child. Should the root be split, and what prediction does the resulting tree make?
The root mean is \((2+2+2+10+10)/5=26/5=5.2\). Its SSE is \[ (2^2+2^2+2^2+10^2+10^2)-\frac{26^2}{5} =212-\frac{676}{5} =\frac{384}{5}=76.8. \]
Without the size constraint, \(x\leq3\) would separate the three responses equal to 2 from the two responses equal to 10, producing child SSE 0. But the child sizes would be three and two, so that split is not eligible. Every possible threshold between the ordered \(x\) values makes child sizes \(1\) and \(4\), \(2\) and \(3\), \(3\) and \(2\), or \(4\) and \(1\). None leaves at least three observations in both children. The tree therefore stops at the root, whose terminal rule is “all observations” and whose prediction is 5.2.
Check. The unsplit tree’s training SSE remains 76.8. The unavailable split would have reduced it by 76.8, but an attractive reduction does not override the specified constraint.
Method and mistake avoided. Check eligibility before accepting a candidate split. Do not report an unconstrained split as the fitted tree when it violates the minimum leaf size.
Common Error: Comparing Only One Child
A split’s score is the sum of the two child SSEs. In the first example, the split \(x\leq2\) has left-child SSE 0, which can look especially attractive if only that child is inspected. Its right-child SSE is \(208/3\), however, so its total is \(208/3\). The split \(x\leq4\) has child SSEs 4 and 4, for a total of 8. The latter split is better because \(8<208/3\).
A related accounting check applies after the tree has been built: reductions from separate splits can be added, but only once for each actual split. The final SSE equals the root SSE minus the sum of reductions. In the first example, there is one root split and two further splits, so the reductions are \(128+4+4=136\), not a root reduction plus three further reductions.
Reading the Finished Tree
A terminal-node rule is the conjunction of all conditions along the path from the root. For example, the rule \(x>4\) and \(x\leq6\) is one terminal region, with prediction 10 in the first example. The equality convention matters: an observation with \(x=6\) follows the \(x\leq6\) branch, not the \(x>6\) branch.
The terminal prediction is a constant: the training response mean in that node. It is not a fitted slope, and a threshold does not imply a smooth change in prediction. New predictor values are assigned by applying the rules in order until they reach one terminal node. If a predictor does not occur in the rules, its value does not change the prediction from this tree.
A tree built by selecting the best available split at each step is a greedy construction. Its choices are local to the current node and need not produce the globally best tree among all possible trees of the same size. Also, a lower training SSE alone does not guarantee better predictions for new observations. The workshop calculations establish the fitted rules and their training error; they do not replace appropriate assessment of predictive performance.
Check Your Understanding
For each problem, use the stated squared-error criterion and respect any stated stopping constraint.
- A node has four responses \(3,3,7,7\). A proposed split puts the first two in the left child and the last two in the right child. Find the parent SSE, child SSE total, and reduction. (Answer: Parent SSE \(=16\); child SSE total \(=0\); reduction \(=16\).)
- A fitted tree first sends \(x\leq10\) left and \(x>10\) right. In the left node it predicts 2 if \(x\leq4\) and 6 otherwise. In the right node it predicts 9 if \(x\leq15\) and 13 otherwise. Give the terminal predictions for \(x=3,8,12,18\). (Answer: \(2,6,9,13\), respectively.)
- For \(x=1,\ldots,6\), the ordered responses are \(2,2,4,4,8,8\). Compare the splits \(x\leq2\) and \(x\leq4\), using both child SSEs. Which is better, and what is its reduction from the root? (Answer: \(x\leq2\) has child SSE 16; \(x\leq4\) has child SSE 8 and is better. The root SSE is \(112/3\), so the reduction is \(112/3-8=88/3\).)
- Five ordered observations have responses \(1,1,1,9,9\). Each child must contain at least three observations. Can any threshold split be used? If not, find the root prediction and SSE. (Answer: No; each possible split has a child of size at most two. The root prediction is \(21/5=4.2\), and its SSE is \(384/5=76.8\).)
- A tree starts with root SSE 75. It makes one root split with reduction 30, then splits each of the two child nodes once, with reductions 4 and 6. Find the final SSE and check the total reduction. (Answer: Final SSE \(=75-(30+4+6)=35\); total reduction \(=40=75-35\).)