Introduction
A regression tree predicts a numerical outcome by dividing observations into groups using predictor-based rules. For claim-cost modeling, a rule might separate vehicles by age, then separate one of those groups by annual mileage. Each final group, called a terminal node, receives a numerical prediction derived from the claim costs of the training observations in that group.
The tree is easy to describe as a sequence of rules, but its construction involves a specific trade-off: a split should reduce variation in the response enough to justify the additional complexity. This tutorial develops that calculation using hypothetical annual claim costs, then compares a deeper tree with a pruned version on separate validation data. All claim-cost figures are invented for teaching purposes.
How a Regression-Tree Split Is Evaluated
Suppose a node contains responses \(y_1,\ldots,y_n\), with mean \(\bar{y}\). Its residual sum of squares, or RSS, is the sum of the squared deviations from that mean. For a proposed split, calculate the RSS separately in the two child nodes and add them. The split reduction is the parent RSS minus this combined child RSS. The tree-growing procedure selects a split with a large reduction among the candidate splits it considers, subject to its growth controls.
This criterion has a direct justification. In either child node, write each response as its deviation from the child mean plus the difference between the child mean and the parent mean. When the squared terms are summed, the cross-products vanish because deviations from the child mean sum to zero. Therefore,
The last two terms are nonnegative, so the in-sample RSS cannot increase when a node is split and its child means are used. That fact alone does not mean every split is useful: a deeper tree can fit training data better while predicting new data worse. This is why pruning and validation matter.
Under squared-error loss, using the mean in a terminal node is appropriate because the mean minimizes the sum of squared deviations. This construction does not require a normal distribution for claim costs. It does require a numerical response and is most useful when squared error is a relevant measure of prediction error and the response has a finite mean and variance. A tree’s stepwise predictions do not imply that claim costs themselves follow a stepwise distribution.
Worked Example: Choosing Claim-Cost Splits
Method and mistake avoided. Compare candidate splits by their child-node RSS, not by looking only at the difference between their child means. Suppose a training sample contains eight hypothetical vehicle policies. The response is annual paid claim cost in thousands of dollars. “New” and “Older” are vehicle-age groups; mileage is either Low or High.
| Vehicle age | Annual mileage | Annual claim cost ($000) |
|---|---|---|
| New | Low | 1 |
| New | Low | 3 |
| New | High | 5 |
| New | High | 7 |
| Older | Low | 9 |
| Older | Low | 11 |
| Older | High | 13 |
| Older | High | 15 |
1. Calculate the parent RSS. The response mean is \((1+3+5+7+9+11+13+15)/8=64/8=8\). The parent RSS is
2. Evaluate the vehicle-age split. The New-group mean is \((1+3+5+7)/4=4\), and its RSS is \(9+1+1+9=20\). The Older-group mean is \((9+11+13+15)/4=12\), and its RSS is \(9+1+1+9=20\). The child RSS is \(20+20=40\), so the reduction is \(168-40=128\).
3. Evaluate the mileage split. The Low-mileage mean is \((1+3+9+11)/4=6\); its RSS is \(25+9+9+25=68\). The High-mileage mean is \((5+7+13+15)/4=10\); its RSS is \(25+9+9+25=68\). The child RSS is \(136\), so the reduction is \(168-136=32\).
Among these two candidates, vehicle age is the better root split because it reduces RSS by \(128\), compared with \(32\) for mileage. The resulting first two terminal predictions are \(4\) and \(12\), in thousands of dollars.
4. Consider splitting each age group again by mileage. Within New vehicles, the age-node mean is \(4\) and the RSS is \(20\). The Low-mileage mean is \((1+3)/2=2\), with RSS \(1+1=2\); the High-mileage mean is \((5+7)/2=6\), also with RSS \(2\). This split reduces RSS by \(20-(2+2)=16\). Within Older vehicles, the node mean is \(12\) and the RSS is \(20\). The two mileage-node means are \(10\) and \(14\); each has RSS \(2\), so this split also reduces RSS by \(16\).
The four resulting terminal predictions are \(2,6,10,\) and \(14\) thousand dollars. Their combined RSS is \(2+2+2+2=8\). The reductions check: the age split reduces RSS from \(168\) to \(40\), and the two later splits together reduce it from \(40\) to \(8\).
Terminal Predictions and Pruning
A terminal-node prediction is the training-response mean for the observations routed to that node. To predict for a new policy, follow its predictor values down the tree and report the mean for the terminal node it reaches. This is a group-level prediction; it is not a promise that the policy’s eventual cost will equal that amount.
A deeper tree generally has lower or equal training RSS because it has more opportunities to fit the training responses. Cost-complexity pruning formalizes the trade-off between fit and size. For a tree \(T\), let \(|T|\) be its number of terminal nodes and let \(R(T)\) be its training RSS. For a chosen nonnegative penalty \(\alpha\), compare trees using the objective below. Larger \(\alpha\) penalizes additional terminal nodes more heavily. The penalty’s scale depends on the scale of the response and on the particular implementation; this expression describes the principle, not a universal software parameter value.
For the worked data, compare the four-leaf tree (RSS \(8\)) with the two-leaf age-only tree (RSS \(40\)). At \(\alpha=20\), their objectives are \(8+20(4)=88\) and \(40+20(2)=80\), respectively. The two-leaf tree has the smaller penalized objective. The root-only tree has RSS \(168\), so its objective is \(168+20(1)=188\). Of these three candidates, the age-only tree is preferred by this penalty calculation.
In practice, the choice of tree size should also be checked against data not used to grow the tree. A validation set provides an estimate of prediction performance on new observations. For a numerical claim-cost target, common summaries include validation mean squared error (MSE) and root mean squared error (RMSE). RMSE is in the same units as the target; if costs are in thousands of dollars, an RMSE of \(2\) means an error scale of \(2{,}000\) dollars.
Worked Example: Using Validation Data to Choose a Pruned Tree
Method and mistake avoided. Compare candidate trees on the same held-out observations using a stated error measure. Do not select the deeper tree just because its training RSS is lower. Suppose the four-leaf tree from the earlier calculation predicts \(2,6,10,14\) thousand dollars for four validation policies. The policies’ actual costs are \(4,4,12,12\) thousand dollars. The pruned age-only tree predicts \(4\) for each New vehicle and \(12\) for each Older vehicle.
1. Score the four-leaf tree. Its residuals, defined as actual minus predicted, are \(4-2=2\), \(4-6=-2\), \(12-10=2\), and \(12-14=-2\). The squared errors are \(4,4,4,4\), so the sum is \(16\), the MSE is \(16/4=4\) in squared thousands of dollars, and the RMSE is \(\sqrt{4}=2\) thousand dollars.
2. Score the age-only tree. The New policies each have prediction \(4\), and the Older policies each have prediction \(12\). The four predictions therefore match the actual costs, giving squared errors \(0,0,0,0\). The MSE is \(0/4=0\), and the RMSE is \(0\) thousand dollars.
3. Decide what the comparison supports. On this validation set, the age-only tree performs better: its RMSE is \(0\), compared with \(2\) thousand dollars for the deeper tree. This supports pruning for this comparison, although a very small validation set gives an uncertain estimate of future performance. The check is meaningful because neither tree was scored on the observations used to calculate its training RSS.
Applying the Selected Tree and Explaining Its Limits
A practical interpretation should state the rule, the predicted amount, and the population to which the model applies. If validation supports the age-only tree above, its rule is simple: predict \(4{,}000\) dollars for a New vehicle and \(12{,}000\) dollars for an Older vehicle. A prediction is an average conditional on the training cases assigned to that node, not an individual claim-cost estimate with certainty.
Worked Example: Summarizing Predictions for a Hypothetical Portfolio
Method and mistake avoided. Route every policy through the selected tree, then add its predicted cost. Do not use the deeper tree’s predictions after deciding to retain the pruned tree. Suppose a hypothetical five-policy portfolio has two New vehicles and three Older vehicles. Their mileage values do not affect predictions under the selected age-only tree.
1. Assign the predictions. Each New vehicle receives a prediction of \(4\) thousand dollars, so the two New-vehicle predictions total \(2(4)=8\) thousand dollars. Each Older vehicle receives \(12\) thousand dollars, so the three Older-vehicle predictions total \(3(12)=36\) thousand dollars.
2. Aggregate and average. The predicted total is \(8+36=44\) thousand dollars, or \(\$44{,}000\). The average predicted cost per policy is \(44/5=8.8\) thousand dollars, or \(\$8{,}800\).
3. State the interpretation carefully. The tree predicts an aggregate cost of \(\$44{,}000\) for these five hypothetical policies, conditional on the model and its input data. It does not say that their realized total will be \(\$44{,}000\), nor does it quantify uncertainty around that total. Such uncertainty can be important when claim costs are variable or a terminal node contains few observations.
A regression tree’s transparency does not remove the need to assess its data and limitations. The vehicle-age rule may reflect patterns specific to the training sample, and another sample could produce different splits. Validation performance can also be imprecise when there are few observations. If the target is highly skewed or includes unusually large costs, squared-error fitting can be strongly affected by those values. Report the target definition and units, the split rules, the size of the terminal groups, and the validation method alongside the predictions.
Check Your Understanding
For each problem, calculate the requested quantity before checking the answer.
- A node has claim costs of \(3,5,9,\) and \(11\) thousand dollars. A proposed split puts \(3,5\) in one child and \(9,11\) in the other. Calculate the parent RSS, the child RSS, and the reduction. (Answer: Parent RSS \(=40\), child RSS \(=4\), reduction \(=36\), in squared thousands of dollars.)
- A terminal node contains three annual costs of \(\$12{,}000\), \(\$15{,}000\), and \(\$18{,}000\). Under squared-error loss, what prediction does the node make? (Answer: \(\$15{,}000\).)
- On three validation policies, a tree predicts \(6,10,14\) thousand dollars and the actual costs are \(8,8,12\) thousand dollars. Calculate its RMSE. (Answer: \(2\) thousand dollars, or \(\$2{,}000\).)
- Tree A has RSS \(18\) and three terminal nodes; Tree B has RSS \(24\) and two terminal nodes. For \(\alpha=3\), compare their cost-complexity objectives and select the smaller one. (Answer: Tree A has objective \(18+3(3)=27\); Tree B has objective \(24+3(2)=30\); select Tree A.)
- A validation comparison gives an age-only tree RMSE of \(\$1{,}400\) and a deeper tree RMSE of \(\$1{,}900\). The deeper tree has lower training RSS. Which tree has the better observed validation RMSE, and what does that comparison support? (Answer: The age-only tree has the better validation RMSE; the comparison supports preferring it for prediction, while the validation sample’s size and representativeness still matter.)