Why Compare Splits Across Predictors?
At a regression-tree node, several predictors may offer several possible splits. One predictor might separate observations into groups with very different response averages; another might divide the data into groups whose averages are similar. To choose between them, the tree needs a common score. For continuous responses with squared-error loss, that score is the reduction in within-node sum of squares.
The comparison must be made fairly: score each candidate using the same observations in the same parent node and the same error criterion. The candidate with the largest reduction is preferred at that step. This is a local, greedy choice; it does not establish that the resulting full tree will be best after later splits.
These definitions assume that a terminal node predicts its sample mean and is scored by squared error. Each child must contain at least one observation. The response values and candidate partitions are taken from the observations currently in the node; the score is not calculated from observations outside it. Constraints such as a minimum permitted child size can rule out a candidate before it is scored.
One Criterion for Every Predictor
For a fixed parent node, its sum of squares is the same for every candidate split. Consequently, maximizing the reduction is equivalent to minimizing the total sum of squares in the two children. The units of the reduction are squared response units, and it measures the absolute decrease in training error at this node.
A threshold for one predictor and a threshold for another are directly comparable when both are evaluated on the same parent observations using this same reduction. The predictor’s name or measurement scale does not change the definition of the score. A categorical predictor can also supply a candidate partition: its categories are assigned to two groups, and the resulting children are scored by the same formula.
Compute its response mean and parent sum of squares once.
For each permitted threshold or category grouping, calculate the two child sums of squares and their reduction.
Select the predictor and split rule with the greatest reduction. If candidates tie, they have equal scores under this criterion.
Worked Example: Comparing the Best Thresholds for Two Predictors
A hypothetical insurer uses a regression tree to predict a continuous claim-handling cost for six files. The table gives each response \(y\) and the rank of each file when sorted by predictor \(A\) or predictor \(B\). For either predictor, consider the thresholds between consecutive ranked values. Which predictor-threshold pair gives the greatest reduction?
| File | Response \(y\) | Rank by \(A\) | Rank by \(B\) |
|---|---|---|---|
| 1 | 1 | 1 | 1 |
| 2 | 2 | 2 | 4 |
| 3 | 3 | 3 | 2 |
| 4 | 7 | 4 | 5 |
| 5 | 8 | 5 | 3 |
| 6 | 9 | 6 | 6 |
Method. Compute the parent sum of squares, then for each threshold compute the two child sums of squares. The best threshold for a predictor is the one with the largest reduction for that predictor.
The parent mean is \((1+2+3+7+8+9)/6=30/6=5\). Thus
For predictor \(A\), the responses in sorted order are \(1,2,3,7,8,9\). At the threshold between ranks 3 and 4, the left mean is \((1+2+3)/3=2\), and the right mean is \((7+8+9)/3=8\). Each child has sum of squares \(2\): the left calculation is \((1-2)^2+(2-2)^2+(3-2)^2=2\), and the right calculation is \((7-8)^2+(8-8)^2+(9-8)^2=2\). The child total is \(4\), so the reduction is \(58-4=54\).
To verify this is the best threshold for \(A\), the child totals at the other four boundaries are \(38.8,\ 21.25,\ 21.25,\ 38.8\). For example, at the boundary after rank 2, the left sum of squares is \(0.5\); the right responses \(3,7,8,9\) have mean \(6.75\) and sum of squares \(20.75\), giving \(21.25\). Therefore the largest reduction for \(A\) is \(54\).
For predictor \(B\), the sorted response order is \(1,3,8,2,7,9\). The child totals at its five boundaries are \(38.8,\ 31,\ 52,\ 31,\ 38.8\). For instance, at the boundary after rank 2, the left responses \(1,3\) have mean \(2\) and sum of squares \(2\). The right responses \(8,2,7,9\) have mean \(6.5\) and sum of squares \(29\), so the total is \(31\) and the reduction is \(58-31=27\). The minimum child total for \(B\) is \(31\), so its best reduction is \(27\).
Decision and check. Predictor \(A\) wins: its best reduction, \(54\), exceeds predictor \(B\)’s best reduction, \(27\). The selected rule is the threshold between the third and fourth sorted \(A\)-values. The corresponding child total is \(4\), and \(58-4=54\), as required.
Common mistake avoided: choosing a predictor from only one arbitrarily selected threshold. The comparison uses each predictor’s best available threshold, not a threshold picked before examining its candidate scores.
Why the Reduction Has a Simple Form
The reduction can also be written using the two child means. This identity provides a quick check and explains what makes a split effective: the children should have different means, with enough observations on both sides to contribute to the separation.
Let \(n_L\) and \(n_R\) be the child sizes, with \(n=n_L+n_R\), and let \(\bar{y}_L\), \(\bar{y}_R\), and \(\bar{y}\) be the left, right, and parent means. Because \(\bar{y}=(n_L\bar{y}_L+n_R\bar{y}_R)/n\), decomposing each response’s deviation from the parent mean into its deviation from its child mean plus its child mean’s deviation from the parent gives
The cross-products vanish because, within each child, deviations from that child’s mean sum to zero. Substituting the weighted-mean expression for \(\bar{y}\) into the last two terms and simplifying yields
This also shows that the reduction cannot be negative for a valid split scored by child means. If a calculation gives a negative reduction, check the means, membership of the children, and sums of squares.
Worked Example: Comparing a Numeric Split With a Category Split
A hypothetical pricing team has four observations in a node, with responses \(1,3,7,9\). A numeric predictor proposes one split, and a two-level categorical predictor proposes another. Which candidate should be selected?
Method. Calculate the parent sum of squares, then score both partitions using exactly the same squared-error criterion.
The parent mean is \((1+3+7+9)/4=5\). The parent sum of squares is \((1-5)^2+(3-5)^2+(7-5)^2+(9-5)^2=16+4+4+16=40\).
The numeric split places \(\{1,3\}\) in the left child and \(\{7,9\}\) in the right child. Their means are \(2\) and \(8\). Their sums of squares are \((1-2)^2+(3-2)^2=2\) and \((7-8)^2+(9-8)^2=2\). Thus the child total is \(4\), and the reduction is \(40-4=36\). The identity gives the same result: \((2\cdot2/4)(2-8)^2=1\cdot36=36\).
The categorical split groups \(\{1,7\}\) in one child and \(\{3,9\}\) in the other. The means are \(4\) and \(6\). Each child sum of squares is \(9+9=18\), so the child total is \(36\), and the reduction is \(40-36=4\). The identity checks this: \((2\cdot2/4)(4-6)^2=1\cdot4=4\).
Decision and check. Choose the numeric split because \(36>4\). Its child total, \(4\), is lower than the categorical split’s child total, \(36\), and subtracting either from the common parent total of \(40\) gives the stated reduction.
Common mistake avoided: applying a different scoring rule to a categorical predictor. Both proposed partitions are compared by the same child sum of squares, regardless of predictor type.
Comparing Each Predictor’s Best Candidate
In practice, it is useful to keep a short score table. It separates two tasks that are easy to confuse: searching for a predictor’s strongest candidate and comparing those strongest candidates across predictors. The following example shows why the overall winner need not be the predictor whose candidates are easiest to interpret.
Worked Example: A Predictor With a Smaller Best Reduction
A hypothetical service operation has five files in a node, with responses \(2,4,6,8,10\). Predictor \(P\) orders the files in that response order. Sorting by predictor \(Q\) gives response order \(2,6,10,4,8\). Consider all four boundaries between consecutive values for each predictor. Find each predictor’s best reduction and select the winning pair.
Method. Compute the parent score, tabulate child totals at each boundary, and compare the minimum child total for each predictor.
The parent mean is \((2+4+6+8+10)/5=6\), and the parent sum of squares is \(16+4+0+4+16=40\).
For \(P\), the sorted order is \(2,4,6,8,10\). The child totals at the four boundaries are as follows. After \(2\), the right-child mean is \(7\), and its sum of squares is \(9+1+1+9=20\), so the total is \(20\). After \(4\), the child sums are \(2\) for \(\{2,4\}\) and \(8\) for \(\{6,8,10\}\), giving \(10\). After \(6\), the sums are \(8\) for \(\{2,4,6\}\) and \(2\) for \(\{8,10\}\), also giving \(10\). After \(8\), the left-child mean is \(5\) and its sum of squares is \(9+1+1+9=20\), giving a total of \(20\). Thus \(P\)’s best reduction is \(40-10=30\), attained at either of its two middle boundaries.
For \(Q\), the sorted order is \(2,6,10,4,8\). After the first value, the right-child responses \(6,10,4,8\) have mean \(7\) and sum of squares \(1+9+9+1=20\), giving a child total of \(20\). After the second value, the left sum is \(8\); the right-child mean for \(10,4,8\) is \(22/3\), and its sum is \(56/3\). The total is \(8+56/3=80/3\), giving a reduction of \(40-80/3=40/3\). After the third value, the child sums are \(32\) for \(\{2,6,10\}\) and \(8\) for \(\{4,8\}\), giving \(40\). After the fourth value, the left responses \(2,6,10,4\) have mean \(5.5\) and sum of squares \(35\), while the singleton child has sum \(0\); the total is \(35\). Therefore \(Q\)’s best reduction is \(40-20=20\).
Decision and check. Choose predictor \(P\), whose best reduction is \(30\), rather than \(Q\), whose best reduction is \(20\). The winning child total is \(10\), and \(40-10=30\). This comparison uses the best boundary for each predictor, rather than comparing a chosen middle boundary for \(P\) with every boundary for \(Q\).
Common mistake avoided: selecting the predictor with the smallest-looking individual child sum. A split is scored by the sum of both child errors, then compared with the parent error.
Interpretation and Common Traps
A large reduction means the split separates the responses into children with relatively distinct means, accounting for the sizes of both children. It does not mean every response is well predicted, nor does it by itself measure performance on new observations. The score is calculated on the data in the current node and is used to choose a split at this stage.
- Do not compare child totals without a common parent. Candidate splits at the same node share one parent total, so comparing their reductions is equivalent to comparing child totals. Scores from different parent nodes do not have that same direct interpretation.
- Do not average the two child errors without weighting by observations. The child sum of squares is a sum across observations. A child with more observations contributes its errors for all of them.
- Do not compare unrelated criteria. The definitions here use squared-error loss for a continuous response. A score calculated using a different loss is not the same reduction.
- Do not assume the largest immediate reduction guarantees the best final tree. This choice ranks candidates at the current step; later tree construction and evaluation raise additional questions.
Check Your Understanding
For each problem, use within-node sums of squares and compare candidate reductions on the stated common parent node.
- A node has responses \(1,3,5,7\). A candidate split forms children \(\{1,3\}\) and \(\{5,7\}\). Find its reduction. (Answer: 16)
- A node has responses \(2,4,8,10\). Candidate \(A\) forms \(\{2,4\}\) and \(\{8,10\}\); candidate \(B\) forms \(\{2,8\}\) and \(\{4,10\}\). Find both reductions and identify the preferred split. (Answer: \(A=36\), \(B=4\); choose \(A\))
- A node has responses \(1,2,4,8,9\). A threshold creates children \(\{1,2,4\}\) and \(\{8,9\}\). Calculate the parent sum of squares, the child total, and the reduction. (Answer: parent \(=50.8\), child total \(=31/6\), reduction \(=1369/30\), approximately \(45.6333\))
- At one node the parent sum of squares is \(52\). The best candidate for predictor \(A\) has child total \(19\), for \(B\) has child total \(12\), and for \(C\) has child total \(17\). Which predictor wins, and what is its reduction? (Answer: \(B\), reduction \(40\))
- A node has responses \(1,3,7,9\). Candidate \(U\) forms \(\{1\}\) and \(\{3,7,9\}\); candidate \(V\) forms \(\{1,3\}\) and \(\{7,9\}\). Calculate both reductions and choose the better candidate. (Answer: \(U=64/3\), \(V=36\); choose \(V\))