Introduction
A regression tree selects a predictor and threshold by comparing how much each candidate split reduces squared prediction error. When two predictors are strongly correlated, they may sort observations similarly and offer almost the same division of the response values. The winning split can then depend on a small difference in the training data or on a tie-breaking convention, even when the competing trees fit nearly equally well.
This is not a contradiction: a tree must choose a particular split, but the data may provide little evidence that one of two correlated predictors is uniquely preferable. To assess such a choice, compare the complete training error after each split, not just the predictor names or the error in one child node.
A candidate split is specified by a predictor and a threshold. For example, \(X\leq c\) sends observations with \(X\leq c\) to one child and observations with \(X>c\) to the other. The tree chooses among admissible candidates by comparing their total within-child sums of squares, or equivalently their reductions from the same parent-node sum of squares. The response is treated as numerical, and the criterion here is ordinary, unweighted squared error. No normality assumption is needed to calculate this criterion.
Why the Reduction Formula Works
The child predictions that minimize squared error are their sample means. This follows because, for any set of responses, the sum of squared deviations from a proposed value is minimized at the sample mean. The error reduction can also be written directly in terms of the child sizes and means:
To justify the identity, let \(\bar y\) be the mean in the parent node. Within either child, expand the squared deviation from \(\bar y\) around that child’s mean. The cross-product sums to zero because the deviations from a child mean sum to zero. Thus the parent SSE equals the two child SSEs plus \(n_L(\bar y_L-\bar y)^2+n_R(\bar y_R-\bar y)^2\). Since \(\bar y=(n_L\bar y_L+n_R\bar y_R)/(n_L+n_R)\), substituting this weighted mean into the last two terms gives the displayed formula. Therefore a split is valuable when it separates children with different response means, while also accounting for the sizes of both children.
Near-Tied Splits From Correlated Predictors
The following hypothetical data illustrate how two predictors can be strongly correlated and yet define slightly different partitions. Consider six policy records, with \(X\) and \(Z\) representing two related measurements and \(Y\) a numerical outcome. The data are ordered by \(X\).
| Record | \(X\) | \(Z\) | \(Y\) |
|---|---|---|---|
| 1 | 1 | 1 | 1 |
| 2 | 2 | 2 | 2 |
| 3 | 3 | 4 | 5 |
| 4 | 4 | 3 | 5.2 |
| 5 | 5 | 5 | 8 |
| 6 | 6 | 6 | 9 |
Worked Example: Two Correlated Predictors, Similar Split Scores
Compare the split \(X\leq3\) with the split \(Z\leq3\). Calculate each split’s child SSE and reduction from the parent SSE, then identify the preferred split.
Method. Calculate the parent SSE once, then sum the SSEs of both children for each candidate. The Pearson correlation of \(X\) and \(Z\) is approximately \(0.9429\): both predictors have mean \(3.5\), their centered cross-product sum is \(16.5\), and each centered sum of squares is \(17.5\), giving \(16.5/\sqrt{17.5(17.5)}=0.9429\).
Parent node. The response sum is \(30.2\), so \(\bar y=30.2/6=5.0333\). The response squared sum is \(202.04\). Therefore,
Split on \(X\). The left responses are \(1,2,5\), with sum \(8\) and squared sum \(30\). Their SSE is \(30-8^2/3=8.6667\). The right responses are \(5.2,8,9\), with sum \(22.2\) and squared sum \(172.04\). Their SSE is \(172.04-22.2^2/3=7.76\). Thus the total child SSE is \(8.6667+7.76=16.4267\), and the reduction is \(50.0333-16.4267=33.6066\), or \(33.6067\) when rounded to four decimal places.
Split on \(Z\). The observations with \(Z\leq3\) have responses \(1,2,5.2\). Their sum is \(8.2\), and their squared sum is \(32.04\), so their SSE is \(32.04-8.2^2/3=9.6267\). The other child has responses \(5,8,9\), with sum \(22\) and squared sum \(170\); its SSE is \(170-22^2/3=8.6667\). Total child SSE is \(9.6267+8.6667=18.2934\), or \(18.2933\) using unrounded component values. The reduction is \(50.0333-18.2933=31.7400\).
The \(X\) split has the lower child SSE and the larger reduction, by about \(1.8667\). The two partitions differ only in whether records 3 and 4 are placed together or on opposite sides; their outcomes, \(5\) and \(5.2\), are close. The gap in split quality is modest relative to either reduction, so the observed preference does not establish that \(X\) is meaningfully superior to \(Z\). Common mistake avoided: treating a small score difference as strong evidence that the selected predictor is uniquely important.
Exact Ties and Equivalent Partitions
If two predictors induce the same partition of the observations, their split scores are identical under this criterion. A simple case occurs when one predictor is a strictly increasing transformation of another and there are no ties that change the possible groupings. For example, multiplying every value of a predictor by a positive constant preserves its ordering and therefore preserves its threshold-based partitions. The tree may still report one variable rather than the other, but that label can reflect a tie-breaking rule rather than a difference in fit.
Worked Example: Duplicate Split Information
In a hypothetical review of four accounts, two recorded predictors are \(U=(10,20,30,40)\) and \(V=(100,200,300,400)\). Their response values in that order are \(Y=(2,4,8,10)\). Compare the candidate splits \(U\leq25\) and \(V\leq250\).
Parent error. The response mean is \((2+4+8+10)/4=6\). The parent SSE is \((2-6)^2+(4-6)^2+(8-6)^2+(10-6)^2=16+4+4+16=40\).
First split. \(U\leq25\) places responses \(2,4\) in the left child and \(8,10\) in the right child. The means are \(3\) and \(9\). The child SSEs are \((2-3)^2+(4-3)^2=2\) and \((8-9)^2+(10-9)^2=2\). Total child SSE is \(4\), giving a reduction of \(40-4=36\).
Second split. \(V\leq250\) places exactly the same two pairs in the two children. Its child means, SSEs, total SSE of \(4\), and reduction of \(36\) are therefore identical. Both splits predict \(3\) for the first pair and \(9\) for the second pair.
This is an exact tie in both training error and fitted values. Common mistake avoided: interpreting the variable name selected by a software implementation as evidence that it predicts better. Here the observed data support the same partition under either name.
High Correlation Does Not Guarantee an Equivalent Split
Correlation summarizes the overall linear association between two numerical predictors; it does not say that every threshold on one predictor reproduces a particular threshold partition on the other. A small local change in ordering near a useful threshold can matter greatly if it separates observations with very different responses. The response arrangement, not predictor correlation alone, determines the squared-error reduction.
Worked Example: Correlated Predictors With Very Different Gains
Use the same \(X\) and \(Z\) values as in the first example, but suppose the hypothetical outcomes are \(Y=(0,0,0,10,10,10)\). Compare \(X\leq3\) and \(Z\leq3\).
Parent error. The response mean is \(5\). Three responses are \(0\) and three are \(10\), so the parent SSE is \(3(0-5)^2+3(10-5)^2=75+75=150\).
Split on \(X\). The left child contains the three zeros and the right child contains the three tens. Each child has SSE \(0\), so total child SSE is \(0\), and the reduction is \(150-0=150\).
Split on \(Z\). The left child, \(Z\leq3\), contains records 1, 2, and 4, with outcomes \(0,0,10\). Its mean is \(10/3\), and its SSE is \(0^2+0^2+10^2-10^2/3=200/3\). The right child has outcomes \(0,10,10\), mean \(20/3\), and SSE \(10^2+10^2-20^2/3=200/3\). Total child SSE is \(400/3\), or \(133.3333\), and the reduction is \(150-400/3=50/3\), or \(16.6667\).
The predictors still have correlation approximately \(0.9429\), but their candidate splits differ substantially in fit. Record 4, with outcome \(10\), is grouped with two zeros by the \(Z\) threshold, while the \(X\) threshold separates all zeros from all tens. Common mistake avoided: assuming a high predictor-predictor correlation by itself guarantees similar split scores. It creates the possibility of redundant split information, not a guarantee that every competing threshold is equally good.
Interpreting a Selected Variable
A tree reports a specific variable and threshold because it needs a concrete rule for assigning observations to children. When correlated predictors are close substitutes for that division, small changes in outcomes, observations, candidate thresholds, or implementation details can change which variable appears. The resulting tree structure may look different even though the training errors and predictions are similar.
A useful interpretation separates two questions:
- Which split minimizes the criterion in this sample? Answer this by computing the complete child SSE or its reduction.
- Does the selected predictor have a uniquely strong role? A near tie with a correlated alternative is a warning that the data do not clearly distinguish the two choices at this node.
The second question cannot be answered from the winning split score alone. A selected predictor is part of the tree’s rule, but a small numerical advantage does not establish a causal relationship or unique predictive relevance. In particular, do not infer that the predictor chosen first is necessarily the most important variable in every population or future data set.
Common Calculation Trap
A split must be assessed using the sum of the errors in both children. Comparing only the left-child SSE can make an unbalanced split look attractive simply because a singleton child has zero error.
For the outcomes in the previous example, consider a split after the first observation: the left child contains one zero and has SSE \(0\), but the right child contains \(0,0,10,10,10\). Its mean is \(6\), so its SSE is \(2(0-6)^2+3(10-6)^2=72+48=120\). Total child SSE is \(120\), and the reduction is \(150-120=30\). By contrast, the split after the third observation has both child SSEs equal to zero and reduction \(150\).
Wrong calculation: “Both splits have left-child SSE zero, so they are equally good.” Correction: include the right-child SSE before comparing. The first split has total error \(120\); the second has total error \(0\). A child’s error is not the split’s error.
Check Your Understanding
For each problem, compare the full split errors before drawing a conclusion about the chosen predictor.
- A node has outcomes \(1,3,7,9\). Split A forms groups \((1,3)\) and \((7,9)\); split B forms groups \((1,7)\) and \((3,9)\). Calculate both total child SSEs and identify the better split. (Answer: Split A has SSE \(4\), split B has SSE \(36\); A is better by \(32\).)
- Two predictors give exactly the same partition of a node, whose parent SSE is \(58\). The shared partition has total child SSE \(13\). What is the reduction for each predictor, and how should a software tie between them be interpreted? (Answer: Each reduction is \(58-13=45\); neither predictor has a better training fit under this split.)
- For two candidate splits from the same parent, the first has child SSEs \(6\) and \(19\), while the second has child SSEs \(11\) and \(12\). Which split has the larger reduction, and by how much? (Answer: Both total SSEs equal \(25\), so the reductions are tied and the difference is \(0\).)
- A parent node has SSE \(80\). A split has left and right child sizes \(4\) and \(6\), with means \(2\) and \(7\). Use the reduction formula to calculate the child SSE and check the result with the parent SSE. (Answer: Reduction \(=(4)(6)(7-2)^2/10=60\); child SSE \(=80-60=20\).)
- Two predictors have a high correlation, but one threshold gives child SSE \(8+12\), while a threshold on the other gives \(5+20\), from a parent SSE of \(50\). Which split is selected by squared-error reduction, and what is the difference in reductions? (Answer: The first split has SSE \(20\) and reduction \(30\); the second has SSE \(25\) and reduction \(25\); the first is preferred by \(5\).)