Repeated Predictor Values and Tied Split Scores

Why Repeated Values and Ties Matter

A regression tree divides the predictor space into regions and predicts a constant response in each terminal region. For a numeric predictor, a split sends observations to one side or the other of a threshold. When predictor values repeat, not every position in a sorted list corresponds to a possible split: observations with the same predictor value must stay together. Even among valid splits, two different thresholds can have exactly the same improvement in squared error. Recognizing both facts is important when interpreting how a tree was built.

This tutorial considers unweighted regression trees that choose splits by reducing the within-node sum of squared errors. For a node containing responses \(y_1,\ldots,y_n\), its prediction is the sample mean, and its within-node sum of squares measures the errors around that mean.

Definition: For a node \(A\) containing \(n_A\) observations, its mean response and within-node sum of squares are

\bar{y}_A=\frac{1}{n_A}\sum_{i\in A}y_i, \qquad \operatorname{SSE}(A)=\sum_{i\in A}(y_i-\bar{y}_A)^2.

A candidate split is a partition of the node into a left child \(L\) and a right child \(R\). Its improvement is the parent sum of squares minus the sum of the two child sums of squares:

\Delta=\operatorname{SSE}(A)-\operatorname{SSE}(L)-\operatorname{SSE}(R).

A larger \(\Delta\) is a better split under this criterion. These definitions assume squared-error splitting with equal observation weights.

Which Thresholds Are Actually Candidates?

Suppose the distinct observed values of a numeric predictor in a node are \(u_1<u_2<\cdots<u_K\). A threshold can separate observations only at a boundary between consecutive distinct values. A common convention places the threshold at their midpoint, \((u_j+u_{j+1})/2\), and sends values below it left and values at or above it right. Other threshold values in the gap give the same partition. The midpoint is a convenient representative.

Repeated observations at a value belong to the same side of every threshold based on that predictor. Thus, if the sorted predictor values are \(1,1,2,2,4,4\), the distinct candidate boundaries are between 1 and 2 and between 2 and 4—not between the two observations with value 1, for example. A threshold is about predictor values, not positions in a list after sorting.

Worked Example: Finding Valid Boundaries and a Score Tie

A hypothetical claim-severity model has predictor values \(x=(1,1,2,2,4,4)\) and responses \(y=(0,2,4,6,0,2)\). Find the valid thresholds, calculate their improvements, and determine whether the best split is unique.

Method. First group the responses by distinct predictor value. The groups are \((0,2)\) at \(x=1\), \((4,6)\) at \(x=2\), and \((0,2)\) at \(x=4\). The only candidate thresholds are \(1.5\) and \(3\).

Parent error. The response total is \(14\), the sum of squared responses is \(60\), and the sample size is \(6\). Using \(\operatorname{SSE}=\sum y_i^2-(\sum y_i)^2/n\),

\operatorname{SSE}(A)=60-\frac{14^2}{6} =60-\frac{98}{3} =\frac{82}{3}.

Threshold \(1.5\). The left child has responses \((0,2)\), mean \(1\), and SSE \(2\). The right child has responses \((4,6,0,2)\), total \(12\), sum of squares \(56\), and mean \(3\). Its SSE is \(56-12^2/4=20\). Therefore the child total is \(2+20=22\), and

\Delta_{1.5}=\frac{82}{3}-22=\frac{16}{3}.

Threshold \(3\). The left child has responses \((0,2,4,6)\), total \(12\), sum of squares \(56\), and mean \(3\), so its SSE is \(56-12^2/4=20\). The right child has responses \((0,2)\), mean \(1\), and SSE \(2\). The child total is again \(22\), giving \(\Delta_3=82/3-22=16/3\).

Both valid thresholds have exactly the same improvement, \(16/3\). The tie follows from the matching child errors: the two resulting pairs of child SSEs are \(2\) and \(20\), in opposite order. The threshold \(1.5\) predicts \(1\) for \(x=1\) and \(3\) for \(x=2\) or \(4\); the threshold \(3\) predicts \(3\) for \(x=1\) or \(2\) and \(1\) for \(x=4\). These are different partitions and different predictions, despite the tied scores. The parent SSE also checks the calculation: \(82/3\) is greater than either child total, \(22\).

Why the Split-Improvement Formula Works

A useful way to calculate a score is to compare the child means. Let the parent contain \(n=n_L+n_R\) observations, with child means \(\bar{y}_L\) and \(\bar{y}_R\), and parent mean \(\bar{y}\). Within each child, write a response’s deviation from the parent mean as its deviation from its child mean plus the difference between the child and parent means. The cross-product terms sum to zero because deviations from each child mean sum to zero. Consequently,

\operatorname{SSE}(A) = \operatorname{SSE}(L)+\operatorname{SSE}(R) +n_L(\bar{y}_L-\bar{y})^2+n_R(\bar{y}_R-\bar{y})^2.

The parent mean is the size-weighted average of the child means: \(n\bar{y}=n_L\bar{y}_L+n_R\bar{y}_R\). Substituting \(\bar{y}=(n_L\bar{y}_L+n_R\bar{y}_R)/n\) into the last two terms and simplifying gives

\Delta =n_L(\bar{y}_L-\bar{y})^2+n_R(\bar{y}_R-\bar{y})^2 =\frac{n_Ln_R}{n}(\bar{y}_L-\bar{y}_R)^2.

This identity explains why a split improves squared error when the child means differ. The improvement is zero if their means are equal, and it is larger when the difference in means is larger, with the factor \(n_Ln_R/n\) accounting for child sizes. It is equivalent to subtracting the child SSEs directly, provided the split and means are calculated correctly.

Worked Example: Comparing Several Candidate Boundaries

A hypothetical risk model has predictor values \(x=(0,0,1,1,2,2,3,3)\) and responses \(y=(1,3,-1,1,-1,1,1,3)\). Calculate the parent SSE and compare all valid thresholds. Identify the tied best candidates.

Method. The distinct predictor values are \(0,1,2,3\), so the thresholds are \(0.5,1.5,2.5\). The response total is \(8\), the sum of squared responses is \(24\), and \(n=8\). Thus the parent mean is \(1\), and

\operatorname{SSE}(A)=24-\frac{8^2}{8}=16.

Threshold \(0.5\). The left responses \((1,3)\) have mean \(2\) and SSE \(2\). The right responses \((-1,1,-1,1,1,3)\) have total \(4\) and sum of squares \(14\), so their mean is \(4/6=2/3\) and their SSE is \(14-4^2/6=14-8/3=34/3\). The child total is \(2+34/3=40/3\); the improvement is \(16-40/3=8/3\).

Threshold \(1.5\). The left responses \((1,3,-1,1)\) have total \(4\), sum of squares \(12\), and mean \(1\). Their SSE is \(12-4^2/4=8\). The right responses \((-1,1,1,3)\) also have total \(4\), sum of squares \(12\), mean \(1\), and SSE \(8\). The child total is \(16\), so the improvement is \(0\).

Threshold \(2.5\). The left responses \((1,3,-1,1,-1,1)\) have total \(4\) and sum of squares \(14\), giving SSE \(14-4^2/6=34/3\). The right responses \((1,3)\) have mean \(2\) and SSE \(2\). The child total is \(40/3\), so the improvement is \(8/3\).

The best score is \(8/3\), attained by both \(0.5\) and \(2.5\). The middle split has equal child means and therefore zero improvement, which also follows from the derived formula. As a check, each child total is no greater than the parent SSE of \(16\).

How a Tied Choice Can Change the Tree

A split score evaluates the immediate reduction in error, not whether two candidate partitions assign the same observations to children. Equal scores therefore do not imply identical trees. The next example isolates the effect of a tie on the root decision. It does not assume that the tied choices will remain tied at later nodes: after a root split, the observations available for a later split differ.

Worked Example: Two Tied Root Choices, Two Prediction Rules

For a hypothetical portfolio study, a numeric rating \(x\) takes values \((1,1,3,3,6,6)\), with responses \((2,4,8,10,2,4)\). Find the best root split and state the prediction rule for each tied choice.

Method. The distinct values are \(1,3,6\), so the only thresholds are \(2\) and \(4.5\). The total response is \(30\), the sum of squared responses is \(204\), and the parent mean is \(30/6=5\). Its SSE is \(204-30^2/6=204-150=54\).

Threshold \(2\). The left responses \((2,4)\) have mean \(3\) and SSE \(2\). The right responses \((8,10,2,4)\) have total \(24\), mean \(6\), and sum of squares \(184\). Their SSE is \(184-24^2/4=184-144=40\). The child total is \(42\), and the improvement is \(54-42=12\).

Threshold \(4.5\). The left responses \((2,4,8,10)\) have total \(24\), mean \(6\), and SSE \(184-24^2/4=40\). The right responses \((2,4)\) have mean \(3\) and SSE \(2\). The child total is \(42\), and the improvement is again \(54-42=12\).

Both thresholds are tied for the largest improvement. If the tree chooses \(2\), it predicts \(3\) when \(x<2\) and \(6\) when \(x\geq2\). If it chooses \(4.5\), it predicts \(6\) when \(x<4.5\) and \(3\) when \(x\geq4.5\). These are different root partitions and different predictions for the observed rating \(x=3\). The arithmetic checks: both child totals, \(42\), are below the parent SSE, \(54\), by \(12\).

Common Mistakes and Practical Interpretation

A frequent error is to sort the rows and try a split between every adjacent pair, even when the predictor values on either side are equal. That creates a partition no threshold on the predictor can implement: identical predictor values would be sent to different children. For example, the two observations with \(x=1\) in the preceding example cannot be separated by an \(x\)-threshold. A valid candidate list uses the distinct ordered values, not the full sorted row list.

A second error is to treat close displayed scores as an exact tie, or to miss an exact tie because of rounded arithmetic. Keep exact fractions or sufficient precision while comparing candidates, and round only when reporting. In the second worked example, the endpoint improvements are exactly \(8/3\), while the middle improvement is exactly \(0\); displaying \(2.67\) for both endpoint scores does not make them approximate or different.

A third error is to compare one child’s SSE in isolation. The split criterion uses the sum of both child errors, subtracted from the parent error. A child with a small SSE may simply contain fewer observations. Always compare the full child total, or use the improvement formula with both child sizes and means.

Key takeaway: Repeated predictor values restrict candidate splits to boundaries between distinct values. Compute each valid split’s total improvement, and retain all candidates with the best exact score. A tie means equal immediate squared-error reduction—not identical partitions, predictions, or subsequent tree structure. If a fitting procedure returns just one tied split, its tie-breaking convention determines which equally scoring tree is selected.

Check Your Understanding

For each problem, identify only realizable thresholds before comparing their scores. Use unweighted squared-error splitting.

A node has \(x=(2,2,5,5)\) and \(y=(1,3,5,7)\). State the valid midpoint threshold and calculate the parent SSE, both child SSEs, and the improvement. (Answer: threshold \(3.5\); parent SSE \(20\); child SSEs \(2\) and \(2\); improvement \(16\).)
A node has distinct predictor values \(0,2,5\), with two observations at each value. How many distinct midpoint candidates are there, and can a numeric threshold separate the two observations that share \(x=2\)? (Answer: two candidates, \(1\) and \(3.5\); no.)
For \(x=(1,1,2,2,3,3)\) and \(y=(0,2,2,4,0,2)\), list the candidate thresholds and calculate the improvement for each. (Answer: thresholds \(1.5\) and \(2.5\); improvements \(4/3\) and \(4/3\), a tie.)
A candidate split has \(n_L=4\), \(n_R=6\), and child means \(2\) and \(5\). Calculate its squared-error improvement without computing individual residuals. (Answer: \((4\cdot6/10)(2-5)^2=21.6\).)
Two valid thresholds have exact improvements \(7.125\) and \(7.1246\). A report rounds both to \(7.12\). Are the candidate scores tied, and which is better under squared-error reduction? (Answer: They are not tied; the candidate with improvement \(7.125\) is better.)

What You'll Learn