Why Numeric Split Points Need a Search
A tree split on a numeric predictor divides observations according to a rule such as \(x \le c\) versus \(x > c\), where \(c\) is a threshold. The challenge is not to test every possible real number for \(c\): almost all thresholds produce exactly the same partition. The useful candidates are determined by the observed predictor values.
Once a candidate threshold is chosen, the resulting child nodes can be compared using the criteria already used for tree fitting: squared error for a regression tree, or a classification impurity such as entropy or Gini impurity. The split criterion must be weighted by the number of observations in each child. A small child and a large child do not contribute equally merely because each is one node.
For example, if the distinct observed values are 2, 5, and 8, the standard midpoint candidates are 3.5 and 6.5. There is no need to test 4, 4.2, or 4.9: each would send the same observations to the same children. The midpoint is a convenient representative of that entire gap, not a uniquely optimal real number within it.
Ties matter. If several observations have \(x=5\), a threshold cannot separate some of those observations from the others using this rule. A split between distinct observed values preserves all tied values in the same child. If the predictor has only one distinct observed value, it provides no candidate split in that node.
What Makes a Candidate Split Better?
For regression, a tree commonly chooses the split with the smallest total within-child sum of squared errors. Each child predicts its own response mean, so the criterion is the sum of the squared deviations from the respective child means. Comparing this sum across candidate splits is equivalent to choosing the split with the largest reduction from the parent-node sum of squared errors.
For classification, the same search over thresholds is used, but the child partitions are evaluated with a classification impurity. For entropy, if a node has class proportions \(p_1,\ldots,p_C\), its entropy is \(-\sum_{r=1}^{C} p_r \log_2(p_r)\), with a zero contribution for any \(p_r=0\). If the parent has \(n\) observations and the children have \(n_L\) and \(n_R\), the weighted child entropy is \((n_L/n)H_L+(n_R/n)H_R\). The entropy gain is parent entropy minus weighted child entropy. Gini impurity can be used in the same weighted comparison, replacing entropy with \(1-\sum_r p_r^2\).
Here \(I_L\) and \(I_R\) are the relevant child criteria: squared-error totals or a classification impurity, depending on the tree. When comparing splits for regression, it is also common to add child squared errors directly. When comparing classification impurities, use the weighted average above, or equivalently the weighted sum \(n_L I_L+n_R I_R\) when every candidate is being compared within the same parent node. Do not compare an unweighted average of child criteria when child sizes differ.
Worked Example: Scanning Regression Cut Points
A hypothetical equipment-maintenance team is using a regression tree to predict a repair-duration score from a numeric sensor reading. Four training observations have readings \(x=1,2,4,7\) and response scores \(y=2,3,8,9\). Find the candidate thresholds, calculate each split's child squared error, and select the best split. Use the sum of squared errors within the children as the criterion.
Method and candidates. The distinct readings are already ordered: 1, 2, 4, 7. The candidate thresholds are the adjacent midpoints: \(c_1=(1+2)/2=1.5\), \(c_2=(2+4)/2=3\), and \(c_3=(4+7)/2=5.5\).
Parent reference. The response mean is \((2+3+8+9)/4=22/4=5.5\). The parent sum of squared errors is \((2-5.5)^2+(3-5.5)^2+(8-5.5)^2+(9-5.5)^2 =12.25+6.25+6.25+12.25=37\).
Threshold 1.5. The left child contains only \(y=2\), so its squared error is 0. The right child contains \(3,8,9\), with mean \(20/3\). Its squared error is \((3-20/3)^2+(8-20/3)^2+(9-20/3)^2 =121/9+16/9+49/9=186/9=62/3\), approximately 20.6667. The total child squared error is therefore \(62/3\).
Threshold 3. The left child contains \(2,3\), with mean \(2.5\) and squared error \((2-2.5)^2+(3-2.5)^2=0.25+0.25=0.5\). The right child contains \(8,9\), with mean \(8.5\) and squared error \((8-8.5)^2+(9-8.5)^2=0.25+0.25=0.5\). The total is \(0.5+0.5=1\).
Threshold 5.5. The left child contains \(2,3,8\), with mean \(13/3\). Its squared error is \((2-13/3)^2+(3-13/3)^2+(8-13/3)^2 =49/9+16/9+121/9=186/9=62/3\). The right child contains only \(9\), with squared error 0. The total is \(62/3\), approximately 20.6667.
Decision and check. The three totals are \(62/3\), 1, and \(62/3\). The best candidate is \(c=3\), with total child squared error 1. Its reduction from the parent error is \(37-1=36\), which is positive and greater than the reductions from either other candidate. The common mistake this calculation avoids is choosing a split because one child looks tight while ignoring the other child: the criterion sums the errors from both children.
Why the Regression Reduction Has This Form
The squared-error reduction has a useful interpretation: it measures the separation between the child means, scaled by the sizes of the children. This explains why a split can improve predictions even when neither child has zero error.
To see why, for an observation in child \(L\), write \(y_i-\bar y=(y_i-\bar y_L)+(\bar y_L-\bar y)\), square, and sum. The cross-product vanishes because \(\sum_{i\in L}(y_i-\bar y_L)=0\). The same expansion applies to child \(R\). Also, the parent mean satisfies \(n_L(\bar y_L-\bar y)+n_R(\bar y_R-\bar y)=0\). Substituting this weighted-mean relation into the final two terms gives
Thus, the reduction in squared error is exactly \((n_Ln_R/n)(\bar y_L-\bar y_R)^2\). It is zero if the child means are equal, and it increases with their separation while still reflecting the sizes of both children. This identity applies when each child prediction is its response mean and the criterion is unweighted squared error. It does not automatically apply to a different loss function or to a model that uses different leaf predictions.
Worked Example: Comparing Entropy Gains for a Numeric Predictor
A hypothetical claims-screening team considers a numeric signal \(x\) to classify cases as class A or class B. A summarized training set has 20,000 cases in three predictor-value groups. The table gives the class counts. Compare the candidate thresholds and choose the one with the larger entropy gain.
| Predictor value | Class A | Class B | Total |
|---|---|---|---|
| 1 | 2,257 | 7,743 | 10,000 |
| 2 | 5,743 | 257 | 6,000 |
| 3 | 2,000 | 2,000 | 4,000 |
Method and parent entropy. The candidate midpoints are \(1.5\) and \(2.5\). The parent contains 10,000 cases of each class, so its class proportions are \(0.5\) and \(0.5\), and its entropy is \(-0.5\log_2(0.5)-0.5\log_2(0.5)=1\) bit.
Threshold 1.5. The left child has 10,000 cases, with class proportions \(0.2257\) and \(0.7743\). The right child has 10,000 cases, with the proportions reversed. Since reversing the class proportions does not change entropy, both child entropies are \(-0.2257\log_2(0.2257)-0.7743\log_2(0.7743)\), approximately \(0.7704\) bits. Both children have weight \(10{,}000/20{,}000=0.5\), so weighted child entropy is \(0.5(0.7704)+0.5(0.7704)=0.7704\). The gain is \(1-0.7704=0.2296\) bits, rounded to four decimal places.
Threshold 2.5. The left child combines predictor values 1 and 2. It has 8,000 cases of class A and 8,000 of class B, so its entropy is 1 bit. The right child has 2,000 cases of each class, so its entropy is also 1 bit. The weighted child entropy is \((16{,}000/20{,}000)(1)+(4{,}000/20{,}000)(1)=0.8+0.2=1\) bit. The gain is \(1-1=0\) bits.
Decision and check. Threshold 1.5 has the larger entropy gain, \(0.2296\) versus 0 bits. The child counts sum to the parent counts: class A has \(2{,}257+5{,}743+2{,}000=10{,}000\), and class B has \(7{,}743+257+2{,}000=10{,}000\). The common mistake avoided here is reporting the child entropy \(0.7704\) as the gain. Gain is parent entropy minus weighted child entropy, so the correct calculation is \(1-0.7704=0.2296\).
Ties, Midpoints, and Equivalent Thresholds
The midpoint rule keeps the candidate list manageable and makes the role of ties explicit. Any threshold strictly between the same two consecutive distinct predictor values produces the same left and right membership. The midpoint is a reproducible choice, but it does not create a different partition from another threshold in that gap.
Worked Example: Respecting Tied Predictor Values
A hypothetical service center predicts a numeric workload score from a sensor reading. The five training observations are \((x,y)=(2,4),(2,6),(5,5),(5,7),(8,10)\). Generate all candidates and choose the one with the smallest total child squared error. In particular, do not divide observations that share a sensor reading.
Candidate thresholds. The distinct predictor values are 2, 5, and 8, so the only candidate midpoints are \((2+5)/2=3.5\) and \((5+8)/2=6.5\). The two observations with \(x=2\) must stay together, as must the two with \(x=5\).
Parent reference. The response total is \(4+6+5+7+10=32\), so the parent mean is \(32/5=6.4\). The parent squared error is \((4-6.4)^2+(6-6.4)^2+(5-6.4)^2+(7-6.4)^2+(10-6.4)^2 =5.76+0.16+1.96+0.36+12.96=21.20\).
Threshold 3.5. The left child contains responses 4 and 6. Its mean is 5 and its squared error is \((4-5)^2+(6-5)^2=2\). The right child contains 5, 7, and 10. Its mean is \(22/3\), and its squared error is \((5-22/3)^2+(7-22/3)^2+(10-22/3)^2 =49/9+1/9+64/9=114/9=38/3\). The total is \(2+38/3=44/3\), approximately 14.6667.
Threshold 6.5. The left child contains 4, 6, 5, and 7, with mean \(22/4=5.5\). Its squared error is \(2.25+0.25+0.25+2.25=5\). The right child contains only 10 and has squared error 0. The total is 5.
Decision and check. The threshold \(6.5\) is preferred because its total child squared error, 5, is less than \(44/3\). Its reduction from the parent error is \(21.20-5=16.20\). No candidate split can put one observation with \(x=5\) on the left and the other on the right: doing so would not be a threshold split on this predictor. The common mistake avoided is treating tied predictor values as though an arbitrary row ordering could separate them.
A Reliable Candidate-Search Process
Within the node being considered, sort the numeric predictor values and identify the distinct values. Keep the response or class label attached to each observation.
For each adjacent pair of distinct values, use the midpoint as a representative cut point. There are no candidate gaps within a tied value.
Apply the same rule, \(x \le c\) versus \(x>c\), and record the size and responses or class counts for each child.
For regression, add child squared errors. For classification, calculate each child impurity and weight it by that child's share of the parent observations.
Select the candidate with the smallest weighted child criterion, or equivalently the greatest reduction or gain from the parent criterion. Check that the child counts add back to the parent.
Common Errors and Practical Checks
- Testing thresholds inside a gap repeatedly. Thresholds in the same gap create the same partition, so they do not need separate evaluation.
- Splitting tied values. A numeric threshold sends all observations with the same predictor value to the same child. Do not invent a split based on their row order.
- Ignoring child sizes. Classification impurities must be weighted by child size. An unweighted average can favor a split simply because it has two children, regardless of how many observations each contains.
- Confusing impurity with improvement. A low child impurity is not itself the gain. Subtract the weighted child criterion from the parent criterion.
- Comparing inconsistent criteria. Use the same response scale, loss, and impurity definition for every candidate within a search. A threshold comparison is meaningful only when the candidate partitions are scored consistently.
In practice, a numeric predictor with \(k\) distinct values in a node has at most \(k-1\) candidate gaps before any additional tree-growing restrictions are applied. The tree does not choose a cut point by numerical proximity to an ideal value; it evaluates the partitions those gaps create. This makes the search interpretable: a selected threshold identifies a boundary between observed predictor values, and its merit comes from the resulting child predictions or class compositions.
Check Your Understanding
For each problem, identify the candidate thresholds and apply the requested comparison. Round entropy and gains to four decimal places where needed.
- A regression node has distinct predictor values 3, 6, and 10, with responses \(2,5\) at \(x=3\), response \(4\) at \(x=6\), and responses \(8,10\) at \(x=10\). List the candidate midpoints. (Answer: 4.5 and 8.)
- A numeric predictor takes values 1, 1, 4, 4, 9 in one node. How many candidate gaps are there, and can a threshold split the two observations with value 4? (Answer: 2 candidate gaps; no.)
- A regression parent has total squared error 25. A proposed split has child squared errors 3 and 10. Calculate the split's total child error and its reduction from the parent. (Answer: total child error 13; reduction 12.)
- A classification parent has entropy 1 bit. A candidate creates two equally sized children, each with entropy 0.5 bits. Calculate weighted child entropy and information gain. (Answer: weighted child entropy 0.5 bits; gain 0.5 bits.)
- A classification node has 12 observations. Candidate A creates children of sizes 3 and 9 with entropies 0 and 0.8 bits. Candidate B creates children of sizes 6 and 6 with entropies 0.4 and 0.4 bits. Which candidate has lower weighted child entropy? (Answer: Candidate A, \( (3/12)(0)+(9/12)(0.8)=0.6000\) bits versus Candidate B, \(0.4000\) bits; therefore Candidate B is lower.)