Threshold Splits for Numeric Predictors

Turning a Numeric Predictor Into a Binary Rule

A classification tree uses a split to divide the observations at a node into two groups. For a numeric predictor, the split is a cutoff: observations at or below the cutoff go to one child, and observations above it go to the other. The challenge is to identify which cutoffs could produce different groups in the training data, then compare those candidate splits using a chosen impurity criterion.

The candidate cutoffs depend on the predictor values observed at the node, not on the response labels. Once the candidates are formed, the response labels are used to assess the resulting child groups. This separation matters: sorting values generates the possible rules, while a split criterion evaluates their usefulness for classification.

Definition: Suppose the observations at a node have numeric predictor values whose distinct sorted values are $v_1 < v_2 < \cdots < v_k$. A threshold split at cutoff $c$ assigns an observation to the left child if its predictor value $x \leq c$, and to the right child if $x > c$. A standard set of candidate cutoffs is $$ c_j = \frac{v_j+v_{j+1}}{2}, \qquad j=1,\ldots,k-1. $$ If there is only one distinct predictor value, no threshold can divide the observations into two nonempty children.

The midpoint is a convenient representative of the gap between adjacent distinct values. Other cutoffs within that same gap would separate the training observations in exactly the same way, but the midpoint gives a simple, reproducible candidate. Equal predictor values must remain together: a rule based only on that predictor cannot send two observations with the same value to opposite children.

Why Midpoints Give the Relevant Candidate Splits

The midpoint rule follows from how a threshold partitions sorted observations. With the rule $x \leq c$, every observed value at or below $c$ goes left, and every value above $c$ goes right. If both children are nonempty, there is a last distinct value on the left, say $v_j$, and a next distinct value on the right, $v_{j+1}$. The cutoff must lie between these values, including $v_j$ but excluding $v_{j+1}$. Any cutoff in that interval creates the same partition of the observed cases.

v_j \leq c < v_{j+1} \quad\Longrightarrow\quad \{x \leq c\}=\{x \leq c_j\}, \qquad c_j=\frac{v_j+v_{j+1}}{2}.

Thus, checking one midpoint for each adjacent pair of distinct values is enough to represent every possible nonempty split of the training observations on that predictor. There are $k-1$ such candidates when there are $k$ distinct values. If some values are repeated, sorting the rows does not create extra candidates between those repeated values.

A candidate can be evaluated with any appropriate classification split criterion. For the examples below, use Gini impurity. If a node contains proportions $p_0$ and $p_1$ of two classes, its Gini impurity is $1-p_0^2-p_1^2$. For a candidate split, the weighted child impurity is the sum of each child’s impurity multiplied by that child’s fraction of the parent observations. The reduction is parent impurity minus weighted child impurity; a larger reduction is preferred.

G=1-p_0^2-p_1^2, \qquad G_{\text{weighted}} =\frac{n_L}{n}G_L+\frac{n_R}{n}G_R, \qquad \text{reduction}=G_{\text{parent}}-G_{\text{weighted}}.

This calculation assumes the observations being assessed belong to the same current node, each has a numeric predictor value and a known class label, and the split criterion is being applied to those observations. The criterion does not change how candidate thresholds are generated. If the chosen criterion is entropy rather than Gini impurity, the candidate cutoffs remain the same, although their ranking can differ.

Worked Example: Comparing Three Candidate Cutoffs

A hypothetical insurer is classifying submitted device claims as routine (R) or requiring additional review (V). For eight claims at one node, the numeric device-risk scores and labels, in sorted order, are: 10:R, 10:R, 20:R, 20:V, 30:V, 30:V, 40:V, 40:R. Generate and compare the candidate thresholds using Gini reduction.

Method. First sort and group the distinct scores. The distinct values are 10, 20, 30, and 40, so the candidate cutoffs are $(10+20)/2=15$, $(20+30)/2=25$, and $(30+40)/2=35$. There are four V claims and four R claims, so the parent Gini impurity is $1-(4/8)^2-(4/8)^2=0.5$.

Cutoff 15. The left child contains the two score-10 claims: 0 V and 2 R, so its Gini impurity is 0. The right child contains six claims: 4 V and 2 R. Its impurity is $1-(4/6)^2-(2/6)^2=1-16/36-4/36=16/36=0.4444$. The weighted child impurity is $(2/8)(0)+(6/8)(16/36)=1/3=0.3333$. The reduction is $0.5-0.3333=0.1667$.

Cutoff 25. The left child contains four claims: 1 V and 3 R. Its impurity is $1-(1/4)^2-(3/4)^2=6/16=0.375$. The right child also contains four claims: 3 V and 1 R, giving impurity $0.375$. The weighted child impurity is $(4/8)(0.375)+(4/8)(0.375)=0.375$. The reduction is $0.5-0.375=0.125$.

Cutoff 35. The left child contains six claims: 3 V and 3 R, with impurity $1-(3/6)^2-(3/6)^2=0.5$. The right child contains the two score-40 claims: 1 V and 1 R, also with impurity $0.5$. The weighted child impurity is $(6/8)(0.5)+(2/8)(0.5)=0.5$. The reduction is $0.5-0.5=0$.

The largest reduction is $0.1667$, so cutoff 15 is selected among these candidates. The resulting rule sends scores of 15 or less left and scores above 15 right. Check: the child counts are 2 and 6, which sum to the original 8 claims.

Common mistake avoided: comparing the two child impurities without weighting them by their sizes. For cutoff 15, the right child’s impurity is 0.4444, but it contains six of the eight claims; the relevant weighted impurity is 0.3333, not the unweighted average of the child impurities.

Tied Values and Valid Partitions

Repeated predictor values affect which partitions are possible. A threshold cannot separate two cases that have the same predictor value, even if their class labels differ. In practice, list the distinct predictor values, not every row, before generating candidate cutoffs.

Worked Example: Keeping Tied Scores Together

A hypothetical benefits administrator classifies applications as standard (S) or requiring review (V). At a node, the sorted score-label pairs are 12:S, 12:S, 18:V, 18:V, 24:S, 31:V, 31:V. Find the valid candidate cutoffs, compare their Gini reductions, and identify the preferred rule.

Generate candidates. The distinct scores are 12, 18, 24, and 31. Therefore, the candidates are 15, 21, and 27. There are 4 V and 3 S applications, so parent impurity is $1-(4/7)^2-(3/7)^2=24/49=0.4898$.

Cutoff 15. The left child has 0 V and 2 S, with impurity 0. The right child has 4 V and 1 S, with impurity $1-(4/5)^2-(1/5)^2=8/25=0.32$. Weighted child impurity is $(2/7)(0)+(5/7)(0.32)=8/35=0.2286$. The reduction is $24/49-8/35=0.2612$, rounded to four decimals.

Cutoff 21. The left child has 2 V and 2 S, with impurity $1-(2/4)^2-(2/4)^2=0.5$. The right child has 2 V and 1 S, with impurity $1-(2/3)^2-(1/3)^2=4/9=0.4444$. Weighted child impurity is $(4/7)(0.5)+(3/7)(4/9)=10/21=0.4762$. The reduction is $24/49-10/21=2/147=0.0136$.

Cutoff 27. The left child has 2 V and 3 S, with impurity $1-(2/5)^2-(3/5)^2=12/25=0.48$. The right child has 2 V and 0 S, with impurity 0. Weighted child impurity is $(5/7)(0.48)+(2/7)(0)=12/35=0.3429$. The reduction is $24/49-12/35=0.1469$.

Cutoff 15 has the largest reduction, so the preferred rule is score $\leq 15$ versus score $>15$. Both applications with score 18 go to the same child, as they must. Check: the three candidate partitions have child sizes 2 and 5, 4 and 3, and 5 and 2, respectively; each pair sums to 7.

Common mistake avoided: placing a candidate boundary between the two score-18 rows because they appear consecutively in a sorted list. Such a row-based boundary is not a valid threshold rule: equal scores cannot be assigned to different children using this predictor alone.

Reading the Selected Rule at the Boundary

A tree’s written rule must specify which child receives equality. With the convention used here, the left child receives $x \leq c$ and the right child receives $x>c$. The cutoff itself may not occur among the training values when it is a midpoint, but the rule still defines what happens for a new observation exactly at that cutoff. Always preserve the stated inequality when interpreting or applying the split.

Worked Example: Applying the Rule to New Scores

A hypothetical screening tree uses a numeric sensor index. At a node, the sorted observations are 0.2:N, 0.2:N, 0.5:V, 0.5:V, 0.9:V, 0.9:N, where N means no alert and V means alert. Use Gini reduction to choose a threshold, then assign new sensor indices 0.35, 0.60, and 0.90 to the resulting children.

Generate candidates and find the parent impurity. The distinct values are 0.2, 0.5, and 0.9, so the candidate cutoffs are $(0.2+0.5)/2=0.35$ and $(0.5+0.9)/2=0.70$. There are 3 N and 3 V observations, so parent impurity is $1-(3/6)^2-(3/6)^2=0.5$.

Evaluate 0.35. The left child contains two N observations and has impurity 0. The right child has three V and one N, so its impurity is $1-(3/4)^2-(1/4)^2=6/16=0.375$. Weighted child impurity is $(2/6)(0)+(4/6)(0.375)=0.25$. The reduction is $0.5-0.25=0.25$.

Evaluate 0.70. The left child contains two N and two V, with impurity 0.5. The right child contains one V and one N, also with impurity 0.5. Weighted child impurity is $(4/6)(0.5)+(2/6)(0.5)=0.5$. The reduction is $0.5-0.5=0$.

The selected cutoff is 0.35. Under the rule $x\leq0.35$ versus $x>0.35$, an index of 0.35 goes left, while 0.60 and 0.90 go right. Check: 0.35 is the midpoint between the observed values 0.2 and 0.5, and the observed training cases on the left are exactly the two cases with index 0.2.

Common mistake avoided: assuming that a midpoint is merely a label for the gap and has no effect on new observations. It represents the training partition, but it also defines a precise rule for future values, including a value equal to the cutoff.

Practical Checks and Common Traps

Before accepting a numeric split, verify that it is an implementable rule and that the comparison used to select it is consistent:

Sort within the current node. Candidate values come from the observations at the node being split, not automatically from the entire training set.
Use distinct values. Repeated values do not create additional candidate partitions.
Check both child sizes. Each candidate between adjacent distinct values should put at least one observed case in each child.
Weight child impurity by child size. A small child and a large child should not contribute equally unless they happen to be the same size.
Keep the inequality convention explicit. The rule $x\leq c$ on the left is different in wording from $x<c$, particularly for a new observation exactly equal to $c$.

A numeric split is a stepwise rule, not a claim that the response changes smoothly with the predictor. Choosing a threshold identifies a useful partition according to the specified criterion and the observations at that node. It does not establish that the cutoff has a causal or continuous-effect interpretation.

Key takeaway: Sort the distinct numeric predictor values at the node, form one midpoint between each adjacent pair, and evaluate each resulting partition with the chosen split criterion. Tied values stay together, and the selected cutoff must be read with its exact inequality.

Check Your Understanding

For each problem, identify the valid threshold rule or compare the candidates as requested.

A node has distinct numeric predictor values 4, 9, 15, and 26. List the midpoint candidates and state how many different nonempty threshold partitions they represent. (Answer: 6.5, 12, and 20.5; three partitions.)
A hypothetical claims node has score-label pairs 21:N, 21:N, 25:V, 30:V, 30:N, 41:V, 41:V. Using Gini reduction, which candidate among 23, 27.5, and 35 is preferred? (Answer: 23, with Gini reduction approximately 0.2612.)
A node has values 8:A, 8:B, 11:A, 19:B, 19:B. What are the valid midpoint candidates, and can a threshold on this predictor separate the two observations with value 19? (Answer: 9.5 and 15; no, the two value-19 observations must stay together.)
A fitted node rule is $x\leq 14.5$ versus $x>14.5$. Which child receives new predictor values 14.5 and 14.6? (Answer: 14.5 goes to the $x\leq14.5$ child; 14.6 goes to the $x>14.5$ child.)
A node has observations 0:N, 0:N, 2:V, 2:V, 5:N, 5:V. Compare the Gini reductions for candidates 1 and 3. (Answer: cutoff 1 is preferred; its reduction is 0.25, while cutoff 3 has reduction 0.)

What You'll Learn