Tutorials › Exam SRM - Statistics for Risk Modeling › Worked Comparison of Candidate Thresholds

Home  /  Tutorials  /  Exam SRM / Regression Trees

Exam SRM · Regression Trees · Tutorial 569 of 926

Worked Comparison of Candidate Thresholds

Learn how to enumerate valid threshold partitions, calculate their squared-error improvements, and choose the best candidate without splitting identical predictor values.

Intermediate 10 min read

What You'll Learn

  • Identify distinct predictor values and enumerate the partitions they permit
  • Calculate a node’s mean prediction and total squared error
  • Compare candidate thresholds using child errors and parent improvement
  • Apply a squared-error decomposition to understand split improvement
  • Recognize why repeated predictor values cannot be separated by a threshold
  • Check a selected threshold by verifying its partition and error calculations

Why Compare Candidate Thresholds?

A regression tree predicts a constant value within each terminal node. When a node is split using a numeric predictor, a threshold sends observations on one side to the left child and observations on the other side to the right. The practical question is not simply where a threshold can be placed, but which valid partition reduces the squared prediction error the most.

Repeated predictor values matter: a threshold rule must send every observation with the same predictor value to the same child. Thus, possible splits are determined by the distinct observed predictor values, not by the number of observations. The comparison below gives a systematic way to enumerate those splits, calculate their errors, and select the best one.

Definition: For observations in a node with responses \(y_1,\ldots,y_n\), the node prediction under squared-error loss is the response mean \(\bar{y}=\frac{1}{n}\sum_{i=1}^n y_i\). The node’s sum of squared errors (SSE) is \(\sum_{i=1}^n(y_i-\bar{y})^2\). For a numeric predictor \(x\), a threshold \(c\) creates the partition \(x\leq c\) and \(x>c\). Its split SSE is the left-child SSE plus the right-child SSE.

The mean is used because it minimizes the sum of squared deviations from a constant prediction. For any proposed prediction \(a\), expand the squared deviations around the sample mean:

$$ \sum_{i=1}^n (y_i-a)^2 = \sum_{i=1}^n\bigl((y_i-\bar{y})+(\bar{y}-a)\bigr)^2 = \sum_{i=1}^n(y_i-\bar{y})^2+n(\bar{y}-a)^2. $$

The cross-product term is zero because \(\sum_i(y_i-\bar{y})=0\). The final term is nonnegative and equals zero when \(a=\bar{y}\), which justifies using the mean within each node. This calculation requires no distributional assumption: it is an exact property of squared-error fitting to the observed responses.

Enumerating Valid Thresholds

Sort the distinct observed values of the predictor. A valid threshold can be placed in each gap between two consecutive distinct values. Any threshold in the same gap gives the same assignment of observations to the children, so it gives the same child means and SSE. A convenient representative is the midpoint of the two values.

Key result: If the distinct predictor values in a node are \(z_1<z_2<\cdots<z_k\), there are at most \(k-1\) distinct threshold partitions. A threshold between \(z_j\) and \(z_{j+1}\) separates all observations with \(x\leq z_j\) from all observations with \(x\geq z_{j+1}\). Repeated observations at one predictor value always remain together.

For each candidate, calculate the mean response in each child and then that child’s SSE. Add the two child errors to obtain the split SSE. The parent SSE provides a common baseline: the reduction in training error, or improvement, is parent SSE minus split SSE. The selected candidate is the valid threshold with the largest improvement, equivalently the smallest split SSE.

$$ \text{Improvement}(c)=\text{Parent SSE}-\bigl(\text{Left SSE}(c)+\text{Right SSE}(c)\bigr). $$

Worked Comparisons

Worked Example: Repeated Values Produce Two Candidate Partitions

A hypothetical claims review uses a numeric complexity score \(x\) to predict review hours \(y\). The six observations are \((1,2),(1,4),(3,6),(3,8),(5,10),(5,12)\). Enumerate the valid thresholds and select the best split by squared-error improvement.

Step 1: Find the parent mean and SSE. The response sum is \(2+4+6+8+10+12=42\), so the parent mean is \(42/6=7\). The parent SSE is \( (2-7)^2+(4-7)^2+(6-7)^2+(8-7)^2+(10-7)^2+(12-7)^2 =25+9+1+1+9+25=70.\)

Step 2: Enumerate the partitions. The distinct predictor values are \(1,3,5\), so the only distinct partitions are between 1 and 3, and between 3 and 5. Use the midpoints \(c=2\) and \(c=4\).

Candidate \(c=2\). The left responses are \(2,4\), with mean \(3\) and SSE \((2-3)^2+(4-3)^2=2\). The right responses are \(6,8,10,12\), with mean \(9\) and SSE \(9+1+1+9=20\). Thus, split SSE is \(2+20=22\), and improvement is \(70-22=48\).

Candidate \(c=4\). The left responses are \(2,4,6,8\), with mean \(5\) and SSE \(9+1+1+9=20\). The right responses are \(10,12\), with mean \(11\) and SSE \(1+1=2\). Thus, split SSE is \(20+2=22\), and improvement is \(70-22=48\).

Conclusion and check. Both candidates tie, each reducing SSE by 48. The equal scores are consistent with the responses being evenly spaced around the center. The method is to enumerate distinct partitions, not to create additional splits within the repeated values.

Worked Example: Compare Three Unequal-Sized Child Partitions

A hypothetical equipment-monitoring team records sensor level \(x\) and repair time \(y\), in hours: \((0,1),(0,3),(2,5),(4,7),(4,9),(7,11)\). Find the threshold with the largest squared-error improvement.

Parent calculation. The response sum is \(36\), so the parent mean is \(36/6=6\). The parent SSE is \(25+9+1+1+9+25=70\). The distinct predictor values are \(0,2,4,7\). The candidate midpoints are \(1,3,5\).

Candidate \(c=1\). The left responses \(1,3\) have mean \(2\) and SSE \(1+1=2\). The right responses \(5,7,9,11\) have mean \(8\) and SSE \(9+1+1+9=20\). Split SSE is \(22\), so improvement is \(70-22=48\).

Candidate \(c=3\). The left responses \(1,3,5\) have mean \(3\) and SSE \(4+0+4=8\). The right responses \(7,9,11\) have mean \(9\) and SSE \(4+0+4=8\). Split SSE is \(16\), so improvement is \(70-16=54\).

Candidate \(c=5\). The left responses \(1,3,5,7,9\) have mean \(5\) and SSE \(16+4+0+4+16=40\). The right response \(11\) has mean \(11\) and SSE \(0\). Split SSE is \(40\), so improvement is \(70-40=30\).

Conclusion and check. The threshold \(c=3\) is best: its split SSE of 16 is the smallest, and its improvement of 54 is the largest. Each improvement also checks against the same parent SSE of 70.

Worked Example: A Threshold That Is Valid Even With a Large Gap

A hypothetical policy-administration team relates processing score \(x\) to handling time \(y\), in minutes: \((1,0),(1,4),(2,5),(3,6),(3,10),(6,20)\). Compare all valid thresholds. In particular, do not assume that a wider gap between predictor values creates extra distinct partitions.

Parent SSE. The response sum is \(45\), giving parent mean \(45/6=7.5\). The parent SSE is \(56.25+12.25+6.25+2.25+6.25+156.25=239.25\). The distinct predictor values are \(1,2,3,6\); the candidates are \(1.5,2.5,4.5\).

Candidate \(c=1.5\). The left responses \(0,4\) have mean \(2\) and SSE \(4+4=8\). The right responses \(5,6,10,20\) have mean \(41/4=10.25\). Their squared deviations are \(27.5625,18.0625,0.0625,95.0625\), summing to \(140.75\). Split SSE is \(148.75\), and improvement is \(239.25-148.75=90.5\).

Candidate \(c=2.5\). The left responses \(0,4,5\) have mean \(3\) and SSE \(9+1+4=14\). The right responses \(6,10,20\) have mean \(12\) and SSE \(36+4+64=104\). Split SSE is \(118\), and improvement is \(239.25-118=121.25\).

Candidate \(c=4.5\). The left responses \(0,4,5,6,10\) have mean \(5\) and SSE \(25+1+0+1+25=52\). The right response \(20\) has mean \(20\) and SSE \(0\). Split SSE is \(52\), and improvement is \(239.25-52=187.25\).

Conclusion and check. The best threshold is \(4.5\), which separates predictor values 1, 2, and 3 from 6. The wide gap from 3 to 6 still gives just one partition; every threshold strictly between them assigns the observations identically.

Why the Improvement Measures Separation

The improvement has a useful exact form. Let the left and right child sizes be \(n_L\) and \(n_R\), their response means be \(\bar{y}_L\) and \(\bar{y}_R\), and the parent mean be \(\bar{y}\), where \(n=n_L+n_R\). Decompose each response’s deviation from the parent mean into its deviation from its child mean plus the child mean’s deviation from the parent mean. Within each child, the cross-product sums to zero because deviations from that child’s mean sum to zero. Therefore:

$$ \text{Parent SSE} = \text{Left SSE}+\text{Right SSE} +n_L(\bar{y}_L-\bar{y})^2+n_R(\bar{y}_R-\bar{y})^2. $$

Since \(n_L(\bar{y}_L-\bar{y})+n_R(\bar{y}_R-\bar{y})=0\), the two between-child terms simplify to \(\frac{n_Ln_R}{n}(\bar{y}_L-\bar{y}_R)^2\). Thus:

$$ \text{Improvement} = \frac{n_Ln_R}{n}(\bar{y}_L-\bar{y}_R)^2. $$

This identity explains why a split can improve fit: the child means differ, and both child sizes affect how much that separation contributes. It also shows that improvement cannot be negative when each child is fitted by its own mean. It does not mean a large improvement proves a stable or useful prediction outside the training data; it describes the reduction in squared error for this node’s observed responses.

Common Threshold Trap

A threshold cannot split observations that have the same predictor value. For example, in the first worked example, a mistaken calculation might put the observation \((3,6)\) on the left and \((3,8)\) on the right. That partition is impossible under a rule of the form \(x\leq c\) versus \(x>c\): both observations have \(x=3\), so they must receive the same assignment for every \(c\).

The correction is to sort the distinct predictor values and consider only gaps between consecutive values. A threshold at \(c=3\) in that example puts both \(x=3\) observations on the left; it creates the same partition as \(c=4\), not a new partition between the two tied observations. Always verify the membership of both children before calculating their means and errors.

Key takeaway: List the distinct predictor values, evaluate one threshold partition per gap, calculate each child mean and SSE, and compare improvements from the same parent SSE. Repeated predictor values must stay together; thresholds within one gap are equivalent.

Check Your Understanding

For each problem, identify valid threshold partitions, calculate their split errors, and use improvement from the parent SSE to choose a split where appropriate.

  1. A hypothetical laboratory records \((x,y)=(0,1),(0,3),(2,5),(2,7),(5,9),(5,11)\). Find the parent SSE and compare all valid threshold partitions. (Answer: Parent SSE \(=70\); thresholds \(1\) and \(3.5\) each have split SSE \(22\) and improvement \(48\), so they tie.)
  2. A hypothetical service desk records \((1,2),(1,6),(4,7),(8,12),(8,14)\). Compare the valid thresholds and select the better split. (Answer: Parent SSE \(=92.8\); thresholds \(2.5\) and \(6\) have split SSEs \(34\) and \(16\), respectively, so \(c=6\) is best with improvement \(76.8\).)
  3. A hypothetical greenhouse records \((0,4),(0,6),(3,8),(9,12),(9,14)\). Find all valid candidate midpoints and select the best. (Answer: Parent SSE \(=68.8\); candidates \(1.5\) and \(6\) have split SSEs \(62/3\) and \(10\), respectively, so \(c=6\) is best with improvement \(58.8\).)
  4. A hypothetical repair unit records \((2,0),(2,2),(5,4),(5,10),(9,12)\). Compare the two valid partitions. (Answer: Parent SSE \(=107.2\); threshold \(3.5\) has split SSE \(110/3\) and improvement \(70.5333\), while threshold \(7\) has split SSE \(56\) and improvement \(51.2\); choose \(c=3.5\).)
  5. For a node with \(n=8\), a candidate split has \(n_L=3\), \(n_R=5\), \(\bar{y}_L=4\), and \(\bar{y}_R=10\). Find its improvement using the child-mean identity. (Answer: \(\frac{3\cdot5}{8}(10-4)^2=67.5\).)

Previous

Next

Back to Tutorials
Clinical Biostats