Why Interactions Need Branch-Specific Rules
A predictor can relate to a response differently in different circumstances. For example, in a hypothetical study of equipment repairs, the effect of operating intensity on repair cost might depend on the equipment type. A regression tree can represent this pattern by splitting first on one predictor and then applying different split rules in different branches.
This is a tree representation of an interaction: the prediction rule for one predictor depends on the value or range of another predictor. The rule need not be a smooth curve or a constant slope. Each terminal node instead has its own prediction, and the path to that node determines which prediction applies.
For a numeric predictor, a split at threshold \(c\) sends observations with predictor value at most \(c\) to one child and observations greater than \(c\) to the other. A new observation follows the same rules down the tree until it reaches a leaf. In this tutorial, thresholds use the convention \(\leq c\) versus \(>c\).
The tree examples use a numeric response and the residual sum of squares (RSS) as the fitting criterion. They make no assumption that the response is normally distributed. A small RSS describes fit to the displayed observations; by itself, it does not establish that the tree will predict well on new data.
Why a Leaf Predicts Its Mean
Suppose the responses in one terminal node are \(y_1,\ldots,y_n\), and a constant prediction \(a\) is assigned to every observation in that node. The node's squared error is \(\sum_{i=1}^n (y_i-a)^2\). The arithmetic mean minimizes this quantity.
To verify the identity, write \(y_i-a=(y_i-\bar{y})+(\bar{y}-a)\), square, and sum. The cross-product term is \(2(\bar{y}-a)\sum_i(y_i-\bar{y})=0\), because deviations from the mean sum to zero. Since \(n(a-\bar{y})^2\geq 0\), the RSS is smallest at \(a=\bar{y}\). This is why a squared-error regression tree predicts the response mean in each leaf.
Worked Examples
Worked Example: Different Thresholds for the Same Predictor
A hypothetical repair team records operating category \(A\), which is either 0 or 1, a sensor band \(B\), and repair cost \(Y\), in thousands of dollars. The proposed tree first splits at \(A\leq 0\). It then splits on \(B\) at 0.5 in the \(A=0\) branch and at 1.5 in the \(A=1\) branch. Calculate the leaf predictions and the training RSS.
| A | B | Y values (thousands of dollars) |
|---|---|---|
| 0 | 0 | 0, 2 |
| 0 | 1 | 8, 10 |
| 0 | 2 | 8, 10 |
| 1 | 0 | 10, 12 |
| 1 | 1 | 10, 12 |
| 1 | 2 | 20, 22 |
Method and routing. Apply the root split first. In the \(A=0\) branch, \(B\leq 0.5\) contains the two responses 0 and 2; the other child contains the four responses 8, 10, 8, and 10. In the \(A=1\) branch, \(B\leq 1.5\) contains 10, 12, 10, and 12; its other child contains 20 and 22.
Leaf predictions. The four leaf means are, respectively, \((0+2)/2=1\), \((8+10+8+10)/4=36/4=9\), \((10+12+10+12)/4=44/4=11\), and \((20+22)/2=21\) thousand dollars.
Training RSS. For the first leaf, the residuals from 1 are \(-1\) and \(1\), giving RSS \(1+1=2\). In the second leaf, deviations from 9 are \(-1,1,-1,1\), giving RSS \(4\). The third leaf has the same deviations around 11 and RSS \(4\). The fourth leaf has deviations \(-1,1\) and RSS \(2\). Thus total RSS is \(2+4+4+2=12\), in squared thousands of dollars.
Check and interpretation. Every observation is assigned to exactly one leaf, and the leaf counts \(2+4+4+2=12\) match the 12 responses in the table. The split threshold for \(B\) depends on the value of \(A\): the tree uses 0.5 in one branch and 1.5 in the other. This is a branch-specific interaction pattern, not one global threshold for \(B\). A common mistake is to use 0.5 in both branches; that would route the \(A=1,B=1\) observations incorrectly.
Worked Example: A Different Predictor Is Useful in Each Branch
In a hypothetical portfolio review, \(A\) is a site score, \(B\) is an indicator taking values 0 or 1, and \(Y\) is a numeric review score. The proposed tree splits first at \(A\leq 1.5\). In the left branch it splits on \(B\leq 0.5\); in the right branch it splits again on \(A\leq 2.5\). Find the leaf predictions and show how the predictor used for the next split differs between branches.
| A | B | Y |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 8 |
| 1 | 0 | 0 |
| 1 | 1 | 8 |
| 2 | 0 | 10 |
| 2 | 1 | 10 |
| 3 | 0 | 20 |
| 3 | 1 | 20 |
Route and average each leaf. The root sends the four observations with \(A=0\) or 1 left and the four with \(A=2\) or 3 right. In the left branch, the \(B\leq 0.5\) leaf contains responses 0 and 0, so its mean is \(0/2=0\). The \(B>0.5\) leaf contains 8 and 8, so its mean is \(16/2=8\). In the right branch, the \(A\leq 2.5\) leaf contains 10 and 10, so its mean is \(20/2=10\); the \(A>2.5\) leaf contains 20 and 20, so its mean is \(40/2=20\).
Check the fit. Each pair in each leaf equals its leaf mean, so all eight residuals are zero and the training RSS is 0. The counts are \(2+2+2+2=8\), as required. This exact fit is a property of this small constructed data set, not evidence that a tree of this size will generalize perfectly.
Interpretation and trap. The tree uses \(B\) for the next split when \(A\leq 1.5\), but uses \(A\) when \(A>1.5\). Thus the role of \(B\) is conditional: it separates the responses in the left branch, while in the right branch the displayed responses do not vary with \(B\). A common mistake is to keep splitting on the predictor used at the root simply because it was useful there. A tree can use a different predictor in each child, and the resulting prediction depends on the full path.
Worked Example: Predicting New Cases With Branch-Dependent Cutoffs
A hypothetical maintenance analyst has fitted the following tree to training data. It splits first at \(A\leq 0.5\). The left branch splits on \(B\leq 1.5\), and the right branch splits on \(B\leq 0.5\). The response is repair duration in hours. The training responses in the leaves are shown below. Calculate each leaf prediction, the training RSS, and predictions for two new cases: \((A,B)=(0,2)\) and \((1,1)\).
| Leaf condition | Training responses (hours) |
|---|---|
| \(A\leq 0.5,\ B\leq 1.5\) | 2, 4, 6, 8 |
| \(A\leq 0.5,\ B>1.5\) | 10, 12 |
| \(A>0.5,\ B\leq 0.5\) | 20, 22 |
| \(A>0.5,\ B>0.5\) | 20, 22, 30, 32 |
Compute the means. The first leaf mean is \((2+4+6+8)/4=20/4=5\). The second is \((10+12)/2=11\). The third is \((20+22)/2=21\). The fourth is \((20+22+30+32)/4=104/4=26\) hours.
Compute the RSS leaf by leaf. In the first leaf, deviations from 5 are \(-3,-1,1,3\); the squared deviations sum to \(9+1+1+9=20\). In the second and third leaves, the deviations are \(-1,1\) in each, giving RSS 2 for each. In the fourth leaf, deviations from 26 are \(-6,-4,4,6\), giving \(36+16+16+36=104\). Total training RSS is \(20+2+2+104=128\) hours squared. The leaf counts \(4+2+2+4=12\) agree with the number of training responses.
Route the new cases. For \((0,2)\), \(A\leq 0.5\), so use the left branch; there \(B=2>1.5\), and the prediction is 11 hours. For \((1,1)\), \(A>0.5\), so use the right branch; there \(B=1>0.5\), and the prediction is 26 hours.
Check and common mistake. The second case uses the right-branch threshold 0.5, not the left-branch threshold 1.5. Using the left threshold globally would send \(B=1\) to the wrong leaf and return 21 rather than 26 hours. The tree's rules are conditional on the branch already reached.
Reading the Interaction in a Tree
A useful way to interpret a fitted tree is to describe its paths in words. For example: “When \(A\leq 0.5\), the prediction changes according to whether \(B\leq 1.5\); when \(A>0.5\), the relevant threshold for \(B\) is 0.5.” That statement preserves the conditional nature of the rule. Saying only “\(B\) is important” would omit how its split depends on \(A\).
Interactions can appear in at least two visible ways in a small tree: the same predictor can have different thresholds in different branches, or different predictors can be selected for the next split in different branches. In either case, a single global description of one predictor's effect may fail to capture the tree's prediction pattern.
A tree's terminal-node prediction is constant within each leaf. It therefore creates a piecewise-constant prediction surface, with boundaries set by its split rules. The boundary and prediction change when a case crosses a split; the tree does not assert that the response changes smoothly between leaves. Interpretation should follow the actual conditions, including which branch is taken first.
Check Your Understanding
Work through the routing and arithmetic before checking each answer.
- A tree splits first at \(A\leq 0\), then splits on \(B\leq 2\) in the left branch and \(B\leq 4\) in the right branch. A new case has \(A=1\) and \(B=3\). Which terminal rule does it satisfy? (Answer: The right branch, then \(B\leq 4\).)
- A terminal node contains responses 3, 7, and 8. Find its squared-error prediction. (Answer: The mean is \((3+7+8)/3=6\).)
- A proposed leaf contains responses 4, 6, 8, and 10. Find its mean and RSS. (Answer: Mean \(=7\); RSS \(=(-3)^2+(-1)^2+1^2+3^2=20\).)
- A tree splits first on \(A\). The left child splits on \(B\), while the right child splits on \(A\) again. In which branch does the next split rule depend on \(B\), and what does the different child predictor indicate? (Answer: The left branch; the tree uses different conditional rules in the two branches, which can represent an interaction.)
- A fitted tree uses \(B\leq 1\) after \(A\leq 0\), and \(B\leq 3\) after \(A>0\). A new case is \((A,B)=(2,2)\). If the leaf reached contains training responses 9, 13, and 14, calculate the prediction. (Answer: It follows the \(A>0\) branch and then \(B\leq 3\); the prediction is \((9+13+14)/3=12\).)