Recursive Binary Partitioning for Continuous Outcomes

How a Regression Tree Produces a Prediction

A regression tree predicts a numerical outcome by applying a sequence of simple, binary rules to an observation’s predictors. Each rule sends the observation down one of two branches. The process ends at a terminal node, where the tree assigns a numerical prediction. Thus, a prediction depends on the full path through the tree, not on a single split viewed in isolation.

For example, a hypothetical insurer might use policy characteristics to predict annual repair cost. A tree could first separate policies by vehicle age, then divide one of those groups by annual mileage. A policy’s predicted cost is determined by which side of each applicable split it falls on and the value assigned to the terminal node it reaches.

Definition: A regression tree is a prediction rule for a numerical response built by recursively partitioning the predictor space with binary, predictor-based splits. The first node is the root. A node that is split is an internal node; each such node has two child nodes. A node that is not split further is a terminal node (or leaf). Each terminal node has an assigned numerical prediction.

The tree is recursive because a split is applied to the observations that reach a particular node, and later splits may be applied to either of that node’s children. A later rule therefore operates within the branch already reached. It does not send an observation back to the root or replace an earlier decision.

A binary split has two outcomes. For a numerical predictor $x_j$ and threshold $s$, one common rule sends observations with $x_j \le s$ to the left child and observations with $x_j > s$ to the right child. The tree’s convention must specify where equality goes. A split can also use a binary condition on a categorical predictor, such as whether a category belongs to a specified group.

From Rules to Terminal Predictions

To use a fitted tree, begin at the root and apply its condition. Follow the selected branch to the next node. Continue until reaching a terminal node, then report that node’s numerical prediction. The split conditions determine the path; the terminal-node value determines the prediction.

The following compact tree uses a convention that values equal to a numerical threshold go left. The symbols $c_1$, $c_2$, and $c_3$ represent numerical predictions already assigned to its three terminal nodes.

\begin{array}{c} \text{Root: } x_1 \le s_1?\\ \begin{array}{cc} \text{Yes} & \text{No}\\ x_2 \le s_2? & \text{Terminal prediction } c_3\\ \begin{array}{cc} \text{Yes} & \text{No}\\ c_1 & c_2 \end{array} \end{array} \end{array}

An observation satisfying $x_1 \le s_1$ must also be checked against the second condition. An observation satisfying $x_1 > s_1$ reaches $c_3$ immediately; the condition on $x_2$ is not evaluated for that observation. This is why the path matters: a node’s rule applies only to cases that arrive there.

Worked Example: Following a Three-Leaf Tree

A hypothetical fitted tree predicts annual repair cost in thousands of dollars. At the root it splits on vehicle utilization $U$: $U \le 0.40$ goes left and $U > 0.40$ goes right. The right child is terminal and predicts $3.10$. The left child splits on vehicle age $A$: $A \le 2$ predicts $1.20$, while $A > 2$ predicts $1.80$. Find predictions for four vehicles and their total predicted cost.

Method. Apply the root condition first, then apply a child condition only if the vehicle reaches that child. The terminal values are given by the fitted tree.

Vehicle 1 has $U=0.35$ and $A=1$. Since $0.35 \le 0.40$, it goes left. Then $1 \le 2$, so it reaches the terminal node predicting $1.20$ thousand dollars.

Vehicle 2 has $U=0.35$ and $A=3$. It goes left because $0.35 \le 0.40$. At the next split, $3 > 2$, so its prediction is $1.80$ thousand dollars.

Vehicle 3 has $U=0.60$ and $A=1$. Since $0.60 > 0.40$, it reaches the right terminal node and is predicted to cost $3.10$ thousand dollars. Its age is not used: there is no age split on this path.

Vehicle 4 has $U=0.40$ and $A=2$. Equality goes left at the root, and equality also goes left at the age split. Its prediction is $1.20$ thousand dollars.

The total predicted cost is $1.20+1.80+3.10+1.20=7.30$ thousand dollars, or $\$7{,}300$. Check: the four predictions correspond to the paths left-left, left-right, right, and left-left, respectively. A common mistake would send Vehicle 4 right at the root by treating equality as $U>0.40$; that contradicts the stated rule.

Recursive Partitioning in a General Form

Let $x=(x_1,\ldots,x_p)$ denote a predictor vector. Each internal node applies one binary rule to the observations that reach it. Repeating such splits produces terminal regions $R_1,\ldots,R_M$, where $M$ is the number of terminal nodes. Each region has an assigned numerical value $c_m$. The tree’s prediction at $x$ is the value attached to the terminal region containing $x$.

\hat{f}(x)=\sum_{m=1}^{M} c_m\,\mathbf{1}(x\in R_m)

Here, $\mathbf{1}(x\in R_m)$ equals 1 if $x$ belongs to $R_m$, and 0 otherwise. This expression is a compact way to write the routing procedure: the observation belongs to one terminal region, so only that region’s prediction contributes.

Justification: At a node with region $R$, a binary rule divides the cases reaching it into $R_L=\{x\in R:x_j\le s\}$ and $R_R=\{x\in R:x_j>s\}$. These sets do not overlap, and every case in $R$ belongs to one of them. Starting from the root, repeated application of this fact means that the terminal regions do not overlap and together cover the predictor values handled by the tree. Therefore exactly one indicator in the prediction formula equals 1, giving precisely the prediction at the reached terminal node.

This argument explains the structure of the prediction rule; it does not specify how the splits or terminal values are selected during training. In a fitted tree, a training procedure determines those choices. The present task is to interpret the resulting rules correctly and apply them to new predictor values.

Worked Example: Applying Splits Only Along the Reached Path

A hypothetical model predicts annual equipment-maintenance cost in thousands of dollars. Its root splits on annual operating hours $H$: $H \le 1{,}000$ goes left, and $H > 1{,}000$ goes right. The left child splits on equipment age $A$: $A \le 5$ predicts $0.80$, and $A > 5$ predicts $1.10$. The right child is terminal and predicts $1.60$. Find each prediction and the total for three machines.

Method. Check operating hours first. Only machines sent left are checked against the age split.

Machine P has $H=900$ and $A=4$. Since $900 \le 1{,}000$, it goes left; since $4 \le 5$, its prediction is $0.80$ thousand dollars.

Machine Q has $H=900$ and $A=7$. It goes left at the root, then right at the age split because $7>5$. Its prediction is $1.10$ thousand dollars.

Machine R has $H=1{,}200$ and $A=2$. It goes right at the root and reaches the terminal prediction $1.60$ thousand dollars. The age value is irrelevant to its path.

The total is $0.80+1.10+1.60=3.50$ thousand dollars, or $\$3{,}500$. Check: P, Q, and R reach three different terminal nodes. A common mistake is to apply the age split to Machine R simply because its age is provided; that split exists only in the left branch.

Reading and Comparing Tree Paths

A tree diagram can be translated into a set of conditional rules. Each terminal prediction is paired with all the conditions along the path from the root to that terminal node. For instance, “go left at the root, then right” is a complete path only when both split conditions are stated. This translation is useful when a diagram is dense or when a question asks which observations receive a particular prediction.

The number of splits on a path is its depth. A terminal node may be reached after one split or several, depending on the tree’s structure. The number of terminal nodes is not necessarily the same as the number of internal nodes: each internal node has two children, and either child can itself be terminal or split again.

A tree’s predictions are constant within each terminal region because the assigned value does not change as long as the observation follows the same path. Predictions can change when a predictor value crosses a split boundary and changes the path. The tree therefore creates a stepwise prediction rule, rather than a single formula that changes smoothly with every small change in a predictor.

Worked Example: Recovering Rules and a Portfolio Total

A hypothetical fitted tree predicts a numerical quarterly service cost, measured in thousands of dollars. Its root splits on workload $W$: $W \le 8$ goes left, and $W > 8$ goes right. The left child splits on response delay $D$: $D \le 3$ predicts $2.4$, and $D > 3$ predicts $3.1$. The right child splits on $W$ again: $W \le 12$ predicts $4.6$, and $W > 12$ predicts $5.2$. Route four cases, identify the conditions for the $4.6$ prediction, and calculate the total predicted cost.

Method. For each case, apply the root rule and then the rule at the child reached. The repeated workload split is relevant only after a case has gone right at the root.

Case A has $W=6$ and $D=2$. It goes left because $6 \le 8$, then reaches the $2.4$ terminal because $2 \le 3$.

Case B has $W=6$ and $D=4$. It goes left at the root, then reaches the $3.1$ terminal because $4>3$.

Case C has $W=10$ and $D=1$. It goes right at the root because $10>8$. The right child checks workload: $10 \le 12$, so the prediction is $4.6$. The response delay is not used on this path.

Case D has $W=14$ and $D=2$. It goes right at the root, then right again because $14>12$, giving a prediction of $5.2$.

The rule for predicting $4.6$ is $8<W\le12$, with no condition on $D$. The total predicted cost is $2.4+3.1+4.6+5.2=15.3$ thousand dollars. Check: all four cases reach different terminal nodes, and their total is $15.3$, or $\$15{,}300$. A common mistake is to describe the $4.6$ rule as $W\le12$ alone; that omits the root condition $W>8$.

Common Interpretation Traps

Ignoring the path. A condition at a child node is relevant only for observations that reach that child.
Reversing a branch condition. Read the exact inequality attached to each branch; do not assume that left always means “less” unless the tree says so.
Misplacing equality. If the rule says $x_j\le s$, an observation with $x_j=s$ follows that branch, not the $x_j>s$ branch.
Confusing a split with a prediction. A threshold decides where an observation goes. The numerical prediction comes from the terminal node reached.
Applying an unused predictor. A predictor may be recorded for an observation but not used on its path through the fitted tree.

Key takeaway: A regression tree is a sequence of local binary decisions. To obtain a prediction, follow the stated conditions from the root to exactly one terminal node and report its numerical value. The full path, including equality conventions, is essential to interpreting the tree.

Check Your Understanding

For each problem, trace the stated rules in order and use the terminal values provided.

A tree splits first on $X\le4$. The left child predicts $2$ if $Z\le6$, and $5$ if $Z>6$; the right child predicts $8$. What does it predict for $(X,Z)=(4,7)$? (Answer: 5)
A tree splits first on $M\le10$. Its left child predicts $1.5$, and its right child predicts $2.5$ if $N\le3$, otherwise $4$. Find the predictions for $(M,N)=(9,4)$ and $(12,3)$, and their total. (Answer: 1.5 and 2.5; total 4.0)
A tree splits on $A\le2$. The left child splits on $B\le5$, predicting $3$ on the left and $6$ on the right. The root’s right child predicts $9$. State the complete path conditions for prediction $6$, and find the prediction for $(A,B)=(1,5)$. (Answer: $A\le2$ and $B>5$; prediction 3)
A tree splits on $T\le7$. The left child predicts $0.7$ if $C\le2$, otherwise $1.1$. The right child splits on $T\le11$, predicting $1.6$ if true and $2.0$ otherwise. Find the total prediction for $(T,C)=(7,3),(9,8),(12,1)$. (Answer: $1.1+1.6+2.0=4.7$)
A tree splits at the root on $R\le100$. The left child predicts $4$. The right child splits on $S\le20$, predicting $6$ if true and $10$ otherwise. For $R=100,S=25$ and $R=101,S=20$, give both predictions and their total. (Answer: 4 and 6; total 10)

What You'll Learn