Regression Trees as Piecewise-Constant Functions

From Split Rules to a Prediction Function

A regression tree can be read as a set of rules, but it can also be viewed as a function of the predictors. The rules divide predictor space into terminal regions. Within each region, the fitted tree returns one constant value, regardless of where a point lies inside that region.

This perspective gives a direct way to find a prediction: identify which terminal region contains the point, then use that region’s prediction. Under the ordinary squared-error regression-tree rule, the prediction is the mean of the training responses assigned to that terminal region.

Definition: A terminal region $R_m$ is the set of predictor values that follow the tree’s split rules to terminal node $m$. If $n_m$ training observations fall in $R_m$, with responses $y_i$, the terminal-node prediction is $$ \hat{c}_m=\frac{1}{n_m}\sum_{i:\,\mathbf{x}_i\in R_m} y_i. $$ The fitted regression-tree function is constant within each terminal region: $\hat{f}(\mathbf{x})=\hat{c}_m$ whenever $\mathbf{x}\in R_m$.

Here, $\mathbf{x}$ denotes the predictor values for an observation, and $y_i$ denotes its observed response. The formula assumes an unweighted regression tree using squared-error loss. It requires a numeric response and training observations assigned to terminal nodes. No particular probability distribution, interest rate, or mortality assumption is needed to evaluate this fitted function; the calculation uses the tree structure and its training responses.

A region is defined by the complete sequence of rules from the root to a terminal node. For example, a path might require $x_1>80$ and $x_2\leq 3$. A point meets that path only if it satisfies both conditions. At a threshold, the tree’s stated inequality determines the assignment: if the rule is $x\leq t$, a point with $x=t$ follows that branch, not the branch $x>t$.

The Piecewise-Constant Function

Suppose a tree has $M$ terminal regions, $R_1,\ldots,R_M$. These regions are disjoint: a point is routed to one terminal node, not two. They also cover the predictor values that can be routed through the tree. Write $\hat{c}_m$ for the prediction in region $R_m$. An indicator function is 1 when its condition is true and 0 otherwise, so the fitted tree can be written as

\hat{f}(\mathbf{x})=\sum_{m=1}^{M}\hat{c}_m\,I(\mathbf{x}\in R_m).

This expression is a compact way to encode the same routing procedure as the tree diagram. For a given $\mathbf{x}$, exactly one indicator is 1 and the rest are 0. Therefore every term except the one for the containing region vanishes, leaving that region’s prediction.

The terminal-node values are estimated from the responses in their own regions. The overall average response is not substituted for a node’s average, and a point’s prediction is not obtained by averaging the predictions of all terminal nodes. The point’s location selects the relevant constant.

Read the split rules.
Start at the root and apply each condition to the point’s predictor values.

Identify the terminal region.
Continue until the point reaches a terminal node. Keep threshold inequalities exactly as stated.

Find that node’s constant.
Use its training-response mean for an unweighted squared-error tree.

Report the prediction in context.
Retain the response’s units and check that the point’s path and selected node agree.

Worked Examples

Worked Example: Finding a Prediction at a Split Threshold

A hypothetical insurer fits an unweighted squared-error regression tree to predict annual repair cost, measured in thousands of dollars, from the number $x$ of service requests in the prior year. The tree sends $x\leq 2$ to one terminal node. For $x>2$, it splits again: $x\leq 5$ goes to a second terminal node, and $x>5$ goes to a third. The training responses in these three nodes are, respectively, $(1,3,5)$, $(7,9,11)$, and $(15,17,19)$. Find the fitted function and predict for $x=2$, $x=5$, and $x=6$.

Method and assumptions. Use the path rules to assign each requested value to a leaf, then calculate each leaf’s response mean. The response values are fixed training data; no interest, mortality, or response-distribution assumption is used.

First node. Its response sum is $1+3+5=9$, and it contains 3 responses. Thus its prediction is $9/3=3$ thousand dollars.

Second node. Its response sum is $7+9+11=27$, and it contains 3 responses. Its prediction is $27/3=9$ thousand dollars.

Third node. Its response sum is $15+17+19=51$, and it contains 3 responses. Its prediction is $51/3=17$ thousand dollars.

The fitted function is $\hat{f}(x)=3$ for $x\leq 2$, $\hat{f}(x)=9$ for $2<x\leq 5$, and $\hat{f}(x)=17$ for $x>5$, with values in thousands of dollars. At $x=2$, the first rule $x\leq2$ applies, so the prediction is 3. At $x=5$, the point follows $x>2$ and then $x\leq5$, so the prediction is 9. At $x=6$, it follows $x>2$ and $x>5$, so the prediction is 17.

Check. The threshold values 2 and 5 are assigned to the branches marked $\leq$, and every prediction matches the mean of the responses in its selected node. The common mistake is to send $x=5$ to the $x>5$ node; the inequality does not allow that.

Worked Example: Applying a Path With Two Predictors

A hypothetical risk model predicts annual claim cost, in thousands of dollars, from insured asset value $x_1$, measured in thousands of dollars, and policy duration $x_2$, measured in years. Its root rule sends $x_1\leq80$ to terminal node A. If $x_1>80$, a second rule sends $x_2\leq3$ to node B and $x_2>3$ to node C. The training responses in A are $(2,4,6)$; in B they are $(8,10)$; and in C they are $(12,14,16,18)$. Find the node predictions and the predictions at $(80,8)$, $(90,3)$, and $(95,4)$.

Method and assumptions. Treat the path conditions as a conjunction: a point must satisfy the root condition and, when applicable, the next condition. Calculate each unweighted node mean. The response values are training data, with no assumed distribution, interest rate, or mortality model.

For A, the response sum is $2+4+6=12$, and the count is 3, giving $12/3=4$. For B, the sum is $8+10=18$, and the count is 2, giving $18/2=9$. For C, the sum is $12+14+16+18=60$, and the count is 4, giving $60/4=15$. All three predictions are in thousands of dollars.

For $(80,8)$, $x_1=80$ satisfies $x_1\leq80$, so the point reaches A. Its prediction is 4; $x_2$ is not used because the point does not enter the second split. For $(90,3)$, $90>80$ and $3\leq3$, so the point reaches B and its prediction is 9. For $(95,4)$, $95>80$ and $4>3$, so the point reaches C and its prediction is 15.

Check. The path rules and all three node means agree with the reported predictions. A common mistake is to apply the $x_2$ threshold to $(80,8)$ anyway. That rule applies only after the point takes the $x_1>80$ branch.

Worked Example: Writing and Evaluating the Indicator Form

A hypothetical tree predicts a response $Y$, measured in thousands of dollars, from a continuous predictor $x$. Its terminal regions are $R_1=\{x:x\leq10\}$, $R_2=\{x:10<x\leq20\}$, and $R_3=\{x:x>20\}$. The nodes contain, respectively, 4, 3, and 2 training observations, and their response sums are 28, 33, and 26. Write the fitted function using indicators, evaluate it at $x=10$, $x=15$, $x=20$, and $x=21$, and check its response-weighted overall average across the training observations.

Method and assumptions. Divide each node’s response sum by its count to obtain its constant, then use the indicator for each region. This is an unweighted squared-error tree; there is no interest, mortality, or distribution assumption.

The node constants are $28/4=7$, $33/3=11$, and $26/2=13$. Therefore,

\hat{f}(x)=7I(x\leq10)+11I(10<x\leq20)+13I(x>20).

At $x=10$, only $I(x\leq10)$ equals 1, so $\hat{f}(10)=7$. At $x=15$, only $I(10<x\leq20)$ equals 1, so $\hat{f}(15)=11$. At $x=20$, the same middle indicator equals 1, so $\hat{f}(20)=11$. At $x=21$, only the final indicator equals 1, so $\hat{f}(21)=13$.

As a check using the training observations, the node predictions sum to $4(7)+3(11)+2(13)=28+33+26=87$, equal to the supplied total response sum. There are $4+3+2=9$ observations, so their overall mean is $87/9=9.666\ldots$ thousand dollars. This overall mean is a check across the training set, not the prediction at every point; the pointwise predictions remain 7, 11, or 13 according to the region.

Check. The four points activate exactly one indicator each, and the total fitted response over the training observations equals the total observed response. Averaging the three node constants without weighting by their node counts would not provide this check.

Common Mistakes and Checks

A piecewise-constant representation makes several errors easier to spot. First, use the full path to find a region; a condition farther down the tree applies only to points that reach that split. Second, honor threshold equality exactly as written. Third, keep the node’s training-response mean distinct from the overall training mean. The overall mean combines observations from all nodes, while a fitted value uses only the node selected by the point.

For example, if a tree has three node predictions $3$, $9$, and $17$, it is incorrect to predict $(3+9+17)/3=9.67$ for a point simply because there are three nodes. That calculation averages node constants without regard to which region contains the point. The correction is to route the point and select exactly one value. If the point is in the region with prediction 17, its fitted value is 17.

The formula also depends on the fitting criterion and weighting convention. For an unweighted squared-error regression tree, terminal predictions are arithmetic means. If observations have unequal weights or a different loss function is used, the terminal-node prediction rule may differ; the arithmetic-mean formula should not be applied automatically.

Key takeaway: A fitted regression tree is a function that assigns one constant to each terminal region. To predict at $\mathbf{x}$, follow its path to one region and use that region’s training-response mean. This representation is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each problem, identify the terminal region before using its response mean.

A tree sends $x\leq4$ to node A and $x>4$ to node B. A contains responses $(5,7,8)$, and B contains $(10,14)$. What is the prediction at $x=4$? (Answer: $20/3$, or $6.666\ldots$)
A tree first sends $z\leq12$ to node A. If $z>12$, it sends $w\leq2$ to B and $w>2$ to C. A has responses $(4,6,8)$, B has $(9,15)$, and C has $(20,22,26)$. Find the predictions at $(12,10)$, $(13,2)$, and $(13,2.1)$. (Answer: $6,\ 12,\ 68/3$, respectively)
A tree has regions $x\leq0$, $0<x\leq5$, and $x>5$. Their response counts and sums are $(3,21)$, $(4,32)$, and $(3,27)$. What is the prediction at $x=5$? (Answer: $8$)
A tree sends $x_1\leq3$ to a node with response mean 4. For $x_1>3$, it sends $x_2\leq7$ to a node containing responses $(6,10)$, and $x_2>7$ to a node containing $(13,17,20)$. Find the predictions at $(3,9)$, $(4,7)$, and $(4,8)$. (Answer: $4,\ 8,\ 50/3$, respectively)
A fitted tree has two terminal nodes: one with 2 training observations and mean response 5, and one with 6 training observations and mean response 11. A new point follows the path to the second node. What is its tree prediction, and what incorrect value results from taking the unweighted average of the two node means? (Answer: Prediction $11$; incorrect unweighted average $8$)

What You'll Learn