Why Check Residuals in a Regression Tree?
A regression tree predicts a constant response for every observation in the same terminal node: the prediction is the mean response among the training observations in that node. That makes residual calculation straightforward, but it also creates a diagnostic trap. The residuals in a terminal node must sum to zero, whether or not the tree has captured the patterns in the data.
Residual plots help assess what remains after the tree’s predictions have been subtracted from the observed responses. In particular, a residual-versus-predictor plot can expose a trend or other structure within a terminal node. A residual-versus-fitted plot can show how residual spreads and unusual values vary across predicted levels. Neither plot, by itself, proves that the tree is adequate; the pattern must be interpreted in context.
These definitions concern the training data used to calculate the terminal-node means. If residuals are calculated for new observations, their fitted values are still assigned by the tree, but the residuals in a terminal node need not sum to zero: the new responses were not used to calculate that node’s mean.
Why Residuals Sum to Zero Within a Training Node
The mean in a terminal node is chosen to minimize the sum of squared errors for observations in that node. This choice gives a precise reason for the zero-sum property.
For any proposed constant prediction \(c\), the node’s squared error is \(Q(c)=\sum_{i\in R_m}(y_i-c)^2\).
Differentiate: \(Q'(c)=-2\sum_{i\in R_m}(y_i-c)\). Setting \(Q'(c)=0\) gives \(c=\frac{1}{n_m}\sum_{i\in R_m}y_i=\hat{y}_m\). Since \(Q''(c)=2n_m>0\), this is the minimum.
\(\sum_{i\in R_m}e_i=\sum_{i\in R_m}(y_i-\hat{y}_m)=\sum_{i\in R_m}y_i-n_m\hat{y}_m=0\), by the definition of \(\hat{y}_m\).
This result is an algebraic consequence of fitting a mean; it is not evidence that residuals are randomly scattered. A node can contain a strong upward or downward pattern across a predictor even though its residuals add to zero. The diagnostic question is therefore not simply whether the residual average is zero, but whether their signs and magnitudes show a pattern.
Worked Examples
Worked Example: Finding a Trend Within Two Terminal Nodes
A hypothetical tree predicts response \(y\) using two terminal nodes. The first contains observations with \(x=1,2,3,4\) and responses \(8,10,12,14\). The second contains observations with \(x=5,6,7,8\) and responses \(6,8,10,12\). Calculate the fitted values and residuals, then interpret a residual-versus-\(x\) plot.
Method. Calculate the mean response separately in each terminal node, then subtract that node’s fitted value from each response.
For the first node, the response total is \(8+10+12+14=44\), so its mean is \(44/4=11\). Its fitted values are all 11. The residuals, in order of increasing \(x\), are \(8-11=-3\), \(10-11=-1\), \(12-11=1\), and \(14-11=3\).
For the second node, the response total is \(6+8+10+12=36\), so its mean is \(36/4=9\). Its residuals are \(6-9=-3\), \(8-9=-1\), \(10-9=1\), and \(12-9=3\).
| \(x\) | \(y\) | Fitted value | Residual |
|---|---|---|---|
| 1 | 8 | 11 | -3 |
| 2 | 10 | 11 | -1 |
| 3 | 12 | 11 | 1 |
| 4 | 14 | 11 | 3 |
| 5 | 6 | 9 | -3 |
| 6 | 8 | 9 | -1 |
| 7 | 10 | 9 | 1 |
| 8 | 12 | 9 | 3 |
Check. The first node’s residuals sum to \(-3-1+1+3=0\); the second node’s residuals also sum to zero. In both nodes, however, residuals increase as \(x\) increases. A residual-versus-\(x\) plot would show a rising pattern within each node, rather than an unstructured scatter around zero. That is evidence of remaining association with \(x\) that the current piecewise-constant predictions have not represented.
Common mistake avoided. A zero residual sum does not mean the node has no remaining trend. Here, the sum is zero precisely because the node mean was used, while the ordered residuals still rise consistently.
Worked Example: Reading a Residual-versus-Fitted Plot
A second hypothetical tree has two terminal nodes. The first has responses \(5,7,9,11\), and the second has responses \(3,5,7\). Find the residuals and describe what a residual-versus-fitted plot displays.
Method. Compute each node mean and pair every observation’s residual with its fitted value.
In the first node, the response total is \(5+7+9+11=32\), so the fitted value is \(32/4=8\). The residuals are \(5-8=-3\), \(7-8=-1\), \(9-8=1\), and \(11-8=3\). Their squared values are \(9,1,1,9\), which sum to \(20\).
In the second node, the response total is \(3+5+7=15\), so the fitted value is \(15/3=5\). The residuals are \(3-5=-2\), \(5-5=0\), and \(7-5=2\). Their squared values are \(4,0,4\), which sum to \(8\). Across both nodes, the total squared residual is \(20+8=28\).
In the residual-versus-fitted plot, the first node contributes four points at fitted value 8, with residuals \(-3,-1,1,3\). The second contributes three points at fitted value 5, with residuals \(-2,0,2\). The points therefore form vertical stacks at 5 and 8. Each stack’s residuals sum to zero, as required.
Interpretation. The plot can show the vertical spread of residuals at each fitted level and draw attention to unusually large residuals. Here, the residual magnitudes reach 3 at fitted value 8 and 2 at fitted value 5. These are descriptive differences in this small example, not sufficient grounds for a broad claim about changing variability.
Check. The calculated residual sums are \(-3-1+1+3=0\) and \(-2+0+2=0\). The squared-error total is \(28\), matching the sum of the two node totals.
Common mistake avoided. Do not interpret the zero average residual at each fitted value as proof that the tree has captured all response structure. Those zeros follow from the way the training fitted values were calculated.
Worked Example: Detecting Structure Against a Different Predictor
Suppose all five observations below are in one terminal node of a hypothetical tree. The tree was built using other predictor information. Their values of a predictor \(z\) are \(1,2,3,4,5\), and their observed responses are \(20,20,22,27,35\). Calculate the node residuals and assess a residual-versus-\(z\) plot.
Method. Because there is one node, use the mean of all five responses as the fitted value for each observation.
The response total is \(20+20+22+27+35=124\). The node mean is \(124/5=24.8\). The residuals at \(z=1,2,3,4,5\) are, respectively, \(20-24.8=-4.8\), \(20-24.8=-4.8\), \(22-24.8=-2.8\), \(27-24.8=2.2\), and \(35-24.8=10.2\).
The residual sum is \(-4.8-4.8-2.8+2.2+10.2=0\). The squared residuals are \(23.04,23.04,7.84,4.84,104.04\), which sum to \(162.80\). The residuals are negative at the smaller \(z\) values and positive at the larger ones. A residual-versus-\(z\) plot would therefore show a rising pattern, including a particularly large positive residual at \(z=5\).
Interpretation and check. The node mean is correct because \(5(24.8)=124\), and the residuals sum to zero. Nevertheless, their ordered signs suggest remaining association with \(z\) within the node. This is a diagnostic clue, not a proof of a particular relationship or a guarantee that changing the tree will improve predictions on new data.
Common mistake avoided. Do not stop after verifying that the residuals sum to zero. That check confirms the arithmetic of the fitted mean; the residual-versus-predictor pattern addresses whether structure remains.
What Each Plot Can and Cannot Show
| Plot | What to inspect | Important limitation |
|---|---|---|
| Residual versus a predictor | Trends, curvature, clusters, or a changing spread across predictor values; identify whether patterns appear within particular terminal nodes. | A visible pattern is a clue about the training data, not by itself proof that a particular alternative tree will predict better. |
| Residual versus fitted value | Vertical spread, unusual residuals, and differences in the scatter across fitted levels. | Training fitted values are constant within a node. Residuals within each node average to zero by construction, so this plot cannot use a nonzero node-average residual to diagnose lack of fit. |
Because fitted values are constant within a terminal node, a residual-versus-fitted plot consists of vertical stacks, one for each distinct fitted value. If different nodes happen to have the same mean, they also share a fitted value; their plotted points occupy the same vertical stack. The plot remains useful for examining spread and individual residuals, but it can hide a within-node pattern across a predictor. For that reason, pair it with residual-versus-predictor plots when assessing the training observations.
A diagnostic pattern should be described specifically. For example, say that residuals tend to be negative at smaller values of a predictor and positive at larger values within a node. Avoid claiming that the plot alone establishes a causal effect, identifies the best next split, or guarantees out-of-sample performance. Residual plots are tools for finding potential structure; conclusions about predictive performance require appropriate assessment beyond the fitted training residuals.
Common Exam Traps
- Reversing the residual sign. The convention here is observed minus fitted. If \(y_i=18\) and \(\hat{y}_i=21\), the residual is \(18-21=-3\), not \(+3\). A negative value means the observation is below the prediction.
- Treating a zero node average as a fit diagnostic. For training data, the node residual sum is zero by the mean calculation. Inspect their order against predictors and their spread instead.
- Expecting a smooth horizontal cloud against fitted values. A regression tree assigns one fitted value per terminal node, so the plot contains vertical stacks rather than a continuum of fitted values.
- Overstating what a pattern proves. A trend in training residuals suggests that the current partition may leave structure uncaptured. The plot alone does not show that a revised tree will perform better on new data.
Check Your Understanding
For each question, use observed response minus fitted value as the residual convention.
- A terminal node has responses \(2,4,8\). Find its fitted value and the three residuals, then verify their sum. (Answer: fitted value \(14/3\); residuals \(-8/3,-2/3,10/3\); sum \(0\).)
- Three training observations in one node have responses \(12,15,18\). Find the fitted value and residuals, and state what a residual-versus-fitted plot shows for these observations. (Answer: fitted value \(15\); residuals \(-3,0,3\); one vertical stack at fitted value \(15\).)
- Within one terminal node, residuals ordered by increasing predictor value are \(-4,-1,2,3\). What is their sum, and what feature should you investigate in the residual-versus-predictor plot? (Answer: sum \(0\); investigate the increasing residual pattern across predictor values.)
- An observation has observed response \(36\) and fitted value \(40\). Calculate its residual and interpret its sign. (Answer: \(-4\); the observed response is 4 units below the fitted value.)
- Two training nodes have means 7 and 10. Each node has residuals that sum to zero. Can those facts alone establish that no predictor pattern remains within either node? Explain. (Answer: No. Zero sums are forced by the node means; residuals may still show a pattern against a predictor within a node.)