Outlier Sensitivity In Regression Trees

Why Outliers Matter in Regression Trees

A regression tree predicts a numerical response by dividing observations into groups and assigning a prediction to each terminal node. In the usual squared-error approach, that prediction is the mean response in the node. A single extreme response can therefore affect both the mean and the split scores used to choose a tree’s branches. The effect can be large because squared deviations grow rapidly with the size of a residual.

An extreme predictor value is different. For a split on a numerical predictor, the predictor value determines which side of a threshold an observation enters; the response values determine how good that division is under the squared-error criterion. Thus, an extreme predictor is not automatically an extreme response or a poor fit. It may still matter by changing the available partitions or by placing an observation in a distinct group.

Definition: For a set of observations $R$ in a regression-tree node, let $n_R$ be its number of observations and let $y_i$ be the response for observation $i$. Under the unweighted squared-error criterion, the node prediction is the mean response $\bar{y}_R$, and the node’s sum of squared errors is $$ \operatorname{SSE}(R)=\sum_{i\in R}(y_i-\bar{y}_R)^2. $$ For a candidate split into child nodes $R_L$ and $R_R$, the split score is the sum $\operatorname{SSE}(R_L)+\operatorname{SSE}(R_R)$. Among eligible splits, the split with the smallest child total (equivalently, the largest reduction from the parent SSE) is preferred.

These formulas describe the standard unweighted regression-tree criterion for a numerical response. They do not by themselves determine whether a split is allowed: tree-growth controls can rule out candidate splits, and pruning can later remove branches. If observation weights are used, the corresponding weighted criterion must be used instead of these unweighted sums.

Calculating a Split Score

A convenient way to calculate a node’s SSE is to use the sum and sum of squares of its responses. For a node with responses $y_1,\ldots,y_n$, the identity is:

\operatorname{SSE} =\sum_{i=1}^{n}y_i^2-\frac{\left(\sum_{i=1}^{n}y_i\right)^2}{n}.

This is not a different criterion; it is a computational form of the definition. It is especially useful when comparing several candidate splits by hand.

Justification: Expand each squared deviation around the node mean: $$ \begin{aligned} \operatorname{SSE} &=\sum_{i=1}^{n}(y_i-\bar{y})^2\\ &=\sum_{i=1}^{n}y_i^2-2\bar{y}\sum_{i=1}^{n}y_i+n\bar{y}^2. \end{aligned} $$ Since $\bar{y}=\sum_i y_i/n$, the middle term is $-2n\bar{y}^2$. The last two terms therefore combine to $-n\bar{y}^2=-(\sum_i y_i)^2/n$, giving the stated identity.

The split reduction is the parent SSE minus the sum of the child SSEs. The parent SSE is the same for every candidate split at that node, so minimizing child SSE and maximizing reduction give the same ranking.

\text{Reduction} =\operatorname{SSE}(R) -\left\{\operatorname{SSE}(R_L)+\operatorname{SSE}(R_R)\right\}.

Worked Example: An Extreme Response Changes the Preferred Split

Six hypothetical policy records are ordered by a numerical predictor $x$. Their responses in order are $2,3,4,5,6,7$. Consider two eligible cut points: after the third record and after the fourth. Find the preferred cut point, then replace the last response by $31$ and repeat the comparison. Use unweighted squared error.

Method. For each cut point, calculate the SSE in each child, add the two child errors, and compare the totals. The split with the smaller total is preferred.

Before the replacement, split after the third record. The left responses are $2,3,4$, with sum $9$ and sum of squares $29$. Thus its SSE is $29-9^2/3=29-27=2$. The right responses are $5,6,7$, with sum $18$ and sum of squares $110$. Its SSE is $110-18^2/3=110-108=2$. The child total is $2+2=4$.

Before the replacement, split after the fourth record. The left responses $2,3,4,5$ have sum $14$ and sum of squares $54$, giving SSE $54-14^2/4=54-49=5$. The right responses $6,7$ have sum $13$ and sum of squares $85$, giving SSE $85-13^2/2=85-84.5=0.5$. The child total is $5+0.5=5.5$. Since $4<5.5$, the split after the third record is preferred.

After replacing $7$ by $31$, split after the third record. The left child is unchanged, with SSE $2$. The right responses $5,6,31$ have sum $42$ and sum of squares $1022$. Their SSE is $1022-42^2/3=1022-588=434$. The child total is $2+434=436$.

After replacing $7$ by $31$, split after the fourth record. The left child is unchanged, with SSE $5$. The right responses $6,31$ have sum $37$ and sum of squares $997$. Their SSE is $997-37^2/2=997-684.5=312.5$. The child total is $5+312.5=317.5$. Since $317.5<436$, the preferred split changes to the cut point after the fourth record.

Check. The changed observation is in the right child under both candidate splits, but the second split places it with only one other response rather than two. That smaller child error makes the second split preferable here. The example shows sensitivity, not a general rule that an outlier should be isolated.

Common mistake avoided. Do not assume a split score changes only by the outlier’s squared distance from a fixed mean. The node means are recalculated for each candidate partition, so calculate each child’s SSE using that child’s own responses.

What an Extreme Response Does to a Leaf Prediction

A terminal node’s prediction is the mean response of the training observations that reach that node. Consequently, an extreme response can pull the prediction toward itself, even when the other responses in the node are close together. This change in the mean is separate from the change in split selection: a response can affect a leaf prediction even if the tree structure does not change.

Worked Example: Sensitivity of a Terminal-Node Prediction

A hypothetical terminal node contains four records with responses $10,12,14,100$. Calculate its prediction and SSE. Then suppose investigation establishes that the value $100$ was recorded in error and the corrected response is $16$. Recalculate both quantities. Use the node mean as the prediction.

Method. The squared-error regression-tree prediction is the arithmetic mean. Calculate it separately for each version of the data, then calculate deviations from that version’s mean.

Recorded data. The sum is $10+12+14+100=136$, so the prediction is $136/4=34$. The deviations from $34$ are $-24,-22,-20,66$. Their squares are $576,484,400,4356$, so the SSE is $576+484+400+4356=5816$.

Corrected data. The sum is $10+12+14+16=52$, so the prediction is $52/4=13$. The deviations from $13$ are $-3,-1,1,3$. Their squares are $9,1,1,9$, so the SSE is $9+1+1+9=20$.

Check. Replacing the verified error changes the fitted leaf prediction from $34$ to $13$, and the SSE from $5816$ to $20$. The calculation illustrates why checking an extreme value’s provenance matters. It does not justify deleting a value merely because it is large; if $100$ is a valid observation, it is part of the response distribution the model is being fitted to.

Common mistake avoided. Do not confuse an observation’s size with evidence that it is erroneous. Data correction requires a defensible reason, such as a confirmed recording or processing mistake, not just a large residual.

Why an Extreme Predictor Is Not the Same Thing

For a numerical predictor, a tree split sends observations to the left or right according to whether their predictor value is below or above a threshold. The response values then determine the squared-error score of those groups. If one predictor value is changed from $100$ to $10{,}000$ but remains the largest value, the ordering of observations is unchanged. For a split search based only on ordered partitions of that predictor, the possible groupings and their response-based scores are unchanged as well.

That does not mean an extreme predictor is always harmless. A large predictor value may belong to a record with an unusual response, may make a useful partition possible, or may expose a data-quality or measurement issue. The key distinction is that the numerical magnitude of the predictor does not enter the response SSE directly. Its role is through the partitions it permits.

Worked Example: An Extreme Predictor With Ordinary Responses

Five hypothetical records have ordered predictor values $x=1,2,3,4,100$ and responses $y=2,2,8,8,8$. Compare a split after the second record with a split after the fourth. Then change the last predictor from $100$ to $10{,}000$, leaving all responses unchanged. Find whether these two split scores change.

Method. Calculate child SSEs from the responses. The predictor values specify which records enter each child; they are not substituted into the response-error formula.

Split after the second record. The child responses are $[2,2]$ and $[8,8,8]$. Each child is constant, so each has SSE $0$. The total is $0+0=0$.

Split after the fourth record. The left responses are $[2,2,8,8]$, with mean $5$. Their squared deviations are $9,9,9,9$, giving SSE $36$. The right child contains $[8]$, with SSE $0$. The total is $36$. The split after the second record is preferred because $0<36$.

Change $x=100$ to $x=10{,}000$. The record remains last in the predictor ordering. Both candidate splits still create exactly the same response groups, so their scores remain $0$ and $36$, respectively. The preferred split does not change.

Check. The response groups, not the gap between $4$ and the largest predictor, determine these two SSE scores. A predictor outlier can still matter if it changes an ordering or a permitted partition; this particular magnitude change does neither.

Common mistake avoided. Do not call a record influential on the regression-tree fit solely because its predictor value is numerically far from the others. Check whether it changes the candidate groupings and whether its response changes the scores.

A Practical Sensitivity Check

When a fitted regression tree appears to depend heavily on one observation, separate three questions. First, is the value a valid response or predictor, or is there evidence of a data error? Second, does the observation materially change a leaf mean or a candidate split score? Third, does the resulting change affect the predictions or decision being made? These questions keep a numerical sensitivity check distinct from a decision to alter the data.

Verify the observation.
Check units, data entry, joins, and the definition of the response. A large but valid claim cost, for example, should not be silently removed.

Recalculate the affected quantities.
For a response outlier, compare node means and split SSEs with and without the observation as a diagnostic. For an extreme predictor, compare the partitions it creates rather than treating its scale as a response error.

Interpret any difference in context.
Report whether the split or predictions are sensitive, and explain the reason. If a value is excluded, the exclusion should follow a documented data-quality rule or a justified modeling decision.

This check is diagnostic, not an alternative fitting objective. Removing a valid observation simply to obtain a preferred tree changes the modeling problem and can make the resulting evaluation misleading. Sensitivity also depends on the allowed split set and tree-growth settings; a candidate split that is not eligible cannot be selected, regardless of its unrestrained SSE reduction.

Common Calculation Traps

Using the wrong child sum of squares. In the identity $\operatorname{SSE}=\sum y_i^2-(\sum y_i)^2/n$, the sum of squares is $\sum y_i^2$, not $(\sum y_i)^2$. The latter is divided by the node size.
Forgetting to recompute a mean. Each child has its own mean. The parent mean is not generally the correct center for calculating either child’s SSE.
Mixing SSE and reduction. A smaller child total is better; a larger reduction is better. These rankings agree because the parent SSE is fixed for candidate splits at the same node.
Equating extreme predictor and response values. A predictor defines partitions; a response contributes to the response-based error. An extreme predictor may affect which partitions are possible, but it is not squared as a residual.
Removing a legitimate extreme value automatically. A high response may be rare and important. Verify it and assess sensitivity before deciding how to handle it.

Key takeaway: In a squared-error regression tree, an extreme response can strongly affect both node means and split scores because the criterion uses squared deviations. An extreme predictor acts through the partitions it permits, not directly through the response SSE. Verify unusual values and distinguish sensitivity from evidence of error.

Check Your Understanding

For each problem, use the unweighted squared-error criterion unless stated otherwise.

A terminal node has responses $4,6,8$. Find its prediction and SSE. Replace $8$ by $20$ and find both quantities again. (Answer: Original mean $6$, SSE $8$; revised mean $10$, SSE $152$.)
Responses in predictor order are $1,3,5,7$. Compare a split after the second observation with a split after the third. Calculate each child total and identify the preferred split. (Answer: After the second, total SSE $4$; after the third, total SSE $8$; prefer after the second.)
Responses in predictor order are $1,2,3,4,20$. Compare splits after the third and fourth observations. Calculate each child total and reduction from the parent SSE. (Answer: Parent SSE $250$; after the third, total $130$ and reduction $120$; after the fourth, total $5$ and reduction $245$; prefer after the fourth.)
Predictor values are $1,2,3,4,100$, with ordered responses $3,3,9,9,9$. Compare splits after the second and fourth observations. What happens to those scores if the last predictor changes to $10{,}000$ and the response remains $9$? (Answer: Totals are $0$ and $36$, respectively; they remain unchanged, and the split after the second is preferred.)
A terminal node has responses $2,4,6$. Replace the last response by $30$. Calculate the original and revised mean and SSE, then state why the revised value alone is not a reason to discard the observation. (Answer: Original mean $4$, SSE $8$; revised mean $12$, SSE $488$; a large valid response is not evidence by itself of a data error.)

What You'll Learn