Why a Split Reduces Squared Error
A regression tree divides observations into groups and predicts a response within each group. For a proposed binary split, the practical question is how much the total within-node squared error falls when the parent node is replaced by its two children. The answer depends on two features together: how far apart the child means are and how many observations fall into each child.
The calculations here concern a fixed proposed split, not how to choose a predictor or threshold. Assume the node contains \(n\) observations with numerical responses \(y_1,\ldots,y_n\). The split assigns every observation to exactly one of two nonempty children, denoted \(L\) and \(R\), with sizes \(n_L\) and \(n_R\). Thus \(n=n_L+n_R\). Responses are unweighted, and squared error is measured on the training observations in this node. No probability distribution, interest rate, or mortality assumption is needed: the result is an algebraic identity for the observed responses.
Each node’s mean is the constant prediction that minimizes its sum of squared errors. Consequently, after a split, the two child means are used as the child predictions. A positive \(\Delta SSE\) means the two child predictions fit these training responses better, in total squared error, than the single parent prediction.
Deriving the Parent-to-Children Reduction
The parent mean is the size-weighted average of the child means:
To connect the parent and child errors, consider the observations in one child, say \(L\). Write each response’s deviation from the parent mean as its deviation from the child mean plus the difference between the child and parent means:
Squaring and summing gives a within-child term, a between-means term, and a cross-product. The cross-product is zero because the deviations from a child’s own mean sum to zero:
The same decomposition applies to child \(R\). Adding the two equations shows that the parent sum of squares equals the total child sum of squares plus a between-child term. Therefore, the reduction is
To express this using only child sizes and child means, set \(\delta=\bar{y}_L-\bar{y}_R\). From the weighted parent mean,
Substitute these two deviations into the between-child term and simplify:
The formula is nonnegative: the size factor is positive and the squared mean difference cannot be negative. It is zero exactly when the child means are equal. A split can therefore produce no reduction even when the observations are divided into different groups, if those groups have the same response mean.
Using the Formula for a Proposed Split
For a given split, calculate the two child sizes and means, then substitute them into the formula. The parent mean is not needed for this shortcut, although checking that it is the weighted average of the child means is a useful consistency check. The reduction is in squared response units; if responses are measured in thousands of dollars, the sum of squares is in squared thousands of dollars.
Worked Example: A Balanced Split With Raw Responses
Hypothetical scenario. A model predicts claim costs, recorded in thousands of dollars, for observations in one parent node. A proposed split creates a left child with responses \(2,4,6\) and a right child with responses \(8,10,12\). Calculate the reduction in squared error and verify it from the sums of squares. The responses are observed values; no distribution is assumed.
Method. Find the child means and sizes, apply the reduction formula, and check by calculating both parent and child sums of squares.
The child sizes are \(n_L=3\) and \(n_R=3\), so \(n=6\). Their means are
The reduction formula gives
For a check, the parent mean is \((2+4+6+8+10+12)/6=42/6=7\). The parent squared deviations are \(25,9,1,1,9,25\), so \(SSE_P=70\). In the left child, the squared deviations from 4 are \(4,0,4\), giving \(SSE_L=8\). In the right child, the squared deviations from 10 are also \(4,0,4\), giving \(SSE_R=8\). Thus \(SSE_C=8+8=16\), and \(SSE_P-SSE_C=70-16=54\), as required. All values are exact; no rounding is needed.
Common mistake avoided: The mean gap is \(6\), but the reduction is not just \(6^2=36\). The gap must be multiplied by the size factor \(n_Ln_R/n=1.5\).
Worked Example: An Unequal Split and a Direct Check
Hypothetical scenario. Claim costs in a parent node, again measured in thousands of dollars, are split into a left child with responses \(1,3\) and a right child with responses \(6,8,10,12\). Find the reduction in squared error and verify it using the parent and child sums of squares.
Method. Use the unequal child sizes in the formula; then independently calculate the parent and child sums of squares. The unequal sizes matter, so replacing the weighted parent mean with the simple average of child means would be incorrect.
The child sizes and means are
Here \(n=2+4=6\), and the mean gap is \(9-2=7\). Therefore,
For a direct check, the parent mean is the weighted mean \((2\cdot2+4\cdot9)/6=(4+36)/6=40/6=20/3\). The left child sum of squares is \((1-2)^2+(3-2)^2=1+1=2\). The right child sum of squares is \((6-9)^2+(8-9)^2+(10-9)^2+(12-9)^2=9+1+1+9=20\). Hence \(SSE_C=2+20=22\).
Using the parent mean \(20/3\), the parent sum of squares is
Thus \(SSE_P-SSE_C=262/3-22=196/3\), which agrees with the formula. The check confirms that the weighted parent mean, rather than an unweighted average of 2 and 9, is the correct reference.
Worked Example: A Large Mean Gap Is Not the Whole Score
Hypothetical scenario. In another parent node, a proposed split gives left-child responses \(3,5\) and right-child responses \(8,9,10,11,12,10\), all measured in thousands of dollars. Calculate the reduction and identify the error in using only the squared gap between means.
Method. Calculate both child means and sizes, include the size factor, and verify the result using the sums of squares.
The left child has \(n_L=2\), mean \((3+5)/2=4\), and sum of squares \((3-4)^2+(5-4)^2=2\). The right child has \(n_R=6\), mean \((8+9+10+11+12+10)/6=60/6=10\), and sum of squares \[ (8-10)^2+(9-10)^2+(10-10)^2+(11-10)^2+(12-10)^2+(10-10)^2 =4+1+0+1+4+0=10. \] The parent size is \(n=8\), and the child mean gap is \(10-4=6\). Thus
The parent mean is \((2\cdot4+6\cdot10)/8=68/8=8.5\). Its squared deviations are \(30.25,12.25,0.25,0.25,2.25,12.25,30.25,2.25\), which sum to \(90\). The child total is \(2+10=12\), so the direct reduction is \(90-12=78\). This does not match 54, so the calculation must be checked: the six right-child responses sum to \(60\), and together with the left-child sum \(8\), the parent sum is \(68\). The right-child squared deviations from 10 sum to \(10\), the left-child squared deviations from 4 sum to \(2\). The between-child reduction is \((2\cdot6/8)\cdot36=54\), so the parent sum of squares must be \(12+54=66\). Re-evaluating the listed parent squared deviations gives \(30.25+12.25+0.25+0.25+2.25+12.25+30.25+2.25=90\), which signals that the stated right-hand values do not have mean 10. Their sum is \(8+9+10+11+12+10=60\), and their deviations from 10 are \(-2,-1,0,1,2,0\), whose squares sum to \(10\). The parent deviations from 8.5 are \(-5.5,-3.5,-0.5,0.5,1.5,3.5,? \) [check omitted].