Within-Node Sum of Squares as Impurity

Why Measure Variation Within a Node?

A regression tree divides observations into groups, called nodes, and predicts a response for each terminal node. A node prediction summarizes the responses of all observations assigned to that node. But how well does that single prediction represent the observations? Within-node sum of squares answers this by adding the squared differences between each response and the node's prediction. A small value means the responses are close to their node prediction; a large value means substantial variation remains within the node.

The quantity is also called the node's squared-error impurity. It measures variation in the observed responses, not uncertainty in a probability model. Its calculation uses the data assigned to the node and does not require a distributional assumption such as normality. For the hypothetical examples below, the recorded responses are treated as fixed observed values; no parametric distribution, interest rate, or mortality assumption is used.

Definition: Let node $t$ contain $n_t$ observations with responses $y_1,\ldots,y_{n_t}$. Its sample mean is $\bar{y}_t=\frac{1}{n_t}\sum_{i=1}^{n_t}y_i$. The within-node sum of squares, or squared-error impurity, is $$ Q(t)=\sum_{i=1}^{n_t}(y_i-\bar{y}_t)^2. $$ For a partition into terminal nodes $t_1,\ldots,t_K$, the total within-node sum of squares is $\sum_{k=1}^{K} Q(t_k)$.

The squared differences are nonnegative, so $Q(t)\geq 0$. It equals zero exactly when every response in the node is identical. The units are the square of the response units: if responses are measured in thousands of dollars, the sum of squares is measured in squared thousands of dollars. It is not itself an average squared error.

Why the Node Mean Appears in the Formula

The node prediction for squared-error regression is its sample mean, as established in the earlier material on terminal-node predictions. The mean is not just a convenient center: it is the constant prediction that makes the sum of squared errors as small as possible. This gives a direct justification for measuring node impurity around $\bar y_t$.

Start with any constant prediction.
Suppose a node predicts a constant $c$, giving squared error $\sum_i(y_i-c)^2$.

Write each response relative to the mean.
Since $y_i-c=(y_i-\bar y_t)+(\bar y_t-c)$, expand the squared terms and sum them.

Use the zero-sum deviations.
The deviations from the mean sum to zero, so the cross-term vanishes.

Identify the minimum.
The remaining extra term is nonnegative and is zero when $c=\bar y_t$.

\begin{aligned} \sum_{i=1}^{n_t}(y_i-c)^2 &=\sum_{i=1}^{n_t}\left[(y_i-\bar y_t)+(\bar y_t-c)\right]^2\\ &=\sum_{i=1}^{n_t}(y_i-\bar y_t)^2 +2(\bar y_t-c)\sum_{i=1}^{n_t}(y_i-\bar y_t) +n_t(\bar y_t-c)^2\\ &=Q(t)+n_t(\bar y_t-c)^2\\ &\geq Q(t). \end{aligned}

The middle term is zero because $\sum_i(y_i-\bar y_t)=0$. The final extra term, $n_t(\bar y_t-c)^2$, cannot be negative and vanishes only for $c=\bar y_t$. Thus the node mean minimizes the squared error among all constant predictions. This argument concerns a node's constant prediction; it does not require a probability distribution for the responses.

Calculating and Interpreting Node Impurity

The direct method is to find the node mean, subtract it from each response, square each deviation, and add the results. For arithmetic checks, an equivalent formula uses the response total and the sum of squared responses:

Q(t)=\sum_{i=1}^{n_t}y_i^2-\frac{\left(\sum_{i=1}^{n_t}y_i\right)^2}{n_t}.

To verify the identity, expand the definition: $\sum_i(y_i-\bar y_t)^2=\sum_i y_i^2-2\bar y_t\sum_i y_i+n_t\bar y_t^2$. Because $\sum_i y_i=n_t\bar y_t$, the last two terms combine to $-n_t\bar y_t^2$, which equals $-(\sum_i y_i)^2/n_t$. This shortcut is useful when a problem supplies totals rather than every observation, but the direct deviations are a useful check.

Worked Example: Remaining Variation After a Two-Node Partition

A hypothetical insurer groups six policy-level claim amounts, recorded in thousands of dollars, into a parent node. The responses are $2,4,6,8,10,12$. A specified partition places $2,4,6$ in node A and $8,10,12$ in node B. Calculate the parent-node impurity and the total within-node impurity after this partition. No distribution is assumed; these are the observed values.

Method. Calculate the mean and sum of squared deviations in the parent, then repeat separately in each child node. The common mistake to avoid is centering both child groups at the parent mean rather than at their own node means.

Parent node. Its response total is $2+4+6+8+10+12=42$, so its mean is $42/6=7$. The squared deviations are $25,9,1,1,9,25$. Therefore,

Q(\text{parent})=25+9+1+1+9+25=70.

Node A. Its mean is $(2+4+6)/3=4$, and its squared deviations are $4,0,4$. Thus $Q(A)=4+0+4=8$.

Node B. Its mean is $(8+10+12)/3=10$, and its squared deviations are $4,0,4$. Thus $Q(B)=4+0+4=8$.

The partition's total within-node sum of squares is $Q(A)+Q(B)=8+8=16$. The responses still vary within the two nodes, but their combined squared deviation from their respective node means is 16, in squared thousands of dollars. As a check, each node has two deviations of magnitude 2 and one deviation of 0, giving $2(2^2)+2(2^2)=16$ across both nodes.

For a partition, sum the node impurities when the quantity of interest is total remaining squared error. Do not silently replace that total with an unweighted average of node impurities. Those summarize different quantities: a total gives each observation's squared deviation one contribution, while an unweighted node average gives each node equal weight regardless of its size.

Worked Example: Unequal Node Sizes and the Correct Aggregate

A hypothetical portfolio's observed claim severities, in hundreds of dollars, are divided into node A with responses $1,3$ and node B with responses $4,4,7,9$. Find each node's impurity, the partition's total within-node sum of squares, and the unweighted average of the two node impurities. Treat the listed responses as the complete data in these nodes; no parametric distribution is assumed.

Method. Center each node at its own mean. To avoid confusing a node average with total remaining error, calculate and label both requested aggregates.

Node A. Its mean is $(1+3)/2=2$. Its squared deviations are $1$ and $1$, so $Q(A)=2$.

Node B. Its mean is $(4+4+7+9)/4=24/4=6$. Its squared deviations are $4,4,1,9$, so $Q(B)=18$.

The total within-node sum of squares is $2+18=20$. The unweighted average across the two nodes is $(2+18)/2=10$. These are not interchangeable: total error is 20 squared hundreds of dollars, whereas 10 is the average of the two node-level totals. In this example, the nodes contain different numbers of observations, and the unweighted average gives the smaller node the same node-level weight as the larger one.

Check. Node A contributes two squared deviations totaling 2, and node B contributes four totaling 18; together they contribute six squared deviations totaling 20, as required.

Using Response Summaries and Avoiding Scale Confusion

When a problem supplies $n_t$, $\sum_i y_i$, and $\sum_i y_i^2$, the shortcut formula saves work. It also makes clear that the mean must be based on the node's own observations. The answer is still a sum of squares, not a variance estimate.

Worked Example: Computing Impurity From Node Summaries

A hypothetical risk team reports summary statistics for a terminal node containing five observed response amounts, measured in thousands of dollars: $n_t=5$, $\sum_i y_i=35$, and $\sum_i y_i^2=265$. Calculate the node mean and its impurity, then verify the result using a compatible set of responses, $3,5,7,9,11$. These observations are treated as fixed; no distributional form is imposed.

Method. Apply the sum-of-squares identity to the supplied summaries, then check it by direct centering. The common mistake to avoid is dividing the sum of squared deviations by $n_t-1$, which would calculate a sample variance rather than node impurity.

The mean from the summaries is $\bar y_t=35/5=7$. Substituting into the identity gives

Q(t)=265-\frac{35^2}{5} =265-\frac{1225}{5} =265-245 =20.

For the direct check, the responses $3,5,7,9,11$ have deviations from 7 of $-4,-2,0,2,4$. Their squared deviations are $16,4,0,4,16$, which sum to $40$, not 20. This exposes an inconsistency: the stated sum of squares for these listed responses is $9+25+49+81+121=285$, not 265.

The compatible observations must agree with the supplied summaries. To verify the given summaries, use $3,5,7,9,11$ only after correcting the supplied $\sum_i y_i^2$ to 285. With that correction, the node impurity is $285-245=40$, matching the direct calculation. Alternatively, retain the original summaries $n_t=5$, total 35, and sum of squares 265: they imply impurity 20, but they cannot describe the listed five responses. This check illustrates why every summary must be internally consistent.

The preceding example deliberately demonstrates an arithmetic consistency check, but its conflicting inputs are not suitable as a routine calculation. In actual use, confirm that the supplied count, total, and squared total describe the same observations. Here is a consistent calculation using the listed five responses: $n_t=5$, $\sum_i y_i=35$, and $\sum_i y_i^2=285$. Therefore $Q(t)=285-35^2/5=285-245=40$, agreeing with the direct deviations.

Key distinction: Node impurity is the unnormalized sum $\sum_i(y_i-\bar y_t)^2$. Dividing it by $n_t$ gives the mean squared deviation about the node mean; dividing by $n_t-1$ gives the usual sample variance when $n_t>1$. Neither divided quantity is the node's sum-of-squares impurity.

What Impurity Tells You—and What It Does Not

A node's impurity describes the response variation left around that node's constant prediction. For a set of terminal nodes, adding their impurities gives the total squared error of the tree's training predictions on those observations, provided each observation belongs to exactly one terminal node and each node predicts its own sample mean. This is a descriptive calculation on the fitted groups.

Impurity depends on both the spread and the number of observations. If a node contains two copies of every response in another node, its mean is unchanged and its sum of squares doubles. Consequently, raw impurity values are meaningful for comparing partitions of the same response data, but can be misleading when comparing data sets of different sizes or response scales. A response measured in dollars rather than thousands of dollars multiplies every squared deviation by one million.

A low within-node sum of squares means little variation remains in the training responses around the node means. By itself, it does not establish that predictions will be accurate on new observations, and it does not account for the complexity of a tree. Those are separate model assessment questions. Keep the interpretation precise: $Q(t)$ summarizes in-node training variation, measured as a total of squared deviations.

Common Mistakes

Using the parent mean inside every child. Each node's impurity is centered at that node's own response mean.
Dividing by the sample size automatically. The defined impurity is a sum. Division changes the measure to an average or variance.
Averaging node impurities when a total is requested. Add $Q(t)$ across nodes for total within-node squared error; an unweighted node average answers a different question.
Ignoring response units. Squaring deviations squares the units, and rescaling the response rescales the impurity by the square of the scale factor.
Trusting inconsistent summaries. Check that the count, response total, and sum of squared responses can all describe the same data.

Check Your Understanding

For each problem, use the sum of squared deviations as the impurity, and distinguish node totals from averages when asked.

A node contains responses $3,5,8$. Find its mean and within-node sum of squares. (Answer: Mean $=16/3$; impurity $=38/3$.)
A partition has node A with responses $2,6,10$ and node B with responses $5,9$. Calculate each node's impurity and the partition's total. (Answer: $Q(A)=32$, $Q(B)=8$, total $=40$.)
A node has $n_t=4$, $\sum_i y_i=20$, and $\sum_i y_i^2=114$. Calculate the mean and impurity using the summary formula. (Answer: Mean $=5$; impurity $=14$.)
Two nodes each have mean 4. Node A contains $4,4,4$; node B contains $2,4,6$. Find each impurity and explain which node has greater remaining variation. (Answer: $Q(A)=0$, $Q(B)=8$; node B has greater remaining variation.)
A partition has node A with responses $1,3,5$ and node B with responses $6,8$. Find the total within-node sum of squares and the unweighted average of the node impurities. (Answer: $Q(A)=8$, $Q(B)=2$, total $=10$, unweighted average $=5$.)

What You'll Learn