Weighted Means and Weighted Split Errors

Why Weights Change a Tree’s Prediction

A regression tree predicts a constant value for every observation in a terminal node. When observations have unequal weights, they do not contribute equally to that constant: a larger-weight observation has more influence on the fitted prediction. The same weights must also be used when evaluating how well a node or a candidate split fits the response values.

This matters whenever the tree-fitting objective assigns different importance to observations, whether the weights represent replicated records, sampling adjustments, or another specified fitting convention. A weighted calculation is not automatically a statement about how common an outcome is in a population; its interpretation depends on how the weights were defined. The focus here is the weighted least-squares calculation itself, aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: In a regression-tree node $A$, let observation $i$ have response $y_i$ and nonnegative weight $w_i$. If the total node weight $W_A=\sum_{i\in A}w_i$ is positive, the weighted terminal-node prediction is $$ \hat{y}_A=\frac{\sum_{i\in A}w_i y_i}{\sum_{i\in A}w_i}. $$ The weighted within-node squared error is $$ \operatorname{SSE}_w(A)=\sum_{i\in A}w_i(y_i-\hat{y}_A)^2. $$

An observation with weight zero contributes nothing. The formulas require positive total weight in any node whose mean is to be calculated. If every observation in a node has the same positive weight, the weighted mean equals the ordinary mean, and the weighted error is that common weight times the ordinary squared error.

Why the Weighted Mean Minimizes the Error

The terminal prediction is chosen to minimize the weighted sum of squared deviations. For a proposed constant prediction $c$, define $$ Q(c)=\sum_{i\in A}w_i(y_i-c)^2. $$ Because the weights are nonnegative and sum to a positive value, this is a quadratic function with a minimum. Differentiating with respect to $c$ gives $$ \frac{dQ(c)}{dc}=-2\sum_{i\in A}w_i(y_i-c). $$ At a minimum, this derivative is zero. Rearranging the resulting equation gives the weighted mean: $$ \sum_{i\in A}w_i y_i-c\sum_{i\in A}w_i=0 \quad\Longrightarrow\quad c=\frac{\sum_{i\in A}w_i y_i}{\sum_{i\in A}w_i}=\hat{y}_A. $$

The second derivative is $2\sum_{i\in A}w_i>0$, confirming that this stationary point is a minimum. Substituting $\hat{y}_A$ into the objective gives the node’s weighted squared error. Expanding the squares also yields a useful computational identity:

\operatorname{SSE}_w(A) =\sum_{i\in A}w_i y_i^2 -\frac{\left(\sum_{i\in A}w_i y_i\right)^2}{\sum_{i\in A}w_i}.

This identity can make hand calculations shorter, but it does not change the definition: the score is the sum of weighted squared deviations from the weighted mean.

Worked Example: Finding a Weighted Node Prediction and Error

A hypothetical regression tree has three observations in one node. Their response values are 4, 10, and 13, with respective weights 1, 2, and 1. Find the terminal-node prediction and weighted squared error. Then check the result using the computational identity.

Step 1: Find total weight and weighted response total. The total weight is $1+2+1=4$. The weighted response total is $1(4)+2(10)+1(13)=4+20+13=37$. Thus the node prediction is $37/4=9.25$.

Step 2: Calculate the weighted squared deviations. The deviations from 9.25 are $-5.25$, $0.75$, and $3.75$. Their weighted squared contributions are $1(5.25^2)=27.5625$, $2(0.75^2)=1.125$, and $1(3.75^2)=14.0625$. Adding them gives $27.5625+1.125+14.0625=42.75$.

Step 3: Check by the identity. The weighted sum of squared responses is $1(4^2)+2(10^2)+1(13^2)=16+200+169=385$. The identity gives $385-37^2/4=385-1369/4=385-342.25=42.75$, matching the direct calculation.

Common mistake and correction. Using the unweighted mean gives $(4+10+13)/3=9$, which is not the least-squares prediction under these weights. It would also give the wrong weighted error: $1(4-9)^2+2(10-9)^2+1(13-9)^2=25+2+16=43$. The correct prediction is 9.25 and its error is 42.75. The weight of 2 on the response 10 must be used in both calculations.

Scoring a Candidate Split

A proposed split divides a parent node into two child nodes. For squared-error regression trees, calculate the weighted mean separately in each child and add the two child errors. The split score is not the average of the two errors, and it is not the error of the parent after keeping its original prediction. The tree’s fitting objective after the split is the sum of the child-node errors.

Split criterion: For child nodes $A_L$ and $A_R$, the weighted split error is $$ \operatorname{SSE}_w(A_L)+\operatorname{SSE}_w(A_R). $$ Among candidate splits being compared, the split with the smaller sum has the lower weighted within-node squared error. The reduction from splitting a parent $A$ is $$ \operatorname{SSE}_w(A)-\left[\operatorname{SSE}_w(A_L)+\operatorname{SSE}_w(A_R)\right]. $$

The parent and both children must use the same observation weights and the same response scale. A positive reduction means the chosen child predictions fit the weighted responses better than the single parent prediction. In standard squared-error splitting, the parent prediction is the weighted mean, so splitting cannot increase the minimum training error: each child could always use the parent prediction, and optimizing its own constant can only maintain or reduce that child’s error.

Partition the observations.
Apply the candidate split rule and assign each observation to exactly one child.

Compute each child’s weighted mean.
For each child, divide its weighted response total by its total weight.

Compute and add child errors.
Use the weighted deviations from each child’s own mean, then sum the two errors.

Compare candidates consistently.
Choose the smaller child-error total, or equivalently the larger reduction from the common parent error.

Worked Example: Comparing All Thresholds for a Weighted Predictor

A hypothetical tree considers splitting on a numeric predictor $x$. The five training observations, listed in increasing $x$, have response values $2,6,7,10,14$ and weights $1,2,1,3,1$, respectively. Evaluate all four thresholds between adjacent predictor values and identify the best split by weighted squared error.

Step 1: Calculate the parent error. The total weight is $1+2+1+3+1=8$, and the weighted response total is $2+12+7+30+14=65$. The parent weighted mean is $65/8=8.125$. The weighted squared-response total is $4+72+49+300+196=621$. Thus the parent error is $621-65^2/8=621-528.125=92.875$.

Step 2: Score the threshold after the first observation. The left child contains only response 2, so its error is zero. In the right child, total weight is 7, weighted response total is 63, and weighted squared-response total is 617. Its error is $617-63^2/7=617-567=50$. The split score is $0+50=50$.

Step 3: Score the threshold after the second observation. The left child has total weight 3, weighted response total 14, and weighted squared-response total 76. Its error is $76-14^2/3=76-196/3=32/3$. The right child has total weight 5, weighted response total 51, and weighted squared-response total 545. Its error is $545-51^2/5=545-520.2=24.8$. The split score is $32/3+24.8=35.4667$ to four decimal places.

Step 4: Score the threshold after the third observation. The left child has total weight 4, weighted response total 21, and weighted squared-response total 125. Its error is $125-21^2/4=125-110.25=14.75$. The right child has total weight 4, weighted response total 44, and weighted squared-response total 496. Its error is $496-44^2/4=496-484=12$. The split score is $14.75+12=26.75$.

Step 5: Score the threshold after the fourth observation. The left child has total weight 7, weighted response total 51, and weighted squared-response total 425. Its error is $425-51^2/7=425-2601/7=374/7=53.4286$ to four decimal places. The right child contains only response 14, so its error is zero. The split score is 53.4286.

Conclusion and check. The four split scores are 50, 35.4667, 26.75, and 53.4286. The smallest is 26.75, so the threshold after the third observation is best among the four candidates. Its reduction from the parent error is $92.875-26.75=66.125$. This method avoids comparing thresholds using only one child’s error; both child errors must be included.

Weights Must Be Carried Through the Whole Comparison

Weights affect both parts of a split calculation: the prediction within each child and the contribution of that child’s observations to its error. A useful equivalent interpretation applies when weights are positive integers: an observation with weight $w_i$ can be viewed as $w_i$ identical copies for purposes of the weighted sum of squares. This explains why a high-weight response has more influence, but it is only a calculation device; noninteger weights are also valid in the formulas.

Multiplying every weight in the data by the same positive constant leaves all weighted means unchanged. It multiplies every node error and every split reduction by that constant, so it does not change which candidate has the smallest error. In contrast, changing only some weights can change both the terminal predictions and the preferred split. Therefore, weights should not be silently normalized, discarded, or applied only after a split has been chosen.

Worked Example: Comparing Two Candidate Splits With Unequal Weights

A hypothetical tree has five observations ordered by predictor $x$. Their responses are $2,5,6,14,16$, and their weights are $2,1,3,1,2$. Compare two candidate thresholds: one after the second observation and one after the third. Use weighted child errors and state the reduction for the better candidate.

Step 1: Find the parent error. The total weight is $2+1+3+1+2=9$. The weighted response total is $2(2)+1(5)+3(6)+1(14)+2(16)=4+5+18+14+32=73$. The weighted squared-response total is $2(2^2)+5^2+3(6^2)+14^2+2(16^2)=8+25+108+196+512=849$. The parent error is $849-73^2/9=849-5329/9=2312/9=256.8889$ to four decimal places.

Step 2: Score the threshold after observation two. The left child has total weight 3, weighted response total 9, and weighted squared-response total 33. Its error is $33-9^2/3=6$. The right child has total weight 6, weighted response total $18+14+32=64$, and weighted squared-response total $108+196+512=816$. Its error is $816-64^2/6=816-2048/3=400/3=133.3333$ to four decimal places. The split score is $6+400/3=418/3=139.3333$.

Step 3: Score the threshold after observation three. The left child has total weight 6, weighted response total $4+5+18=27$, and weighted squared-response total $8+25+108=141$. Its error is $141-27^2/6=141-121.5=19.5$. The right child has total weight 3, weighted response total $14+32=46$, and weighted squared-response total $196+512=708$. Its error is $708-46^2/3=708-2116/3=8/3=2.6667$ to four decimal places. The split score is $19.5+8/3=133/6=22.1667$ to four decimal places.

Conclusion and check. The threshold after observation three is better because $22.1667<139.3333$. Its reduction is $256.8889-22.1667=234.7222$ to four decimal places. The exact reduction is $21125/90$, or $4225/18$. As expected, the chosen split has a positive reduction from the parent error.

Common Errors to Avoid

Weighted split calculations are mechanical once the node membership is clear, but a few shortcuts can change the answer. Check each of the following before reporting a split score:

Using the ordinary mean. Use $\sum w_i y_i/\sum w_i$, not the unweighted average, unless all weights are equal.
Using the parent mean in both children. Each child gets its own weighted mean when evaluating the split.
Forgetting to weight squared deviations. The child error is $\sum w_i(y_i-\hat y)^2$, not merely the sum of squared deviations.
Scoring only one child. Add the left and right errors; a small error in one child says nothing by itself about the total split score.
Dividing by node weight when the criterion is SSE. The weighted squared-error criterion is a sum, not a mean squared error. Dividing by a common total weight across all candidate splits would preserve their ranking, but dividing each child separately and then averaging changes the criterion.

Key takeaway: For each node, use the weighted mean as its constant prediction and sum the weighted squared deviations from that mean. For a candidate split, add the two child errors and compare that total with the other candidate totals.

Check Your Understanding

For each problem, use the stated weights in both the prediction and error calculations. Give errors to two decimal places where needed.

A node contains responses 5 and 11 with weights 2 and 1. Find its weighted prediction and weighted squared error. (Answer: prediction $7$; error $24$.)
A node contains responses 1, 4, and 8 with weights 1, 2, and 1. Find the weighted mean and weighted squared error. (Answer: mean $4.25$; error $24.75$.)
Four observations are ordered by a predictor, with responses $1,4,7,10$ and weights $1,1,2,1$. Compare the split after observation two with the split after observation three. Which is better, and what is its weighted child-error total? (Answer: the split after observation two; total error $10.50$.)
A node contains responses 2 and 6 with weights 1 and 3. Calculate the weighted mean and error, then multiply both weights by 5. What changes? (Answer: the mean stays $5$; the error changes from $12$ to $60$.)

What You'll Learn