Compare Regression Trees And Neighbors For Nonlinear Risk

Why Compare These Two Prediction Methods?

A continuous risk measure, such as a hypothetical repair cost or claim amount, may change nonlinearly with a predictor. A straight-line model can miss a sharp change or a curved pattern. Regression trees and k-nearest neighbors (KNN) can represent nonlinear relationships without requiring a single linear equation, but they adapt in different ways.

A regression tree divides the predictor space into regions and assigns a constant prediction to each region. KNN instead predicts at a particular point by averaging the responses of nearby training observations. This distinction matters: a tree’s boundaries are determined by splits, while a neighbor prediction depends on distances and the chosen number of neighbors, \(k\). Both methods must be compared on observations not used to fit them if the goal is to assess prediction on new data.

Definition: In regression, a terminal node is a final region of predictor space in a fitted tree. Its prediction is the mean response of the training observations in that node. For KNN regression with \(k\) neighbors, the prediction at a new predictor value is the mean response among its \(k\) closest training observations, using the specified distance measure.

Regression-Tree Predictions and Split Error

For a node containing responses \(y_1,\ldots,y_m\), the prediction that minimizes the sum of squared errors within that node is their arithmetic mean. If a proposed split creates left and right nodes, its training error is the sum of squared deviations from the two node means. A tree-building procedure can compare candidate splits using this quantity, subject to any specified restrictions such as a minimum node size.

\bar{y}_{\text{node}}=\frac{1}{m}\sum_{i=1}^{m}y_i, \qquad \operatorname{SSE}_{\text{split}} =\sum_{i\in L}(y_i-\bar{y}_L)^2+ \sum_{i\in R}(y_i-\bar{y}_R)^2

For a predictor \(x\), a split at threshold \(c\) commonly places observations with \(x\le c\) on one side and observations with \(x>c\) on the other. Within a terminal node, every observation receives the same prediction. As a result, the fitted function is piecewise constant: it can describe changes across regions, but does not interpolate smoothly within a region.

Justification: For responses \(y_1,\ldots,y_m\), consider the squared-error total at a proposed constant prediction \(a\). Write \(y_i-a=(y_i-\bar{y})+(\bar{y}-a)\). Since \(\sum_i(y_i-\bar{y})=0\), expanding and summing gives \(\sum_i(y_i-a)^2=\sum_i(y_i-\bar{y})^2+m(a-\bar{y})^2\). The second term is nonnegative and is zero at \(a=\bar{y}\). Thus the node mean minimizes squared error.

Worked Example: Choosing a Split for a Nonlinear Pattern

A hypothetical training sample has one predictor \(x=1,2,3,4,5,6\) and responses \(y=0,1,4,9,16,25\). Consider a tree with one split and squared-error criterion. Compare thresholds \(c=0.5,1.5,2.5,3.5,4.5\), using \(x\le c\) for the left node. Which threshold has the smallest training SSE?

Method. For each threshold, calculate the mean response in each resulting node, then add the squared deviations from the corresponding node means. No tree-size penalty is included in this comparison.

At \(c=0.5\), the left node contains \(0\), with mean \(0\). The right node contains \(1,4,9,16,25\), with mean \(55/5=11\). Therefore, \( \operatorname{SSE}=0+(1-11)^2+(4-11)^2+(9-11)^2+(16-11)^2+(25-11)^2 =0+100+49+4+25+196=374.\)

At \(c=1.5\), the left node contains \(0,1\), with mean \(0.5\); its SSE is \((0-0.5)^2+(1-0.5)^2=0.25+0.25=0.5\). The right node contains \(4,9,16,25\), with mean \(54/4=13.5\). Its SSE is \(90.25+20.25+6.25+132.25=249\). The total is \(249.5\).

At \(c=2.5\), the left node contains \(0,1,4\), with mean \(5/3\). Its SSE is \(25/9+4/9+49/9=78/9=26/3\). The right node contains \(9,16,25\), with mean \(50/3\). Its SSE is \(529/9+4/9+625/9=1158/9=386/3\). The total is \(412/3=137.3333\ldots\).

At \(c=3.5\), the left node contains \(0,1,4,9\), with mean \(3.5\), and its SSE is \(12.25+6.25+0.25+30.25=49\). The right node contains \(16,25\), with mean \(20.5\), and its SSE is \(20.25+20.25=40.5\). The total is \(89.5\).

At \(c=4.5\), the left node contains \(0,1,4,9,16\), with mean \(30/5=6\); its SSE is \(36+25+4+9+100=174\). The right node contains \(25\), so its SSE is \(0\). The total is \(174\).

The five total SSE values, in threshold order, are \(374,\ 249.5,\ 137.3333\ldots,\ 89.5,\ 174\). The smallest is \(89.5\), so \(c=3.5\) is best among these candidates. The calculation is checked by adding the two node errors at that threshold: \(49+40.5=89.5\).

K-Nearest-Neighbor Regression

KNN requires a distance rule, a value of \(k\), and a convention for ties at the \(k\)th distance. In a one-predictor problem, Euclidean distance is the absolute difference in predictor values. With several predictors, distance can be dominated by variables measured on larger numerical scales unless the predictors are appropriately scaled. Any scaling parameters must be estimated using training data, not the holdout data.

\widehat{f}_{\text{KNN}}(x_0) =\frac{1}{k}\sum_{i\in N_k(x_0)} y_i

Here \(N_k(x_0)\) is the set of indices for the \(k\) training observations closest to the new point \(x_0\). This is an unweighted average; variants can use distance weights, but the examples here use the unweighted definition. A small \(k\) makes predictions depend on a small local sample and can be sensitive to individual responses. A larger \(k\) averages over a wider neighborhood and may smooth over local variation.

Worked Example: Finding a Local Risk Estimate

A hypothetical training set records a predictor \(x\) and a continuous risk measure \(y\): \((0,0),(1,1),(2,4),(4,16),(5,25)\). Predict at \(x_0=3.2\) using unweighted KNN with \(k=3\). Also calculate the prediction for \(k=1\) to show the effect of neighborhood size.

Method. Compute the absolute predictor distances from \(3.2\), sort them, and average the responses for the required nearest observations.

The distances for \(x=0,1,2,4,5\), respectively, are \(3.2,2.2,1.2,0.8,1.8\). In increasing order they are \(0.8\) at \(x=4\), \(1.2\) at \(x=2\), \(1.8\) at \(x=5\), \(2.2\) at \(x=1\), and \(3.2\) at \(x=0\). There is no tie at the third-neighbor boundary.

For \(k=3\), the selected responses are \(16,4,25\). Their sum is \(45\), so the prediction is \(45/3=15\). For \(k=1\), only the closest observation, \((4,16)\), is used, giving prediction \(16\).

The calculations are consistent with the specified distance rule: the nearest three predictor values are \(4,2,5\), not the three observations with responses closest to one another. A common mistake is to select neighbors by response similarity, which is unavailable when predicting a new response.

Comparing Adaptation and Holdout Error

A tree and KNN can both fit nonlinear patterns, but their fitted shapes differ. A tree creates regions with constant predictions, and a split can produce an abrupt jump at a boundary. KNN produces a local average that can change as the neighborhood changes; it is not necessarily smooth, especially for small \(k\), but it does not impose the same fixed rectangular partition as a tree.

Feature	Regression tree	KNN regression
How prediction is formed	Mean response in the terminal node	Mean response among the \(k\) closest training observations
How local structure is chosen	Split thresholds define regions	Distance and \(k\) define the neighborhood at each prediction point
Typical sensitivity	Small changes in data can change a split or node assignment	Small \(k\) can be sensitive to nearby observations; large \(k\) can smooth over local features
Important setup choice	Tree complexity and split restrictions	\(k\), distance rule, and predictor scaling

For a holdout set of \(n\) observations, mean squared error is the average squared difference between the observed response and the prediction made without using that response to fit the model. The models must be assessed on the same holdout cases and with the same loss measure for a direct comparison. Training SSE is not a substitute for holdout error: a flexible fit can reduce training error without improving prediction on new cases.

Worked Example: Correctly Comparing Holdout Predictions

A hypothetical holdout set has actual responses \(0,1,4,9,16,25\). A previously fitted tree predicts \(0.5,13.5,13.5,13.5,13.5,13.5\), while a previously fitted KNN model predicts \(0.5,2.5,6.5,12.5,20.5,24\). Calculate each model’s holdout SSE and MSE, then identify the lower-error model.

Method. For each model, subtract its prediction from each actual response, square each residual, sum the six squared residuals to obtain SSE, and divide by \(n=6\) to obtain MSE.

For the tree, the squared residuals in order are \((0-0.5)^2=0.25\), \((1-13.5)^2=156.25\), \((4-13.5)^2=90.25\), \((9-13.5)^2=20.25\), \((16-13.5)^2=6.25\), and \((25-13.5)^2=132.25\). Thus the holdout SSE is \(0.25+156.25+90.25+20.25+6.25+132.25=405.5\), and the holdout MSE is \(405.5/6=67.5833\ldots\).

For KNN, the squared residuals are \((0-0.5)^2=0.25\), \((1-2.5)^2=2.25\), \((4-6.5)^2=6.25\), \((9-12.5)^2=12.25\), \((16-20.5)^2=20.25\), and \((25-24)^2=1\). The holdout SSE is \(0.25+2.25+6.25+12.25+20.25+1=42\), and the holdout MSE is \(42/6=7\).

KNN has the smaller holdout MSE in this comparison. To three decimal places, the MSEs are \(67.583\) for the tree and \(7.000\) for KNN. The arithmetic is checked by multiplying each MSE by six to recover its SSE. This comparison says which model predicted these six holdout responses better; it does not establish that KNN will always perform better.

Common Traps in a Continuous-Risk Comparison

Confusing training SSE with holdout SSE. A candidate split is usually evaluated on its training observations. Holdout error uses separate observations and the predictions made for them.
Using the wrong node mean or neighbor set. A tree prediction is the mean response in the relevant terminal node. A KNN prediction uses responses of the closest training observations, selected by predictor distance.
Failing to recompute the arithmetic. For example, the squared-error sum \(0.25+156.25+90.25+20.25+6.25+132.25\) is \(405.5\), not \(249.5\). The two totals refer to different calculations when one is a candidate-split training SSE and the other is the stated holdout residual sum; they must not be substituted for one another.
Choosing \(k\) or tree complexity from the final test set. These are tuning choices. Use training and validation procedures for selection, then reserve the test set for final assessment.
Ignoring predictor scale in KNN. When predictors have substantially different units, a distance measure can give one predictor disproportionate influence. Fit any scaling using training data and apply that same transformation to later data.

Key takeaway: A regression tree predicts with terminal-node means; KNN predicts with local response averages. Compare their predictions on the same untouched holdout observations, calculate the chosen error measure carefully, and interpret the result as evidence for that evaluation setting rather than a universal ranking. This treatment is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

Work each problem using the stated prediction rule and show the intermediate calculations.

A one-split tree places responses \(2,4\) in its left node and \(10,12\) in its right node. Find both terminal-node predictions and the total training SSE. (Answer: left mean \(3\), right mean \(11\), total SSE \(4\).)
A training set has predictor-response pairs \((1,3),(2,5),(4,9),(7,15)\). Using absolute distance and \(k=2\), predict at \(x_0=3\). Identify the two neighbors before averaging. (Answer: neighbors at \(x=2\) and \(x=4\); prediction \(7\).)
On three holdout cases, actual responses are \(4,8,12\). A tree predicts \(5,10,10\), and KNN predicts \(4,9,11\). Calculate both MSEs and state which is lower. (Answer: tree MSE \(=9/3=3\); KNN MSE \(=2/3\); KNN is lower.)
For a two-predictor KNN problem, use distance \(d=\sqrt{(x_1-x_{01})^2+((x_2-x_{02})/2)^2}\). The query is \((1.2,0.2)\); training points \(A=(1,0)\), \(B=(0,2)\), and \(C=(2,1)\) have responses \(10,30,22\), respectively. Find the two nearest points and their unweighted prediction. (Answer: distances \(d_A=\sqrt{0.05}\), \(d_C=\sqrt{0.80}\), \(d_B=1.5\); the two nearest are \(A,C\), and the prediction is \(16\).)

What You'll Learn