Why the Response Scale Matters
A regression tree divides observations into groups and assigns a constant prediction to each terminal node. For the usual squared-error criterion, a candidate split is judged by how much it reduces variation within its child nodes. The response values determine that variation, so changing their scale can change which observations contribute most to the score—and potentially which split is selected.
This is different from changing the units of a predictor. A change such as measuring distance in metres rather than centimetres can preserve the order of predictor values and hence the available partitions. Transforming the response, for example by taking logarithms, changes the distances between response values used in the squared-error calculation.
This definition assumes the ordinary, unweighted squared-error criterion: every observation contributes one squared deviation. It also assumes that the response values being compared are on the same scale. If the response is transformed, the tree uses squared deviations of the transformed values, not of the original values. The discussion here concerns split selection and terminal predictions; it does not imply that a transformation is automatically appropriate for every response.
How a Response Transformation Changes the Criterion
Let \(Y\) be the response and let \(g(Y)\) be a transformed response. A tree fitted to \(g(Y)\) minimizes squared deviations of \(g(y_i)\) from the mean transformed response in each node. A nonlinear \(g\) can expand some gaps and compress others. Consequently, a group of observations that looks compact on the original scale may be relatively spread out on the transformed scale, or vice versa.
For positive outcomes, a common choice is \(g(y)=\ln(y)\). The logarithm often makes multiplicative differences easier to compare: the gap between 2 and 4 is the same on the log scale as the gap between 10 and 20, because both ratios are 2. On the original scale, those differences are 2 and 10. Logarithms require positive responses; they cannot be applied directly to zero or negative values.
Worked Example: A Log Transformation Reverses the Preferred Cut
A hypothetical insurer records four positive claim amounts, ordered by a predictor \(x\). The amounts, in thousands of dollars, are \(1,10,11,100\). Three possible cuts separate the first observation, the first two observations, or the first three observations from the rest. Compare the child squared-error totals on the original and log scales.
Method. For each candidate, calculate the sum of squared deviations from each child’s own mean. Then repeat using the natural logarithms of the responses. The split scores are totals of within-child errors, not differences between the two child means.
Original scale, cut after the first observation. The first child is \(\{1\}\), with score \(0\). The second child is \(\{10,11,100\}\), whose mean is \(121/3\). Its score is \[ (10-121/3)^2+(11-121/3)^2+(100-121/3)^2 =16022/3\approx 5340.6667. \]
Original scale, cut after the second observation. The children are \(\{1,10\}\) and \(\{11,100\}\). For two observations, the score is half their squared difference. Thus the total is \[ (10-1)^2/2+(100-11)^2/2=40.5+3960.5=4001. \]
Original scale, cut after the third observation. The first child is \(\{1,10,11\}\), with mean \(22/3\), and the second child is the singleton \(\{100\}\), with score \(0\). The first child’s score is \[ 1^2+10^2+11^2-\frac{22^2}{3} =222-\frac{484}{3} =\frac{182}{3}\approx 60.6667. \] The smallest original-scale score is \(60.6667\), so the original-scale tree cuts after the third observation.
Log scale. The transformed values are approximately \(0,\ 2.302585,\ 2.397895,\ 4.605170\). For the cut after the first observation, the three-value child has mean \(3.101883\), giving a score of approximately \[ (2.302585-3.101883)^2+(2.397895-3.101883)^2+(4.605170-3.101883)^2 \approx 3.3943. \] The cut after the second observation has two pairs. Its score is \[ \frac{(2.302585-0)^2}{2} +\frac{(4.605170-2.397895)^2}{2} \approx 2.6510+2.4360=5.0870. \] For the cut after the third observation, the first child’s mean is approximately \(1.566827\); its score is approximately \(3.6880\), while the singleton’s score is zero. The smallest log-scale score is therefore approximately \(3.3943\), so the log-scale tree cuts after the first observation.
Check. The original-scale choice isolates the largest response, while the log-scale choice isolates the smallest. The scores have different units and should not be compared numerically across scales; compare candidate cuts within each scale. The common mistake is to assume that the preferred cut must be unchanged just because the observations and predictor are unchanged.
Terminal Predictions Depend on the Scale Too
For squared-error fitting, the prediction in a terminal node is the mean of the response values used to fit the tree. If the tree is fitted to \(g(Y)\), its terminal prediction is the mean of \(g(y_i)\). Applying \(g^{-1}\) to that prediction gives a value on the original response scale, but it generally does not give the original-scale arithmetic mean.
Worked Example: Reporting a Log-Scale Terminal Prediction
A hypothetical terminal node contains two positive payment amounts, \(8\) and \(20\) thousand dollars. Calculate its fitted prediction if the tree is fitted to the natural logarithm of payment, and compare it with the prediction from a tree fitted directly to payment.
Method. On the original scale, the squared-error prediction is the arithmetic mean: \[ (8+20)/2=14. \] Thus the direct-scale tree predicts \(14\) thousand dollars.
On the log scale, the transformed observations are \(\ln(8)\approx2.079442\) and \(\ln(20)\approx2.995732\). Their mean is \[ (2.079442+2.995732)/2=2.537587. \] Reversing the transformation gives \[ \exp(2.537587)=\sqrt{8\cdot20}=\sqrt{160}\approx12.6491. \] Thus the back-transformed log-scale prediction is about \(12.6491\) thousand dollars, not \(14\) thousand dollars. The geometric-mean calculation checks the result directly. The common mistake is to exponentiate the mean of the logs and call the result the arithmetic mean.
The same pair also illustrates a split-score calculation. If \(8\) and \(20\) form one child and a separate singleton forms the other, the pair’s log-scale squared-error score is \[ \frac{(\ln(20)-\ln(8))^2}{2} =\frac{(2.995732-2.079442)^2}{2} \approx 0.4198. \] The singleton contributes zero. Rounding this score to four decimal places gives \(0.4198\), not \(0.4197\). This calculation concerns a within-child score; it is not the difference between child means.
A back-transformed log prediction is not automatically an estimate of the original-scale conditional mean. It is the geometric mean of the observed responses in that node. If the practical goal is to estimate an arithmetic mean payment, the analyst must consider whether fitting and evaluating predictions on the log scale answers that goal. The transformation changes both the fitting criterion and the direct interpretation of the fitted values.
Why a Change of Units Is Different
A positive affine transformation of the response, \(y_i^*=a+b y_i\) with \(b\ne0\), preserves the ranking of candidate splits under squared error. It changes the numerical scores, but every score is multiplied by \(b^2\). This contrasts with a nonlinear transformation such as a logarithm, which need not multiply every candidate score by the same constant.
For a node, \(y_i^*=a+b y_i\), and its transformed mean is \(\bar{y}^*=a+b\bar{y}\).
\(y_i^*-\bar{y}^*=b(y_i-\bar{y})\), because the common shift \(a\) cancels.
Squaring and summing gives \(\sum_i(y_i^*-\bar{y}^*)^2=b^2\sum_i(y_i-\bar{y})^2\). The same factor applies to every child and every candidate split.
Since \(b^2\) is positive, multiplying all candidate scores by it cannot change their order. A shift in response units alone has no effect on the split order; a nonzero change of scale changes score magnitudes but preserves the minimizing split. This result concerns the ordinary unweighted squared-error score and the same set of candidate partitions.
Worked Example: Converting Response Units Without Changing the Split
A hypothetical data set has predictor-ordered responses \(3,5,9,13\). Candidate cuts occur after the first, second, or third observation. A second analyst records the response as \(y^*=2y+7\). Determine the preferred cut on both scales and check the score transformation.
Original-scale scores. After the first observation, the remaining values \(5,9,13\) have mean \(9\) and score \(16+0+16=32\). After the second, the two child scores are \((5-3)^2/2=2\) and \((13-9)^2/2=8\), for a total of \(10\). After the third, the first child \(\{3,5,9\}\) has mean \(17/3\) and score \[ 3^2+5^2+9^2-\frac{17^2}{3} =115-\frac{289}{3} =\frac{56}{3}\approx18.6667. \] The second-observation cut is preferred, with score \(10\).
Transformed scale. The responses become \(13,17,25,33\). For the same second-observation cut, the child scores are \((17-13)^2/2=8\) and \((33-25)^2/2=32\), totaling \(40\). This equals \(2^2(10)=40\). The first- and third-observation cut scores likewise become \(128\) and \(224/3\), respectively, so the second-observation cut remains preferred.
Check. The predictions for the selected leaves change from \(4\) and \(11\) to \(15\) and \(29\), exactly as \(y^*=2y+7\) requires. The common mistake is to treat any response rescaling as if it could change the split order; that is true for a nonlinear transformation in general, not for this affine change.
Common Mistakes and Practical Interpretation
- Comparing scores from different scales directly. A dollar-scale score and a log-scale score have different meanings and units. Use each set of scores to rank candidate splits on its own scale.
- Using the parent score as the candidate split score. The split score is the sum of the child-node errors. The parent score is useful for computing the reduction, but it is the same for every candidate cut of that parent.
- Assuming the transformed prediction is an arithmetic mean. A log-scale squared-error fit predicts the mean of the logs; exponentiating gives a geometric mean.
- Applying a logarithm outside its domain. The natural logarithm is defined only for positive values. Zero and negative responses require a different approach if a transformation is needed.
- Confusing response transformations with predictor-unit changes. A monotone rescaling of a predictor can preserve available partitions, whereas changing the response changes the errors used to score those partitions.
Check Your Understanding
For each problem, distinguish the scale used to calculate the score from the scale used to report the prediction.
- A node contains responses \(3\) and \(7\). Calculate its ordinary squared-error score and its squared-error score after taking natural logarithms. (Answer: Original-scale score \(=8\); log-scale score \(=(\ln(7/3))^2/2\approx0.3590\).)
- A terminal node contains positive responses \(4\) and \(12\). What prediction does a squared-error tree fitted to their logs give after exponentiation, and what is the direct-scale arithmetic-mean prediction? (Answer: Log-scale prediction \(\sqrt{48}\approx6.9282\); direct-scale prediction \(8\).)
- For the ordered responses \(1,4,9,36\), compare cuts after observations 2 and 3 on the original scale. Which has the smaller score? (Answer: Cut after observation 3; its score is \(98-196/3=98/3\approx32.6667\), compared with \(369\) after observation 2.)
- For the same ordered responses \(1,4,9,36\), compare cuts after observations 2 and 3 on the log scale. Which has the smaller score? (Answer: Cut after observation 2; its score is \((\ln4)^2\approx1.9218\), compared with approximately \(5.318\) after observation 3.)
- Every response is changed from \(y\) to \(5-3y\), while the candidate partitions stay fixed. By what factor does each squared-error score change, and can the preferred split change? (Answer: Each score is multiplied by \(9\); the preferred split cannot change under the ordinary squared-error criterion.)