Reading Error Along a Pruning Sequence
A classification tree can fit its training observations more closely as it grows. That does not guarantee better predictions for new observations. Reading training and validation error together helps distinguish improved fit to the training data from evidence that added branches are fitting patterns that do not generalize.
The focus here is a sequence of nested subtrees, from a simpler pruned tree to a more complex tree. We will track the fraction of observations misclassified at each size. The validation error can help compare candidate sizes, but it is an estimate based on a finite sample, not a guarantee about future performance.
For example, if a tree misclassifies 9 of 60 validation observations, its validation error is \(9/60=0.15\), or 15%. The denominator matters: compare error rates, not raw error counts, whenever the sets being compared have different sizes. In a pruning comparison, every candidate should ordinarily be evaluated on the same training set and the same validation set.
Why Training Error Usually Falls as a Tree Grows
A pruning sequence consists of nested trees: a more complex tree can retain the distinctions made by a simpler tree and may add further splits. When each terminal node is assigned its majority class, adding a split cannot increase the minimum possible training misclassification count. The children can, at worst, predict the same class as their parent; they may instead classify some training observations correctly that the parent misclassified.
This is a statement about training performance under the usual majority-class leaf predictions, not a promise about validation performance. Validation observations were not used to determine the tree's splits or leaf classes, so additional branches can help, do nothing, or hurt their classification.
The practical pattern of concern is therefore not merely that training error decreases. That is expected. It is that training error continues to fall while validation error stops improving and begins to rise. Such a divergence is evidence consistent with overfitting: increasing complexity improves fit to the training sample but worsens performance on the held-out sample.
Worked Example: Finding the Validation Error Minimum
A hypothetical insurer develops a tree to classify whether a submitted claim requires additional review. The same 100 training claims and 50 separate validation claims are scored at each stage of a pruning sequence. The table gives the number of errors. Calculate both error rates, identify the validation minimum, and interpret the trend.
| Terminal nodes | Training errors out of 100 | Validation errors out of 50 |
|---|---|---|
| 1 | 24 | 10 |
| 3 | 16 | 4 |
| 6 | 9 | 8 |
| 10 | 3 | 13 |
Method. Divide each error count by its own sample size. For the one-node tree, training error is \(24/100=0.24=24\%\), and validation error is \(10/50=0.20=20\%\).
For the three-node tree, the rates are \(16/100=16\%\) on training data and \(4/50=8\%\) on validation data. For the six-node tree, they are \(9/100=9\%\) and \(8/50=16\%\). For the ten-node tree, they are \(3/100=3\%\) and \(13/50=26\%\).
The validation error is lowest at three terminal nodes: \(4/50=8\%\). After that point, training error falls from 16% to 9% to 3%, while validation error rises from 8% to 16% to 26%. This pattern is evidence that the larger trees are increasingly fitting features of these training claims that do not carry over to this validation set.
Check. The validation counts are 10, 4, 8, and 13 out of the same 50 claims, so their rates are 20%, 8%, 16%, and 26%. The minimum is indeed 8%, at three nodes. Common mistake avoided: choosing the ten-node tree because its training error is smallest. That criterion rewards fit to the training data alone and ignores the validation results.
Interpreting the Shape, Not Just the Minimum
A validation curve need not fall smoothly and then rise smoothly. It may fluctuate because it is computed on a finite set of observations. The useful questions are: where is the lowest observed validation error, how large are the differences among nearby candidates, and does the broader pattern suggest that added complexity is helping or harming held-out performance?
A validation error higher than training error is common, because the tree was built using the training observations. A widening difference can be a warning sign, but it is not a formal diagnosis by itself. In particular, a validation set that is small or unrepresentative may give a noisy or misleading comparison. The training and validation observations should be separated appropriately, and the validation data should not be used to grow the tree.
Worked Example: A Rising Validation Error After Extra Splits
A hypothetical clinic uses a classification tree to flag appointments at risk of being missed. A fixed set of 120 appointments is used for training, and a separate set of 80 appointments is used for validation. Compare three nested trees and explain whether the added complexity is supported by the validation results.
| Terminal nodes | Training errors out of 120 | Validation errors out of 80 |
|---|---|---|
| 2 | 30 | 16 |
| 5 | 18 | 10 |
| 9 | 9 | 17 |
Method. For the two-node tree, the training rate is \(30/120=25\%\), and the validation rate is \(16/80=20\%\). For the five-node tree, the rates are \(18/120=15\%\) and \(10/80=12.5\%\). For the nine-node tree, they are \(9/120=7.5\%\) and \(17/80=21.25\%\).
The five-node tree has the lowest observed validation error, 12.5%. Growing from five to nine nodes reduces training error by \(15\%-7.5\%=7.5\) percentage points but increases validation error by \(21.25\%-12.5\%=8.75\) percentage points. The validation results do not support that extra complexity for this comparison; the pattern is consistent with overfitting at nine nodes.
Check. The denominator for training remains 120 and for validation remains 80 throughout, so the stated rates follow directly from each row's counts. Common mistake avoided: comparing 9 training errors with 10 validation errors as though they were directly comparable counts. They come from sets of different sizes; the corresponding rates, 7.5% and 12.5%, are the meaningful comparison.
Why Training Error Alone Cannot Choose the Subtree
Training error measures how well a tree classifies the observations used to construct it. Because a larger nested tree has more opportunities to separate observations into leaves, its training error is generally no larger than that of a smaller tree. Consequently, minimizing training error over a pruning sequence will tend to favor the largest tree, even when its validation performance is worse.
Validation error provides a comparison on observations not used to grow the tree. If the validation observations reasonably represent the cases to which the model will be applied, their error rates offer evidence about comparative predictive performance. They do not prove which tree will perform best on every future sample. A lower observed rate can reflect both genuine improvement and chance variation.
Worked Example: A Tie in Validation Error
A hypothetical equipment provider classifies service requests as routine or urgent. It compares two nested trees on the same 120 training requests and the same 60 validation requests. The four-node tree makes 15 training errors and 9 validation errors. The seven-node tree makes 8 training errors and also 9 validation errors. Calculate the error rates and interpret the tie.
Method. For the four-node tree, training error is \(15/120=12.5\%\), and validation error is \(9/60=15\%\). For the seven-node tree, training error is \(8/120=6.67\%\) to two decimal places, and validation error is again \(9/60=15\%\).
The larger tree fits the training set better: its training error is lower by \(12.5\%-6.67\%=5.83\) percentage points. But it has no observed validation-error advantage. Based only on these two validation results, the data do not show that the additional splits improve classification on held-out requests. If other considerations are equal, the smaller tree may be preferred for simplicity, but the tie does not establish that it has lower future error.
Check. Both validation rates are \(9/60=15\%\), while the training rates use denominator 120 and differ. Common mistake avoided: treating the lower training error of the seven-node tree as proof that it generalizes better. The validation comparison is tied, so that conclusion is not supported by these results.
Common Traps in Reading Error Tables
- Selecting the smallest training error. In a nested sequence, training error usually favors greater complexity. It cannot, by itself, establish which tree predicts best on new cases.
- Comparing counts instead of rates. Convert each count to a proportion using the correct sample size before comparing results.
- Calling every rise in validation error proof of overfitting. A rise is evidence to interpret, not certainty. Finite validation samples can produce fluctuations.
- Ignoring how validation data were used. Validation error is most useful as a held-out comparison when those observations were not used to grow the tree. Repeatedly adapting decisions to the same validation set can make its apparent performance optimistic.
- Overinterpreting a tiny difference. Nearby observed rates can be close enough that the sample does not give a clear practical distinction. Report the counts and denominators, not just a ranking.
Check Your Understanding
For each problem, calculate the requested rates before interpreting the tree comparison.
- A tree makes 12 errors among 80 validation observations. What is its validation error rate? (Answer: \(12/80=15\%\).)
- Two nested trees have training errors of 22, 14, and 7 out of 100 observations, and validation errors of 8, 6, and 11 out of 40 observations, respectively. Which tree has the lowest observed validation error, and what happens after that point? (Answer: The second tree has the minimum, \(6/40=15\%\); the third tree's validation error rises to \(11/40=27.5\%\) while its training error falls to 7%.)
- A four-node tree makes 9 validation errors out of 75 cases; a six-node tree makes 10 errors out of the same 75 cases. What are the rates, and which has the lower observed validation error? (Answer: \(9/75=12\%\) and \(10/75=13.33\%\); the four-node tree is lower by about 1.33 percentage points.)
- A tree has 18 training errors out of 120 observations and 13 validation errors out of 50. Calculate both rates and their difference, with validation minus training in percentage points. (Answer: Training error is 15%; validation error is 26%; the difference is \(26\%-15\%=11\) percentage points.)
- A smaller tree has training error 6% and validation error 19%; a nested larger tree has training error 2% and validation error 28%. Which tree has lower validation error, and what does the pattern suggest? (Answer: The smaller tree has lower validation error, 19% versus 28%. The larger tree's lower training error paired with higher validation error is evidence consistent with overfitting.)