Class Errors Across Terminal Nodes

Why Examine Errors by Terminal Node?

A classification tree assigns a predicted class to each terminal node, also called a leaf. An overall error rate summarizes how often the tree is wrong, but it does not show where those errors occur. Errors concentrated in one leaf may indicate that a particular path through the tree leaves important variation unexplained. Errors spread across leaves may point to a broader limitation of the model. Either way, examining the terminal nodes provides a more useful diagnosis than accuracy alone.

For a professional analysis, the distinction between training and held-out data matters. Counts calculated from the observations used to fit a tree describe its apparent, or training, errors. They can help explain the fitted model, but they do not by themselves establish how well it will classify new observations. Held-out observations can show whether a pattern continues beyond the fitting data.

The empirical class proportions describe the composition of the observations reaching a leaf. Under the usual interpretation, they estimate the conditional class probabilities for observations with predictor values in that region. They are not guarantees that a future observation in the same region belongs to the most common class. A node can have a majority class and still have a substantial error rate.

Calculating Node Errors

Suppose there are \(K\) possible classes and the tree uses equal cost for every incorrect classification. The predicted class in node \(j\) is the class with the largest observed count. If there is a tie, the prediction depends on the software’s tie-breaking convention; the calculations below assume there is a unique largest count.

Here, \(\widehat{p}_{jk}\) is the observed proportion of class \(k\) in node \(j\), \(\widehat{c}_j\) is the node’s predicted class, and \(e_j\) is its number of errors. The node’s apparent error rate is \(e_j/n_j\). Across all terminal nodes, the overall apparent error rate is the total number of errors divided by the total number of observations.

This overall rate weights each node according to its size. That is appropriate when the goal is the fraction of all observations classified incorrectly. It differs from the unweighted average of the node error rates, which gives a small leaf the same influence as a large one. Neither summary replaces the node-by-node counts: a leaf with the largest error rate need not contribute the largest number of errors.

Class-Specific Errors Reveal the Direction of Mistakes

An overall error count treats all incorrect classifications alike. For a two-class problem, it can also be useful to identify the direction of errors. If the tree predicts A in a leaf, observations actually in B are errors of the B-to-A type. If it predicts B, observations actually in A are errors of the A-to-B type. These errors can have different practical consequences, so report them separately when the decision context makes that distinction important.

For several terminal nodes, add the relevant counts across the leaves. For example, the number of actual B observations predicted as A is the sum of \(n_{jB}\) over leaves whose prediction is A. A class-specific error rate uses the number of actual observations of that class as its denominator. State the denominator clearly: a rate among actual B cases is not the same as the fraction of predictions of A that are wrong.

Why the Majority-Count Formula Works

The node error formula follows directly from the equal-cost classification rule. Consider a fixed terminal node containing \(n_j\) observations. If the tree predicts class \(k\), then every observation in that node whose actual class is not \(k\) is an error. The number of errors is therefore \(n_j-n_{jk}\). To minimize that count, choose the class with the largest \(n_{jk}\).

Dividing by \(n_j\) gives the node’s minimum apparent error rate under this rule:

The same reasoning applies to conditional probabilities: if the true class probabilities within a region were \(p_{jk}\), predicting the most probable class would have conditional error probability \(1-\max_k p_{jk}\), when every wrong class prediction has the same cost. The observed proportions replace those unknown probabilities when describing training data. This result does not apply unchanged when different errors have different costs or when the prediction rule uses a threshold other than the majority-class rule.

Check Whether the Pattern Persists on Held-Out Data

A tree’s training counts are shaped by the data used to construct it. To assess whether a node-level pattern generalizes, send held-out observations through the already fitted tree and count actual classes and predictions in each leaf. Keep the tree’s fitted split rules and predicted class labels fixed during this evaluation. Re-estimating the leaf labels from the held-out outcomes would use the evaluation data to alter the model being evaluated.

For example, suppose the fitted tree predicts No in leaf 1 and Yes in leaves 2 and 3. A held-out count table can reveal whether leaf 1 still contains many actual Yes cases, or whether its training errors were not repeated in the new data. Compare rates as well as counts, and note the sample size behind each rate. A high rate based on very few held-out cases is less stable than a similar rate based on many cases.

A node pattern is evidence of a localized weakness, not proof of its cause. The predictor path may leave relevant differences unrepresented, the observed sample may be variable, or the relationship may differ between the fitting and evaluation data. Describe what the counts show before proposing an explanation. In particular, do not infer that a predictor causes the error merely because the leaf is defined by that predictor’s split rules.

Practical Reading and Common Traps

A concise terminal-node error audit should make it possible to answer four questions: what class the tree predicts in each leaf, what actual classes occur there, how many observations are misclassified, and whether the same pattern appears on held-out data. A compact table is often clearer than a single accuracy statistic.

• Do not equate the most errors with the highest error rate. A large leaf can contribute many errors even when its error rate is moderate; a small leaf can have a high rate but few errors.
• Do not average leaf rates without considering their sizes. The overall error rate is total errors divided by total observations. An unweighted average answers a different question.
• Do not treat a majority prediction as certainty. A leaf with proportions 55% and 45% has a majority class, but the observed composition is close to a tie.
• Do not confuse training and held-out errors. Training errors describe the fitted sample; held-out errors evaluate fixed predictions on observations not used to fit the tree.
• State the error direction and denominator. “Errors among actual Yes cases” and “the fraction of Yes predictions that are wrong” are different quantities.

In R, the essential task is to compare each observation’s actual class with its predicted class and group those comparisons by the terminal node it reached. For example, if a data frame called audit contains the columns leaf, actual, and predicted, a grouped cross-tabulation such as with(audit, table(leaf, actual, predicted)) displays the counts needed for this audit. Ensure that the predictions were produced by the fitted tree and, for a held-out assessment, that the held-out outcomes were not used to change the tree.