Why Prune an Fitted rpart Tree?
A tree grown to capture many patterns in its training data may be harder to interpret and may fit noise. Pruning removes branches from an already fitted tree to produce a smaller subtree. In R, the rpart package stores information about candidate subtrees in a complexity table, and the prune function uses a complexity parameter to extract one of them.
Pruning does not refit a new, unrelated tree from scratch. It selects a smaller tree from the nested sequence represented by the original fit. Once pruned, that object can be used with predict just like the original fitted object. The main tasks are to identify a suitable row of the complexity table, apply its CP value, and confirm that the resulting tree has the expected structure.
rpart complexity table, CP is the complexity parameter associated with a point in the tree’s pruning sequence; nsplit is the number of splits in the corresponding subtree; rel error is its training error relative to the root-only tree; xerror is a cross-validated relative-error estimate; and xstd is the estimated standard error of that cross-validated value.
Reading the Complexity Table
The table is available as fit$cptable, where fit is an object created by rpart. The command printcp(fit) displays it in a readable format. The first row generally describes the root-only tree, with zero splits. Subsequent rows describe successively larger subtrees in the pruning sequence. A binary tree with \(k\) splits has \(k+1\) terminal nodes.
The relative-error columns have different roles. rel error describes fit to the data used to grow the tree. xerror estimates relative error using the tree-fitting procedure’s cross-validation. A tree can have lower training error simply because it has more splits, so the smallest rel error is not automatically the best row to select. A common simple selection rule is to choose the row with the lowest xerror.
CP is not a probability, a classification threshold, or a measure of prediction accuracy on its own. It is a complexity control. In particular, the CP used in prune acts on the existing tree. It cannot restore branches that were never grown. If the original tree was restricted during fitting, for example by a high growth-control CP in rpart.control, the sequence of subtrees available to prune is restricted too.
Worked Example: Selecting a Row From a Complexity Table
A hypothetical insurer fits a regression tree to predict a renewal-cost index. Its complexity table contains the following values.
| CP | nsplit | rel error | xerror | xstd |
|---|---|---|---|---|
| 0.240 | 0 | 1.00 | 1.08 | 0.10 |
| 0.110 | 1 | 0.62 | 0.78 | 0.09 |
| 0.055 | 2 | 0.41 | 0.64 | 0.08 |
| 0.025 | 3 | 0.33 | 0.69 | 0.09 |
| 0.010 | 4 | 0.29 | 0.73 | 0.09 |
Method. Apply the minimum-xerror rule. The xerror values are 1.08, 0.78, 0.64, 0.69, and 0.73. Their minimum is 0.64, in the row with CP \(=0.055\) and nsplit \(=2\). The corresponding subtree therefore has two splits and three terminal nodes.
In R, the selection and pruning can be written as tab <- fit$cptable, best_row <- which.min(tab[, "xerror"]), best_cp <- tab[best_row, "CP"], and small_fit <- prune(fit, cp = best_cp). Inspect the result rather than assuming the requested subtree was obtained: count its internal nodes, for example with sum(small_fit$frame$var != "<leaf>"). The count should be two for this selected row.
Check. The selected row has fewer splits than the four-split row with the lowest training relative error. The selection uses xerror, not rel error. This avoids the common mistake of choosing the largest tree merely because it fits the training data best.
Why the Pruning Sequence Uses CP
Cost-complexity pruning balances fit against size. Let \(R(T)\) denote a tree’s scaled error and let \(|T|\) denote its number of terminal nodes. For a nonnegative penalty \(\alpha\), consider the criterion
The penalty makes larger trees pay a greater complexity cost. To see why a CP value marks a change in the preferred subtree, compare a smaller tree \(T_s\) with a larger tree \(T_l\). The larger tree is preferred when
Rearranging gives
Thus, the larger tree is retained only when its reduction in error is sufficiently large to justify its additional terminal nodes. At a boundary between two candidate subtrees, the two criteria are equal, so the penalty at that boundary is
The CP column records the complexity scale used by rpart for its pruning sequence. It is therefore useful as a control for extracting a subtree, not as an independent claim that one tree is universally best. The cross-validated xerror column supplies a common rule for choosing among the rows.
Pruning and Predicting for a Regression Tree
After pruning a regression tree, each new observation follows the retained split rules to a terminal node. Its predicted response is the fitted value for that node. For a standard squared-error regression tree without unusual case weights, that fitted value is the mean response of the training observations in the terminal node.
Worked Example: Predicting Renewal Costs From a Pruned Tree
Suppose a hypothetical fitted regression tree predicts annual service costs in dollars from annual mileage. Its table has a minimum xerror at CP \(=0.08\), the row for a one-split subtree. After pruning, suppose its retained rule is “mileage less than 8,000” and its two terminal-node fitted means are $420 and $760, respectively. The prediction data contain mileage values of 7,000, 9,000, and 12,000 miles.
Method. Use the selected CP to prune the fitted tree, then send each row of new data down its retained split rule. In R, the steps are small_fit <- prune(fit, cp = 0.08), new_data <- data.frame(mileage = c(7000, 9000, 12000)), and predict(small_fit, newdata = new_data).
For 7,000 miles, \(7{,}000<8{,}000\), so the observation reaches the left terminal node and receives prediction $420. For 9,000 miles, \(9{,}000\) is not less than 8,000, so it reaches the other terminal node and receives $760. The same comparison holds for 12,000 miles, so its prediction is also $760.
The predictions, in input-row order, are therefore $420, $760, and $760. The result is checked by verifying that the first observation follows the less-than branch and the other two follow the alternative branch. This is a prediction of the fitted response, not a guarantee that the eventual observed cost will equal that value.
Common mistake. Do not use the original, unpruned tree’s additional branches to route these observations after pruning. Predictions must follow the rules in small_fit. Also check that prediction data use the predictor names and units expected by the fitted tree.
Classification Predictions: Classes and Probabilities
For a classification tree, predict can return predicted classes or class probabilities. Use type = "class" for class labels and type = "prob" for probabilities. Under the usual default classification rule, the predicted class is the class with the largest predicted probability. If the model uses non-default priors or loss settings, class assignment may reflect those settings, so do not infer every prediction from raw terminal-node counts alone.
A probability prediction and a class prediction answer different questions. A probability such as 0.35 for “Yes” is an estimated class probability; it does not mean that the model has assigned the label “Yes.” Changing the decision threshold for a business decision is a separate step from obtaining the probabilities.
Worked Example: Obtaining Class Probabilities From a Pruned Tree
A hypothetical claims-screening tree predicts whether a submitted claim requires additional review. Its selected complexity-table row has CP \(=0.06\) and one split, based on the number of prior incidents. After pruning, suppose the terminal-node probabilities are 0.90 for “No” and 0.10 for “Yes” when prior incidents are fewer than two, and 0.35 for “No” and 0.65 for “Yes” otherwise. Three new claims have 0, 1, and 3 prior incidents.
Method. Prune using the selected row, then request probabilities and classes separately: small_fit <- prune(fit, cp = 0.06), predict(small_fit, newdata = new_data, type = "prob"), and predict(small_fit, newdata = new_data, type = "class").
The first claim has 0 incidents, which is fewer than two, so its probability pair in the order No, Yes is (0.90, 0.10). The second claim has 1 incident and reaches the same node, giving (0.90, 0.10). The third has 3 incidents, reaches the other node, and receives (0.35, 0.65). Under the default largest-probability rule, the first two are classified No because \(0.90>0.10\), and the third is classified Yes because \(0.65>0.35\).
The check is that each probability pair sums to 1.00, and the class label agrees with the larger probability. If a decision-maker instead specifies a Yes threshold of 0.30, the third claim remains Yes, while the first two remain No because their Yes probabilities are 0.10. The threshold-based decision is not a different probability prediction.
Common mistake. Requesting type = "class" does not return the probability column. Request type = "prob" when the numerical class probabilities are needed.
Practical Checks Before Using the Pruned Model
Identify the selection rule and the row it selects. For a minimum-xerror rule, locate the smallest xerror and read CP and nsplit from that same row.
Pass the selected CP value to
prune. A CP value from the table is a reproducible choice; an arbitrary value may correspond to a different subtree.Check the pruned object’s number of splits and inspect its retained rules. If the count does not match the selected row, investigate the CP value and the object being pruned.
Use the pruned object and suitable new data. For classification, explicitly request class labels or probabilities according to the question being answered.
Cross-validation in the complexity table helps compare candidate subtrees within the tree-fitting process. It does not make predictions on a separate test set, and it should not be described as an independent assessment on data that played no role in model selection. For this task, the essential operational check is that the chosen row, the CP passed to prune, and the resulting subtree are consistent.
rpart fit, and verify the resulting subtree. Then use predict on the pruned object, distinguishing regression values, classification labels, and class probabilities.
Check Your Understanding
Work through each question by identifying the relevant row or prediction rule before giving the result.
- A hypothetical complexity table has rows (CP, nsplit, xerror): (0.20, 0, 1.10), (0.09, 1, 0.82), (0.04, 2, 0.71), and (0.015, 3, 0.76). Under the minimum-xerror rule, which CP is selected, and how many terminal nodes does the selected subtree have? (Answer: CP = 0.04; 3 terminal nodes.)
- A fitted binary classification tree is pruned using CP from a row with nsplit = 4. How many terminal nodes should the resulting binary subtree have? (Answer: 5 terminal nodes.)
- A pruned regression tree has the rule “account age less than 5 years” with fitted terminal values $310 and $540. What predictions are made for account ages of 2, 5, and 8 years? (Answer: $310, $540, and $540, respectively.)
- A pruned classification tree gives a new case probabilities of 0.42 for “Continue” and 0.58 for “Review.” What does the default largest-probability rule predict, and what should be requested to obtain the two probabilities in R? (Answer: It predicts Review; use type = "prob".)
- The training relative error is smallest in the four-split row, but xerror is smallest in the two-split row. If the stated selection rule is minimum xerror, which row should be selected, and why? (Answer: Select the two-split row because the stated criterion is xerror, not training relative error.)