Fitting Regression And Classification Trees With rpart

Fitting Trees with rpart

A tree can be understood as a sequence of rules, but fitting one in R requires translating the modeling question into the right response variable, formula, and prediction request. In rpart, the response type matters: a numeric response calls for regression, while a categorical response calls for classification. A model may run without an error even when the response has been coded in a way that answers the wrong question.

This tutorial focuses on fitting a small regression tree and a small classification tree, then using each to predict for new observations. The examples make the tree structure deliberately shallow so the fitted rules and returned predictions can be checked directly.

Definition: In the formula interface to rpart, a model formula has the form response ~ predictor_1 + predictor_2, with the response on the left and the predictors on the right. For regression, the response should be numeric and the tree method is "anova". For classification, the response should be a factor and the tree method is "class". A fitted model can be used with predict() and a data frame of new predictor values.

Formulas, Responses, and Fitting Controls

A formula such as cost ~ age tells rpart to model cost using age. With a data frame supplied through data, the formula ordinarily uses column names rather than expressions such as claims$cost. For example, rpart(cost ~ age, data = claims, method = "anova") fits a regression tree.

For classification, the response should represent categories, not measurements. If renewed contains labels such as "No" and "Yes", store it as a factor and use method = "class". Setting the method explicitly makes the intended model clear. In the examples, rpart.control() is used to limit the tree to one split; this is for clarity in the worked calculations, not a general recommendation about how deep a practical model should be.

Identify the response.
Decide whether the target is a numeric quantity or a category. Check its R class as well as its meaning.

Write the formula.
Place the response to the left of ~ and the predictors to the right. Supply the data frame with data =.

Fit the intended method.
Use "anova" for a numeric regression response or "class" for a factor classification response.

Predict with the requested output.
Supply a data frame containing the predictor columns used in the formula, then choose a prediction type appropriate to the model and question.

Worked Regression Tree

Worked Example: Predicting a Numeric Service Cost

A hypothetical service team records the number of annual maintenance visits, visits, and a service cost in dollars, cost, for six equipment units. Fit a one-split regression tree and predict the costs for units with 2 and 5 visits.

The data are visits = 1, 2, 3, 4, 5, 6 and cost = 8, 10, 12, 28, 30, 32, where the costs are in hundreds of dollars. The response is numeric, so the model uses method = "anova".

Fit the model. Setting maxdepth = 1 restricts the tree to one split below the root. Setting cp = 0 allows the best available split to be considered; the other controls permit this small data set to be split.

library(rpart)

service <- data.frame(visits = 1:6, cost = c(8, 10, 12, 28, 30, 32))

fit_cost <- rpart(cost ~ visits, data = service, method = "anova",
control = rpart.control(cp = 0, minsplit = 2,
minbucket = 1, maxdepth = 1, xval = 0))

Check the split and leaf predictions. The candidate split after 3 visits places costs $8,10,12$ in the left node and $28,30,32$ in the right node. The left mean is $(8+10+12)/3=10$, and the right mean is $(28+30+32)/3=30$. The within-node sum of squared errors is

\bigl((8-10)^2+(10-10)^2+(12-10)^2\bigr) +\bigl((28-30)^2+(30-30)^2+(32-30)^2\bigr) = (4+0+4)+(4+0+4)=16.

For comparison, the root mean is $120/6=20$, giving root sum of squared errors $144+100+64+64+100+144=616$. The split reduces this total to 16, a reduction of 600. The split therefore gives a substantial improvement over the unsplit prediction. The terminal-node predictions are 10 and 30, in hundreds of dollars.

Predict for new units. Use a data frame whose predictor column has the same name as in the formula. The explicit prediction type "vector" returns the numeric fitted value.

new_service <- data.frame(visits = c(2, 5))

predict(fit_cost, newdata = new_service, type = "vector")

The unit with 2 visits follows the left rule and receives prediction 10; the unit with 5 visits follows the right rule and receives prediction 30. In dollars, these are $10 \times 100=\$1{,}000$ and $30 \times 100=\$3{,}000$. The method is regression-tree prediction by the terminal-node mean. The calculation also checks the split directly: the two leaf means reproduce the stated total squared error of 16.

Worked Classification Tree

Worked Example: Predicting a Renewal Category

A hypothetical portfolio records an application score from 1 to 8 and whether each application was renewed. The outcomes in score order are No, No, Yes, No, Yes, Yes, Yes, Yes. Fit a one-split classification tree and obtain both class probabilities and predicted classes for scores 3 and 6.

Make the response categorical and fit. The outcome is a factor, so this is classification rather than regression. With one numeric predictor, the formula is renewed ~ score.

applications <- data.frame(
score = 1:8,
renewed = factor(c("No", "No", "Yes", "No", "Yes", "Yes", "Yes", "Yes"),
levels = c("No", "Yes"))
)

fit_renewal <- rpart(renewed ~ score, data = applications, method = "class",
control = rpart.control(cp = 0, minsplit = 2,
minbucket = 1, maxdepth = 1, xval = 0))

Check the selected split. The overall class proportions are 3/8 No and 5/8 Yes. The root Gini impurity, $1-\sum_k p_k^2$, is $1-(3/8)^2-(5/8)^2=30/64=0.46875$. A split at 4.5 leaves scores 1 through 4 on the left, with three No and one Yes, and scores 5 through 8 on the right, all Yes. The left impurity is $1-(3/4)^2-(1/4)^2=0.375$; the right impurity is 0. The weighted child impurity is $(4/8)(0.375)+(4/8)(0)=0.1875$, which is lower than the root impurity. Thus the split separates the all-Yes right node from the mixed left node.

Request probabilities. For scores 3 and 6, use type = "prob". The score 3 observation enters the left node, which contains three No and one Yes, so its probabilities are $P(\text{No})=3/4=0.75$ and $P(\text{Yes})=1/4=0.25$. Score 6 enters the right node, with probabilities 0 for No and 1 for Yes.

new_applications <- data.frame(score = c(3, 6))

predict(fit_renewal, newdata = new_applications, type = "prob")

Request classes. The class prediction selects the category with the largest estimated probability. The left-node probabilities favor No, and the right-node probabilities favor Yes.

predict(fit_renewal, newdata = new_applications, type = "class")

The predicted classes are No for score 3 and Yes for score 6. The method is classification by terminal-node class proportions, with the most probable class returned for type = "class". The calculation avoids confusing a predicted class with a probability: score 3 has predicted class No, but its estimated probability of Yes is still 0.25.

Why Classification Probabilities Are Leaf Proportions

A classification tree sends each training observation to a terminal node. Within a node containing $n$ observations, let $n_k$ be the count in class $k$. The fitted class probability is the observed fraction $n_k/n$. This is not a separate probability model fitted across the predictor scale; it is the empirical class distribution among the observations in that leaf.

One justification comes from the node's multinomial likelihood. If the class probabilities are $p_1,\ldots,p_K$, the log-likelihood, apart from a constant, is $\sum_k n_k\log(p_k)$, subject to $\sum_k p_k=1$. Maximizing this expression gives $p_k=n_k/n$. Therefore a leaf with three No and one Yes has fitted probabilities $3/4$ and $1/4$. This interpretation describes the fitted training-node proportions; it does not guarantee that predicted probabilities are calibrated for new observations.

Numeric Coding Is Not the Same as Classification

Worked Example: A Binary Outcome Stored as Numbers

Suppose the renewal outcomes from the previous example are stored as 0 for No and 1 for Yes. The question is whether fitting that numeric column as a regression response returns a class prediction.

Create a numeric response and fit an ANOVA tree with the same one-split controls. Since the response is numeric, rpart estimates a numeric mean in each leaf; this is not a classification tree.

applications$renewed_num <- c(0, 0, 1, 0, 1, 1, 1, 1)

fit_numeric <- rpart(renewed_num ~ score, data = applications, method = "anova",
control = rpart.control(cp = 0, minsplit = 2,
minbucket = 1, maxdepth = 1, xval = 0))

At the split after score 4, the left numeric responses are $0,0,1,0$, with mean $1/4=0.25$. The right responses are $1,1,1,1$, with mean $4/4=1$. Thus for scores 3 and 6, type = "vector" returns numeric predictions 0.25 and 1. These numbers can be interpreted as leaf averages of the 0/1 outcome, but the fitted model is still a regression tree.

To fit a classification tree instead, use the factor column renewed and method = "class", as in the previous example. Then request probabilities with type = "prob" or labels with type = "class". The method is to match the response's meaning and storage type to the intended prediction task. The common mistake is to assume that numeric codes automatically tell rpart to classify.

Using Predictions Reliably

The structure of a newdata frame must match the predictors in the fitted formula. If a model was fitted with cost ~ visits, prediction data must contain a column named visits. Extra response columns are unnecessary; omitting a required predictor or using a different column name can prevent prediction or lead to an unintended input.

Model and question	Response used for fitting	Prediction request
Regression: estimate a numeric outcome	Numeric response; typically `method = "anova"`	`type = "vector"` for numeric fitted values
Classification: estimate class probabilities	Factor response; `method = "class"`	`type = "prob"` for a probability column per class
Classification: assign a category	Factor response; `method = "class"`	`type = "class"` for predicted labels

Before interpreting output, check the model method, response type, and requested prediction type. Also keep the prediction scale clear: in the regression example, the model was fitted in hundreds of dollars, so a prediction of 10 represents $1,000, not $10. Fitting a tree is only one step in predictive modeling; the small controls used here illustrate tree mechanics and do not establish that a particular tree is suitable for deployment.

Key takeaway: Use a numeric response and regression method for numeric predictions; use a factor response and classification method for class probabilities or labels. Supply new observations as a data frame with the predictor names used in the formula, and request the output type that matches the question.

Check Your Understanding

For each problem, identify the fitting or prediction choice and check the final result.

A numeric response is measured in dollars, and the predictor is the number of service calls. Which method and prediction type should be used to request a numeric fitted value? (Answer: method = "anova" and type = "vector".)
A classification response has 6 observations in one leaf: 4 in class A and 2 in class B. What probabilities should type = "prob" return for that leaf? (Answer: A = 4/6 = 0.6667 and B = 2/6 = 0.3333.)
A fitted formula is delay ~ distance. A new observation has distance 12. What predictor column must the newdata frame contain? (Answer: a column named distance with value 12.)
A binary outcome is stored as 0 and 1, and the analyst fits method = "anova". In one leaf, the outcomes are 0, 0, 1, 1, 1. What is the regression prediction for that leaf, and is it a class label? (Answer: 3/5 = 0.6; it is a numeric regression prediction, not a class label.)
A factor classification tree gives probabilities No = 0.40 and Yes = 0.60 for a new record. What does type = "class" predict, and what distinction should be retained? (Answer: Yes; the predicted class is a label, while the estimated probability of Yes is 0.60.)

What You'll Learn