Fitting A Regression Forest With Ranger

Fitting a Regression Forest in R

A regression forest converts the random-forest ideas of bootstrap sampling, random feature selection, and tree averaging into a fitted prediction rule for a numeric outcome. In practice, a sound fit starts before calling the software: identify the response, select predictors available when a prediction will be made, and check that the response is numeric and the analysis data are suitable for fitting.

This tutorial uses the R package ranger. The examples focus on numeric predictors so that the response, formula, predictions, and fitted-model information remain the main issues. A reproducible seed makes the random steps repeatable in a given computational environment, but does not turn a forest into a deterministic model across all software versions or settings. The implementation skills here are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: A regression forest is an ensemble of regression trees whose prediction for an observation is the average of the predictions from the fitted trees. In a standard squared-error regression tree, a terminal node predicts the mean response among the training observations assigned to that node. The response is numeric; predictors are the variables used to form the tree splits.

In ranger’s formula interface, the response goes on the left of the tilde and the predictors go on the right. For example, annual_cost ~ age + prior_visits specifies annual_cost as the response and age and prior_visits as predictors. The formula annual_cost ~ . uses every other column in the supplied data frame as a predictor. That shorthand is convenient, but it can accidentally include identifiers or variables that would not be available at prediction time.

A Basic Fit and Its Main Controls

A minimal fit can rely on defaults, but an exam-ready analysis should be able to state what is being fitted and recognize the controls that affect it. A typical explicit call is shown below. It assumes that the data frame contains the numeric response annual_cost and numeric predictors age and prior_visits, with no missing values in the selected columns.

R: fit = ranger::ranger(annual_cost ~ age + prior_visits, data = train, num.trees = 500, mtry = 1, min.node.size = 5, seed = 24)

Here, num.trees is the number of trees, mtry is the number of candidate predictors considered at each split, and min.node.size controls how small a node can be before it is no longer eligible for further splitting. These settings are choices to record and justify, not universal best values. When setting mtry explicitly, it must be a sensible positive integer no greater than the number of predictors. The example has two predictors, so mtry = 1 is permitted.

For a regression forest, ranger identifies the task from the numeric response. The default regression splitting rule is variance reduction, which selects splits to reduce within-node response variation. A seed controls the random-number stream used in fitting. Changing the seed can change the fitted trees and their predictions, especially when the data allow several plausible splits.

Key distinction: The response is the quantity the forest predicts. Predictors are inputs to the prediction rule. Do not include a quantity that is only known after the outcome occurs, or a version of the response itself, among the predictors: doing so creates information leakage rather than a valid predictive fit.

Why the Forest Prediction Is an Average

Let $T_b(x)$ denote the prediction from tree $b$ for predictor values $x$, where $b=1,\ldots,B$ and $B$ is the number of trees. The regression-forest prediction is the arithmetic mean of those tree predictions:

\hat f(x)=\frac{1}{B}\sum_{b=1}^{B}T_b(x).

The averaging rule follows directly from the definition of the forest prediction: each tree supplies one numeric prediction, and the forest combines them with equal weight. For example, if three trees predict 4, 7, and 10, the forest prediction is $(4+7+10)/3=7$. This does not mean the forest fits one large tree. Each tree is fitted using its own randomization, and the average is taken after the trees have made their predictions.

In ranger, a call to predict returns a list; for a regression forest, its predictions element contains the numeric predictions. Pass prediction data using the data argument. The new data must contain the predictor columns used in the fit, with compatible types. Do not include the response in that new-data frame unless it was itself used as a predictor, which would usually indicate a modelling error.

Worked Fitting and Prediction Examples

Worked Example: Verify a Small Regression-Forest Fit

A hypothetical insurer records annual repair cost in dollars for five vehicles. The only predictor is vehicle age in years. Fit a small forest that makes it possible to check the response, predictors, and predictions by hand. This deliberately restricted fit is an implementation check, not a recommended production model.

Step 1: Enter the data. The responses are 12, 18, 27, 33, and 60 dollars. Their sum is $12+18+27+33+60=150$, and there are five observations. The predictor values can be 1, 2, 3, 4, and 5 years.

Step 2: Fit the model. Use a response formula with vehicle_age as the predictor. Set min.node.size to 6, which exceeds the five training observations. With the root node too small to split, each tree remains a single terminal node. Sampling all observations without replacement makes the calculation especially transparent.

R: train = data.frame(repair_cost = c(12, 18, 27, 33, 60), vehicle_age = 1:5)

R: fit = ranger::ranger(repair_cost ~ vehicle_age, data = train, num.trees = 3, mtry = 1, min.node.size = 6, replace = FALSE, sample.fraction = 1, seed = 31)

Step 3: Calculate each tree’s prediction. Each tree’s root contains all five responses, so its terminal-node mean is $150/5=30$ dollars. All three trees predict 30 dollars for every observation.

Step 4: Average the trees. For any supplied vehicle age, the forest prediction is $(30+30+30)/3=30$ dollars. For example, obtain predictions at ages 2 and 6 with predict(fit, data = data.frame(vehicle_age = c(2, 6)))$predictions; the predictions are 30 and 30.

Check and common mistake. The formula has one response and one predictor, and the response mean is exactly 30. The value at age 6 is not a claim that a useful forest would predict a constant across ages; it is a consequence of deliberately preventing every tree from splitting. A common mistake is to interpret this diagnostic stump forest as evidence that age has no predictive relationship.

Worked Example: Compare Predictions With New Outcomes

A hypothetical maintenance team fits a deliberately shallow forest to four training records. Repair cost is measured in dollars. The fitted forest is used to predict costs for two new records whose observed costs are later available for an error calculation.

Step 1: Fit using training records only. The training responses are 100, 120, 140, and 160, with predictor mileage equal to 10, 20, 30, and 40 thousand miles. Their total response is $100+120+140+160=520$, so their mean is $520/4=130$. Set min.node.size to 5 to keep the four-observation root unsplit.

R: train = data.frame(repair_cost = c(100, 120, 140, 160), mileage_thousands = c(10, 20, 30, 40))

R: fit = ranger::ranger(repair_cost ~ mileage_thousands, data = train, num.trees = 2, mtry = 1, min.node.size = 5, replace = FALSE, sample.fraction = 1, seed = 42)

Step 2: Predict new records. Suppose the new records have mileage values 15 and 45, and their later-observed repair costs are 125 and 150 dollars. Supply only the predictor in the new-data frame.

R: newdata = data.frame(mileage_thousands = c(15, 45))

R: pred = predict(fit, data = newdata)$predictions

Step 3: Calculate predictions and errors. Both trees predict the training mean 130, so the forest predictions are 130 and 130. Define prediction error as observed minus predicted. The two errors are $125-130=-5$ and $150-130=20$ dollars.

Step 4: Calculate RMSE. The mean squared error for these two new observations is $((-5)^2+20^2)/2=(25+400)/2=212.5$ dollars squared. Therefore, RMSE is $\sqrt{212.5}\approx14.5774$ dollars, or $14.58 to two decimal places.

Check and common mistake. The RMSE calculation uses only the two new outcomes and predictions, not the training responses. A frequent error is to average the signed errors, $(-5+20)/2=7.5$, and call that RMSE. RMSE squares the errors before averaging and then takes the square root, so it cannot be computed by simply averaging signed errors.

Worked Example: Inspect the Fit and Interpret the Units

A hypothetical analyst models repair cost in thousands of dollars using two numeric predictors, vehicle age and annual distance driven. There are six training observations with costs $2, 4, 6, 8, 10,$ and $12$ thousand dollars. Fit a small no-split forest to check the returned model information and the scale of its predictions.

Step 1: Check the response and predictor count. The response is repair_cost_thousands, and the predictors are vehicle_age and annual_distance. There are $p=2$ predictors. Their response sum is $2+4+6+8+10+12=42$, so the response mean is $42/6=7$ thousand dollars.

R: train = data.frame(repair_cost_thousands = c(2, 4, 6, 8, 10, 12), vehicle_age = c(1, 2, 3, 4, 5, 6), annual_distance = c(8, 10, 12, 14, 16, 18))

R: fit = ranger::ranger(repair_cost_thousands ~ vehicle_age + annual_distance, data = train, num.trees = 4, mtry = 2, min.node.size = 7, replace = FALSE, sample.fraction = 1, seed = 9)

Step 2: Inspect core fields. Use fit$treetype to check the task type, fit$num.trees for the number of trees, fit$mtry for the candidate-predictor count, fit$num.independent.variables for the predictor count, and fit$num.samples for the number of training observations. In this fit these are, respectively, Regression, 4, 2, 2, and 6.

Step 3: Predict and convert units. Since min.node.size exceeds six, each tree predicts the root mean, 7 thousand dollars. A new record with vehicle_age = 3 and annual_distance = 12 is therefore predicted to cost 7 thousand dollars, or $7 \times 1000=$$7,000. The number 7 is not a prediction of $7; it is a prediction in the response’s stated units of thousands of dollars.

Check and common mistake. The predictor count is two, so mtry = 2 is allowable, and the formula makes the response explicit. A common interpretation error is to report the prediction as $7 instead of $7,000. Always carry the response units from the data through to the reported prediction.

Inspecting Fitted-Model Information

After fitting, inspect enough information to confirm that the model matches the intended analysis. The following fields are useful for a basic fit review:

fit$treetype identifies the fitted task as Regression.
fit$num.trees and fit$mtry report the corresponding forest settings.
fit$num.independent.variables and fit$num.samples help check the predictor and training-row counts.
fit$prediction.error reports ranger’s fitted prediction-error measure when it is available. For regression, this is an error on the response scale squared; its interpretation depends on how the fit generated predictions.

With the usual bootstrap setup, ranger can calculate an out-of-bag prediction error from observations omitted from the relevant tree samples. That quantity is not the same as error on a separate test set, and it may be unavailable or uninformative under settings that leave no usable out-of-bag predictions. For instance, the deliberately transparent examples above sample all rows without replacement, so they should not be used to interpret an out-of-bag error field. The earlier material on out-of-bag error explains why its precise construction and use matter.

For new cases, inspect the prediction vector and check its length and ordering against the new-data rows. Before interpreting values, confirm that the predictor columns are present and that the response units are understood. These small checks often reveal a formula mistake or a unit mismatch before it reaches a report.

Common Fitting Errors

Reversing response and predictor. In y ~ x, y is predicted from x. Confirm the left-hand side names the outcome of interest.
Using every column without checking. The dot in y ~ . includes every other data column. Remove identifiers, post-outcome information, and any variable unavailable at prediction time.
Passing the wrong columns to predict. The data argument to predict should contain the predictors used by the fitted formula. It does not need the response column.
Assuming a seed guarantees identical results everywhere. A seed supports reproducibility in a given setup; software versions, inputs, and settings still matter.
Reporting a value without its units. A response recorded in thousands of dollars produces predictions in thousands of dollars unless explicitly converted.
Calling training or OOB error test error. Name the data and method used to calculate each reported error. A numeric error value alone is not enough to identify what it measures.

Key takeaway: A clear ranger regression fit states the numeric response, the predictors, and the forest controls; prediction uses compatible new-data columns; and inspection verifies the fitted task, settings, dimensions, prediction scale, and error interpretation.

Check Your Understanding

For each problem, identify the fitting or calculation step before checking the final answer.

A no-split regression forest is fitted to responses 3, 5, 7, and 9. Each tree contains all four observations in its root. What does it predict for any new case? (Answer: 6 response units.)
A data frame has one numeric response and nine eligible numeric predictors. An analyst explicitly sets mtry to 3 and num.trees to 250. How many predictors are considered as candidates at each split, and how many trees are fitted? (Answer: 3 candidate predictors per split and 250 trees.)
A regression response is recorded in thousands of dollars. A prediction from ranger is 4.6. State the corresponding dollar prediction. (Answer: $4,600.)
A no-split forest is fitted to the eight training responses 1 through 8, using all eight rows in every tree. Calculate the forest prediction and explain which data determine it. (Answer: 4.5 response units; it is the mean of the eight training responses.)
For two new records, observed responses are 8 and 17, while predictions are 10 and 13. Calculate RMSE, rounding to two decimal places. (Answer: $\sqrt{((8-10)^2+(17-13)^2)/2}=\sqrt{10}\approx3.16$ response units.)

What You'll Learn