Worked Comparison Of Forest Sampling Strategies

Why Compare Forest Sampling Strategies?

A random forest’s predictions depend on more than the predictors in the data. Each tree is built using a sample of rows, and at each split the algorithm considers a subset of predictors. Changing either choice can affect predictive error, the variation between fits, and computation time. A useful comparison therefore evaluates these outcomes together rather than selecting a setting from one attractive metric.

This tutorial uses a hypothetical insurer’s task of predicting annual claim cost. The response is measured in thousands of dollars, and all numerical forest results below are invented for instruction; they are not company or industry results. The comparison methods are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: In a random forest, $mtry$ is the number of predictors made eligible for consideration at a split. A row-sampling setting specifies how the training rows are selected for each tree, including whether sampling is with or without replacement and, where applicable, the sampling fraction. An out-of-bag (OOB) prediction for a training observation is formed using trees whose tree-building samples did not include that observation.

These controls have different jobs. Changing $mtry$ changes the candidate predictors available at a split. Changing row sampling changes which observations contribute to each tree and which observations can serve as OOB cases for that tree. Neither change guarantees a lower OOB error. The right comparison keeps other choices—such as the response, predictors, tree-size controls, and number of trees—fixed.

Define the Comparison Before Fitting

For a regression forest, let $y_i$ be the observed response for training observation $i$, and let $\hat{y}_i^{\text{OOB}}$ be its OOB prediction. The OOB root mean squared error is the square root of the mean squared OOB prediction error:

\operatorname{RMSE}_{\text{OOB}} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \left(y_i-\hat{y}_i^{\text{OOB}}\right)^2}.

This formula assumes that each observation has an OOB prediction and that the squared errors are being averaged over the same $n$ observations for every candidate. A fit with missing OOB predictions, or a comparison that silently uses different sets of observations, does not meet that condition; first establish a common evaluation set. RMSE has the same units as the response, so here its units are thousands of dollars.

A fair experiment specifies a short list of candidate settings in advance. For example, with eight predictors, one might compare $mtry=2$ and $mtry=5$ under both a full-size bootstrap sample and a 70% subsample without replacement. Use the same data and other forest controls, and record the OOB metric and elapsed fit time for every candidate. Then repeat the fits with several random seeds to assess how much the observed comparison depends on forest randomness.

Choose the target and metric.
State whether the task is regression or classification and select a metric that matches the task and decision. This example uses regression RMSE.

Set candidate controls.
Vary the $mtry$ and row-sampling settings of interest while holding other modeling choices fixed.

Fit and record.
Use consistent data, tree counts, and computing conditions. Record OOB error and elapsed time, and repeat with different seeds.

Compare and qualify.
Consider the size and stability of error differences alongside runtime. Treat OOB performance as an internal estimate, not as proof of future performance.

Worked Example: Calculating and Comparing OOB RMSE

Two candidate forests predict annual claim cost, in thousands of dollars, for the same six training observations. Forest A uses $mtry=2$ with full-size bootstrap samples; Forest B uses $mtry=5$ with 70% subsampling without replacement. Suppose their squared OOB errors, in squared thousands of dollars, are A: $1,4,9,16,4,5$, and B: $1,1,4,4,1,4$. Calculate and compare the OOB RMSE values.

Method. Sum the six squared errors for each forest, divide by six, and take the square root. This puts both candidates on the response scale and uses the same six observations.

For A, the squared-error sum is $1+4+9+16+4+5=39$. Its mean squared error is $39/6=6.5$, so

\operatorname{RMSE}_{\text{OOB,A}}=\sqrt{6.5} \approx 2.5495\text{ thousand dollars} = \$2{,}549.51.

For B, the squared-error sum is $1+1+4+4+1+4=15$. Its mean squared error is $15/6=2.5$, so

\operatorname{RMSE}_{\text{OOB,B}}=\sqrt{2.5} \approx 1.5811\text{ thousand dollars} = \$1{,}581.14.

To the nearest cent, B’s OOB RMSE is $\$968.37$ lower: $\$2{,}549.51-\$1{,}581.14=\$968.37$. This is a better OOB result for B in this illustration, not by itself evidence that its future prediction error will be lower. The common mistake is to report $\sqrt{39}$ and $\sqrt{15}$ as RMSE; those are square roots of the sums, not of the mean squared errors. Dividing by six before taking the square root is essential.

Why OOB Predictions Work—and What Sampling Changes

The OOB idea follows directly from how a tree’s row sample is formed. For a given observation, a tree can contribute to its OOB prediction only if that observation was absent from the sample used to fit that tree. Thus, the sampling scheme determines the chance of being OOB for any one tree, and the number of trees determines how many OOB predictions are available on average.

For a bootstrap sample of size $n$ drawn with replacement from $n$ rows, the probability that a particular row is not selected on any one draw is $1-1/n$. The $n$ draws are independent, so the chance that the row is absent from the entire sample is:

\left(1-\frac{1}{n}\right)^n.

With subsampling without replacement at fraction $s$, exactly a fraction $s$ of rows is selected for each tree (assuming the sample size is $sn$, an integer). A particular row is therefore absent with probability $1-s$. If there are $B$ trees, the expected number of trees that can contribute an OOB prediction for one row is $B$ times its probability of being absent from one tree’s sample. This expectation does not say that every row receives exactly that number of OOB predictions.

OOB predictions are useful because each observation is evaluated by trees that excluded it from their own fitting samples. However, the observations used to train the forest are still involved in building the overall ensemble. OOB error is a convenient internal estimate, but it is not a substitute for a separate test set when one is required for the analysis.

Worked Example: Expected OOB Coverage Under Two Sampling Schemes

A forest has $n=1{,}000$ training rows and $B=200$ trees. Compare the expected number of trees contributing to one row’s OOB prediction under (a) a bootstrap sample of 1,000 draws with replacement and (b) a subsample of 70% of the rows without replacement.

Method. Calculate the probability that the specified row is omitted from one tree’s sample, then multiply by 200. For the bootstrap, the probability is $(1-1/1000)^{1000}=0.999^{1000}\approx0.3677$. Thus, the expected OOB tree count is $200(0.3677)\approx73.54$, or about 73.5 trees.

For the 70% subsample without replacement, the row is omitted with probability $1-0.70=0.30$. The expected OOB tree count is $200(0.30)=60$ trees.

The bootstrap scheme therefore gives about $73.54-60=13.54$ more OOB trees per observation on average in this example. This is an expectation, not a guarantee for an individual row. The trap is to treat “70% sampled” as meaning “70% OOB”; the selected fraction is 70%, so the omitted fraction is 30%.

Compare Stability and Runtime, Not Only the Best Error

A forest’s OOB score can vary across random seeds even when the data and controls are unchanged. A repeated-fit comparison summarizes that variation. For $K$ repeated fits with OOB RMSE values $r_1,\ldots,r_K$, the mean RMSE is $\bar r=\sum_{k=1}^{K}r_k/K$. The sample standard deviation is:

s_r=\sqrt{\frac{\sum_{k=1}^{K}(r_k-\bar r)^2}{K-1}}.

This sample standard deviation describes variation among the observed repeated fits; it is not a confidence interval for future prediction error. Runtime should also be measured under comparable conditions, because hardware load, parallel settings, and software implementation can affect elapsed time.

Worked Example: Balancing Repeated-Fit Stability and Runtime

Two settings are each fitted five times using different seeds on the same hypothetical training data. Their OOB RMSE values are A: $7,8,9,10,11$, and B: $8,9,9,10,9$, in thousands of dollars. The corresponding runtime measurements in seconds are A: $42,44,43,45,46$, and B: $31,32,30,33,34$. Compare mean RMSE, sample standard deviation of RMSE, and mean runtime.

Method. Calculate the mean and sample standard deviation of each list of five errors, then calculate the arithmetic mean of each runtime list. For A, the RMSE mean is $(7+8+9+10+11)/5=45/5=9$. Its deviations from 9 are $-2,-1,0,1,2$, whose squared sum is $4+1+0+1+4=10$. Therefore its sample standard deviation is $\sqrt{10/(5-1)}=\sqrt{2.5}\approx1.5811$ thousand dollars.

For B, the RMSE mean is $(8+9+9+10+9)/5=45/5=9$. Its deviations are $-1,0,0,1,0$, with squared sum $1+0+0+1+0=2$. Its sample standard deviation is $\sqrt{2/4}=\sqrt{0.5}\approx0.7071$ thousand dollars.

A’s mean runtime is $(42+44+43+45+46)/5=220/5=44$ seconds. B’s is $(31+32+30+33+34)/5=160/5=32$ seconds. In this illustration the settings have equal mean OOB RMSE, while B has lower observed run-to-run variation and a 12-second lower mean runtime. The sample standard deviation calculation divides by $K-1=4$, not by $K=5$; using the wrong denominator would understate the sample standard deviation.

A lower standard deviation across seeds can make a setting more reproducible in this experiment, but it does not establish that the setting will remain best on new data. Similarly, a shorter runtime is useful only in context: it may matter greatly when many fits are required, but a small time saving may not justify a meaningful increase in prediction error.

Implement a Controlled Comparison

In R, a practical comparison can be organized as a table of candidate settings and repeated fits. For example, the ranger package can fit regression forests with explicit values for the number of trees, $mtry$, replacement, and sampling fraction. In a hypothetical data frame named claims, with response annual_cost and eight predictors, a single fit could be specified as ranger::ranger(annual_cost ~ ., data = claims, num.trees = 500, mtry = 2, replace = FALSE, sample.fraction = 0.70, seed = 101). The candidate settings should be changed deliberately, not left to an undocumented default.

For an interpretable comparison, record the software and version, formula, candidate settings, seed, number of trees, OOB error, and elapsed time. When repeating fits, vary the seed but retain the same data and controls. If using a package that reports an OOB prediction for each training row, verify how it handles rows with no OOB trees; do not assume that every package defines or stores the predictions identically.

A useful summary table should distinguish the single-fit result from repeated-fit summaries. For example, report the mean and sample standard deviation of repeated OOB RMSE values, along with mean runtime. If an OOB difference is very small relative to variation between repeated fits, describe the evidence as inconclusive rather than declaring a clear winner. If the goal is to choose settings, make that choice within a validation plan that does not reuse a final test set for repeated tuning.

Common Comparison Traps

Changing several controls at once without a plan. If $mtry$, sampling fraction, tree size, and number of trees all change together, the comparison cannot isolate which change is associated with the result. A joint comparison can be useful, but specify its candidates and interpret it as a comparison of complete settings.
Comparing metrics from different observations or units. RMSE must use the same response scale and evaluation cases for a direct numerical comparison. State explicitly whether a value is in dollars or thousands of dollars.
Selecting the minimum from one run. Random variation can make one fit look unusually favorable. Repeat fits and report their distributional summary rather than relying only on the smallest observed error.
Treating runtime as a universal constant. Time depends on hardware and execution conditions. Compare candidates in the same environment and report the conditions relevant to the decision.
Calling OOB performance external validation. OOB predictions exclude each evaluated row from the corresponding trees, but they arise from the training data and the fitting process. State what OOB results do and do not establish.

Key takeaway: Compare forests using a controlled set of settings and a common OOB evaluation basis. Report predictive error, variation across repeated fits, and runtime together. A difference in OOB scores is evidence about the fitted comparison, not proof of a universal ranking of sampling strategies.

Check Your Understanding

Work each problem through before checking the final answer.

A regression forest has six OOB squared errors of $1,1,1,4,4,5$, measured in squared thousands of dollars. Calculate its OOB RMSE and convert it to dollars, rounding the dollar result to the nearest cent. (Answer: $\sqrt{16/6}\approx1.6330$ thousand dollars, or $\$1{,}632.99$.)
A forest uses 500 trees, each based on a 60% subsample without replacement. What is the expected number of OOB trees for one training row? (Answer: $500(1-0.60)=200$ trees.)
Two settings have repeated RMSE values A: $4,5,6$, and B: $5,5,5$, in the same units. Find each mean and sample standard deviation. Which has less observed run-to-run variation? (Answer: Both means are 5; A has sample SD $1$, B has sample SD $0$, so B has less observed variation.)
With $n=200$, a tree uses a bootstrap sample of 200 draws with replacement. Write the probability that a specified row is OOB for that tree and give its approximate value to four decimals. (Answer: $(1-1/200)^{200}=0.995^{200}\approx0.3670$.)
A candidate has mean OOB RMSE 12.0 and mean runtime 20 seconds; another has mean OOB RMSE 11.9 and mean runtime 45 seconds. Repeated fits show standard deviations of 0.4 and 0.5, respectively. What is a defensible interpretation? (Answer: The second setting has a 0.1 lower mean RMSE but greater observed variability and more than twice the runtime; the small difference alone does not establish a clear practical advantage.)

What You'll Learn