Why Tune Forest Controls Together?
A random forest’s performance depends on more than the number of trees. The controls mtry, minimum node size, and sample fraction can each affect the fitted trees, and the effect of one control can depend on the settings of the others. A setting that performs well when trees are shallow, for example, need not be the best setting when trees are allowed to grow further.
Joint tuning means comparing specified combinations of these controls under a common experimental design. It does not mean changing several settings at once and attributing the result to whichever change seems most plausible. The experiment should make it possible to distinguish a parameter’s average effect from an interaction, while recording the computational cost of producing each comparison.
The three factors have distinct meanings. In a forest with \(p\) candidate predictors, mtry is the number made available for consideration at a split, and must be an admissible value no greater than \(p\). The minimum node size controls how small a node may become under the fitting procedure’s stated definition. The sample fraction controls the proportion of training observations used to grow a tree; whether those observations are drawn with or without replacement must be held fixed when comparing fractions.
For a fair experiment, also hold the response, predictors, preprocessing, tree count, split rule, class handling, and validation folds fixed. Use the same performance measure for every configuration. For regression, one possible measure is root mean squared error (RMSE); for classification, use a measure suitable for the problem and its class balance. The goal here is not to select a particular metric, but to ensure that the chosen metric is applied consistently.
Constructing a Controlled Tuning Experiment
A full grid evaluates every combination of the selected values. For example, three mtry values, two node sizes, and two sample fractions require \(3\times2\times2=12\) configurations. A grid is balanced when every setting of one factor occurs equally often alongside the settings of the other factors. Balance makes average comparisons interpretable: a setting is not accidentally judged only in combination with favorable values of another control.
Use a modest range for mtry, node size, and sample fraction, respecting the software’s valid values and the data available.
Use the same folds, metric, tree count, predictors, sampling-with-or-without-replacement choice, and other fitting settings for all configurations.
Save validation performance for each fold, along with fitting time under a consistent timing procedure.
Calculate marginal effects across the balanced grid, then check whether rankings change across settings of the other controls.
Choose a configuration using the stated performance-cost rule. Keep any final test data out of tuning and use it only for an appropriate final evaluation.
In R, a loop over a pre-specified grid can pass each configuration’s values to a forest-fitting function such as ranger. The important design feature is not the loop itself: it is that each fit receives the intended settings and the same evaluation data. Record the configuration, fold-level metric values, their summary, and elapsed fitting time so that the comparison can be audited.
Estimating Effects and Detecting Interactions
Let \(E(a,b,c)\) denote the mean validation error for a configuration with mtry setting \(a\), node-size setting \(b\), and sample-fraction setting \(c\). For a balanced grid, the average effect of changing mtry from \(a_1\) to \(a_2\) can be summarized by averaging the difference across the other factors. With two node sizes and two sample fractions, this contrast is
If error is the target, a negative contrast means that \(a_2\) has lower average error. This is a descriptive comparison over the selected grid, not proof that the same effect will hold for every possible setting or data set.
The justification for the average is straightforward. In a balanced grid, each mtry value is paired with every combination of the other factors. Taking the difference within each such pair removes the shared setting of node size and sample fraction from that particular comparison. Averaging those paired differences then summarizes the mtry change over the chosen combinations. If the differences vary substantially across combinations, report that variation: it is evidence that the mtry effect depends on the other controls.
When all configurations use the same validation folds, fold-wise differences can also be informative. For each fold, subtract one configuration’s error from the other’s, then summarize those differences. This paired comparison avoids treating a fold that happens to be difficult for every model as though it were evidence against just one configuration. It does not eliminate uncertainty, and a small observed difference should not automatically be treated as a reliable improvement.
Worked Example: Separating mtry and Node-Size Effects
In a hypothetical regression study, a team evaluates four forest configurations on the same three validation folds. Tree count, sample fraction, and all other settings are fixed. The reported fold RMSEs are:
| mtry | Minimum node size | Fold RMSEs |
|---|---|---|
| 2 | 5 | 12, 10, 11 |
| 4 | 5 | 10, 9, 11 |
| 2 | 15 | 13, 11, 12 |
| 4 | 15 | 11, 10, 12 |
Method. Calculate each configuration’s mean fold RMSE, then compare averages across the other factor. The common folds make the comparison controlled; the fold values shown here are hypothetical.
At node size 5, the mean for mtry 2 is \((12+10+11)/3=33/3=11\). The mean for mtry 4 is \((10+9+11)/3=30/3=10\). At node size 15, the corresponding means are \((13+11+12)/3=36/3=12\) and \((11+10+12)/3=33/3=11\).
The average mtry 2 error is \((11+12)/2=11.5\); the average mtry 4 error is \((10+11)/2=10.5\). Thus the average contrast for changing mtry from 2 to 4 is \(10.5-11.5=-1.0\) RMSE unit. For node size, the average error at size 5 is \((11+10)/2=10.5\), and at size 15 it is \((12+11)/2=11.5\), a contrast of \(+1.0\).
The mtry improvement is 1 RMSE unit at each node size: \(11-10=1\) at size 5 and \(12-11=1\) at size 15. In this grid, that contrast is consistent across the two node sizes; the data do not show an mtry-by-node-size interaction in these means. The lowest mean is 10, for mtry 4 and node size 5. The common mistake would be to compare only the mtry 4, node-size 5 fit with the mtry 2, node-size 15 fit and credit the entire 2-unit difference to mtry. The controlled contrasts show why that attribution would be unjustified.
Including Sample Fraction and Recording Cost
Adding sample fraction creates more combinations and gives another opportunity for interactions. A smaller fraction can reduce fitting time, but it can also change the trees and their predictive performance. Its effect should therefore be measured rather than assumed. To interpret a fraction comparison, keep the replacement rule and the training data fixed.
Worked Example: Reading a Balanced Three-Factor Grid
A hypothetical team evaluates the following mean cross-validation RMSEs. The grid crosses two mtry values, two node sizes, and two sample fractions. Every other modeling choice is held fixed.
| mtry | Node size | Sample fraction | Mean RMSE |
|---|---|---|---|
| 2 | 5 | 0.6 | 10.8 |
| 2 | 5 | 1.0 | 11.0 |
| 4 | 5 | 0.6 | 9.7 |
| 4 | 5 | 1.0 | 10.0 |
| 2 | 15 | 0.6 | 11.5 |
| 2 | 15 | 1.0 | 11.8 |
| 4 | 15 | 0.6 | 10.6 |
| 4 | 15 | 1.0 | 10.8 |
Method. Average each factor level across the four combinations of the other factors. For mtry 2, the sum is \(10.8+11.0+11.5+11.8=45.1\), so the average is \(45.1/4=11.275\). For mtry 4, the sum is \(9.7+10.0+10.6+10.8=41.1\), giving \(41.1/4=10.275\). The average contrast for mtry 4 versus 2 is \(10.275-11.275=-1.000\).
For node size 5, the average is \((10.8+11.0+9.7+10.0)/4=41.5/4=10.375\). For node size 15, it is \((11.5+11.8+10.6+10.8)/4=44.7/4=11.175\). The contrast is \(11.175-10.375=0.800\), favoring size 5 on average.
For fraction 0.6, the average is \((10.8+9.7+11.5+10.6)/4=42.6/4=10.65\). For fraction 1.0, it is \((11.0+10.0+11.8+10.8)/4=43.6/4=10.90\). The average contrast is \(10.65-10.90=-0.25\), favoring fraction 0.6 in this grid. The lowest individual mean is 9.7 at mtry 4, node size 5, and fraction 0.6. A marginal average alone would not identify that combination; the cell-level results are needed too.
Suppose the team also records that the complete fraction-0.6 set of fits took 44 seconds and the fraction-1.0 set took 67 seconds under the same timing procedure. Those totals suggest a useful performance-cost comparison, but elapsed time depends on the computing environment and is not a measure of predictive quality. The defensible conclusion is limited to this experiment: fraction 0.6 had lower average RMSE and lower recorded fitting time in the tested grid.
Balancing Performance, Stability, and Computation
The lowest observed validation error is not always the best operational choice. If two configurations perform similarly, a much slower one may offer little practical value. One possible rule is the one-standard-error rule: identify the configuration with the lowest estimated validation error, then consider configurations whose error is no more than one estimated standard error above that minimum. Among those, choose the simplest or least costly configuration according to a stated criterion. This is a practical selection heuristic, not a guarantee of equal future performance.
When using this rule, be precise about which standard error is being used and how it was calculated. Fold scores can be dependent because their training sets overlap, so a naive standard error across folds should not be presented as a universal uncertainty measure. A comparison across repeated splits or another suitable uncertainty assessment may be preferable when the decision warrants it.
Worked Example: Choosing a Faster Configuration Within a Tolerance
A hypothetical tuning run produces the following cross-validation summaries. The team has decided in advance to use the one-standard-error rule and, among eligible settings, choose the one with the lowest recorded fitting time.
| Configuration | Mean RMSE | Estimated SE | Fit time |
|---|---|---|---|
| A | 8.40 | 0.12 | 60 seconds |
| B | 8.46 | 0.10 | 29 seconds |
| C | 8.51 | 0.08 | 18 seconds |
| D | 8.54 | 0.07 | 12 seconds |
Method. The lowest mean RMSE is 8.40, from configuration A. Its estimated SE is 0.12, so the one-SE cutoff is \(8.40+0.12=8.52\). A, B, and C are eligible because \(8.40\leq8.52\), \(8.46\leq8.52\), and \(8.51\leq8.52\). D is not eligible because \(8.54>8.52\).
Among the eligible configurations, C is fastest at 18 seconds, compared with 29 seconds for B and 60 seconds for A. The rule therefore selects C. This does not establish that C’s true predictive error equals A’s; it makes explicit the tradeoff the team chose. The common trap is to select D solely because it is fastest, ignoring that it falls outside the stated performance tolerance.
Common Experimental Traps
- Changing several settings without a grid. A better result cannot be attributed to one factor if mtry, node size, and sample fraction all changed together.
- Using different validation folds for different candidates. A difficult fold can make a model look worse for reasons unrelated to its settings. Apply the same folds to every configuration.
- Ignoring interactions. A favorable average effect may hide a setting that helps at one node size and hurts at another. Inspect cell-level results, not just marginal means.
- Changing the sampling rule along with the fraction. If replacement status changes too, the comparison no longer isolates sample fraction.
- Tuning on the test set. Repeatedly choosing settings based on test performance makes that set part of the tuning process. Reserve it for final assessment, not selection.
- Reporting time without conditions. State how fitting time was recorded and compare fits made under consistent computing conditions. Treat time as an operational measure, not a model-quality metric.
Check Your Understanding
For each problem, use the stated summaries and comparison rule. Treat all data as hypothetical.
- A controlled two-fold experiment reports mean RMSEs of 14 and 12 for mtry 2 and 4 at node size 6, and 16 and 13 at node size 18. Calculate the average mtry contrast for changing from 2 to 4, and identify the lower-error mtry at each node size. (Answer: average contrast \(=(12-14+13-16)/2=-2.5\); mtry 4 has lower error at both node sizes.)
- At fixed mtry and sample fraction, node size 4 has errors 9 and 11 across two folds, while node size 12 has errors 10 and 10 on those same folds. Calculate both mean errors and the paired differences, defined as size 12 minus size 4 for each fold. What does the comparison show? (Answer: both means are 10; paired differences are \(+1\) and \(-1\), so the average difference is 0 and the fold-wise direction is not consistent.)
- A balanced grid has four mean errors at sample fraction 0.5: 7.8, 8.2, 8.6, and 8.4. At fraction 1.0 the corresponding values are 8.0, 8.1, 8.9, and 8.6. Calculate the average error at each fraction and the contrast for 0.5 minus 1.0. (Answer: fraction 0.5 average \(33.0/4=8.25\); fraction 1.0 average \(33.6/4=8.40\); contrast \(=-0.15\).)
- The minimum cross-validation error is 5.70 with estimated SE 0.08. Three configurations have errors and times of 5.70 and 40 seconds, 5.76 and 22 seconds, and 5.80 and 10 seconds. Under the one-SE rule, which is the fastest eligible configuration? (Answer: the cutoff is \(5.70+0.08=5.78\); the first two are eligible, so select the second configuration at 22 seconds.)
- A modeling team tries six settings, chooses the one with the lowest test-set error, and reports that same error as its final performance estimate. Identify the design problem and the correction. (Answer: the test set was used for tuning, so its error is no longer an untouched final assessment; select settings using training-data validation, then evaluate the selected model on the reserved test set.)