Why Random-Forest Controls Matter for Severity
A random forest combines predictions from many decision trees. For claim severity, each tree predicts a claim cost from available characteristics, such as vehicle age, repair type, or territory; the forest averages the tree predictions. The controls determine how much the trees resemble one another, how finely they divide the data, and how much computation the fit requires.
These choices matter because claim severity can be highly variable. A forest with very small terminal groups may react strongly to a few unusual claims, while a forest with overly large groups may miss meaningful differences. Increasing the number of trees can stabilize an average, but it does not automatically correct a poor target, leakage, or unrepresentative data. The goal is to compare defensible settings on data not used to fit the trees, while keeping the intended use and computational budget in view.
For a severity model, first define the target precisely. For example, a hypothetical analysis might predict the paid amount for each closed claim, in dollars. Fitting amounts in dollars is not the same modeling choice as fitting log amounts: the target scale affects what the tree predictions represent and how errors are judged. Here, examples use dollar-scale predictions and squared-error calculations. A few large claims can have substantial influence on squared error, so the selected scale and evaluation measure should match the intended decision.
Controls to Compare
A useful tuning plan varies controls for a reason rather than trying combinations without a question in mind. Common controls include the number of trees, the number of candidate predictors at each split, and restrictions on tree size. Sampling options can also change the fitted forest. In R, names differ by package; for example, tree-count, candidate-predictor, and minimum-node-size arguments may have different names or defaults.
| Control | What it changes | Practical implication |
|---|---|---|
| Number of trees, \(B\) | How many tree predictions are averaged | More trees can stabilize the average, usually with more fitting and scoring time. |
| Candidate predictors per split, often called \(mtry\) | How many randomly selected predictors can compete at each split | Smaller values encourage tree diversity; larger values give strong predictors more chances to be considered. |
| Minimum terminal-node size | How few training observations a leaf can contain | Larger minimums generally produce less locally detailed predictions; smaller minimums can fit finer patterns. |
| Maximum depth or other tree-size restrictions | How far a tree can keep splitting | Restrictions can limit complexity and computation, but may prevent useful structure from being represented. |
| Sampling fraction and replacement | How training rows are sampled for each tree | These settings affect the observations available to each tree and can change both tree similarity and out-of-bag predictions. |
The controls do not all serve the same purpose. Increasing \(B\) adds more trees to the average; it does not, by itself, make an individual tree shallower. Changing \(mtry\) affects which splits are available to each tree, not the number of predictors in the data set. Increasing a minimum node size restricts local detail, but does not guarantee better prediction. Compare these settings using a validation design that reflects the intended scoring situation, and keep repeated observations from the same policy or other related unit from leaking across partitions.
Worked Example: Comparing Tree Counts on a Validation Set
Suppose a hypothetical insurer has reserved four claims for validation. Their observed severities are \(1{,}000\), \(2{,}000\), \(3{,}000\), and \(4{,}000\) dollars. Two otherwise comparable forests use 100 and 400 trees. Their respective validation predictions are \([1{,}200, 1{,}800, 3{,}300, 3{,}700]\) and \([1{,}100, 1{,}900, 3{,}100, 3{,}900]\). Compare their root mean squared errors (RMSE), defined as the square root of the mean squared prediction error.
Method. Calculate the error for each claim as prediction minus observed severity, square each error, average the squared errors, and take the square root. This evaluates the forest’s averaged predictions, not the average of separate tree-level RMSE values.
100-tree forest. Its errors are \(200\), \(-200\), \(300\), and \(-300\) dollars. Their squares are \(40{,}000\), \(40{,}000\), \(90{,}000\), and \(90{,}000\) dollars squared. The sum is \(260{,}000\), so the mean squared error is \(260{,}000/4=65{,}000\). Thus, RMSE is \(\sqrt{65{,}000}=\$254.95\), rounded to the nearest cent.
400-tree forest. Its errors are \(100\), \(-100\), \(100\), and \(-100\) dollars. Their squares sum to \(4(10{,}000)=40{,}000\). The mean squared error is \(40{,}000/4=10{,}000\), so RMSE is \(\sqrt{10{,}000}=\$100.00\), rounded to the nearest cent.
On this tiny hypothetical validation set, the 400-tree forest has lower RMSE. That comparison alone does not establish that 400 trees will be better on future claims; the difference could be sensitive to which claims were held out. More trees usually increase computation roughly in proportion to the number of trees when other settings are fixed. They are most useful when the forest average is still changing materially as trees are added.
Trap and correction. If two trees predict \(800\) and \(1{,}400\) dollars for a claim whose observed severity is \(1{,}000\), their forest prediction is \((800+1{,}400)/2=1{,}100\). The forest’s squared error is \((1{,}100-1{,}000)^2=10{,}000\). Averaging the trees’ separate squared errors gives \((40{,}000+160{,}000)/2=100{,}000\), which is not the squared error of the forest prediction. Evaluate the ensemble prediction after averaging the tree predictions.
Why More Trees Can Stabilize Predictions
Averaging reduces the effect of variation among individual trees, but the reduction depends on how correlated the trees are. The following result makes that dependence explicit.
To justify the formula, let \(T_b=\hat{f}_b(x)\). Each tree has variance \(\sigma^2\), and the covariance of any two distinct trees is \(\rho\sigma^2\). The variance of a sum is the sum of all individual variances and all ordered pairwise covariances. Therefore,
If the trees were uncorrelated, \(\rho=0\), the variance would be \(\sigma^2/B\). If they were perfectly correlated, \(\rho=1\), averaging would leave the variance at \(\sigma^2\). In practice, trees are neither perfectly independent nor generally identical in variance. The key implication is qualitative: adding trees can reduce variation in the ensemble average, but highly similar trees limit the gain. Randomly selecting candidate predictors at each split helps create diversity; it is not merely a way to reduce the number of calculations.
Choosing Candidate Predictors and Limiting Tree Size
At each split, \(mtry\) is the number of predictors sampled as candidates from the \(p\) available predictors. The best split is selected from those candidates according to the tree-fitting criterion. A small \(mtry\) can reduce the chance that a particular strong predictor is available at a given split, but it can also make trees less alike. A large \(mtry\) gives strong predictors more opportunities to compete, while potentially making trees more similar.
Worked Example: How \(mtry\) Changes Split Opportunities
Suppose a hypothetical severity data set has \(p=8\) candidate predictors. At a particular node, assume two of them could provide useful splits. The candidate subset is drawn uniformly without replacement. Compare the chance that at least one of these two predictors is available when \(mtry=2\) and when \(mtry=6\).
Method. Count all possible subsets of size \(mtry\). The chance of including at least one useful predictor is one minus the chance of selecting only from the other six predictors.
For \(mtry=2\), there are \(\binom{8}{2}=28\) possible subsets. There are \(\binom{6}{2}=15\) subsets containing neither useful predictor. Thus, the chance of at least one useful predictor being available is \(1-15/28=13/28=0.4643\), to four decimal places.
For \(mtry=6\), there are \(\binom{8}{6}=28\) subsets in total and \(\binom{6}{6}=1\) subset containing neither useful predictor. The chance is \(1-1/28=27/28=0.9643\), to four decimal places.
The larger setting gives a much higher chance that a useful predictor is considered at this node. It may also increase similarity among trees if the same predictors repeatedly dominate splits. The calculation is about availability, not guaranteed selection or validation performance: a predictor being offered does not mean its split will be chosen.
Tree-size controls address a separate question: how finely should the data be partitioned? A small minimum terminal-node size allows predictions based on few observations. This can represent local variation, but it can also make predictions unstable, particularly when a few large claims fall into a leaf. A larger minimum size pools more observations and usually smooths predictions. A maximum depth can constrain complexity more directly. Neither control has a universally best value; their effects depend on the number of training observations, predictor structure, and severity distribution.
Sampling settings also deserve care. In a conventional bootstrap with \(n\) draws from \(n\) training rows, the probability that one particular row is omitted from a tree’s sample is \((1-1/n)^n\), which approaches \(e^{-1}\), about 0.368, as \(n\) grows. Such omitted rows are called out-of-bag (OOB) for that tree. OOB predictions combine predictions from trees that omitted each row and can provide a useful internal performance check. They are not a substitute for a final independent evaluation when the modeling process has repeatedly used the same data to choose controls.
Variable Importance: Useful Signal, Limited Meaning
Random-forest software may report importance using split-based measures or permutation. A split-based measure aggregates how much the fitted trees improve their splitting criterion when using a predictor. Such measures can favor predictors with more possible split points or many categories, depending on the implementation and data. Permutation importance instead measures how much a chosen loss worsens when one predictor’s values are shuffled while the fitted forest is otherwise held fixed.
Permutation importance answers a predictive question: how much did the model’s performance rely on the observed information in this predictor, under this evaluation procedure? If predictors are correlated, shuffling one may leave a similar predictor available to carry much of the same information. Its importance can then appear small even when the underlying information is useful. Conversely, importance does not establish that changing a predictor would change claim severity. It is not a causal effect, nor a measure of fairness or business value by itself.
Worked Example: Calculating Permutation Importance
For a hypothetical five-claim holdout set, observed severities are \(1{,}000\), \(2{,}000\), \(3{,}000\), \(4{,}000\), and \(5{,}000\) dollars. The fitted forest’s baseline predictions are \(1{,}100\), \(1{,}800\), \(3{,}000\), \(4{,}300\), and \(4{,}600\). After permuting one predictor and rescoring the same rows, suppose predictions are \(700\), \(1{,}600\), \(2{,}800\), \(3{,}500\), and \(4{,}400\). Calculate the increase in mean squared error.
Baseline loss. The prediction errors are \(100\), \(-200\), \(0\), \(300\), and \(-400\) dollars. The squared errors are \(10{,}000\), \(40{,}000\), \(0\), \(90{,}000\), and \(160{,}000\). Their sum is \(300{,}000\), so the baseline mean squared error is \(300{,}000/5=60{,}000\) dollars squared.
Permuted loss. The errors after permutation are \(-300\), \(-400\), \(-200\), \(-500\), and \(-600\) dollars. The squared errors are \(90{,}000\), \(160{,}000\), \(40{,}000\), \(250{,}000\), and \(360{,}000\). Their sum is \(900{,}000\), so the permuted mean squared error is \(900{,}000/5=180{,}000\) dollars squared.
The estimated importance on this loss scale is the increase \(180{,}000-60{,}000=120{,}000\) dollars squared per claim. This one permutation is illustrative, not a stable importance estimate; in practice, repeat permutations and summarize the variation. The result describes this model’s reliance on the predictor in this evaluation set. It does not prove that the predictor causes severity or that it should be used in a business decision.
A Practical Comparison Plan
A defensible comparison changes a small number of controls at a time and records both predictive performance and computational consequences. For instance, first check whether validation performance is still changing as \(B\) increases. Then compare a modest range of \(mtry\) values and tree-size settings, keeping the data partition and target definition consistent. If many combinations are tested, avoid choosing and reporting a result as though the validation set had been used only once; repeated tuning can make its apparent performance optimistic.
Specify the severity outcome, measurement unit, observation unit, and which claims are eligible. Keep the target scale consistent across candidate fits.
Vary tree count, candidate predictors per split, and a tree-size control separately enough to understand their effects. Record software-specific settings rather than assuming package defaults are equivalent.
Use the same evaluation rows and loss definition for each fit. Record runtime or scoring burden as well as predictive performance, and treat small differences cautiously.
Use importance to generate predictive insights, check correlated alternatives, and explain limitations. Do not describe a ranking as a causal effect.
Common exam trap. A lower training error is not evidence that a forest will predict new claims better: a flexible forest can adapt to its training observations. Also, a larger tree count is not a general remedy for weak validation results. It can stabilize an average, but it cannot repair target leakage, a mismatched severity definition, or a validation set that does not represent the intended use. A well-reasoned comparison states what each control changes, what evidence supports the choice, and what uncertainty remains. This treatment is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
Work each problem through before checking the final answer.
- A forest has five tree predictions for one hypothetical claim: \(1{,}200\), \(1{,}000\), \(1{,}400\), \(1{,}600\), and \(800\) dollars. The observed severity is \(1{,}000\) dollars. Find the forest prediction and its squared error. (Answer: prediction \(=\$1{,}200\); squared error \(=\$40{,}000\).)
- There are ten candidate predictors, two of which are considered useful at a node. A uniformly sampled subset of three predictors is considered. What is the probability that at least one useful predictor is included? (Answer: \(1-\binom{8}{3}/\binom{10}{3}=8/15=0.5333\), to four decimal places.)
- Three validation severities, in thousands of dollars, are \(2\), \(4\), and \(6\); their predictions are \(3\), \(3\), and \(5\). Calculate RMSE in thousands of dollars. (Answer: \(\sqrt{(1+1+1)/3}=1\) thousand dollars.)
- A conventional bootstrap tree draws 500 rows with replacement from 500 training rows. Using \((1-1/n)^n\) for the omission probability, approximately how many trees in a 500-tree forest omit a particular row? (Answer: omission probability is approximately \(0.3675\), so \(500(0.3675)\approx184\) trees.)
- On a holdout set, a forest has baseline MSE of \(25{,}000\) dollars squared. Permuting a predictor raises MSE to \(70{,}000\) dollars squared. State the permutation importance on the MSE-increase scale and one conclusion it does not support. (Answer: increase \(=45{,}000\) dollars squared per claim; it does not establish that the predictor causes severity.)