Why Diagnose Tree Stability?
A decision tree can be easy to explain: its splits create regions, and each terminal node assigns a prediction to the cases that reach it. But a small change in the training sample can change which split is selected, where a numeric cutpoint falls, or which cases share a terminal node. A single fitted tree does not show whether its rules are dependable under plausible changes in the data.
A stability assessment refits a tree on multiple perturbed versions of the training data and compares the results. It asks two related but distinct questions: Does the tree structure recur? and Do the resulting predictions remain similar? A tree can have different-looking rules but produce similar predictions for the cases of practical interest. Conversely, trees can agree on many splits yet disagree for a case near a decision boundary.
Make the Comparison Fair
A useful comparison changes the data in a defined way while holding other modeling choices fixed. Use the same response, predictors, preprocessing, split criterion, stopping controls, and pruning approach for each refit. Otherwise, observed differences may reflect changed modeling choices rather than sensitivity to the sample. If the question is specifically about sensitivity to tuning choices, vary those choices deliberately and report that this is a different comparison.
The prediction comparison also needs a consistent set of cases. For example, fit each tree on a perturbed training sample, then predict the same held-out cases with every tree. Comparing predictions on each tree's own training sample confounds changes in the fitted tree with changes in the cases being predicted.
Several levels of structural comparison are useful. Root splits are straightforward to compare because every tree has a root. Deeper splits require more care: a split in one tree is only directly comparable with a split in another if the parent regions are meaningfully alike. A split on age within a particular tenure group is not necessarily comparable with a split on age applied to the whole sample. Record the context as well as the variable and cutpoint.
State whether fits use bootstrap samples, subsamples, or another defined change. The assessment applies to that choice, not to every possible change in data.
Keep the response, predictors, preprocessing, and tree controls fixed unless the purpose is explicitly to examine sensitivity to those choices.
Record split variables, cutpoints, and node context. Summarize how often a split or predictor appears, and report threshold variation only among comparable splits.
Predict the same cases with each fit. Summarize variation for numeric predictions or the class and probability patterns for classification predictions.
Separate recurring conclusions from unstable details, and state the perturbation scheme and number of refits.
Summarizing Structural Stability
For a particular split feature, a simple descriptive summary is its selection frequency: the number of fitted trees containing that feature in the specified position, divided by the total number of fits. For example, root selection frequency measures how often a feature is selected at the root. This is not the same as counting its appearance anywhere in a tree; state which definition is being used.
For a numeric predictor selected at a comparable node, record the cutpoints as well as the frequency. A feature may recur while its threshold varies substantially. Do not average cutpoints from unrelated nodes or from different predictors: those values do not describe a common split.
A frequency is descriptive, not a formal probability that the predictor is “truly” important. Its meaning depends on the chosen perturbations, sample size, tree controls, and available candidate predictors. It does not establish a causal relationship, and it does not by itself show that the tree predicts accurately on new data.
Worked Example: Comparing Root Splits Across Six Fits
A hypothetical insurer fits six regression trees to perturbed versions of a training sample, using identical controls. The root splits are: age at 24, age at 26, region grouped as North versus other regions, age at 25, tenure at 3 years, and tenure at 4 years. Compare the root predictors and the age cutpoints. Name the method and a mistake to avoid.
Method. Tabulate root-predictor frequencies across all six fits. Then summarize thresholds only for the fits that selected age at the root.
Age is selected in fits 1, 2, and 4, so its root frequency is \(3/6=0.50\). Tenure is selected in fits 5 and 6, so its root frequency is \(2/6\approx0.33\). Region is selected in fit 3, so its frequency is \(1/6\approx0.17\). The frequencies sum to \(6/6=1\), as exactly one root split is recorded for each fit.
For age, the three comparable cutpoints are 24, 26, and 25 years. Their mean is \((24+26+25)/3=75/3=25\) years, and their range is \(26-24=2\) years. Thus, age is the most frequent root predictor in these fits, but it is not selected in every fit. When selected, its root cutpoint varies by 2 years.
Common mistake. Averaging all six cutpoints is not meaningful: the region split is categorical, and the tenure cutpoints concern a different predictor. The correction is to summarize age thresholds only among fits with an age root split, while reporting the predictor-selection frequencies separately. These results describe this perturbation exercise; they do not prove that age is the most important predictor in the underlying population.
Comparing Predictions Across Fits
Structural variation matters most when it changes conclusions or decisions. For a regression tree, predict the same case with every fitted tree, then examine the collection of predictions. The range, mean, median, and standard deviation can summarize different aspects of that variation. The mean can be pulled by an extreme prediction, so it is often useful to look at the individual predictions or a robust summary as well.
For a classification tree, record the predicted class for each fit and, where available, the predicted probability for a specified class. The fraction of fits predicting that class is a descriptive vote frequency. A majority vote can hide uncertainty when the vote is close, and predicted probabilities can vary even when the assigned class stays the same. Use the probability threshold actually specified for the task; do not assume every decision rule uses 0.5.
For a fixed case, the sample standard deviation of its \(B\) numeric predictions \(z_1,\ldots,z_B\) is
This is a descriptive measure of variation across the fitted predictions. It is not a standard error for a population parameter, nor a guarantee about future prediction error. The latter requires an appropriate evaluation against outcomes not used to fit the model.
Worked Example: Finding a Case With Unstable Cost Predictions
Five trees fitted to perturbed hypothetical training samples predict annual repair cost, in hundreds of dollars, for two fixed properties. Property A receives predictions 40, 42, 38, 40, and 70. Property B receives 40, 42, 38, 40, and 41. For each property, calculate the mean and range. For Property A, also calculate the median and sample standard deviation. Identify the more stable prediction and a common mistake to avoid.
Method. Compare predictions case by case, not tree by tree. Calculate the range as maximum minus minimum and use the sample standard deviation formula with denominator \(B-1=4\).
For Property A, the sum is \(40+42+38+40+70=230\), so the mean is \(230/5=46\) hundreds of dollars, or $4,600. The sorted predictions are 38, 40, 40, 42, 70, so the median is 40 hundreds of dollars, or $4,000. The range is \(70-38=32\) hundreds of dollars, or $3,200.
For the sample standard deviation of Property A, deviations from the mean 46 are \(-6,-4,-8,-6,24\). Their squared deviations sum to \(36+16+64+36+576=728\). The sample variance is \(728/4=182\), and the sample standard deviation is \(\sqrt{182}\approx13.49\) hundreds of dollars, or approximately $1,349.
For Property B, the sum is \(40+42+38+40+41=201\), giving a mean of \(201/5=40.2\) hundreds of dollars, or $4,020. Its range is \(42-38=4\) hundreds of dollars, or $400. Property B has the more stable predictions: its range is much smaller, and its predictions are tightly grouped.
Common mistake. Reporting only Property A’s mean of $4,600 would conceal that four predictions are between $3,800 and $4,200 while one is $7,000. The median and range reveal this sensitivity. The standard deviation calculation is also checked against the units: predictions are in hundreds of dollars, so the resulting standard deviation must be in hundreds of dollars, not dollars.
What a Stability Assessment Can and Cannot Establish
A stable feature or prediction is one that recurs under the particular perturbations examined. That is useful evidence when deciding how strongly to rely on an individual tree's detailed rules. However, stability is not the same as accuracy. All refitted trees might make similar but poor predictions. Conversely, the root split can vary while predictions for a relevant group of cases remain close.
Interpret the results at the level that matches the decision. If an analyst needs to explain a rule, recurring splits and comparable cutpoints may matter. If the goal is to estimate a cost for a particular case, the spread of that case's predictions is more direct. When conclusions change across fits, report that sensitivity rather than presenting one tree as uniquely determined by the data.
A further trap is to treat every deeper split as a repeated independent confirmation. Branches are conditional: a split only applies to cases that have reached its node. If the earlier split changes, a later split may describe a different subset or may not exist at all. Compare deeper rules only when their node contexts are comparable, and avoid counting a feature's appearances without specifying its position or context.
Check Your Understanding
For each question, make the comparison using the specified perturbations and report what the calculation supports.
- Across 10 refitted trees, a predictor appears at the root in 7 fits, while another appears at the root in 3. What is the first predictor’s root selection frequency, and what is the second predictor’s frequency? (Answer: \(7/10=0.70\) and \(3/10=0.30\), respectively.)
- A numeric predictor is selected at a comparable root node in five fits, with cutpoints 18, 20, 22, 19, and 21. Find the mean and range of these cutpoints. (Answer: Mean \(=100/5=20\); range \(=22-18=4\).)
- Four trees predict a hypothetical claim cost of 12, 14, 13, and 21 thousand dollars for the same policy. Find the mean and range, then identify the value that makes the mean less representative of the cluster of predictions. (Answer: Mean \(=60/4=15\) thousand dollars; range \(=21-12=9\) thousand dollars; 21 is the separated high prediction.)
- For one case, five classification trees assign probabilities of 0.71, 0.63, 0.52, 0.48, and 0.39 to class “renewal.” Using a 0.50 class threshold, find the positive-class vote frequency, mean probability, and probability range. (Answer: \(3/5=0.60\); mean \(=2.73/5=0.546\); range \(=0.71-0.39=0.32\).)
- Two trees both use age as a split variable, but one splits the full sample at the root and the other splits only the subset with tenure below 2 years. Can their age thresholds be treated as directly comparable? State the key reason. (Answer: Not without qualification; the splits apply to different parent regions, so they need not describe the same cases or decision.)