Why Subsample Rows During Boosting?
Gradient boosting builds a model in stages: each new tree is fitted to information about the current model’s errors, then added to the existing prediction. Fitting every tree with every training observation can be computationally demanding. It can also make successive trees respond strongly to the same details in the training data. Stochastic gradient boosting changes the tree-fitting step by selecting a random subset of training rows for each stage.
Row subsampling is not the same as fitting one model to a permanently reduced data set. A fresh subset is typically selected at each boosting iteration, so different observations can contribute to different trees. The trees are still added sequentially, and each tree is fitted using the current model’s gradients on the rows selected for that iteration.
For a differentiable loss \(L(y_i,F(x_i))\), define the negative gradient for observation \(i\) at stage \(t\) as
Here \(F_{t-1}\) is the model before the new tree is added. With row subsampling, the tree \(h_t\) is fitted using pairs \((x_i,g_{it})\) for \(i\in S_t\), rather than for every training row. The stagewise update is
where \(\nu\) is the learning rate. The subsampling choice determines which observations inform the new tree; it does not change this additive update rule. In particular, an unselected row does not supply a gradient to fit that stage’s tree, but the fitted tree can still change the model’s prediction for that row.
What Is Random, and What Is Updated?
In the common setup described here, the subset is a simple random sample of a fixed size \(m\), selected without replacement within each iteration. A new subset is drawn at each iteration. The gradient values are recalculated from the current fitted model; they are not residuals frozen at the start of training. This distinction matters because boosting is sequential: the target for a later tree depends on the trees already fitted.
This procedure differs from random-forest row sampling in its role in the model. In boosting, each tree is fitted in sequence to gradient information from the current model. A random forest’s trees are not successive gradient corrections to one another. Both methods can use randomized row selection, but the resulting trees and their purpose differ.
Worked Example: Fitting a Subsampled Squared-Error Tree
Method: calculate the current residuals, use only selected rows to estimate the terminal-node values, then apply the shrunken tree to all rows. This hypothetical six-observation regression problem uses squared-error loss \(L(y,F)=\frac{1}{2}(y-F)^2\), current predictions \(F_0=10\), subsampling fraction \(q=4/6\), and learning rate \(\nu=0.2\). The selected rows are 1, 2, 4, and 5. A one-split tree has left terminal node \(x\leq 3\) and right terminal node \(x>3\).
| Row | \(x\) | \(y\) | Current prediction \(F_0\) | Residual \(y-F_0\) | Selected? |
|---|---|---|---|---|---|
| 1 | 1 | 12 | 10 | 2 | Yes |
| 2 | 2 | 10 | 10 | 0 | Yes |
| 3 | 3 | 8 | 10 | -2 | No |
| 4 | 4 | 14 | 10 | 4 | Yes |
| 5 | 5 | 11 | 10 | 1 | Yes |
| 6 | 6 | 9 | 10 | -1 | No |
For this loss, the negative gradient is the residual: \(g_{i1}=y_i-F_0\). Among the selected rows in the left node, rows 1 and 2 have gradients 2 and 0. Their mean is \((2+0)/2=1\). Among selected rows in the right node, rows 4 and 5 have gradients 4 and 1. Their mean is \((4+1)/2=2.5\). Thus the fitted tree’s terminal-node values are 1 and 2.5.
After shrinkage, the left-node update is \(0.2(1)=0.2\), and the right-node update is \(0.2(2.5)=0.5\). The new predictions are therefore 10.2 for rows 1, 2, and 3, and 10.5 for rows 4, 5, and 6. Rows 3 and 6 were not used to fit this tree, but they still receive its prediction because they fall into one of its terminal nodes.
Check: the unshrunk left value is the selected-row average residual, 1, and the unshrunk right value is 2.5. Multiplying each by the stated learning rate gives the updates 0.2 and 0.5. Common mistake avoided: omitting a row from a stage’s tree fit does not mean its prediction must remain unchanged after that tree is added.
Why the Subsample Gradient Average Is Unbiased
A useful justification for row subsampling concerns the average gradient, not the entire fitted tree. Treat the gradients \(g_{1t},\ldots,g_{Nt}\) as fixed, conditional on the current model and training data. Suppose a simple random sample of exactly \(m\) rows is selected without replacement. Let \(I_i\) equal 1 if row \(i\) is selected and 0 otherwise. Each row has inclusion probability \(P(I_i=1)=m/N\).
The sample mean gradient therefore has the full-data mean gradient as its expectation. This explains why a fresh subset can provide a useful, less expensive view of the current gradient pattern. It does not prove that the tree fitted from the subset equals the full-data tree on average. Tree fitting includes choices such as split selection, and those choices are nonlinear in the observed gradients; a split chosen on one subset may differ from the split chosen on another.
There is also sampling variability around the full-data gradient pattern. Smaller \(m\) generally means less information for fitting each tree and greater uncertainty in the tree’s splits or leaf estimates. Whether that extra randomness helps or harms prediction depends on the data and the other modeling choices.
Worked Example: Inclusion and Repeated Omission
Method: use simple-random-sample probabilities, then assume independent fresh samples between iterations. A hypothetical claims model has \(N=25\) training rows and uses \(q=0.4\), so each stage selects \(m=qN=0.4(25)=10\) rows.
A particular row has probability \(m/N=10/25=0.4\) of being selected at one stage, and probability \(1-0.4=0.6\) of being omitted. Two specified rows are both selected with probability
The second factor is \(9/24\), not \(10/25\), because after the first specified row is selected, there are 9 remaining sample places among 24 remaining rows. Over four independently sampled stages, the probability that one particular row is omitted every time is
Consequently, the probability that it is selected at least once over those four stages is \(1-0.1296=0.8704\). This does not guarantee that the row influences the final model in a useful way; it only describes whether it contributes gradient information to at least one of these four tree fits.
Check: the sample size is exactly 10, the within-stage probability for two specified rows is 0.15, and the across-stage calculation uses fresh independent samples. Common mistake avoided: the two rows’ inclusion events within a fixed-size sample are dependent, so their joint probability is not \(0.4^2\).
Effects on Variance, Speed, and Fit
| Aspect | Potential effect of a smaller subsampling fraction | What to check |
|---|---|---|
| Tree-fitting cost | Fewer rows are used to find splits and estimate leaf values, which can reduce the work per tree. | Account for fixed overhead and the number of trees; total runtime need not fall in direct proportion to \(q\). |
| Randomness | Different samples can produce different trees. The extra randomness may discourage repeatedly fitting the same training details. | Compare results across repeat fits; increased randomness does not guarantee lower prediction variance. |
| Fit | A moderate amount of randomness may improve generalization in some settings. | Very small samples can miss important patterns, produce unstable splits, or underfit. |
A crucial distinction is between variation between fitted models and overfitting to the training data. Subsampling introduces random variation across runs with different seeds. At the same time, that randomness can sometimes reduce a model’s tendency to repeatedly follow idiosyncrasies of the training sample. One effect does not imply the other: a subsampled model can be more variable across runs and still have better average validation performance, or it can simply fit less well.
The computational saving is also conditional. If tree-fitting work were exactly proportional to the number of rows, reducing the fraction to \(q\) would reduce that part of the cost to roughly \(q\) of its full-data amount. Real implementations also incur work that does not shrink in that way. In addition, a smaller fraction may require a different number of boosting iterations to achieve a comparable fit. Do not infer a total runtime or a final predictive advantage from \(q\) alone.
Worked Example: Comparing Runtime and Repeat-Fit Variability
Method: calculate the stated approximate fitting costs, then summarize the repeated validation RMSE values using their mean and sample variance. A hypothetical model has \(N=1{,}000\) rows. Assume, solely for this cost calculation, that fitting one tree takes \(2+0.01n\) seconds when it uses \(n\) rows. Compare 80 trees fitted with all rows against 80 trees fitted with \(q=0.5\), or 500 rows per tree.
With all rows, the estimated cost per tree is \(2+0.01(1{,}000)=12\) seconds, so 80 trees take \(80(12)=960\) seconds. At \(q=0.5\), the estimated cost per tree is \(2+0.01(500)=7\) seconds, so 80 trees take \(80(7)=560\) seconds. The estimated saving is \(960-560=400\) seconds, or \(400/960=0.4167\), approximately 41.67%.
Now suppose repeated fits with the same data split and other settings produce validation RMSE values of 2.10, 2.20, and 2.30 for full-row fitting, and 2.00, 2.20, and 2.40 for subsampling. The full-row mean is \((2.10+2.20+2.30)/3=2.20\). Its sample variance is
The subsampling mean is also \((2.00+2.20+2.40)/3=2.20\). Its sample variance is
In these hypothetical results, average validation RMSE is the same, while the observed repeat-fit sample variance is higher for subsampling. The calculations do not establish that subsampling always increases variability or that it has no predictive value; they show why runtime and validation behavior should be assessed separately. Check: the estimated costs use the stated overhead as well as the row-dependent term, and the variance denominators are \(3-1=2\). Common mistake avoided: multiplying the full-row cost by 0.5 would incorrectly assume that all runtime is proportional to the number of rows.
Practical Interpretation and Common Traps
For an actuarial prediction task, such as a hypothetical model of claim severity, row subsampling means that a given stage’s tree is informed by only the selected training policies or claims. The loss gradients must correspond to the model as it stands at that stage. Training rows not selected for the tree fit are still subject to the fitted tree’s prediction when the model is updated. Validation observations, by contrast, should be used to assess predictions rather than to fit that stage’s tree.
- Do not sample once and then treat the reduced data as the whole training set. The method discussed here draws a fresh subset for each boosting iteration.
- Do not freeze gradients across stages. Recalculate them from the current model before fitting each new tree.
- Do not equate subsampling fraction with a guaranteed reduction in total runtime. Fixed costs, implementation details, and the number of stages matter.
- Do not claim that subsampling always improves generalization. A small subset can add too much noise or omit useful patterns.
- Do not confuse row subsampling with changing the learning rate. The learning rate controls the size of the tree’s contribution; the subsampling fraction controls which rows inform its fit.
A sound assessment considers the subsampling fraction together with validation performance and repeatability. Hold the data split and other settings fixed when comparing alternatives, and use repeated fits or suitable validation procedures when random seed effects could change the conclusion. A single favorable run may reflect a fortunate subset rather than a stable improvement. This treatment of stochastic fitting is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
Work through each calculation or interpretation before checking the final answer.
- A hypothetical training set has 40 rows, and each tree uses a simple random sample of 10 rows without replacement. Find the probability that one specified row is omitted at a given stage, and the probability it is omitted at all three of four independently sampled stages. (Answer: \(0.7500\) for one stage; \(0.3164\) for all four stages, since \(0.75^4=0.31640625\).)
- A data set has 30 rows, with 9 selected for each tree. What is the probability that two specified rows are both selected in one stage? (Answer: \((9/30)(8/29)=12/145\approx 0.0828\).)
- Under squared-error loss, a terminal node contains selected-row residuals \(-2, 1, 3\). Find its fitted leaf value and the prediction update when the learning rate is \(0.1\). (Answer: leaf value \((-2+1+3)/3=2/3\); update \(0.1(2/3)=1/15\approx 0.0667\).)
- For a hypothetical tree-fitting cost of \(1+0.02n\) seconds, compare the estimated cost of 50 trees using all 600 rows with 50 trees using \(q=0.25\). State the estimated saving as a percentage of the full-row cost. (Answer: full-row cost \(50[1+0.02(600)]=650\) seconds; subsampled cost \(50[1+0.02(150)]=200\) seconds; saving \(450/650\approx 69.23\%\).)
- At a fixed boosting stage, suppose 20 training observations have fixed gradients with full-data mean 1.5. A simple random sample of 5 is used. What is the expected sample mean gradient, and does this establish that the sample-fitted tree equals the full-data tree? (Answer: expected sample mean gradient is 1.5; no, because the fitted tree’s split and leaf choices are not generally linear functions of the sample gradients.)