Two Different Ways to Build an Ensemble
Boosting and bagging both combine multiple learners, often trees, but their fitting strategies differ. Boosting builds in sequence: each new learner is fitted after the current ensemble and is directed toward errors that remain. Bagging builds in parallel: each learner is fitted to a different bootstrap sample, and predictions are then aggregated. This distinction helps explain why boosting is often used to reduce systematic underfit, while bagging is especially useful for reducing the variability of unstable learners.
These are tendencies, not guarantees. The effect of either method depends on the base learner, the data, the loss function, and choices such as the number of learners and the learning rate. The comparison is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Boosting constructs an additive ensemble sequentially. At each stage, it fits a new learner to improve the current model under a chosen loss. In squared-error boosting, a common fitting target is the current residual; other boosting methods use loss-specific quantities such as weights or gradients.
A bootstrap sample has the same size as the original training sample and is drawn with replacement. Consequently, some original observations can appear multiple times in a bootstrap sample, while others are left out. Bagging fits its learners separately on these samples; it does not normally change one fit in response to the errors of another. In boosting, by contrast, the next fit depends on the current ensemble’s performance.
Fitting Strategy and the Bias–Variance Tradeoff
For squared-error prediction at a fixed input \(x\), an estimator \(\hat f(x)\) has expected squared prediction error that can be expressed as the square of its bias, plus its variance, plus irreducible error:
Here \(f(x)=E[Y\mid X=x]\), and the expectation and variance of \(\hat f(x)\) are over repeated training samples, with the method and tuning choices held fixed. The decomposition assumes squared-error loss and finite second moments. It separates three sources of error; it does not say that either ensemble method must improve every term.
Averaging can reduce variance when the learners do not make exactly the same errors. If the individual fits are unstable, small changes in the training data can produce materially different predictions. Averaging their predictions can smooth out some of this sampling variation. If the learners are very similar, their errors tend to move together and the benefit is smaller.
Boosting’s sequential fitting can reduce bias by adding learners that address what the current ensemble has not captured. For example, a sequence of simple learners may collectively represent a more complex pattern than any one learner. But later learners can also fit noise or unusual observations, increasing variance. A small learning rate, restrictions on the learners, and a suitable stopping point can help control this risk. None makes boosting inherently low-variance in every setting.
Why Bagging Can Reduce Variance
Suppose each of \(B\) fitted learners has prediction variance \(\sigma^2\) at a particular input, and suppose every pair of learners has the same correlation \(\rho\). These are simplifying assumptions: real bootstrap fits need not have exactly equal variances or equal correlations. Under these assumptions, the variance of their average is:
The result shows why the number of fits alone does not determine the variance reduction. As \(B\) grows, the term \((1-\rho)/B\) shrinks, but the shared component \(\rho\sigma^2\) remains. Increasing the number of bootstrap fits therefore reduces the finite-ensemble averaging component; it cannot average away errors that are perfectly shared across fits.
Worked Example: A Shrunk Boosting Update
Worked Example: A Shrunk Boosting Update
A squared-error boosting model currently predicts \((10,10,10)\) for three training cases with responses \((8,11,14)\). A newly fitted learner predicts the current residuals exactly, \((-2,1,4)\), and the learning rate is \(\nu=0.2\). Find the updated predictions and the new training sum of squared errors (SSE). This is a deterministic calculation; no sampling distribution is assumed.
Method. Add the new learner’s prediction multiplied by the learning rate to each current prediction:
For the first case, \(10+0.2(-2)=9.6\). For the second, \(10+0.2(1)=10.2\). For the third, \(10+0.2(4)=10.8\). The updated predictions are therefore \((9.6,10.2,10.8)\).
The new residuals are response minus updated prediction: \(8-9.6=-1.6\), \(11-10.2=0.8\), and \(14-10.8=3.2\). Thus:
Check. The initial SSE was \((-2)^2+1^2+4^2=21\), so the update reduced it to \(13.44\). The new residuals are also \(0.8\) times the old residuals, as expected when the fitted learner equals the residual and the update uses \(\nu=0.2\). A common mistake is to report the wrong sum of the three squared updated residuals; calculating each square before adding avoids that error.
Worked Example: Variance of Correlated Bagged Fits
Worked Example: Variance of Correlated Bagged Fits
At one fixed input, suppose \(B=40\) bootstrap-fitted regression trees each have prediction variance \(25\), and every pair has correlation \(0.2\). Under the equal-variance, equal-correlation assumptions, calculate the variance of their average and its limiting variance as the number of fits increases. The assumptions specify the relevant second moments; no probability distribution beyond them is needed.
Method. Use the variance formula for an average of equally variable, equally correlated fits:
Since \(0.8/40=0.02\), the quantity in parentheses is \(0.22\). Therefore the variance is \(25(0.22)=5.5\).
As \(B\) grows without bound, the term \((1-\rho)/B\) tends to zero. The limiting variance is \(25(0.2)=5\).
Check. The finite-ensemble variance \(5.5\) exceeds the limiting value \(5\), as it should under this formula. If the fits were independent, their average variance at \(B=40\) would instead be \(25/40=0.625\); the positive correlation explains why the actual value here is much larger. A common trap is to compute \(25(0.2+0.8/40)\) as \(5.4\); evaluating the parentheses first gives \(0.22\), and the product is \(5.5\).
Unstable Learners and What Averaging Can—and Cannot—Do
A learner is called unstable when modest changes in its training data can lead to substantial changes in the fitted model or its predictions. Deep, unpruned trees are a standard example: a small change in the data can alter a split near the top of the tree, which then changes many downstream predictions. Bagging is most useful when the base learner has this kind of sampling variability and the bootstrap fits are not perfectly correlated.
For a stable learner, different bootstrap samples tend to produce similar predictions. Averaging those predictions may then do little to reduce variance. Even with an unstable learner, bagging is not guaranteed to improve predictive performance: the average can retain bias, bootstrap fits may be strongly correlated, and the observed data may not represent the target population well.
Bagging does not ordinarily use a later learner to correct a particular earlier learner’s mistakes. Its main mechanism is aggregation across resampled fits. Boosting does use the current model’s errors or loss-related information to guide later fits. That sequential adaptation can represent patterns that a single base learner misses, but it also means that later fits are not independent replicas whose sole role is to average away variation.
Worked Example: Comparing Variation Across Repeated Fits
A hypothetical analyst evaluates two methods at one fixed input over three repeated training samples. The target conditional mean is \(12\). The bagged method gives estimates \((10,12,14)\), while the boosted method gives \((8,12,16)\). Calculate each method’s average estimate, empirical variance across these three runs (using divisor \(3\)), and empirical squared error of the average. Treat the listed runs as the complete set for this calculation; no parametric sampling distribution is assumed.
Method. First calculate the average estimates. For bagging, \((10+12+14)/3=36/3=12\). For boosting, \((8+12+16)/3=36/3=12\). Both averages equal the target \(12\), so each has squared error \((12-12)^2=0\).
For the bagged estimates, deviations from their average are \(-2,0,2\). The empirical variance is \(((-2)^2+0^2+2^2)/3=(4+0+4)/3=8/3\).
For the boosted estimates, deviations are \(-4,0,4\). The empirical variance is \(((-4)^2+0^2+4^2)/3=(16+0+16)/3=32/3\).
Check. The two methods have the same average and hence the same squared error of the average in this small illustration, but the boosted estimates vary more across the three runs. This is not evidence that boosting always has higher variance: it illustrates how repeated-sample variability is measured and why the actual comparison must be determined from the setting. A common mistake is to compare only one run from each method rather than comparing their distributions of estimates across repeated training samples.
Exam Traps and a Practical Comparison
When a question asks which method is more appropriate, identify the mechanism in the problem rather than relying on a slogan. A highly variable base learner with diverse bootstrap fits points toward bagging as a variance-reduction strategy. A model that is systematically missing structure may benefit from boosting’s sequential additions, provided its loss and tuning choices suit the task. Neither description alone establishes which will have lower test error.
| Feature | Bagging | Boosting |
|---|---|---|
| How fits are formed | Separate fits on bootstrap samples | Sequential fits, each guided by the current ensemble |
| Main aggregation idea | Average predictions or combine votes | Add learners to an evolving model |
| Typical emphasis | Reduce variance through averaging | Improve fit by addressing remaining loss |
| Role of learner instability | Often creates an opportunity for averaging to help | Can make the evolving fit more sensitive to data and tuning |
| Important limitation | Correlated fits limit variance reduction | Excessive fitting can increase variance or fit noise |
A frequent exam error is to treat “bagging reduces variance” and “boosting reduces bias” as universal laws. They are useful descriptions of common behavior, not unconditional theorems. Check what is being averaged or updated, whether the learners’ errors are correlated, what loss is used, and whether the question asks about training fit or generalization. For example, a lower training error after adding boosted learners does not by itself establish lower test error.
Check Your Understanding
Work each problem before checking its answer. The numerical variance problems use the stated equal-correlation assumptions.
- A squared-error boosting model predicts \((5,5)\) for responses \((3,9)\). Its next learner predicts residuals \((-2,4)\), and the learning rate is \(0.25\). Find the updated predictions and training SSE. (Answer: predictions \((4.5,6)\); SSE \(11.25\))
- Ten bagged fits each have variance \(16\) and common pairwise correlation \(0.1\). Find the variance of their average and the limiting variance as the number of fits grows. (Answer: \(3.04\); limiting variance \(1.6\))
- Twenty-five bagged fits each have variance \(9\) and common pairwise correlation \(0.4\). Calculate the variance of their average and its limiting value. (Answer: \(3.816\); limiting variance \(3.6\))
- Across four hypothetical repeated training runs, a bagged method gives predictions \((2,4,8,10)\) at one input, whose target conditional mean is \(5\). Using divisor \(4\), find the average prediction, empirical variance, and squared error of the average. (Answer: average \(6\); empirical variance \(10\); squared error \(1\))
- A base learner changes substantially across bootstrap samples, but the fitted predictions from those samples are highly correlated. Explain what this implies for bagging’s variance reduction and whether a large number of fits removes the shared variation. (Answer: instability creates potential for averaging, but high correlation limits the reduction; with fixed positive common correlation, the variance approaches \(\rho\sigma^2\), not zero.)