Bagging as Bootstrap Aggregation

From Bootstrap Samples to a Bagged Predictor

A fitted model can change substantially when a few training observations change. This is especially likely for procedures with abrupt decisions, such as a small decision tree: a slightly different sample can produce a different split and noticeably different predictions. Bagging, short for bootstrap aggregation, addresses this instability by fitting the same learning procedure repeatedly to bootstrap samples and aggregating its predictions.

Suppose the training data consist of $n$ observed pairs, $D=\{(x_i,y_i):i=1,\ldots,n\}$. A standard bootstrap sample $D_b^*$ is formed by drawing $n$ records independently, with replacement, from the $n$ observed records. Thus, a record can appear more than once in a sample, while another record might not appear at all. Each replicate sample is used to fit the same specified learning procedure, producing a predictor $f_b^*(x)$, for $b=1,\ldots,B$.

Definition: The bagged predictor is the aggregate of the $B$ predictors fitted to bootstrap samples. For a numerical response, the usual aggregate is the arithmetic mean, $\bar f_{\text{bag}}(x)=B^{-1}\sum_{b=1}^B f_b^*(x)$. For a categorical response, common choices are majority voting among predicted classes or averaging predicted class probabilities and then choosing a class.

The bootstrap resamples contain repeated whole records, so each predictor is fitted using the observed relationship between predictors and response. The learning procedure is held fixed across replicates: it could, for example, be a specified tree-building algorithm with fixed settings. If data-dependent steps are part of that procedure, such as estimating a transformation from the training data, those steps belong inside each fit.

Aggregating Numerical Predictions

For regression, each fitted model supplies a numerical prediction at the same input $x$. Bagging averages those predictions; it does not average the bootstrap data sets or select the single replicate that appears best. The result can be a smoother prediction than any one replicate, particularly when small changes to the data cause the base fitting method to change sharply.

Worked Example: Bagging a Sample-Median Predictor

A hypothetical claims analyst has five observed claim amounts, in thousands of dollars: $1,2,3,8,10$. The specified base procedure predicts a constant equal to the sample median. Three bootstrap samples of size five are drawn: $D_1^*=(1,1,2,8,10)$, $D_2^*=(2,3,3,8,10)$, and $D_3^*=(1,2,8,8,10)$. Find the bagged prediction for a new record.

Method. Fit the constant-median predictor separately to each sample, then average the three predictions. Since each sample has five ordered values, its median is its third value.

For $D_1^*$, the ordered values are $1,1,2,8,10$, so the fitted prediction is $2$ thousand dollars. For $D_2^*$, they are $2,3,3,8,10$, so the prediction is $3$ thousand dollars. For $D_3^*$, they are $1,2,8,8,10$, so the prediction is $8$ thousand dollars.

The aggregate is $(2+3+8)/3=13/3=4.3333$ thousand dollars. The original data median is $3$ thousand dollars, which differs from this finite-bootstrap aggregate. That difference is possible: the bagged value averages the particular three fitted predictors, not every possible bootstrap fit. In dollars, the bagged prediction is $\$4,333.33$, rounded to the nearest cent.

Check and common mistake avoided. The three replicate predictions sum to $13$, and $13/3$ is approximately $4.3333$. Selecting the original-data median, or taking the median of the three replicate predictions, would use a different rule; the specified regression bagging rule is their arithmetic mean.

Aggregating Class Predictions

For classification, bagging can combine either class labels or class probabilities. With majority voting, each fitted classifier casts one vote for its predicted class. With probability aggregation, the fitted probabilities for a class are averaged, and a decision rule is then applied to the average. These procedures need not agree: a probability average uses how confident each fit is, while a vote uses only each fit’s selected class.

For two classes, a common probability rule selects class $C$ when its averaged probability is at least a specified threshold, often $0.50$. The threshold is part of the rule and must be read carefully. For multiple classes, a common rule selects the class with the largest averaged probability. A tie requires a stated tie-breaking rule.

Worked Example: Averaging Class Probabilities

Four bootstrap-fitted classifiers give probabilities for class $C$ of $0.20, 0.55, 0.65,$ and $0.40$ for the same new record. The decision rule selects $C$ only if the average probability is greater than $0.50$. Determine the bagged probability and predicted class.

Method. Average the four probabilities first, then compare that average with the stated strict threshold.

Their sum is $0.20+0.55+0.65+0.40=1.80$. The bagged probability for $C$ is $1.80/4=0.45$. Since $0.45$ is not greater than $0.50$, the bagged classifier does not select $C$.

For comparison, applying a $0.50$ threshold separately to each replicate gives two votes for $C$ and two against it: $0.20$ and $0.40$ are below the threshold, while $0.55$ and $0.65$ are above it. Majority voting therefore ties and cannot determine a class without a tie-breaking rule. The averaged-probability rule does give a result here: it selects the other class.

Check and common mistake avoided. The total is $1.80$, so the mean is $0.45$, not a value above $0.50$. Do not confuse “two of four individual probabilities exceed $0.50$” with “the average probability exceeds $0.50$.”

Why Averaging Can Stabilize Predictions

The basic variance calculation explains one way aggregation stabilizes predictions. Fix the observed training data $D$ and a prediction point $x$. Regard the fitted prediction from a random bootstrap sample as a random quantity with conditional variance $\sigma_*^2$. If the $B$ bootstrap samples are drawn independently, their predictions are conditionally independent and have the same conditional variance. The variance of their average is then:

\operatorname{Var}\!\left(\bar f_{\text{bag}}(x)\mid D\right) =\operatorname{Var}\!\left(\frac{1}{B}\sum_{b=1}^B f_b^*(x)\,\middle|\,D\right) =\frac{1}{B^2}\sum_{b=1}^B \operatorname{Var}\!\left(f_b^*(x)\mid D\right) =\frac{\sigma_*^2}{B}.

The second equality uses conditional independence: the covariance terms between distinct replicate predictions are zero. The conclusion concerns variability due to the random bootstrap draws, conditional on the observed data. It does not say that the prediction has zero variance across new training data, nor that its prediction error on future cases must decrease by a factor of $B$.

In practice, replicate predictions can be positively correlated, especially when the fitted procedure is stable and different bootstrap samples produce similar models. If each replicate prediction has variance $\sigma_*^2$ and each distinct pair has correlation $\rho$, the variance of their average is:

\operatorname{Var}\!\left(\bar f_{\text{bag}}(x)\mid D\right) =\sigma_*^2\left(\rho+\frac{1-\rho}{B}\right).

To see why, expand the variance of a sum into $B$ individual variances and $B(B-1)$ pairwise covariances. Each variance is $\sigma_*^2$, and each covariance is $\rho\sigma_*^2$. Dividing their sum by $B^2$ gives $\{\!B\sigma_*^2+B(B-1)\rho\sigma_*^2\}/B^2$, which simplifies to the displayed expression. When $\rho=0$, it reduces to $\sigma_*^2/B$; when $\rho=1$, averaging does not reduce this variability at all.

This calculation describes the effect of averaging correlated bootstrap fits, not a guarantee of better predictive performance. As $B$ grows, the finite-$B$ average approaches the average prediction over the bootstrap distribution conditional on $D$. It does not remove all uncertainty from having observed only $D$, and it does not automatically correct bias in the base learning procedure. Bagging is most useful when the base procedure is sufficiently unstable that resampling produces meaningfully different predictions.

Worked Example: Interpreting the Variance Reduction

At a fixed prediction point, suppose bootstrap-fitted predictions have conditional variance $\sigma_*^2=9$. Compare the conditional variance of a bagged prediction using $B=4$ independent fits with the variance when the pairwise correlation between fits is $\rho=0.50$.

Method. First apply the independent-fit result, then use the correlated-fit formula. The values are variances in squared prediction units.

With independent fits, the variance is $9/4=2.25$. With correlation $0.50$, it is $9\{0.50+(1-0.50)/4\}=9(0.50+0.125)=9(0.625)=5.625$.

Thus, the conditional variance is $2.25$ under the independence assumption and $5.625$ under the stated positive correlation. Both are below the single-fit variance $9$, but the correlated fits give a smaller reduction.

Check and common mistake avoided. Substituting $\rho=0.50$ and $B=4$ gives a multiplier of $0.625$, so $9(0.625)=5.625$. Dividing by $B$ alone would incorrectly assume independent replicate predictions.

Bagging Is Not Bootstrap Inference

Bootstrap methods can serve different purposes. In bagging, bootstrap samples are a way to generate fitted predictors; their outputs are combined to make a prediction. In bootstrap inference, replicate values of a statistic are used to study its sampling variability, bias, or interval estimates. An average of fitted predictions is not, by itself, a standard error or confidence interval.

Use	What is computed from each bootstrap sample?	How the replicates are used
Bagging	A fitted predictor evaluated at a specified input	Aggregate predictions, commonly by a mean or vote
Bootstrap inference	A replicate estimate of a target statistic	Use the replicate distribution to assess uncertainty or construct an interval

Worked Example: A Bagged Value and a Bootstrap Standard Error

Use the observed values $2,4,6,8$ and a constant-mean fitting procedure. Four bootstrap samples of size four produce sample means $4,4,6,$ and $6$. Find the bagged constant prediction and the bootstrap standard error of the mean estimate from these replicates.

Method. The bagged constant prediction is the average of the four replicate means. For the bootstrap standard error, calculate the sample standard deviation of those replicate means, using divisor $B-1$ for the replicate variance.

The bagged prediction is $(4+4+6+6)/4=20/4=5$. The mean of the replicate estimates is therefore $5$. Their deviations from $5$ are $-1,-1,1,1$; the squared deviations sum to $1+1+1+1=4$. The replicate sample variance is $4/(4-1)=4/3$, and the bootstrap standard error is $\sqrt{4/3}\approx1.1547$, rounded to four decimals.

The value $5$ is the aggregate point prediction; $1.1547$ estimates the variability of the statistic across these bootstrap replicates. They answer different questions. The standard error alone is not a confidence interval, and the replicate values are not four independent observations from the original population.

Check and common mistake avoided. The replicate estimates average to $5$, while their sample standard deviation is approximately $1.1547$. Reporting the standard error as the bagged prediction, or treating the mean of the replicates as an uncertainty interval, confuses prediction aggregation with bootstrap inference.

Practical Interpretation and Common Traps

Bagging changes the fitted prediction rule by averaging or voting across bootstrap-fitted versions of that rule. It is not equivalent to fitting one model to a larger data set, and it does not mean selecting the replicate with the smallest apparent error. Each bootstrap sample has the same size as the original sample under the standard procedure, but its repeated records give some observed cases greater weight and others no weight in that particular fit.

Do not confuse the two averaging operations. Regression bagging averages predictions at the same input. It does not average model coefficients and assume that the resulting coefficients define the same predictor.
Do not silently switch classification rules. Majority voting counts labels; probability aggregation averages class probabilities. State which rule is being used and how ties or thresholds are handled.
Do not claim that more replicates fix every problem. Increasing $B$ reduces Monte Carlo fluctuation in the aggregate, but cannot guarantee lower bias or better future prediction.
Do not use the replicate spread as though it were automatically a performance estimate. Prediction uncertainty from bootstrap fits and predictive error on new observations are distinct quantities.

Bagging is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus because it combines bootstrap resampling with prediction from fitted models. The key exam distinction is the role of the replicate outputs: bagging aggregates predictions, whereas bootstrap inference uses replicate statistics to describe uncertainty.

Check Your Understanding

For each problem, identify whether the task is prediction aggregation or uncertainty assessment before calculating.

Three bootstrap-fitted regression models predict $8,10,$ and $16$ at the same input. Using the usual bagging rule, calculate the aggregate prediction. (Answer: $34/3=11.3333$, rounded to four decimals.)
Four classifiers give probabilities $0.35,0.45,0.80,$ and $0.40$ for class $C$. The rule selects $C$ if the average probability is at least $0.50$. Is $C$ selected? (Answer: No; the average is $2.00/4=0.50$—actually, because the rule is at least $0.50$, $C$ is selected.)
Five fitted classifiers vote for class A, three for class B, and two for class C. Under majority voting, which class is predicted, and what proportion of the votes does it receive? (Answer: Class A, with $5/10=0.50$ of the votes.)
Four bootstrap estimates of a statistic are $1,3,5,$ and $7$. Calculate their mean and their sample standard deviation, using divisor $B-1$ for the variance. (Answer: Mean $=4$; sample standard deviation $=\sqrt{20/3}\approx2.5820$, rounded to four decimals.)
At one input, bootstrap-fitted predictions have conditional variance $4$, pairwise correlation $0.25$, and there are $B=5$ fits. Use the correlated-fit formula to find the conditional variance of their average. (Answer: $4\{0.25+0.75/5\}=4(0.40)=1.60$.)

What You'll Learn