Why Averaging Tree Predictions Can Help
A regression tree can change its prediction when its training data or bootstrap sample changes. Bagging uses this variability by fitting multiple trees and averaging their predictions. The key question is precise: if each tree has the same expected prediction and variance, what happens to the expected value and variance of their average?
The answer depends on how the predictions vary together. Averaging preserves their common expected value. It reduces variance by a factor of the number of trees when the predictions are independent; more generally, the same formula applies when they have zero pairwise covariances. Independence is sufficient, but it is not necessary.
Suppose the tree predictions have a common expected value \(\mu_T\) and common variance \(\sigma_T^2\). These conditions say that each tree is centered at the same value and is equally variable. They do not, by themselves, say how the predictions from different trees are related.
Deriving the Mean and Variance
The mean result follows from linearity of expectation. The variance result follows by expanding the variance of a sum; this expansion also makes the required condition visible.
By linearity, \(E[\bar{T}_M(x)]=\frac{1}{M}\sum_{b=1}^M E[T_b(x)]=\frac{1}{M}(M\mu_T)=\mu_T\). No independence assumption is needed for this step.
The variance of a sum equals the sum of the variances plus twice the sum of the covariances over distinct pairs. Therefore, \(\operatorname{Var}(\bar{T}_M(x))=\frac{1}{M^2}\left(\sum_{b=1}^M\operatorname{Var}(T_b(x))+2\sum_{b<c}\operatorname{Cov}(T_b(x),T_c(x))\right)\).
The variance sum is \(M\sigma_T^2\). If every covariance in the second sum is zero, then \(\operatorname{Var}(\bar{T}_M(x))=\frac{M\sigma_T^2}{M^2}=\frac{\sigma_T^2}{M}\).
The variance expansion is important even when the zero-covariance condition does not hold. It shows why equal individual variances alone are not enough to conclude that the average has variance \(\sigma_T^2/M\): the covariance terms must also be accounted for. Independence guarantees zero covariance when the variances exist, but zero covariance does not generally imply independence.
Worked Example: Averaging Independent Claim-Cost Predictions
At a fixed policy profile, each fitted regression tree predicts a hypothetical annual claim cost. Across repeated training and bootstrap samples, suppose each tree prediction has mean \(\$1{,}200\) and variance \(40{,}000\) dollars squared. The 25 tree predictions are mutually independent. Find the expected bagged prediction, its variance, and its standard deviation.
Method. Use the common-mean result and the variance formula for independent, equally variable predictions.
Each tree has \(E[T_b(x)]=1{,}200\), so \(E[\bar{T}_{25}(x)]=1{,}200\) dollars.
Independence implies zero covariance for every distinct pair. Thus \(\operatorname{Var}(\bar{T}_{25}(x))=40{,}000/25=1{,}600\) dollars squared.
The standard deviation is the square root of the variance: \(\sqrt{1{,}600}=40\) dollars.
Check. The expected prediction remains \(\$1{,}200\), while its standard deviation is one-fifth of a single tree's standard deviation, since the single-tree standard deviation is \(\sqrt{40{,}000}=\$200\) and \(\sqrt{25}=5\). The variance is divided by 25; the standard deviation is divided by 5.
Common mistake avoided. Dividing the standard deviation by 25 would give \(\$8\), which is incorrect. The variance is divided by 25, so the standard deviation is divided by \(\sqrt{25}\).
Zero Covariance Does Not Require Independence
The formula \(\sigma_T^2/M\) is valid whenever all pairwise covariances are zero and the individual variances are equal. Some dependent random variables have zero covariance. The following construction verifies this distinction explicitly.
Worked Example: Dependent but Uncorrelated Tree Deviations
At a fixed predictor value, define two tree predictions by \(T_1=100+X\) and \(T_2=100+Y\), where \(X\) is uniform on \(\{-1,0,1\}\) and \(Y=\sqrt{3}(X^2-2/3)\). Find the mean and variance of each prediction, the covariance between them, and the variance of their average.
Method. Compute the moments directly, then use the two-variable variance expansion. All probabilities for the three values of \(X\) are \(1/3\).
First, \(E[X]=(-1+0+1)/3=0\), and \(E[X^2]=(1+0+1)/3=2/3\). Hence \(\operatorname{Var}(X)=2/3\), so \(E[T_1]=100\) and \(\operatorname{Var}(T_1)=2/3\).
For \(Y\), its value is \(-2/\sqrt{3}\) when \(X=0\), and \(1/\sqrt{3}\) when \(X=-1\) or \(X=1\). Therefore, \(E[Y]=\frac{1}{3}(-2/\sqrt{3})+\frac{2}{3}(1/\sqrt{3})=0\). Also, \(E[Y^2]=\frac{1}{3}(4/3)+\frac{2}{3}(1/3)=2/3\), so \(\operatorname{Var}(Y)=2/3\). Consequently, \(E[T_2]=100\) and \(\operatorname{Var}(T_2)=2/3\).
The covariance is \(\operatorname{Cov}(T_1,T_2)=\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]\). Here \(E[X]E[Y]=0\), and \(E[XY]=\frac{1}{3}(-1)(1/\sqrt{3})+\frac{1}{3}(0)(-2/\sqrt{3})+\frac{1}{3}(1)(1/\sqrt{3})=0\). Thus the covariance is zero.
The two predictions are dependent: \(Y\) is determined by \(X\), so knowing \(X\) determines \(Y\). Nevertheless, their covariance is zero. For their average, \(\operatorname{Var}((T_1+T_2)/2)=\frac{1}{4}(2/3+2/3+2(0))=1/3\). This equals the common variance \(2/3\) divided by \(M=2\).
Check. Both predictions have mean 100 and variance \(2/3\); their covariance is zero; and the average has mean 100 and variance \(1/3\). The calculation confirms that independence is sufficient, not necessary.
Common mistake avoided. It would be incorrect to say that the variance formula applies only under independence. The relevant condition in the derivation is zero covariance for every distinct pair.
When the Covariance Condition Fails
A shared random fluctuation can make tree predictions move together. In that case, dividing the individual variance by the number of trees understates the variance of the average. This is not a different mean calculation: linearity still preserves the common mean, while the covariance terms change the variance.
Worked Example: A Shared Fluctuation Between Two Predictions
Suppose two tree predictions for a hypothetical annual benefit amount are \(T_1=500+Z\) and \(T_2=500+Z\), where \(E[Z]=0\) and \(\operatorname{Var}(Z)=9\) dollars squared. Find the mean and variance of their average, and compare the correct variance with the result from incorrectly assuming independence.
Method. Use the covariance expansion, because the predictions share the same random quantity.
Both predictions have mean 500 and variance 9. Also, since both equal \(500+Z\), their covariance is \(\operatorname{Var}(Z)=9\).
Their average is \(\bar{T}_2=(T_1+T_2)/2=((500+Z)+(500+Z))/2=500+Z\). Therefore, \(E[\bar{T}_2]=500\) and \(\operatorname{Var}(\bar{T}_2)=9\).
The covariance expansion gives the same result: \(\operatorname{Var}(\bar{T}_2)=\frac{1}{4}(9+9+2(9))=36/4=9\).
If one incorrectly treated the predictions as independent, one would calculate \((9+9)/4=4.5\). That omits the covariance contribution \(2(9)/4=4.5\), so it is not the variance of this average.
Check. Averaging identical predictions leaves the random fluctuation unchanged, so the variance remains 9. The expected value is still 500.
Common mistake avoided. The equal-variance condition does not justify the formula \(\sigma_T^2/M\) on its own. Verify independence or, more generally, zero pairwise covariances before applying it.
What Variance Reduction Says About Prediction Error
At a fixed predictor value \(x\), let \(f(x)\) denote a nonrandom target, such as the conditional mean response the regression procedure aims to estimate. For any prediction \(U\) with finite second moment, \(E[(U-f(x))^2]=\operatorname{Var}(U)+(E[U]-f(x))^2\). This identity follows by writing \(U-f(x)=(U-E[U])+(E[U]-f(x))\), squaring, and taking expectations. The cross-product has expectation zero because \(E[U-E[U]]=0\).
If each tree has the same mean, averaging leaves that mean unchanged. It therefore leaves the squared bias relative to \(f(x)\) unchanged as well. When the tree predictions are pairwise uncorrelated and equally variable, the variance component of mean squared error falls from \(\sigma_T^2\) for one tree to \(\sigma_T^2/M\) for the average. This conclusion is about the specified prediction randomness and assumptions; it does not assert that averaging removes bias.
The distinction matters in practice. A lower variance does not by itself establish that a model has lower error under every data-generating process or evaluation criterion. The result is a precise statement about averaging predictions with a common mean and equal variance under the stated covariance condition. It is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus because it requires candidates to connect a model-building procedure to expectation, variance, and covariance.
Common Mistakes and a Reliable Checklist
- Confusing variance and standard deviation. If variance is divided by \(M\), standard deviation is divided by \(\sqrt{M}\), not by \(M\).
- Assuming equal variances are enough. The covariance terms also matter. Equal variances do not imply zero covariance.
- Requiring independence unnecessarily. Independence is a sufficient condition for zero covariance when the variances exist, but pairwise uncorrelated predictions also satisfy the variance formula.
- Claiming the average must equal each realized prediction. The preserved quantity is the expected value. A particular realized average can differ from the common expected prediction.
- Claiming averaging removes bias. If all component predictions have the same mean, their average has that mean too. Any bias of that common mean relative to the target remains.
For a calculation, state the common mean and variance, identify the number of predictions, and check whether the distinct pairs are independent or at least uncorrelated. Then use linearity for the mean and the covariance expansion for the variance. This avoids silently assuming a condition that the problem has not supplied.
Check Your Understanding
For each problem, work at the stated fixed predictor value and distinguish the mean calculation from the variance calculation.
- Each of 16 mutually independent tree predictions has mean 30 and variance 144. Find the mean and variance of their average. (Answer: mean \(30\); variance \(9\).)
- Six equally variable predictions each have variance 81 and every distinct pair has covariance zero. They need not be independent. Find the variance of their average. (Answer: \(13.5\).)
- Four tree predictions each have mean 70 and variance 25, and all four predictions are identical random variables. Find the mean and variance of their average. (Answer: mean \(70\); variance \(25\).)
- Two predictions have common mean 10, each has variance 16, and their covariance is 4. Find the mean and variance of their average. (Answer: mean \(10\); variance \(10\).)
- Each of 25 pairwise uncorrelated predictions has mean 8 and variance 100. A fixed target value is 6. Find the average prediction's variance, squared bias, and mean squared error. (Answer: variance \(4\); squared bias \(4\); mean squared error \(8\).)