Conditional and Unconditional Risk of Randomized Ensembles

Why the Conditioning Layer Matters

A randomized learning algorithm can produce different fitted predictors from the same training sample. Its reported risk may therefore refer to more than one target: the risk of a particular fit, the average risk over the algorithm’s random choices, or the average over both new training samples and random choices. These quantities answer different questions. Keeping their conditioning explicit is necessary when assessing reproducibility, comparing algorithms, or interpreting a risk estimate.

Let $D$ denote the training data, and let $U$ denote the algorithmic randomization, such as random feature subsampling or randomized tree construction. Write $\hat f_{D,U}(x)$ for the resulting predictor. Let $Z=(X,Y)$ be a new test observation, independent of the training process, with the population distribution of interest. For a loss function $L(y,a)$, the conditional risk of a realized fit is its expected loss on a fresh test observation.

Definition: The conditional risk of the fit produced from $D$ and $U$ is $R(D,U)=E_Z[L(Y,\hat f_{D,U}(X))\mid D,U]$. The randomization-averaged conditional risk is $\bar R(D)=E_U[R(D,U)\mid D]$. The unconditional risk is $R_{\mathrm{uncond}}=E_{D,U}[R(D,U)]$. The first quantity concerns a particular realized fit; the second averages randomization while holding the data fixed; the third averages over both sources of training-time variation.

These definitions require the corresponding expectations to exist. In particular, assume the loss is measurable and integrable under the joint distribution being considered. The randomization can be represented by a conditional distribution for $U$ given $D$; it need not be independent of $D$. The test observation is drawn independently of the training procedure. The tower property of conditional expectation then gives the first basic relationship.

Theorem: If $L(Y,\hat f_{D,U}(X))$ is integrable, then $R_{\mathrm{uncond}}=E_D[\bar R(D)]$. If $R(D,U)$ is square-integrable, then $\operatorname{Var}_{D,U}(R(D,U))=\operatorname{Var}_D(\bar R(D))+E_D[\operatorname{Var}_U(R(D,U)\mid D)]$.

Proof. By the definition of conditional risk and iterated expectation, $E_{D,U}[R(D,U)]=E_{D,U}[E_Z\{L(Y,\hat f_{D,U}(X))\mid D,U\}]$, which is the unconditional expected test loss. Applying iterated expectation once more, first over $U$ given $D$, gives $E_D[E_U\{R(D,U)\mid D\}]=E_D[\bar R(D)]$.

For the variance identity, write $R-\!E[R]=(R-\bar R(D))+(\bar R(D)-E[R])$. On squaring and taking expectation, the cross term is zero because $E[R-\bar R(D)\mid D]=0$. The first squared term has expectation $E_D[\operatorname{Var}_U(R\mid D)]$, and the second has expectation $\operatorname{Var}_D(\bar R(D))$. Adding them proves the identity. $\square$

Expected Risk Is Not Always Risk of an Averaged Predictor

A randomized fit can be assessed by averaging the risk of its realized predictors, or it can be turned into a deterministic predictor by averaging predictions over randomization and then assessed. Those are not generally the same operation: loss is applied before averaging in the first target and after averaging in the second. Under squared error, averaging predictions can reduce risk, but this fact does not make the two targets interchangeable.

Worked Example: Loss Before or After Averaging Randomization

Hold the training sample fixed. Suppose the test response is always $Y=0$, and a random seed produces prediction $0$ or $2$, each with probability $1/2$. Under squared-error loss, the two realized risks are $0^2=0$ and $(0-2)^2=4$. Thus the expected risk of a randomized realized fit is $(0+4)/2=2$.

The randomization-averaged prediction is $(0+2)/2=1$, whose squared-error risk is $(0-1)^2=1$. Therefore, expected risk of the randomized fit is $2$, whereas risk of its averaged prediction is $1$. The latter is a different predictor and a different risk target.

The reduction here follows directly from averaging predictions under squared loss. It does not imply that every loss or every aggregation rule has the same relationship.

A Layered Decomposition Under Squared Error

A more detailed separation is available when the target is regression prediction under squared-error loss. Assume $Y=m(X)+\varepsilon$, where $m(x)=E[Y\mid X=x]$ and $E[\varepsilon\mid X]=0$. Let $\sigma^2(x)=\operatorname{Var}(Y\mid X=x)<\infty$. Assume the test observation is independent of $(D,U)$, and the predictions have finite second moments over $X,D,U$. Define the prediction averaged over randomization for a fixed data set by $\bar f_D(x)=E_U[\hat f_{D,U}(x)\mid D]$.

Theorem: Under these assumptions, the unconditional squared-error risk satisfies $$ E_{D,U,Z}\big[(Y-\hat f_{D,U}(X))^2\big] = E_X[\sigma^2(X)] + E_X\left[ \big(E_D[\bar f_D(X)]-m(X)\big)^2 +\operatorname{Var}_D(\bar f_D(X)) +E_D\{\operatorname{Var}_U(\hat f_{D,U}(X)\mid D)\} \right]. $$ The terms after the irreducible noise are, respectively, squared bias of the overall mean prediction, variation across training data sets in their randomization-averaged predictions, and variation due to algorithmic randomization.

Proof. Fix $x$, and abbreviate $\hat f_{D,U}(x)$ by $F$ and $m(x)$ by $m$. Conditional on $X=x,D,U$, the test response has mean $m$ and variance $\sigma^2(x)$, so $E[(Y-F)^2\mid X=x,D,U]=\sigma^2(x)+(F-m)^2$. It remains to decompose $E_{D,U}[(F-m)^2\mid X=x]$.

Let $\mu(x)=E_{D,U}[F\mid X=x]=E_D[\bar f_D(x)]$. The identity $F-m=(F-\mu)+(\mu-m)$ yields $E[(F-m)^2]=E[(F-\mu)^2]+(\mu-m)^2$, because $E[F-\mu]=0$, so the cross term vanishes. Next, condition on $D$: write $F-\mu=(F-\bar f_D)+(\bar f_D-\mu)$. The conditional expectation of $F-\bar f_D$ given $D$ is zero, so the cross term again vanishes. Consequently, $E[(F-\mu)^2]=E_D[\operatorname{Var}_U(F\mid D)]+\operatorname{Var}_D(\bar f_D)$. Combining these identities with the test-response variance formula and integrating over $X$ proves the stated decomposition. $\square$

The decomposition separates prediction error into three distinct contributions beyond test-response noise. Randomization variance measures sensitivity to the algorithm’s seed at fixed data. Data-set variation measures how the randomization-averaged prediction changes when training data are resampled. Neither is the variance of the realized conditional risk $R(D,U)$; that is a separate random quantity, covered by the earlier variance identity.

Worked Example: Separating Data Variation and Randomization

Consider one fixed covariate value $x$, with regression mean $m(x)=10$ and test-response variance $\sigma^2(x)=4$. There are two equally likely training data sets, $D=A$ and $D=B$. Given $A$, randomization yields predictions $7$ and $11$ equally often. Given $B$, it yields predictions $10$ and $12$ equally often.

For $A$, the randomization-averaged prediction is $\bar f_A=(7+11)/2=9$, and its randomization variance is $\{(7-9)^2+(11-9)^2\}/2=4$. For $B$, $\bar f_B=(10+12)/2=11$, and its randomization variance is $\{(10-11)^2+(12-11)^2\}/2=1$. The overall mean prediction is $(9+11)/2=10$, so squared bias is $(10-10)^2=0$.

The data-set variance of the randomization-averaged prediction is $\{(9-10)^2+(11-10)^2\}/2=1$. The mean randomization variance is $(4+1)/2=2.5$. The decomposition therefore gives total risk $4+0+1+2.5=7.5$.

Check directly: the four equally likely predictions are $7,11,10,12$. Their squared prediction errors relative to $m(x)=10$ are $9,1,0,4$, averaging to $3.5$. Adding response variance $4$ gives $7.5$, as above. The conditional risks for the realized fits are $13,5,4,8$, since each equals $4+(\hat f-10)^2$; their average is also $7.5$.

Finite Randomized Ensembles

An ensemble that averages $B$ randomized fits has its own algorithmic randomization: the collection of seeds used to generate its members. The squared-error decomposition applies to the ensemble prediction as it does to any predictor. When the member predictions are conditionally independent and identically distributed given $D$ at a fixed $x$, averaging reduces their conditional randomization variance by a factor of $B$. Independence is an assumption here, not a consequence of calling the procedure an ensemble.

Worked Example: Risk of an Average of Four Random Fits

Fix a training data set and covariate value. Let the regression mean be $m(x)=0$, and let test-response variance be $1$. Each randomized member predicts $1+\xi_b$, where $\xi_b$ is independently $2$ or $-2$, each with probability $1/2$, for $b=1,\ldots,4$. Each member has mean prediction $1$ and randomization variance $E[\xi_b^2]=4$.

The average prediction is $1+\bar\xi$, where $\bar\xi=(\xi_1+\xi_2+\xi_3+\xi_4)/4$. Its conditional mean is $1$, so its squared bias is $1^2=1$. Independence and zero means give $\operatorname{Var}(\bar\xi)=\{\sum_{b=1}^4\operatorname{Var}(\xi_b)\}/16=16/16=1$. Thus the ensemble risk is test-response variance $1$, plus squared bias $1$, plus randomization variance $1$, for a total of $3$. A single randomized member instead has risk $1+1+4=6$.

The factor-of-four variance reduction follows because the four random offsets are independent. If their seeds or fitted outputs are dependent, covariance terms enter the variance of their average, and the calculation must change.

Common Mistakes and Practical Interpretation

A frequent mistake is to treat conditional risk as if it must be smaller than unconditional risk. Conditioning does not guarantee this: $R(D,U)$ can be either above or below its average for a particular realized fit. The unconditional risk is an average of conditional risks, not a bound on each one. Likewise, $\bar R(D)$ averages randomization at fixed $D$; it need not equal the risk of the prediction averaged over randomization, as the squared-loss example showed.

The conditioning level should match the question. To describe the realized model that will actually be deployed, report or estimate its conditional risk. To assess how much the algorithm’s seed affects performance for a fixed training set, examine variation over $U$ conditional on $D$. To describe expected performance under repeated data collection and repeated algorithm runs, use unconditional risk. When reporting an estimate, state which of these targets it is intended to estimate.

Key takeaway: For randomized ensembles, distinguish test-sample averaging, algorithmic randomization, and training-data variation. Under squared error, these layers can be separated exactly, but only after specifying the predictor being evaluated and the assumptions governing its randomness.

Check Your Understanding

Use the conditioning layers and squared-error identities to answer these questions.

In words, distinguish $R(D,U)$, $\bar R(D)$, and $R_{\mathrm{uncond}}$.
Why does the law of total variance for $R(D,U)$ describe variability in risk values rather than the squared-error prediction decomposition?
For a fixed data set, a randomized predictor is $0$ or $4$ with equal probability and the test response is always $1$. Calculate the expected risk of a randomized fit and the risk of the randomization-averaged prediction.
In the layered squared-error result, which term measures variation across training data sets after averaging over algorithmic randomness?
What additional assumption justifies dividing a single member’s randomization variance by $B$ when calculating the variance of an average of $B$ members?