Estimands for Validation Under Training Randomness

Why the Risk Target Must Be Specified

A learning algorithm does not usually return the same predictor for every training sample. Even when the data-generating distribution stays fixed, a new sample can produce a different fitted predictor and therefore a different prediction risk. A statement such as “the risk is 0.2” is incomplete unless it is clear whether it describes the fitted predictor from one realized sample or an average over possible training samples.

This distinction matters when assessing a procedure intended for repeated use. The risk of the predictor currently in hand is a conditional target: the training sample has been observed and held fixed. The unconditional target averages over the samples the procedure might have received. Neither target is automatically the risk of a separately specified, fixed predictor.

Setup and Definitions

Let $Z=(X,Y)$ be a new observation with distribution $P$ on a measurable space $\mathcal Z=\mathcal X\times\mathcal Y$. The training sample is $S=(Z_1,\ldots,Z_n)$, where $Z_1,\ldots,Z_n$ are independent and identically distributed according to $P$. Let $\mathcal F$ be a class of predictors $f:\mathcal X\to\mathcal A$, and let $\ell:\mathcal A\times\mathcal Y\to[0,\infty]$ be a measurable loss function. A deterministic learning algorithm is a measurable map $A$ from training samples to predictors; write its output as $f_S=A(S)$. Let $Z_0=(X_0,Y_0)$ be a fresh observation, independent of $S$, with distribution $P$.

For each fixed predictor $f$, its population prediction risk is the expected loss on a new observation. If this expectation is finite, it is a fixed, nonrandom number.

Definition: The population risk of a fixed predictor $f$ is $$ R(f)=E_{Z_0\sim P}\!\left[\ell\{f(X_0),Y_0\}\right] =\int_{\mathcal Z}\ell\{f(x),y\}\,dP(x,y). $$ The conditional prediction risk of the fitted predictor, given the training sample, is $$ R_S=E\!\left[\ell\{f_S(X_0),Y_0\}\mid S\right]. $$ Because $Z_0$ is independent of $S$, this is the risk of the predictor $f_S$ held fixed at the realized training sample. The unconditional prediction risk of the learning algorithm is $$ R_{\mathrm{alg}}=E_S[R_S] =E_{S,Z_0}\!\left[\ell\{f_S(X_0),Y_0\}\right]. $$

The conditional risk $R_S$ is generally a random variable: its value depends on the training sample. For an observed sample $s$, its realized value is $R_s=R(f_s)$. By contrast, $R_{\mathrm{alg}}$ averages over the distribution of training samples and is a single number, possibly infinite when the loss is not integrable. Nonnegative loss makes the expectations well-defined as extended nonnegative values; when using variance identities or finite-valued comparisons, additional moment assumptions will be stated.

A randomized learning algorithm can be handled by including its internal random seed $U$, independent of the training and test data. Write the output as $f_{S,U}$. Conditional on both $S$ and $U$, the risk is the risk of the realized fitted predictor. Conditional on $S$ alone, the risk averages over the algorithm’s internal randomness as well as the new observation. The conditioning convention should therefore be stated whenever algorithmic randomization is present. The examples below use deterministic algorithms.

Worked Examples

Worked Example: A Normal-Mean Predictor Under Squared Error

Suppose $Y_1,\ldots,Y_n,Y_0$ are independent with distribution $N(\mu,\sigma^2)$, where $\sigma^2>0$, and use the constant predictor $f_S(x)=\bar Y$, with $\bar Y=n^{-1}\sum_{i=1}^nY_i$. The loss is squared error, $\ell(a,y)=(y-a)^2$. Find the conditional and unconditional prediction risks.

Given $S$, the value $\bar Y$ is fixed and $Y_0$ is independent with mean $\mu$ and variance $\sigma^2$. Write $Y_0-\bar Y=(Y_0-\mu)+(\mu-\bar Y)$. The cross term has conditional expectation zero, so

R_S=E[(Y_0-\bar Y)^2\mid S] =\sigma^2+(\bar Y-\mu)^2.

Thus the conditional risk varies with the realized sample. Since $\bar Y\sim N(\mu,\sigma^2/n)$, its squared deviation has expectation $E[(\bar Y-\mu)^2]=\sigma^2/n$. Averaging the conditional risk gives

R_{\mathrm{alg}}=E[R_S] =\sigma^2+\frac{\sigma^2}{n} =\sigma^2\left(1+\frac1n\right).

For comparison, if a fixed constant predictor $f_c(x)=c$ is specified in advance, its risk is $R(f_c)=\sigma^2+(c-\mu)^2$. Once the sample is observed, the fitted predictor has this fixed-predictor risk with $c=\bar Y$; before observing the sample, its risk is random.

Worked Example: A Sample-Selected Binary Classifier

Let $Y\sim\operatorname{Bernoulli}(p)$, let $X$ carry no predictive information, and use zero-one loss. With a training sample of size one, define the algorithm to predict the observed label: $f_{Y_1}(x)=Y_1$. Take $p=0.8$. Compute the conditional and unconditional risks, and compare them with the risk of the fixed classifier that always predicts 1.

If $Y_1=1$, the fitted classifier is correct on a new observation with probability $0.8$, so its conditional error risk is $0.2$. If $Y_1=0$, its conditional error risk is $0.8$. The first case occurs with probability $0.8$, and the second with probability $0.2$. Therefore

R_{\mathrm{alg}} =P(Y_1=1)(0.2)+P(Y_1=0)(0.8) =(0.8)(0.2)+(0.2)(0.8)=0.32.

The fixed classifier that always predicts 1 has risk $P(Y_0=0)=0.2$. It is not the algorithm in this example: the algorithm sometimes returns the classifier that predicts 0. Its larger unconditional risk reflects that training-sample randomness, rather than test-label randomness alone, affects the fitted predictor.

Worked Example: Risk of a Bernoulli Sample-Mean Predictor

Again let $Y\sim\operatorname{Bernoulli}(p)$, but now use squared-error loss and fit the constant predictor $f_S(x)=\bar Y$. For $n=2$ and $p=0.25$, calculate the conditional risks for each possible sample mean and then the unconditional risk.

For a fixed prediction $c$, $E[(Y_0-c)^2]=p(1-p)+(c-p)^2$. Here $p(1-p)=0.25(0.75)=0.1875$. The sample mean is 0, $0.5$, or 1. The corresponding conditional risks are

R_S= \begin{cases} 0.1875+(0-0.25)^2=0.25, & \bar Y=0,\\ 0.1875+(0.5-0.25)^2=0.25, & \bar Y=0.5,\\ 0.1875+(1-0.25)^2=0.75, & \bar Y=1. \end{cases}

The probabilities of these sample means are $P(\bar Y=0)=0.75^2=0.5625$, $P(\bar Y=0.5)=2(0.25)(0.75)=0.375$, and $P(\bar Y=1)=0.25^2=0.0625$. Hence

R_{\mathrm{alg}} =(0.5625)(0.25)+(0.375)(0.25)+(0.0625)(0.75) =0.140625+0.09375+0.046875 =0.28125.

The predictor’s conditional risk is not constant across samples, even though the unconditional risk is one number. For a fixed predictor that always predicts 0, the risk is $E[Y_0^2]=p=0.25$; that fixed-predictor target differs from the sample-averaged risk of the fitted algorithm.

Two Identities for Risk Targets

The first identity formalizes why unconditional risk is the average conditional risk. It requires measurability so that the loss evaluated at the fitted predictor and fresh observation is a random variable. Nonnegativity is enough for the identity in extended expectations; integrability makes the quantities finite.

Theorem: Let $L=\ell\{f_S(X_0),Y_0\}$ be measurable and nonnegative, or integrable. Then $$ E[L]=E\!\left[E[L\mid S]\right]=E[R_S]. $$ If $L$ is integrable, both sides are finite and the conditional expectation is defined up to almost-sure equality.

Proof. By definition, $R_S=E[L\mid S]$. The law of total expectation gives $E[E[L\mid S]]=E[L]$ when $L$ is integrable. For nonnegative $L$, the same equality follows from the conditional-expectation identity for nonnegative random variables, with both expectations allowed to be infinite. This proves the stated identity. $\square$

This identity does not say that every realized conditional risk equals the unconditional risk. It says that averaging the random conditional risk over the training-sample distribution recovers the expected loss of the full training-and-test experiment.

Theorem: Suppose $L=\ell\{f_S(X_0),Y_0\}$ has finite second moment. Then $$ \operatorname{Var}(L) =E[\operatorname{Var}(L\mid S)]+\operatorname{Var}(R_S). $$

Proof. Set $M=E[L\mid S]=R_S$. Decompose the centered loss as

$$ L-E[L]=(L-M)+(M-E[L]). $$

Squaring and taking expectations gives the sum of the two squared terms and a cross term. The cross term is zero, since $M-E[L]$ is a function of $S$ and

E[(L-M)(M-E[L])] =E\!\left[(M-E[L])E[L-M\mid S]\right]=0.

The first squared term has expectation $E[\operatorname{Var}(L\mid S)]$, by the definition of conditional variance. The second has expectation $\operatorname{Var}(M)=\operatorname{Var}(R_S)$, since $E[M]=E[L]$. Adding them proves the result. $\square$

The decomposition separates variation in a new test loss even when the fitted predictor is held fixed from variation in conditional risk across training samples. The second term is zero exactly when the conditional risk is almost surely constant; the first need not be zero in that case.

Why the Distinction Matters

A risk statement should identify its conditioning information and its averaging distribution. If a particular training sample has already produced a model, its conditional risk answers how that fitted model performs on new observations. If the learning procedure will be repeated with new training samples, unconditional risk answers how the procedure performs on average over those repetitions. These are different scientific questions, not competing definitions of one number.

A fixed predictor’s risk is a useful comparator, but it must be treated as fixed when the expectation is taken. A realized fitted predictor can be evaluated using the same formula after conditioning on its training sample. Before conditioning, however, the predictor and its risk both vary with that sample. In particular, the risk of an average predictor is generally not the average risk of random predictors; the loss and the fitted rule determine how those quantities relate.

Common Mistake

Wrong reasoning: “The fitted predictor has risk $R(f_S)$, so that is a fixed number and equals the algorithm’s risk.” The error is treating the random training sample as if it had already been fixed while simultaneously claiming an unconditional target.

Correction: For each realized sample $s$, $R(f_s)$ is a number. Before the sample is observed, $R(f_S)$ is a random variable. Its expectation $E_S[R(f_S)]$ is the algorithm’s unconditional risk. If the fitted predictor is randomized, also specify whether the conditional risk conditions on the random seed or averages over it.

Key takeaway: State what is random and what is conditioned on. The conditional risk describes a fitted predictor at a realized training sample; the unconditional risk averages that conditional risk over training randomness; a fixed predictor has its own population risk.

Check Your Understanding

For each question, identify the relevant randomness and explain which risk target is being described.

For a deterministic learning algorithm, is $R_S$ generally a number fixed before the training sample is drawn, or a random variable? Why?
Suppose a predictor $f_c$ is fixed in advance. Write its population risk in terms of the loss and a fresh observation.
In the normal-mean example, why does the conditional risk include $(\bar Y-\mu)^2$ in addition to $\sigma^2$?
What does the law of total expectation say about $R_{\mathrm{alg}}$ and $R_S$?
In the variance decomposition, which term measures variation in risk across training samples?