Squared-Error Prediction and the Conditional Mean

Introduction

A prediction is made with information available at the time of decision. If $X$ denotes that information and $Y$ the outcome to be predicted, a rule $g(X)$ produces a prediction from the observed $X$. Under squared-error loss, its penalty is $(Y-g(X))^2$. The optimal prediction can therefore depend on $X$, rather than being a single summary of the marginal distribution of $Y$.

The key fact is exact: among predictions based on $X$, the conditional mean of $Y$ minimizes conditional expected squared error. Moreover, the amount by which another prediction loses is precisely its squared distance from that mean. This tutorial derives both statements and clarifies the conditions under which uniqueness holds.

Conditional Risk and the Conditional Mean

Assume that $Y$ is a real-valued random variable with $E[Y^2]<\infty$, and let $X$ be the observed predictor (which may be a vector). Write $m(X)=E[Y\mid X]$, where conditioning on $X$ means conditioning on the information generated by $X$. The conditional mean is defined up to events of probability zero. When the predictors take values in a standard Borel space, it can be represented as a measurable function $m(x)$, defined up to sets having probability zero under the distribution of $X$.

Definition: For a prediction $a\in\mathbb{R}$, the conditional squared-error risk given $X$ is $E[(Y-a)^2\mid X]$. For a prediction rule $g(X)$, its conditional risk is $E[(Y-g(X))^2\mid X]$. The conditional variance is $\operatorname{Var}(Y\mid X)=E[(Y-m(X))^2\mid X]$.

Since $E[Y^2]<\infty$, the conditional second moment is finite almost surely, and so are $m(X)$ and the conditional variance. For any finite prediction measurable with respect to $X$, the conditional risk is finite almost surely. These qualifications matter: the results concern predictions based on the observed information, not a prediction that uses the unobserved value of $Y$.

Worked Example: Two Predictor Groups

Let $X$ take values 0 and 1 with probability $1/2$ each. Given $X=0$, let $Y$ be 0 or 2 with equal probability. Given $X=1$, let $Y$ be 2 or 6 with equal probability. Find the conditional means and variances, then compare the rule $g(0)=2,\ g(1)=3$ with the conditional mean rule.

When $X=0$, the conditional mean is $(0+2)/2=1$. The conditional variance is $\{(0-1)^2+(2-1)^2\}/2=1$. The proposed prediction is 2, so its conditional risk is $\{(0-2)^2+(2-2)^2\}/2=2$. Its excess risk is $2-1=1=(2-1)^2$.

When $X=1$, the conditional mean is $(2+6)/2=4$, and the conditional variance is $\{(2-4)^2+(6-4)^2\}/2=4$. Predicting 3 gives conditional risk $\{(2-3)^2+(6-3)^2\}/2=5$. Its excess risk is $5-4=1=(3-4)^2$.

The conditional mean rule has average risk $(1+4)/2=2.5$. The proposed rule has average risk $(2+5)/2=3.5$, an excess of 1. The calculation shows the same squared-distance penalty separately in each predictor group.

The Conditional Risk Identity

The reason the conditional mean is optimal is that the residual $Y-m(X)$ has conditional mean zero. When squared error is expanded around $m(X)$, this makes the cross term vanish.

Theorem: Suppose $E[Y^2]<\infty$. For any finite, $X$-measurable prediction $A$, almost surely, $$ E[(Y-A)^2\mid X] = \operatorname{Var}(Y\mid X)+(A-m(X))^2. $$ Consequently, the conditional risk is minimized by $A=m(X)$. This minimizer is unique up to almost-sure equality: if another $X$-measurable prediction has the same conditional risk almost surely, then it equals $m(X)$ almost surely.

To justify the identity for predictions that need not be globally square-integrable, it is enough to reason conditionally. Given $X$, $A$ is fixed, and the conditional second moment of $Y$ is finite almost surely. Thus the following expansion and conditional expectations are well-defined almost surely:

\begin{aligned} E[(Y-A)^2\mid X] &=E[((Y-m(X))+(m(X)-A))^2\mid X]\\ &=E[(Y-m(X))^2\mid X]\\ &\quad +2(m(X)-A)E[Y-m(X)\mid X]\\ &\quad +(m(X)-A)^2\\ &=\operatorname{Var}(Y\mid X)+(m(X)-A)^2. \end{aligned}

The last equality uses $E[Y-m(X)\mid X]=E[Y\mid X]-m(X)=0$. For each realized value of $X$, the variance term does not depend on the chosen prediction, and the remaining squared term is nonnegative. It equals zero exactly when $A=m(X)$. Therefore no other prediction has smaller conditional risk, and equality of conditional risks requires $A=m(X)$ almost surely. $\square$

This uniqueness statement is about predictions as random variables. It does not require two functions $g(x)$ and $m(x)$ to agree at every possible input value. If a value $x$ occurs with probability zero under the distribution of $X$, the data distribution does not determine the optimal prediction there.

Worked Example: A Conditional Normal Model

Suppose that, given $X=x$, $Y\sim N(1+2x,9)$. The conditional mean is $m(x)=1+2x$, and the conditional variance is 9. Compare the optimal prediction with the rule $g(x)=1+x$.

At $x=2$, the conditional mean is $1+2(2)=5$. The rule predicts $g(2)=3$, so its conditional regret is $(3-5)^2=4$. Its conditional risk is therefore $9+4=13$, whereas predicting 5 has conditional risk 9.

For any $x$, the same calculation gives conditional risk $9+\{(1+x)-(1+2x)\}^2=9+x^2$. The excess risk is $x^2$, which is zero at $x=0$ and positive at any nonzero $x$ that occurs with positive probability.

From Conditional Regret to Population Risk

The conditional identity also gives an exact expression for the difference in overall expected squared error. Here the rules compared are assumed to have finite second moments, so their population risks are finite. Taking expectations of the conditional identity yields the following decomposition.

Result: If $E[Y^2]<\infty$ and $g(X)$ is square-integrable, then $$ E[(Y-g(X))^2] = E[\operatorname{Var}(Y\mid X)] + E[(g(X)-m(X))^2]. $$ In particular, the population excess risk relative to the conditional mean is $E[(g(X)-m(X))^2]$.

For the derivation, the identity just proved applies with $A=g(X)$. Taking expectations on both sides and using the law of total expectation gives

\begin{aligned} E[(Y-g(X))^2] &=E\!\left[E[(Y-g(X))^2\mid X]\right]\\ &=E[\operatorname{Var}(Y\mid X)] +E[(g(X)-m(X))^2]. \end{aligned}

The integrability assumptions ensure these expectations are finite: $E[Y^2]<\infty$ implies $E[m(X)^2]\leq E[Y^2]$, and both $g(X)$ and $m(X)$ are square-integrable. The decomposition also proves uniqueness for population risk among square-integrable rules. If $g$ has the same risk as $m$, then $E[(g(X)-m(X))^2]=0$; a nonnegative random variable has expectation zero only if it is zero almost surely. Hence $g(X)=m(X)$ almost surely. $\square$

Worked Example: The Cost of Ignoring the Predictor

Let $P(X=0)=P(X=1)=1/2$, and suppose that, given either value of $X$, $Y$ has variance 1. Let the conditional means be 0 when $X=0$ and 4 when $X=1$. The conditional mean rule has risk 1 in each group, so its overall risk is 1.

The marginal mean of $Y$ is $E[Y]=\{0+4\}/2=2$. Consider the constant rule $g(X)=2$, which ignores $X$. In the first group, its conditional risk is $1+(2-0)^2=5$. In the second group, its conditional risk is $1+(2-4)^2=5$. Its overall risk is therefore $(5+5)/2=5$.

The difference between the risks is $5-1=4$. The population decomposition gives the same answer: $E[(g(X)-m(X))^2]=\{(2-0)^2+(2-4)^2\}/2=4$. The predictor is useful because the conditional means differ, not because squared error rewards a particular modeling method.

Interpretation and Common Pitfall

Conditional variance and regret describe distinct sources of prediction error. The term $\operatorname{Var}(Y\mid X)$ is the remaining uncertainty in $Y$ given the observed predictors. The term $(g(X)-m(X))^2$ is the additional loss from using a prediction other than the conditional mean. Even the optimal rule generally has positive risk when the outcome remains variable after conditioning on $X$.

A common mistake is to say, “The mean minimizes squared error, so the best prediction is always $E[Y]$.” This confuses a single constant prediction with a rule that can use $X$. If predictions must be constant, the optimal constant is indeed $E[Y]$ (when $E[Y^2]<\infty$). But when predictions can depend on $X$, the relevant minimizer is $E[Y\mid X]$, which may vary across predictor values.

A second pitfall is to treat uniqueness as pointwise equality at every imaginable input. The conditional mean, as a conditional expectation, is specified only almost surely. Two prediction functions that differ only on values of $X$ having probability zero represent the same rule under the population distribution. The appropriate uniqueness claim is therefore almost-sure uniqueness.

Key takeaway: Under squared-error loss, conditional risk equals conditional variance plus squared distance from the conditional mean. The mean is the unique optimal prediction almost surely, and that squared distance is exactly the conditional regret.

Check Your Understanding

Use the conditional-risk identity to answer the following.

If $\operatorname{Var}(Y\mid X)=3$ and a rule predicts 2 units above $E[Y\mid X]$, what are its conditional risk and conditional regret?
Why does the conditional mean minimize squared-error risk even when the conditional variance is positive?
What does equal conditional risk imply about another $X$-measurable prediction, and in what sense does it imply this?
When does a constant prediction of $E[Y]$ coincide with the optimal rule that can use $X$?
In the population decomposition, what does $E[\operatorname{Var}(Y\mid X)]$ measure, and what does $E[(g(X)-m(X))^2]$ measure?