Least-Squares Characterization

Prediction Under a Measurability Constraint

Suppose $X$ is a square-integrable real random variable, and the information available for prediction is represented by a sigma-algebra $\mathcal G$. We seek a predictor that is measurable with respect to $\mathcal G$: it may use the information in $\mathcal G$, but no more.

Among such predictors, which one has the smallest mean squared error? The answer is the conditional expectation $\mathbb E[X\mid\mathcal G]$. The key point is stronger than the claim that it is a good predictor: its squared error is no larger than that of any other square-integrable $\mathcal G$-measurable predictor, and equality is possible only when the predictors agree almost surely.

Least-squares characterization: For $X\in L^2$, the conditional expectation $\mathbb E[X\mid\mathcal G]$ minimizes $\mathbb E[(X-Y)^2]$ over all $Y\in L^2$ that are $\mathcal G$-measurable.

The Admissible Predictors

Write $L^2$ for the real random variables $U$ satisfying $\mathbb E[U^2]<\infty$, with random variables identified when they agree almost surely. The admissible predictors form the class $L^2(\mathcal G)$: the elements of $L^2$ that have a $\mathcal G$-measurable version.

The squared-error risk of a candidate $Y\in L^2(\mathcal G)$ is $\mathbb E[(X-Y)^2]$. This quantity is finite: since $(X-Y)^2\leq 2X^2+2Y^2$, its expectation is finite whenever $X,Y\in L^2$.

Set $Z=\mathbb E[X\mid\mathcal G]$. The conditional expectation is $\mathcal G$-measurable, and the $L^2$ contraction property ensures that $Z\in L^2(\mathcal G)$. Thus $Z$ is itself an admissible predictor. We will compare every candidate directly with this $Z$.

The Squared-Error Identity

Theorem: If $X\in L^2$, $Z=\mathbb E[X\mid\mathcal G]$, and $Y\in L^2(\mathcal G)$, then $$ \mathbb E[(X-Y)^2] = \mathbb E[(X-Z)^2]+\mathbb E[(Z-Y)^2]. $$

Proof. The defining property of conditional expectation gives $\mathbb E[X-Z\mid\mathcal G]=0$. Consequently, for every bounded $\mathcal G$-measurable random variable $H$, $\mathbb E[(X-Z)H]=0$. This follows first for indicators of events in $\mathcal G$, then for simple functions by linearity, and then for bounded measurable $H$ by bounded approximation.

Here $H=Z-Y$ need not be bounded, but it is $\mathcal G$-measurable and belongs to $L^2$. Define $H_n=\max(-n,\min(H,n))$. Each $H_n$ is bounded and $\mathcal G$-measurable, and $H_n\to H$ in $L^2$. By Cauchy–Schwarz,

\left|\mathbb E[(X-Z)(H_n-H)]\right| \leq \bigl(\mathbb E[(X-Z)^2]\bigr)^{1/2} \bigl(\mathbb E[(H_n-H)^2]\bigr)^{1/2} \longrightarrow 0.

Since $\mathbb E[(X-Z)H_n]=0$ for every $n$, it follows that $\mathbb E[(X-Z)(Z-Y)]=0$. Expand $X-Y=(X-Z)+(Z-Y)$, square, and take expectations. The cross term vanishes by this orthogonality, giving the claimed identity. $\square$

The identity is a precise comparison of risks. The second term on its right is nonnegative, so no admissible $Y$ can have smaller risk than $Z$. It also records exactly how much additional squared error is incurred by using $Y$ instead of the conditional expectation.

Existence and Uniqueness of the Minimizer

Theorem: For $X\in L^2$, $Z=\mathbb E[X\mid\mathcal G]$ minimizes $\mathbb E[(X-Y)^2]$ over $Y\in L^2(\mathcal G)$. If $Y$ also attains this minimum, then $Y=Z$ almost surely.

Proof. For any admissible $Y$, the squared-error identity gives

\mathbb E[(X-Y)^2] = \mathbb E[(X-Z)^2]+\mathbb E[(Z-Y)^2] \geq \mathbb E[(X-Z)^2].

Because $Z\in L^2(\mathcal G)$, it is an admissible choice, and taking $Y=Z$ attains the lower bound. If $Y$ attains the same minimum, the identity implies $\mathbb E[(Z-Y)^2]=0$. A nonnegative random variable with expectation zero is zero almost surely; hence $Y=Z$ almost surely. $\square$

Uniqueness here is necessarily almost sure, not pointwise. Changing a predictor on an event of probability zero does not change its squared-error risk. Thus the theorem identifies a unique element of $L^2(\mathcal G)$, where variables equal almost surely are treated as the same element.

Worked Examples on Finite Partitions

Worked Example: Two equally likely outcomes in each information group

Let $\Omega=\{\omega_1,\omega_2,\omega_3,\omega_4\}$, with each outcome having probability $1/4$. Let $\mathcal G$ be generated by the partition $A=\{\omega_1,\omega_2\}$, $B=\{\omega_3,\omega_4\}$. Suppose $X$ takes the values $0,2,1,5$, respectively.

On each partition atom, the conditional expectation is the average of $X$ on that atom. Therefore $Z=\mathbb E[X\mid\mathcal G]$ equals $1$ on $A$ and $3$ on $B$. Any $\mathcal G$-measurable predictor is constant on each atom; write its values as $u$ on $A$ and $v$ on $B$. Its risk is

\mathbb E[(X-Y)^2] = \frac14\left(u^2+(2-u)^2+(1-v)^2+(5-v)^2\right).

Completing squares gives

\mathbb E[(X-Y)^2] = \frac12(u-1)^2+\frac12(v-3)^2+\frac52.

The minimum is $5/2$, attained at $u=1$ and $v=3$, precisely the values of $Z$. The squared-error identity also gives the excess risk directly: it is $\mathbb E[(Z-Y)^2]=\tfrac12(u-1)^2+\tfrac12(v-3)^2$.

Worked Example: Unequal probabilities within an atom

Let $\Omega=\{\omega_1,\omega_2,\omega_3\}$ have probabilities $1/5,3/10,1/2$, respectively. Let $\mathcal G$ be generated by $A=\{\omega_1,\omega_2\}$ and $B=\{\omega_3\}$, and suppose $X$ takes values $0,4,2$, respectively.

The probability of $A$ is $1/2$, and the conditional probabilities of $\omega_1,\omega_2$ given $A$ are $2/5$ and $3/5$. Thus the conditional mean on $A$ is

\frac25\cdot 0+\frac35\cdot 4=\frac{12}{5}.

On $B$, the conditional expectation is $2$, since $B$ contains just $\omega_3$. For a predictor taking values $a$ on $A$ and $b$ on $B$, the risk is

\mathbb E[(X-Y)^2] = \frac15a^2+\frac{3}{10}(4-a)^2+\frac12(2-b)^2.

The first two terms can be written as

\frac15a^2+\frac{3}{10}(4-a)^2 = \frac12\left(a-\frac{12}{5}\right)^2+\frac{24}{25}.

Therefore the risk is minimized at $a=12/5$ and $b=2$. The minimum risk is $24/25$. The unequal probabilities matter: taking the unweighted average of $0$ and $4$ would give the wrong predictor on $A$.

Worked Example: Minimization across three groups

Let $\Omega$ consist of six equally likely outcomes. Take $\mathcal G$ to be generated by $A=\{\omega_1,\omega_2,\omega_3\}$ and $B=\{\omega_4,\omega_5,\omega_6\}$. Suppose $X$ takes values $0,0,3,2,2,8$, in that order.

The average on $A$ is $1$, and the average on $B$ is $4$, so $Z$ takes these two values on the corresponding atoms. A general admissible predictor has values $a$ on $A$ and $b$ on $B$. Its risk equals

\begin{aligned} \mathbb E[(X-Y)^2] &=\frac16\left(2a^2+(3-a)^2+2(2-b)^2+(8-b)^2\right)\\ &=\frac12(a-1)^2+\frac12(b-4)^2+5. \end{aligned}

For the second equality, the terms from $A$ satisfy $2a^2+(3-a)^2=3(a-1)^2+6$, and those from $B$ satisfy $2(2-b)^2+(8-b)^2=3(b-4)^2+24$. Hence the minimum risk is $5$, attained at $a=1$ and $b=4$. Any departure from these two conditional means adds exactly $\tfrac12(a-1)^2+\tfrac12(b-4)^2$ to the risk.

Reading the Finite-Partition Formula

For a finite partition $\{A_1,\dots,A_m\}$ generating $\mathcal G$, each predictor measurable with respect to $\mathcal G$ is constant on each atom, up to null sets. On an atom $A_i$ with positive probability, the conditional expectation is the probability-weighted mean

\mathbb E[X\mid\mathcal G]\big|_{A_i} = \frac{\mathbb E[X\mathbf 1_{A_i}]}{\mathbb P(A_i)}.

This value minimizes the contribution to mean squared error from that atom. If $\mathbb P(A_i)=0$, the displayed quotient is undefined, but the value assigned to the conditional expectation on $A_i$ is immaterial: the atom has probability zero. This is another instance of why conditional expectations, like other $L^2$ variables, are identified up to almost-sure equality.

Why the Characterization Matters

The result turns conditional expectation into a principled prediction rule. The conditioning sigma-algebra specifies what information a predictor may use; the least-squares theorem then identifies the best predictor under squared-error loss. The conclusion does not say that the conditional expectation reproduces $X$. Rather, it says that no predictor using only that information can improve its expected squared error.

A common pitfall is to minimize over the wrong class. If the only allowed predictors are constants, the best constant is $\mathbb E[X]$. If predictors may use all the information in $\mathcal G$, the minimizer is $\mathbb E[X\mid\mathcal G]$, which can vary with the information. The permitted measurability is part of the optimization problem, not an incidental detail.

A second pitfall is to interpret uniqueness pointwise. The optimization criterion is an expectation, so changes on null events have no effect. The mathematically correct uniqueness statement is equality almost surely.

Check Your Understanding

Use the squared-error identity and the finite-partition calculations to check your understanding.

Why is every $Y\in L^2(\mathcal G)$ a legitimate candidate with finite squared-error risk when $X\in L^2$?
In the squared-error identity, which term measures the additional risk of using $Y$ instead of $Z=\mathbb E[X\mid\mathcal G]$?
Why does attaining the minimum imply $Y=Z$ almost surely rather than necessarily at every outcome?
In a finite partition, why must the conditional mean on a positive-probability atom use probability weights when outcomes are not equally likely?
If the only admissible predictors are constants, what is the best constant predictor, and how does that constraint differ from allowing all of $L^2(\mathcal G)$?

What You'll Learn