Predicting Responses Versus Recovering Latent Signals

Two Different Questions

In regression, a rule that outputs a value at a feature value $x$ can serve two different purposes. It may be used to predict a future response, which includes whatever random variation remains after observing $x$. Or it may be used to recover a latent regression signal, a systematic component in a model for the response. These purposes sound similar, but the quantities being targeted are not automatically the same.

For predicting a future response under squared-error loss, the relevant target is its conditional mean given the features. Recovering a latent signal instead targets the signal specified by the data-generating model. An additive-noise model makes the distinction precise: if the noise has conditional mean zero, the targets agree; if not, they can differ. Even when they agree as functions, predicting the noisy response has additional error that cannot be removed by knowing the signal exactly.

Throughout, take $Y$ to be a real-valued response and $X$ the features. Assume the conditional expectations and second moments used below exist. Write the conditional mean as $m(X)=E[Y\mid X]$, understood as a function of $X$ up to equality with probability one. Statements at a particular $x$ apply where the conditional distribution and the stated quantities are defined.

Definition: A response-prediction target is the quantity a prediction rule is meant to approximate for a future response at given features. Under squared-error loss, the target is the conditional mean $E[Y\mid X=x]$. A latent regression signal is a function $s(x)$ specified as a systematic component in a model for $Y$, such as $Y=s(X)+\varepsilon$. It is a model-defined quantity, not necessarily the conditional mean of $Y$.

The Squared-Error Target for a Future Response

Suppose a rule outputs a number $a$ when the features equal $x$. Its conditional squared-error risk for a future response is $E[(Y-a)^2\mid X=x]$. The rule cannot choose the future response's random noise. It can choose only a value that minimizes the expected squared discrepancy.

Theorem: If $E[Y^2]<\infty$, then for any square-integrable prediction $g(X)$, $$ E[(Y-g(X))^2] = E[\operatorname{Var}(Y\mid X)] + E[(E[Y\mid X]-g(X))^2]. $$ Consequently, among square-integrable predictions that are functions of $X$, the conditional mean $E[Y\mid X]$ minimizes mean squared prediction error. Any two minimizing predictions agree with it almost surely.

Proof. Let $m(X)=E[Y\mid X]$. Conditional on $X$, write $Y-g(X)=(Y-m(X))+(m(X)-g(X))$, and expand the square. The cross term has conditional expectation zero because $E[Y-m(X)\mid X]=0$, while $E[(Y-m(X))^2\mid X]=\operatorname{Var}(Y\mid X)$. Therefore $$ E[(Y-g(X))^2\mid X] = \operatorname{Var}(Y\mid X)+(m(X)-g(X))^2. $$ Taking expectations gives the displayed identity. Its first term does not depend on $g$; the second is nonnegative and equals zero exactly when $g(X)=m(X)$ almost surely. The finite second-moment assumption makes the squared-error quantities finite for square-integrable $g(X)$. $\square$

The first term in the identity is the response variability left after conditioning on $X$. Even the optimal prediction $m(X)$ has mean squared prediction error $E[\operatorname{Var}(Y\mid X)]$. This is not an estimation failure: it remains when the conditional mean is known exactly.

Worked Example: Predicting a Response With Two Possible Outcomes

At a particular feature value $x$, suppose a future response is $8$ with probability $1/2$ and $12$ with probability $1/2$. For a prediction $a$, its conditional mean squared error is $$ E[(Y-a)^2\mid X=x] = \frac{1}{2}(8-a)^2+\frac{1}{2}(12-a)^2. $$ Expanding and collecting terms gives $$ \frac{1}{2}(a^2-16a+64)+\frac{1}{2}(a^2-24a+144) = a^2-20a+104 = (a-10)^2+4. $$

This expression is minimized at $a=10$, the conditional mean of the response. The minimum prediction error is $4$, not zero: the future response still differs from $10$ by $2$ in either direction. A prediction of $9$ has error $(9-10)^2+4=5$, so its excess error over the best prediction is $1$.

What the Additive-Noise Model Says

Now suppose the response has the form $Y=s(X)+\varepsilon$, where $s(X)$ is the latent signal and $\varepsilon$ is the noise. The model describes how the response is produced, but it does not by itself say that the noise averages to zero at each feature value. Define its conditional mean by $b(X)=E[\varepsilon\mid X]$. When these conditional expectations exist, $$ E[Y\mid X]=s(X)+b(X). $$ Thus the response-prediction target is $s(X)+b(X)$, whereas the signal-recovery target is $s(X)$. They coincide if $E[\varepsilon\mid X]=0$ almost surely. That condition is called zero conditional mean noise.

A zero unconditional mean, $E[\varepsilon]=0$, is weaker. Positive noise averages at some feature values can cancel negative noise at others, leaving the overall average zero while the conditional means remain nonzero. It is the conditional, not merely overall, mean that determines whether the signal equals the squared-error response target.

Theorem: Suppose $Y=s(X)+\varepsilon$, $E[\varepsilon^2]<\infty$, and $E[\varepsilon\mid X]=0$. For any square-integrable function $h(X)$, $$ E[(Y-h(X))^2] = E[\varepsilon^2]+E[(s(X)-h(X))^2]. $$ Therefore $s(X)$ is both the latent signal and the squared-error response-prediction target, but the minimum mean squared prediction error is $E[\varepsilon^2]$, whereas the signal-recovery error can be zero.

Proof. Under the stated model, $Y-h(X)=\varepsilon+(s(X)-h(X))$. Expanding the square and taking expectations yields $$ E[(Y-h(X))^2] = E[\varepsilon^2]+E[(s(X)-h(X))^2] +2E[\varepsilon(s(X)-h(X))]. $$ Because $s(X)-h(X)$ is a function of $X$, the last expectation is $$ E[\varepsilon(s(X)-h(X))] = E\!\left[(s(X)-h(X))E[\varepsilon\mid X]\right] = 0. $$ This proves the identity. The second term is nonnegative and is zero for $h(X)=s(X)$; the first term remains. Also $E[Y\mid X]=s(X)+E[\varepsilon\mid X]=s(X)$, so this same function is the response-prediction target. $\square$

Worked Example: Zero-Mean Noise and an Unavoidable Prediction Error

Suppose $X$ is either $A$ or $B$, with probability $1/2$ each. The signal is $s(A)=0$ and $s(B)=4$. At $A$, let the noise be $1$ or $-1$, each with probability $1/2$; at $B$, let it be $3$ or $-3$, each with probability $1/2$. At both feature values, the conditional noise mean is zero. Hence the response-prediction target equals the signal: $0$ at $A$ and $4$ at $B$.

Using the signal itself as the prediction, the conditional squared-error risk at $A$ is $\{(1-0)^2+(-1-0)^2\}/2=1$. At $B$, it is $\{(7-4)^2+(1-4)^2\}/2=9$. With the two feature values equally likely, the overall prediction risk is $(1+9)/2=5$. By contrast, the signal-recovery error of the exact function $s$ is zero. The two targets coincide, but the response prediction is not perfect because the noise has conditional variance $1$ at $A$ and $9$ at $B$.

When the Signal and Response Targets Differ

If the conditional noise mean is not zero, a predictor aimed at future responses under squared error should target $s(x)+b(x)$, not $s(x)$. A rule aimed at recovering the specified latent signal should instead target $s(x)$. Neither target is automatically wrong; they answer different questions. The decomposition must also be meaningful for the application: the observed response distribution alone need not determine which component should be called the signal.

Worked Example: A Nonzero Conditional Noise Mean

Let $X$ take values $0$ and $1$. Specify a signal by $s(0)=2$ and $s(1)=5$. Let $U$ take values $-1$ and $1$ with equal conditional probabilities at each value of $X$, and define the noise by $\varepsilon=1+U$ when $X=0$, and $\varepsilon=-2+U$ when $X=1$. Then $E[\varepsilon\mid X=0]=1$ and $E[\varepsilon\mid X=1]=-2$.

At $X=0$, the noise is $0$ or $2$, so the response $Y=2+\varepsilon$ is $2$ or $4$, each with probability $1/2$. Its conditional mean is $3$, while the latent signal is $2$. At $X=1$, the noise is $-3$ or $-1$, so $Y=5+\varepsilon$ is $2$ or $4$, again equally likely. Its conditional mean is $3$, while the signal is $5$. Thus squared-error prediction targets $3$ at either feature value, whereas signal recovery targets $2$ at $0$ and $5$ at $1$. In each case, predicting $3$ has conditional mean squared error $1$, because the outcomes $2$ and $4$ are each one unit away.

Common Mistake and Practical Interpretation

Wrong reasoning: “The response is the signal plus noise, so predicting the signal is always the best way to predict the response.” This silently assumes that the noise averages to zero conditional on the features. In the last example it does not: predicting the signal would mean predicting $2$ at $X=0$ and $5$ at $X=1$, even though the conditional response mean is $3$ at both values.

Correction: First identify the goal and its criterion. For squared-error prediction of a future response, use the conditional response mean. For recovery of a latent signal, use the signal defined by the model or scientific question. Under an additive-noise model, check the conditional mean of the noise before treating these targets as equal. Even when it is zero, report response-prediction error separately from signal-estimation error: the former includes irreducible noise, while the latter measures discrepancy from the latent function.

This distinction matters when interpreting fitted regression functions. A function can predict conditional averages well without recovering a scientifically intended latent component, particularly if the model permits systematic conditional noise. Conversely, a good estimate of the latent signal need not reproduce any one future noisy response. Stating the target makes clear what a claim of accuracy actually means.

Check Your Understanding

Use squared error unless a question says otherwise.

At a fixed feature value, $Y$ is $1$ or $5$ with equal probability. Find the response-prediction target and its minimum conditional mean squared error.
In $Y=s(X)+\varepsilon$, which condition guarantees that the latent signal is the conditional mean response?
Why does zero unconditional mean noise not suffice to conclude that $s(X)=E[Y\mid X]$?
In the nonzero-conditional-mean example, what are the response-prediction target and latent signal at $X=1$?
When zero conditional mean noise holds, why can the minimum prediction error still be positive?