Additive and Heteroscedastic Noise Models

Why Describe Noise Separately?

A regression model can describe a response as a systematic part that depends on the features, plus a remaining random part. This separation is useful because the systematic part and the noise answer different questions: the first describes how the response changes with the features, while the second describes how responses vary around that pattern. In particular, the noise model determines whether the conditional response distribution has the same spread at different feature values.

Write \(X\) for a feature and \(Y\) for a response. An additive-noise model has the form \(Y=m(X)+\varepsilon\), where \(m\) is a signal function and \(\varepsilon\) is a random error. The equation alone does not determine the conditional distribution of \(Y\) given \(X\): that also depends on how \(\varepsilon\) is distributed and how it relates to \(X\). We will make those assumptions explicit.

Here “almost surely” means that a property can fail only for feature values having probability zero under the distribution of \(X\). Thus, a variance function may vary at feature values that never occur without making the model heteroscedastic in this definition. The function \(m\) is not automatically the conditional mean: that identification requires a condition on the error.

Conditional Mean and Variance in an Additive Model

Suppose \(E[\varepsilon\mid X]=0\) and \(E[\varepsilon^2\mid X]<\infty\) almost surely. Then \(m(X)\) is the conditional mean of \(Y\), and the conditional variance of \(Y\) is the conditional variance of the error.

Proof. Conditional expectation is linear, and \(m(X)\) is determined by \(X\). Therefore $$ E[Y\mid X]=E[m(X)+\varepsilon\mid X] =m(X)+E[\varepsilon\mid X] =m(X). $$ For the variance, subtract this conditional mean from \(Y\): $$ Y-E[Y\mid X]=m(X)+\varepsilon-m(X)=\varepsilon. $$ Consequently, $$ \operatorname{Var}(Y\mid X) =E[(Y-E[Y\mid X])^2\mid X] =E[\varepsilon^2\mid X] =\operatorname{Var}(\varepsilon\mid X), $$ where the last equality uses \(E[\varepsilon\mid X]=0\). This gives the stated result. \(\square\)

This calculation shows why homoscedasticity is about conditional variance, not about whether the error is “small” or whether the signal is linear. A model may have a nonlinear \(m\) and constant conditional variance, or a linear \(m\) and feature-dependent conditional variance. The shape of \(m\) and the variance structure are separate modeling choices.

When the Error Is Independent of the Features

A stronger assumption is that \(\varepsilon\) is independent of \(X\). Independence says the full distribution of the error does not change with the features; conditional mean zero alone says only that its conditional average is zero. If the error is independent of \(X\), the conditional response distribution is a shifted copy of the error distribution.

Derivation. For a fixed \(x\), the event \(Y\leq y\), given \(X=x\), is the event \(m(x)+\varepsilon\leq y\), or equivalently \(\varepsilon\leq y-m(x)\). Independence means conditioning on \(X=x\) leaves the distribution of \(\varepsilon\) unchanged. Thus $$ P(Y\leq y\mid X=x) =P(\varepsilon\leq y-m(x)\mid X=x) =P(\varepsilon\leq y-m(x)) =F_{\varepsilon}(y-m(x)). $$ When a density exists, differentiating this translated distribution with respect to \(y\) gives the stated density. Conditional distributions given \(X=x\) are understood as a chosen regular conditional version; the formula is required only for almost every \(x\). \(\square\)

Independence is sufficient for this translation formula, but it is not required for the additive equation itself. If the error depends on the features, then its conditional distribution may change with \(x\), and the conditional response law must reflect that change. A useful special case makes the dependence explicit through a scale.

To see the variance formula, condition on \(X=x\). The signal \(m(x)\) and scale \(s(x)\) are then fixed, while \(Z\) retains its distribution by independence. Hence \(E[Y\mid X=x]=m(x)+s(x)E[Z]=m(x)\), and \(\operatorname{Var}(Y\mid X=x)=s(x)^2\operatorname{Var}(Z)=s(x)^2\). If \(s(x)\) varies on feature values of positive probability, this model is heteroscedastic.

Worked Examples

What Homoscedasticity Does and Does Not Say

Homoscedasticity means that the conditional variance is constant, not that the conditional response distribution is identical at every feature value. In the discrete example, the distribution shifts with \(X\), while its variance stays the same. More generally, conditional distributions can differ in skewness or tail behavior even when their variances agree. Conversely, heteroscedasticity specifically records variation in conditional spread; it does not by itself say whether that spread increases or decreases with a feature.

A common mistake is to assume that writing \(Y=m(X)+\varepsilon\) automatically makes \(\varepsilon\) independent of \(X\), or automatically makes \(m(X)=E[Y\mid X]\). Neither conclusion follows from the equation. The correction is to state the required assumptions: \(E[\varepsilon\mid X]=0\) makes \(m(X)\) the conditional mean, while independence gives a feature-invariant error distribution and the shifted-law formula. A conditional mean-zero error can still have a distribution, including a variance, that changes with \(X\).

The distinction matters when specifying a model for prediction or inference. A constant-variance assumption can simplify calculations, but if the actual conditional spread changes with the features, it gives the wrong description of uncertainty for some feature values. A conditional response law, not just its mean function, describes the variation a learning procedure encounters.