Scalar Gaussian Conditioning With Numerical Examples

Why Scalar Gaussian Conditioning Is Useful

When two measurements are jointly normal, learning the value of one changes what is reasonable to expect for the other. The direction and size of that change depend on their covariance, while the remaining uncertainty depends on how strongly the variables are associated. Scalar conditioning makes these two effects explicit and gives a direct way to calculate the resulting density.

The calculations below concern a pair of jointly normal scalar random variables. The pair must be jointly normal: knowing only that each variable separately has a normal distribution is not enough to justify the conditional normal formula.

Equivalently, because \(c=\rho\sigma_X\sigma_Y\), the conditional mean is \(\mu_X+\rho(\sigma_X/\sigma_Y)(y-\mu_Y)\), and the conditional variance is \(\sigma_X^2(1-\rho^2)\). The variance is positive when \(|\rho|<1\). The statement “given \(Y=y\)” uses a conditional distribution defined as a function of \(y\); even when \(P(Y=y)=0\), its density formula is well-defined.

Deriving the Conditional Density

For \(|\rho|<1\), the joint density of \(X\) and \(Y\), written using their standardized values \(u=(x-\mu_X)/\sigma_X\) and \(v=(y-\mu_Y)/\sigma_Y\), is

The key algebraic step is completing the square in the terms involving \(u\):

The marginal density of \(Y\) is \(f_Y(y)=(\sigma_Y\sqrt{2\pi})^{-1}\exp(-v^2/2)\). Dividing the joint density by this positive marginal density gives the conditional density. The \(v^2\) terms cancel, leaving a normal density in \(x\).

Proof. Substitute the completed-square identity into the joint density. The factor involving \(v\) becomes \(\exp(-v^2/2)\), which is exactly the exponential part of \(f_Y(y)\). After dividing by \(f_Y(y)\), the remaining exponential is \(\exp\{-(u-\rho v)^2/[2(1-\rho^2)]\}\). Since \(u-\rho v\) equals \[ \frac{x-\mu_X-\rho(\sigma_X/\sigma_Y)(y-\mu_Y)}{\sigma_X}, \] the exponent has the stated conditional mean and variance. The remaining constant is \(1/(\sigma_X\sqrt{2\pi(1-\rho^2)})\), which is the normalizing constant for that variance. Thus the displayed expression integrates to one as a density in \(x\). \(\square\)

Regression Residuals and Conditional Moments

The same formula has a useful representation: after accounting for the linear change associated with \(Y\), the leftover part of \(X\) is normal and independent of \(Y\). This gives another way to understand why conditioning shifts the mean but leaves a constant conditional variance.

Proof. The pair \((R,Y)\) is jointly normal because it is an affine transformation of the jointly normal pair \((X,Y)\). Direct calculation gives

and

To verify independence directly, transform from \((X,Y)\) to \((R,Y)\). The inverse transformation is \(x=r+\mu_X+\beta(y-\mu_Y)\), and its Jacobian determinant is \(1\). By the conditional density already derived, \(X\mid Y=y\) has mean \(\mu_X+\beta(y-\mu_Y)\) and variance \(\tau^2\). Therefore, at the corresponding \(x\), the conditional density is the \(N(0,\tau^2)\) density evaluated at \(r\). Multiplying this conditional density by \(f_Y(y)\) gives \(f_{R,Y}(r,y)=f_R(r)f_Y(y)\). This factorization proves independence. Since \(X=\mu_X+\beta(Y-\mu_Y)+R\), conditioning on \(Y=y\) leaves only the independent \(R\) as random variation; its mean is zero and its variance is \(\tau^2\). \(\square\)

Worked Numerical Calculations

Interpreting the Formula and Avoiding Common Errors

The conditional mean is a linear function of the observed value \(y\). Its slope is \(c/\sigma_Y^2\), or \(\rho\sigma_X/\sigma_Y\). Positive correlation gives a positive slope; negative correlation gives a negative slope. The conditional variance does not depend on the particular value observed: it is \(\sigma_X^2(1-\rho^2)\). As \(|\rho|\) approaches one, this variance approaches zero, reflecting that \(Y\) nearly determines \(X\).

A common mistake is to subtract \(\rho^2\) directly from \(\sigma_X^2\). The correct formula is \(\sigma_X^2(1-\rho^2)\), or equivalently \(\sigma_X^2-c^2/\sigma_Y^2\). The correlation is dimensionless, while a variance has squared measurement units; the incorrect subtraction may even combine quantities with incompatible units. Another mistake is to use \(c/\sigma_X^2\) for the slope when predicting \(X\) from \(Y\). The denominator must be the variance of the variable being conditioned on, \(\sigma_Y^2\).

If \(\rho=0\), the conditional mean and variance reduce to \(\mu_X\) and \(\sigma_X^2\), respectively. For a jointly normal pair, the conditional law is then unchanged by observing \(Y\). If \(|\rho|=1\), the pair is degenerate: \(X\) is an exact affine function of \(Y\), so the conditional distribution is a point mass rather than a density with positive variance. The density formula above therefore requires \(|\rho|<1\).