Normal Conditional Response Models for GLMs

Introduction

A Gaussian GLM describes how a response varies with covariates by specifying its distribution conditional on those covariates. That qualification matters: responses at different covariate values can have different means even though they share the same conditional variance. If covariate patterns are then pooled, the resulting distribution need not be normal. Keeping the conditional model separate from any pooled or population distribution prevents a common misreading of what the model assumes.

Here, $Y_i$ is the response for observation $i$, and $x_i$ denotes its covariate vector. We treat the covariates as fixed, or equivalently condition on their observed values. The response model concerns the distribution of each $Y_i$ given $x_i$; it does not, by itself, specify how covariates are distributed in a broader population.

The Conditional Gaussian Response Model

Definition: A Gaussian GLM for $n$ responses specifies, for each $i=1,\ldots,n$, $$ Y_i\mid x_i \sim N(\mu_i,\sigma^2), \qquad \sigma^2>0, $$ where $\mu_i=E[Y_i\mid x_i]$. The mean is related to the covariates through a link function $g$: $$ g(\mu_i)=\eta_i=x_i^\mathsf{T}\beta. $$ Here $\beta$ is a vector of regression coefficients, $g$ is defined and invertible over the relevant mean range, and $\eta_i$ is the linear predictor. Unless stated otherwise, the responses are assumed conditionally independent given the covariates.

The normal distribution specifies more than a mean: for each covariate pattern, it specifies a full conditional distribution, including its shape and spread. Its conditional density at $y_i$ is

f(y_i\mid x_i)=\frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left\{-\frac{(y_i-\mu_i)^2}{2\sigma^2}\right\}.

The link governs how the conditional mean changes with covariates. The normal assumption concerns the distribution around that mean. For example, with the identity link, $\mu_i=x_i^\mathsf{T}\beta$; with a log link, $\mu_i=\exp(x_i^\mathsf{T}\beta)$. In either case the conditional response distribution is normal with variance $\sigma^2$. A normal response can take negative values, even when its mean is positive; a positive mean alone does not restrict the support of a normal distribution.

Worked Example: Specifying Conditional Responses With a Log Link

Suppose $x_i$ is a treatment indicator, with $x_i=0$ for control and $x_i=1$ for treatment. Consider the log-link mean model $ \log(\mu_i)=0.4+0.3x_i $, with $\sigma^2=4$. Find the conditional distributions for the two groups.

For control, $x_i=0$, so $\log(\mu_i)=0.4$ and $\mu_i=\exp(0.4)\approx1.492$. Thus $Y_i\mid x_i=0\sim N(1.492,4)$.

For treatment, $x_i=1$, so $\log(\mu_i)=0.7$ and $\mu_i=\exp(0.7)\approx2.014$. Thus $Y_i\mid x_i=1\sim N(2.014,4)$.

Both conditional distributions have standard deviation $\sqrt{4}=2$. Their means differ because the covariate pattern differs. The model says nothing here about how many control and treatment observations occur in a population; that would require a model or sampling description for treatment status.

From Individual Responses to a Joint Model

When responses are conditionally independent given all the covariates, their joint conditional density is the product of their individual conditional densities. The condition is important: specifying normal conditional distributions one observation at a time does not alone assert independence.

Result: If $Y_1,\ldots,Y_n$ are conditionally independent given $x_1,\ldots,x_n$, and each $Y_i\mid x_i\sim N(\mu_i,\sigma^2)$, then $$ f(y_1,\ldots,y_n\mid x_1,\ldots,x_n) =\prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left\{-\frac{(y_i-\mu_i)^2}{2\sigma^2}\right\}. $$

Proof. Conditional independence means that the conditional joint distribution factors into its conditional marginal distributions. Each marginal distribution has the normal density displayed above. Substituting those densities into the factorization gives the stated product. The factorization is not justified by normality alone; it follows from the additional conditional independence assumption. $\square$

This joint model is the sampling model for the observed response vector at the specified covariates. If two observations have the same covariate pattern, their conditional means are the same under the model, but they remain distinct random responses. Under conditional independence, they are independent draws from that same normal distribution.

Worked Example: A Joint Conditional Density at Two Patterns

Let two independent responses have covariate patterns with conditional distributions $Y_1\mid x_1\sim N(2,1)$ and $Y_2\mid x_2\sim N(4,1)$. Find their joint density at $y_1=3$, $y_2=3$.

For a normal distribution with variance 1, the density at $y$ given mean $\mu$ is $(2\pi)^{-1/2}\exp\{-(y-\mu)^2/2\}$. Therefore,

f(3,3\mid x_1,x_2) =\frac{1}{\sqrt{2\pi}}e^{-(3-2)^2/2} \frac{1}{\sqrt{2\pi}}e^{-(3-4)^2/2} =\frac{e^{-1}}{2\pi} \approx 0.0585.

Each observation is one unit from its own conditional mean, so each individual density factor is $(2\pi)^{-1/2}e^{-1/2}\approx0.2420$. Their product is approximately $0.0585$, as shown. The fact that both observed values equal 3 does not give them the same conditional distribution: the conditioning covariates, and hence the means, differ.

Conditional Distributions and Pooled Distributions

A statement such as $Y_i\mid x_i\sim N(\mu_i,\sigma^2)$ is conditional: it describes the response distribution after the covariate value is fixed. A marginal distribution across covariate patterns is a different object. To define one, the covariates must themselves be treated as random, with a specified distribution. The marginal response distribution then averages the conditional distributions over that covariate distribution.

Result: Let $X$ be a random covariate pattern, and suppose $E[Y^2]$ is finite. If $Y\mid X$ has conditional mean $\mu(X)$ and conditional variance $\sigma^2$, then $$ E[Y]=E[\mu(X)], \qquad \operatorname{Var}(Y)=\sigma^2+\operatorname{Var}(\mu(X)). $$

Proof. By the law of iterated expectation, $E[Y]=E[E[Y\mid X]]=E[\mu(X)]$. For the variance, write $Y-E[Y]=(Y-\mu(X))+(\mu(X)-E[Y])$. Squaring and taking expectations gives

\begin{aligned} \operatorname{Var}(Y) &=E[(Y-\mu(X))^2]+E[(\mu(X)-E[Y])^2]\\ &\quad+2E[(Y-\mu(X))(\mu(X)-E[Y])]. \end{aligned}

The first term is $E[\operatorname{Var}(Y\mid X)]=\sigma^2$, and the second is $\operatorname{Var}(\mu(X))$. For the cross term, condition on $X$: since $\mu(X)-E[Y]$ is fixed given $X$ and $E[Y-\mu(X)\mid X]=0$,

E[(Y-\mu(X))(\mu(X)-E[Y])\mid X] =(\mu(X)-E[Y])E[Y-\mu(X)\mid X]=0.

Its unconditional expectation is therefore zero. Combining the terms proves the variance formula. $\square$

The extra variance term $\operatorname{Var}(\mu(X))$ records the variation in conditional means across covariate patterns. It does not contradict the assumption that each conditional variance equals $\sigma^2$: the conditional and marginal variances refer to different distributions.

Worked Example: A Normal Conditional Model That Is Not Marginally Normal

Let a random covariate $X$ equal 0 or 1 with probability $1/2$ each. Suppose $Y\mid X=0\sim N(0,1)$ and $Y\mid X=1\sim N(2,1)$. Find the marginal mean and variance, and verify that the marginal response is not normal.

The marginal distribution is an equal mixture of the two conditional distributions. Its mean is $E[Y]=\{0+2\}/2=1$. The conditional variance is 1, and the two conditional means have variance $\{(0-1)^2+(2-1)^2\}/2=1$. Thus $\operatorname{Var}(Y)=1+1=2$.

To check normality, let $Z=Y-1$. Conditional on $X=0$, $Z\sim N(-1,1)$; conditional on $X=1$, $Z\sim N(1,1)$. For a normal random variable with mean $m$ and variance 1, its fourth raw moment is $m^4+6m^2+3$. With $m=-1$ or $m=1$, this is $1+6+3=10$. Hence $E[Z^4]=10$. If $Z$ were normal with its actual mean 0 and variance 2, its fourth moment would be $3(2^2)=12$. Since $10\ne12$, the marginal distribution is not normal, despite both conditional distributions being normal.

What an Observed Data Set Does and Does Not Represent

In an analysis with fixed covariates, the observed vector $Y_1,\ldots,Y_n$ is modeled conditionally on the design values $x_1,\ldots,x_n$. If the $x_i$ differ, the $Y_i$ can have different conditional means. The collection of responses is then generally not identically distributed, even though each response is conditionally normal with the same variance. Simply pooling the recorded values does not turn them into draws from a single normal population.

If instead the study samples covariates from a population, a marginal response distribution can be meaningful. It depends on both the conditional response model and the distribution of covariates in that population. Changing the population mix of covariate patterns can change the marginal response distribution even when the conditional model stays the same.

Worked Example: Same Conditional Model, Different Population Mix

Suppose the conditional distributions are $Y\mid X=0\sim N(10,4)$ and $Y\mid X=1\sim N(14,4)$. In population A, $P(X=1)=1/4$; in population B, $P(X=1)=3/4$. Find each marginal mean.

In population A, $E[Y]=10(3/4)+14(1/4)=7.5+3.5=11$. In population B, $E[Y]=10(1/4)+14(3/4)=2.5+10.5=13$.

The conditional response distributions have not changed, but the marginal means differ because the covariate-pattern proportions differ. This is why a pooled description of responses should not be mistaken for the conditional model itself.

Common Mistake

Wrong reasoning: “Every response is conditionally normal with variance $\sigma^2$, so the pooled responses are normal with variance $\sigma^2$.” This ignores that conditional means can vary, and that the distribution of covariate patterns affects any marginal distribution.
Correction: State the conditioning explicitly. Each $Y_i\mid x_i$ has its specified normal distribution; the joint conditional model also requires an independence assumption if its density is to factor into a product. A marginal distribution across patterns requires a distribution for the covariates and is generally a mixture, not necessarily normal.

Key takeaway: A Gaussian GLM specifies normal response distributions conditional on covariates, with means linked to a linear predictor and a common conditional variance. It does not assert that responses pooled across differing covariate patterns have one normal distribution.

Check Your Understanding

Use the conditional-versus-marginal distinction in each answer.

In a Gaussian GLM, what does the statement $Y_i\mid x_i\sim N(\mu_i,\sigma^2)$ specify, and what does it leave unspecified?
Why is conditional independence a separate assumption from conditional normality?
If conditional means vary across a random covariate $X$, how does that affect the marginal variance of $Y$?
Can responses at different fixed covariate patterns be identically distributed when their conditional means differ? Explain.
Why does the example with equal mixtures of $N(0,1)$ and $N(2,1)$ not have a normal marginal distribution?