Residuals in GLMs as Conditional Errors

From Conditional Errors to Response Residuals

A GLM describes the distribution of an outcome $Y_i$ conditional on its covariates. Its mean is $\mu_i=E[Y_i\mid x_i]$, and a link function relates that mean to a linear predictor $\eta_i$ through $g(\mu_i)=\eta_i$. After fitting the model, the fitted mean $\hat{\mu}_i$ gives the model's predicted average response at the observed covariates. A response residual compares the particular observed outcome with that average.

This comparison is useful, but it needs careful interpretation. Outcomes vary around their conditional means even when the model is correct, so a nonzero residual does not by itself indicate a model failure. Also, the response residual is measured in the units of the outcome; it is not a discrepancy on the linear-predictor scale.

Definition: For observation $i$, let $\mu_i=E[Y_i\mid x_i]$ be the true conditional mean under the model, and define the conditional error by $\varepsilon_i=Y_i-\mu_i$. Let $\hat{\mu}_i$ be the fitted response mean and $y_i$ the observed response. The response residual is $$ r_i=y_i-\hat{\mu}_i. $$ Thus $\varepsilon_i$ is defined using the true conditional mean, whereas $r_i$ uses the observed response and an estimated mean.

In a GLM with coefficient estimate $\hat{\beta}$, the fitted linear predictor is typically $\hat{\eta}_i=x_i^{T}\hat{\beta}$, and the fitted response mean is $\hat{\mu}_i=g^{-1}(\hat{\eta}_i)$. These are different quantities: $\hat{\eta}_i$ is on the link scale, while $\hat{\mu}_i$ is on the response scale. The response residual subtracts $\hat{\mu}_i$, not $\hat{\eta}_i$.

What a Response Residual Measures

The residual $r_i$ measures how far the observed response lies above or below the fitted conditional mean, in the response's units. A positive value means $y_i>\hat{\mu}_i$; a negative value means $y_i<\hat{\mu}_i$. Its meaning therefore depends on the response: it may be measured in the original units of a continuous measurement, in success units for a binary response, or in counts for a count response.

The conditional error $\varepsilon_i$ has a useful theoretical property. Conditional on the covariates, its expectation is zero because the conditional mean has been subtracted. Its variance is the conditional variance of the response. These facts describe the model's random outcomes; they do not say that the fitted residuals $r_i$ have conditional mean zero after the same observations have been used to estimate the model.

Result: If $E[Y_i\mid x_i]=\mu_i$ and $\operatorname{Var}(Y_i\mid x_i)=V_i<\infty$, then $$ E[\varepsilon_i\mid x_i]=0 \qquad\text{and}\qquad \operatorname{Var}(\varepsilon_i\mid x_i)=V_i. $$

Proof. Since $\mu_i$ is fixed conditional on $x_i$, $$ E[\varepsilon_i\mid x_i] =E[Y_i-\mu_i\mid x_i] =E[Y_i\mid x_i]-\mu_i =0. $$ Subtracting a conditional constant does not change variance, so $$ \operatorname{Var}(\varepsilon_i\mid x_i) =\operatorname{Var}(Y_i-\mu_i\mid x_i) =\operatorname{Var}(Y_i\mid x_i) =V_i. $$ This result concerns the true conditional error, not the residual formed with an estimated mean. $\square$

Worked Example: Gaussian Response Residuals

Suppose three independent outcomes follow a Gaussian model with an intercept only, $Y_i\mid x_i\sim N(\mu,\sigma^2)$, and the observed values are $2,4,6$. The fitted mean is the sample average: $$ \hat{\mu}=\frac{2+4+6}{3}=4. $$

The response residuals are the observations minus this fitted mean: $$ r_1=2-4=-2,\qquad r_2=4-4=0,\qquad r_3=6-4=2. $$ They are expressed in the same units as $Y_i$. Under the model, a true conditional error has mean zero and variance $\sigma^2$; an individual observed residual need not be zero, even when the Gaussian model is appropriate.

Worked Examples Across GLM Families

The definition $r_i=y_i-\hat{\mu}_i$ does not change across GLM families. What changes is the response distribution and, consequently, the scale and possible values of the residual.

Worked Example: Bernoulli Response With a Logit Link

Suppose one observation is Bernoulli, with fitted success probability $\hat{p}_i=\hat{\mu}_i=0.7$. Its fitted logit is $$ \hat{\eta}_i=\log\left(\frac{0.7}{1-0.7}\right) =\log\left(\frac{7}{3}\right)\approx 0.847. $$ If the observed outcome is $y_i=1$, then $$ r_i=1-0.7=0.3. $$ If instead the outcome is $y_i=0$, then $$ r_i=0-0.7=-0.7. $$

These values are differences in success-probability units, not differences in logit units. In particular, applying the logit to the observed binary outcome does not supply a useful observed predictor: the logit of $1$ is infinite and the logit of $0$ is negative infinite. The response residual remains well-defined for either outcome.

Worked Example: Poisson Response With a Log Link

Suppose a count observation has fitted mean $\hat{\mu}_i=2.5$ in a Poisson regression with a log link. Its fitted linear predictor is $$ \hat{\eta}_i=\log(2.5)\approx 0.916. $$ If the observed count is $y_i=5$, the response residual is $$ r_i=5-2.5=2.5. $$

The residual is 2.5 counts above the fitted mean. It is not a difference of 2.5 on the log scale: the fitted predictor is approximately $0.916$, a quantity on another scale. Nor is the residual itself a probability or a ratio. The same definition applies if the observed count is zero: then $r_i=0-2.5=-2.5$.

Why Fitted Residuals Are Not the Errors

The true conditional error $Y_i-\mu_i$ uses the model's true mean, which is generally unknown. The response residual $y_i-\hat{\mu}_i$ substitutes an estimated mean. It is therefore an observable, fitted-model quantity that can be used to examine how each observation compares with its fitted mean; it is not a direct observation of the true error.

This distinction matters because the fitted mean is estimated from data. Fitting can impose relationships among residuals. For example, in a Bernoulli model with an intercept only and a finite maximum-likelihood estimate, the fitted probability is the sample proportion of successes, so the response residuals sum to zero. That constraint is a consequence of fitting the intercept; it does not mean the underlying conditional errors in a new sample must sum to zero.

Worked Example: A Fitting Constraint Is Not a Zero-Error Rule

Consider three Bernoulli observations $y_1=1$, $y_2=0$, and $y_3=1$ in an intercept-only model with success probability $p$. The likelihood is $L(p)=p^2(1-p)$, for $0<p<1$. Its log-likelihood derivative is $$ \frac{d}{dp}\ell(p)=\frac{2}{p}-\frac{1}{1-p}. $$ Setting this equal to zero gives $2(1-p)=p$, hence $\hat{p}=2/3$. The response residuals are $$ r_1=1-\frac{2}{3}=\frac{1}{3},\qquad r_2=0-\frac{2}{3}=-\frac{2}{3},\qquad r_3=1-\frac{2}{3}=\frac{1}{3}. $$ Their sum is $1/3-2/3+1/3=0$, because the fitted probability equals the sample mean. This exact cancellation is a property of this fitted intercept-only model. It does not change the definition of the true errors $Y_i-p$, and it should not be assumed to hold for every GLM fit.

Response Scale and Linear-Predictor Scale

A linear predictor describes the conditional mean through the link; it is not a transformed version of each observed response. The response residual compares $y_i$ with $\hat{\mu}_i$, while $\hat{\eta}_i$ is obtained by applying the link to $\hat{\mu}_i$. For nonlinear links, subtracting the fitted mean and comparing predictor values are not interchangeable operations.

Result: Let $m_i$ be a fixed prediction at covariates $x_i$, and assume $E[Y_i\mid x_i]=\mu_i$ and $\operatorname{Var}(Y_i\mid x_i)=V_i<\infty$. Then $$ E[(Y_i-m_i)^2\mid x_i] =V_i+(\mu_i-m_i)^2. $$

Proof. Write $Y_i-m_i=(Y_i-\mu_i)+(\mu_i-m_i)$. Squaring and taking the conditional expectation gives $$ E[(Y_i-m_i)^2\mid x_i] =E[(Y_i-\mu_i)^2\mid x_i] +2(\mu_i-m_i)E[Y_i-\mu_i\mid x_i] +(\mu_i-m_i)^2. $$ The middle term is zero because $E[Y_i-\mu_i\mid x_i]=0$. The first term is $V_i$, by the definition of conditional variance. Therefore the displayed identity follows. $\square$

This identity separates two contributions: conditional outcome variation $V_i$, and squared difference between the true mean and a fixed prediction. It explains why a large response residual can occur even when a fitted mean is close to the true mean: the outcome itself is random. The fixed-prediction condition matters. An in-sample fitted mean can depend on $Y_i$, so the identity should not be applied to it as though it were fixed without further justification.

Common Mistakes and Practical Interpretation

Calling the response residual the true error. The true error subtracts $\mu_i$; the observed residual subtracts $\hat{\mu}_i$. The latter is a plug-in quantity because the mean is estimated.
Reading residual size without its units. A residual of 2.5 for a count is 2.5 counts above the fitted mean. It cannot be compared directly with a probability-scale residual or a residual in measurement units.
Transforming the outcome as if it were the mean. The link relates $\mu_i$ to $\eta_i$, not each $Y_i$ to $\eta_i$. For discrete responses, transforming the observation may even be undefined or infinite.
Assuming fitted residuals must average to zero in every model. Particular fitting procedures impose particular constraints. A zero sum in an intercept-only example is not a general property of all GLM response residuals.

Key takeaway: A GLM response residual is the observed response minus its fitted conditional mean. It estimates a conditional error in response units, but it is not the unobserved error itself and is not a discrepancy on the linear-predictor scale.

Check Your Understanding

Use the definitions and distinctions from this tutorial to answer the following.

For a Poisson observation $y_i=3$ with fitted mean $\hat{\mu}_i=4.2$, calculate the response residual and state its units.
What is the difference between the conditional error $Y_i-\mu_i$ and the response residual $y_i-\hat{\mu}_i$?
A Bernoulli observation has fitted success probability $0.25$. Find its response residual for $y_i=0$ and for $y_i=1$.
Why is the fitted logit not obtained by taking the logit of an observed binary outcome?
Under what condition on $m_i$ does the stated conditional squared-error decomposition apply directly?