From an Exponential-Family Model to a GLM Likelihood
A GLM coefficient is chosen by fitting the model to the observed responses. The likelihood makes that fitting principle precise: it measures how plausible the observed data are under a proposed set of coefficients. Differentiating its logarithm gives the score equations, which describe the conditions that an interior maximum-likelihood estimate must satisfy.
This tutorial derives those equations from the exponential-family form and connects them to fitted means and residuals. The derivation assumes independent observations and a correctly specified response distribution. No interest-rate or mortality assumptions are involved; the examples specify their response distributions and parameters directly.
The exponential-family representation gives \(E[Y_i]=b'(\theta_i)\) and \(\operatorname{Var}(Y_i)=a(\phi)b''(\theta_i)\). These identities connect the response distribution to the GLM mean and variance functions. The likelihood for independent observations is the product of their individual likelihoods; taking logarithms turns that product into a sum.
The term \(c(y_i,\phi)\) does not depend on \(\boldsymbol{\beta}\) when \(\phi\) is fixed. It must be included in the likelihood itself, but it contributes zero to the coefficient score. The parameter \(\theta_i\) depends on \(\boldsymbol{\beta}\) through the mean and link, so differentiating requires the chain rule.
Deriving the Score for Regression Coefficients
Let \(\mathbf{x}_i\) be the column vector of predictor values for observation \(i\), including an intercept if the model has one. First differentiate the \(i\)th log-likelihood contribution with respect to \(\theta_i\). Then use the dependence of \(\theta_i\) on \(\eta_i\), and of \(\eta_i\) on \(\boldsymbol{\beta}\).
The derivative of \(y_i\theta_i-b(\theta_i)\) with respect to \(\theta_i\) is \(y_i-b'(\theta_i)=y_i-\mu_i\).
Since \(d\mu_i/d\theta_i=b''(\theta_i)=V(\mu_i)\) and \(d\mu_i/d\eta_i=1/g'(\mu_i)\), the chain rule gives \(d\theta_i/d\eta_i=1/[V(\mu_i)g'(\mu_i)]\).
For \(\eta_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}\), \(d\eta_i/d\boldsymbol{\beta}=\mathbf{x}_i\).
Combining these derivatives gives the score vector. In the expression, \(V(\mu_i)=b''(\theta_i)\) is the variance function, with \(\operatorname{Var}(Y_i)=a(\phi)V(\mu_i)\). The link must be differentiable at the fitted mean, and the mean must lie in the range where the inverse link and the corresponding canonical parameter are defined.
For the canonical link, \(\theta_i=\eta_i\), so \(d\theta_i/d\eta_i=1\). The relationship above then implies \(V(\mu_i)g'(\mu_i)=1\). The score simplifies to a predictor-weighted sum of response residuals:
The equations say that the residuals are orthogonal to each predictor column in the design matrix. With an intercept, the residuals sum to zero. With an additional predictor, their predictor-weighted sum is also zero. This does not generally mean that every fitted mean equals its observed response; it means the residuals satisfy these aggregate conditions.
Worked Likelihoods and Score Equations
Worked Example: Grouped Binomial Data With an Intercept
A hypothetical portfolio is divided into two groups. In group A, 3 of 5 policies have the event of interest; in group B, 1 of 3 policies have it. Assume independent Bernoulli trials with a common event probability \(p\). Fit an intercept-only logistic GLM. No interest-rate or mortality assumptions enter this model; the response counts are binomial, with trial counts 5 and 3 and common probability \(p\).
Method. For a grouped binomial count \(Y_j\sim\operatorname{Binomial}(m_j,p)\), the likelihood contribution is proportional to \(p^{y_j}(1-p)^{m_j-y_j}\). The common intercept \(\beta_0\) has \(p=\exp(\beta_0)/[1+\exp(\beta_0)]\). The logistic link is canonical for the binomial family.
The total number of events is \(3+1=4\), and the total number of trials is \(5+3=8\). Ignoring the binomial coefficients, which do not depend on \(p\), the log-likelihood is \(\ell(p)=4\log(p)+4\log(1-p)\). Since \(dp/d\beta_0=p(1-p)\), differentiation gives \(d\ell/d\beta_0=4(1-p)-4p=4-8p\).
Set the score to zero: \(4-8p=0\), so \(\widehat p=4/8=0.5\). The intercept estimate is \(\widehat\beta_0=\log[\widehat p/(1-\widehat p)]=\log(1)=0\). At this estimate, the score is \(4-8(0.5)=0\), as required.
Check. The second derivative with respect to \(\beta_0\) is \(-8p(1-p)\), which is negative at \(p=0.5\). Thus the log-likelihood is locally concave there, consistent with a maximum. The common mistake is to average the two group proportions equally: \((3/5+1/3)/2\approx0.4667\). That incorrectly gives the groups equal weight despite their different trial counts. The correct pooled estimate is the total events divided by total trials, \(4/8=0.5\).
Worked Example: Two-Predictor Poisson GLM
For a hypothetical count study, observe independent counts \(1,3\) at \(x=0\), and \(2,6\) at \(x=1\). Assume \(Y_i\sim\operatorname{Poisson}(\mu_i)\), with log link \(\log(\mu_i)=\beta_0+\beta_1x_i\). The Poisson dispersion is fixed at 1, and the canonical parameter is \(\theta_i=\log(\mu_i)\).
For a Poisson response, \(b(\theta_i)=e^{\theta_i}\) and \(a(\phi)=1\). Therefore, the canonical-link score equations are \(\sum_i(y_i-\mu_i)=0\) and \(\sum_i x_i(y_i-\mu_i)=0\). The second equation contains only observations with \(x_i=1\), so it requires their fitted means to sum to their observed counts: \(2\mu_1=2+6=8\). Hence their common fitted mean is \(\mu_1=4\).
The first score equation requires the fitted means across all four observations to sum to \(1+3+2+6=12\). The two observations at \(x=1\) account for \(2(4)=8\), leaving 4 for the two observations at \(x=0\). Thus \(\mu_0=4/2=2\).
From \(\log(\mu_0)=\beta_0\), \(\widehat\beta_0=\log(2)\). From \(\log(\mu_1)=\beta_0+\beta_1=\log(4)\), \(\widehat\beta_1=\log(4)-\log(2)=\log(2)\). The fitted means are 2 at \(x=0\) and 4 at \(x=1\).
Check. The intercept score is \(1+3+2+6-[2+2+4+4]=12-12=0\). The slope score is \(2+6-[4+4]=0\). The common mistake is to set each individual residual to zero; with repeated observations at each predictor value, the score requires the group residual sums to be zero, not every residual separately.
Worked Example: Gaussian GLM With an Identity Link
A hypothetical response study records \((x,y)=(0,1),(1,2),(2,4)\). Assume independent \(Y_i\sim N(\mu_i,1)\), so the variance \(\sigma^2=1\) is known. Use the Gaussian identity-link model \(\mu_i=\beta_0+\beta_1x_i\). For this family the canonical parameter is \(\theta_i=\mu_i\), and the identity link is canonical.
With known variance, maximizing the Gaussian likelihood is equivalent to minimizing \(\sum_i(y_i-\mu_i)^2\). The canonical score equations are \(\sum_i(y_i-\mu_i)=0\) and \(\sum_i x_i(y_i-\mu_i)=0\). Write \(r_i=y_i-\beta_0-\beta_1x_i\) for each residual. The first equation gives \(7-3\beta_0-3\beta_1=0\); the second gives \(10-3\beta_0-5\beta_1=0\).
Subtract the first equation from the second: \(3-2\beta_1=0\), so \(\widehat\beta_1=3/2\). Substitute this into \(7-3\beta_0-3\beta_1=0\): \(7-3\beta_0-9/2=0\), so \(3\beta_0=5/2\) and \(\widehat\beta_0=5/6\).
The fitted means are \(5/6\), \(7/3\), and \(23/6\). The residuals are \(1/6\), \(-1/3\), and \(1/6\). Their sum is \(1/6-1/3+1/6=0\), and their predictor-weighted sum is \(0(1/6)+1(-1/3)+2(1/6)=0\).
Check. Both score equations are satisfied. The common mistake is to use only the intercept equation and conclude that the residuals sum to zero is enough to estimate both coefficients. The slope equation is also required. Here the residual sum of squares is \(1/36+1/9+1/36=1/6\); with the stated variance, the log-likelihood is \(-\frac{3}{2}\log(2\pi)-\frac{1}{12}\).
What the Score Equations Do—and Do Not—Establish
The derivation explains why GLM coefficient estimates are tied to residual patterns. For canonical links, every predictor column has zero inner product with the response residual vector at an interior solution. For noncanonical links, the residuals instead receive factors depending on the variance function and the derivative of the link. In either case, the score follows from the likelihood and the specified model; it is not a separate fitting rule that can be chosen independently.
Solving the score equations is often algebraically possible for simple intercept-only or saturated examples. In more general models, the equations are nonlinear in the coefficients and may not have a closed-form solution. A numerical method is then used to find a coefficient vector with score near zero. The score condition is a stopping or checking criterion, but it does not alone establish that the result is the global maximum. One must also consider whether a finite estimate exists and whether the likelihood has the appropriate curvature.
- Keep the model assumptions visible. The product likelihood and sum log-likelihood rely on independent responses. Dependence changes the likelihood construction.
- Separate the coefficient score from constants. Terms independent of the coefficients disappear when differentiating for \(\boldsymbol{\beta}\); terms depending on the coefficients do not.
- Do not confuse residual balance with exact fit. Score equations constrain sums or weighted sums of residuals. They do not usually make each residual zero.
- Use the correct likelihood for the response structure. Grouped binomial counts carry trial counts; treating group proportions as equally weighted can change the estimate.
- Check whether the solution is interior. A zero score is a necessary condition for a finite interior maximum, not a guarantee of existence or global optimality.
Check Your Understanding
For each problem, identify the response distribution and use its likelihood score to obtain the requested estimate or check.
- Six independent Bernoulli responses contain 3 successes. Fit an intercept-only logistic model. Find the fitted probability and intercept. (Answer: \(\widehat p=0.5\), \(\widehat\beta_0=0\).)
- Four independent Poisson counts are \(0,2,4,2\). Fit an intercept-only log-link model. Write the score equation and find the fitted mean and intercept. (Answer: \(8-4\mu=0\), \(\widehat\mu=2\), \(\widehat\beta_0=\log 2\).)
- Two independent binomial groups have \(y_1=6\) successes out of \(m_1=8\) trials and \(y_2=3\) out of \(m_2=4\) trials. Under a common-probability intercept-only logistic model, find the score equation and fitted probability. (Answer: \(9-12p=0\), \(\widehat p=0.75\).)
- Assume independent \(Y_i\sim N(\mu_i,4)\) with known variance 4 and identity link \(\mu_i=\beta_0+\beta_1x_i\). For \((x,y)=(0,2),(1,3),(2,5)\), find the coefficient estimates and verify both residual score equations. (Answer: \(\widehat\beta_0=11/6\), \(\widehat\beta_1=3/2\); residuals \(1/6,-1/3,1/6\) have sum zero and predictor-weighted sum zero.)