Building Parameter-Dependent GLM Log-Likelihoods

From Observation Distributions to a GLM Log-Likelihood

A GLM likelihood is built from the probability model for each response, not just from the part of a formula that contains the fitted mean. In particular, a term that is constant in the regression coefficients may still depend on a dispersion parameter. Dropping such a term can leave regression estimation unchanged while changing the likelihood for dispersion.

Suppose \(Y_1,\ldots,Y_n\) are independent responses, conditional on fixed design information. For an exponential-dispersion model, write the contribution for observation \(i\) as

f_i(y_i;\theta_i,\phi) = \exp\left\{ \frac{y_i\theta_i-b(\theta_i)}{a_i(\phi)} +c(y_i,\phi;w_i) \right\}.

Here \(\theta_i\) is the canonical parameter, \(b\) is the cumulant function, \(\phi\) is a dispersion parameter when one is present, and \(a_i(\phi)\) may account for a known weight \(w_i\). The term \(c\) contains the remaining part of the density or mass function, including normalizing terms. In the common weighted convention \(a_i(\phi)=\phi/w_i\), so the first term is \(w_i\{y_i\theta_i-b(\theta_i)\}/\phi\). A model with known dispersion can instead set \(\phi\) to its known value.

Definition: For independent responses, the likelihood is the product \(L(\beta,\phi)=\prod_{i=1}^n f_i(y_i;\theta_i(\beta),\phi)\), viewed as a function of the unknown parameters for the observed responses. Its log-likelihood is \(\ell(\beta,\phi)=\sum_{i=1}^n \log f_i(y_i;\theta_i(\beta),\phi)\). The regression structure specifies \(\eta_i=x_i^\mathsf{T}\beta\), \(\mu_i=g^{-1}(\eta_i)\), and the corresponding \(\theta_i\).

A likelihood is defined relative to a common dominating measure, and any parameter-dependent support restrictions must be retained. In the usual GLM examples below, the response support is fixed, so the displayed density or mass function gives the full contribution directly. The key construction is to substitute the model for \(\theta_i\) and \(\phi\) into every observation-level term before deciding whether anything can be omitted.

What Counts as a Parameter-Dependent Term?

In the exponential-dispersion form, the joint log-likelihood is

\ell(\beta,\phi) = \sum_{i=1}^n \frac{y_i\theta_i(\beta)-b\{\theta_i(\beta)\}}{a_i(\phi)} + \sum_{i=1}^n c(y_i,\phi;w_i).

The first sum generally depends on the regression coefficients through \(\theta_i(\beta)\), and it may also depend on \(\phi\) through \(a_i(\phi)\). The second sum may depend on dispersion even when it does not depend on \(\beta\). For a full likelihood, retain every term involving any unknown parameter being estimated. A term may be omitted from a likelihood kernel only when it is constant with respect to all those unknown parameters.

Theorem: Suppose, on a common support, a log-likelihood can be written as \(\ell(\psi)=\ell_K(\psi)+C(y)\), where \(\psi\) is the full vector of unknown parameters and \(C(y)\) does not depend on \(\psi\). Then \(\ell\) and \(\ell_K\) have the same maximizers and the same likelihood ratios between any two parameter values. If the omitted term depends on even one component of \(\psi\), those conclusions are not guaranteed.

Proof. For any two parameter values \(\psi_1,\psi_2\),

\ell(\psi_1)-\ell(\psi_2) = \{\ell_K(\psi_1)+C(y)\}-\{\ell_K(\psi_2)+C(y)\} = \ell_K(\psi_1)-\ell_K(\psi_2).

Thus every ordering of parameter values by log-likelihood is unchanged, so the maximizing sets coincide. Exponentiating the same identity gives \(L(\psi_1)/L(\psi_2)=L_K(\psi_1)/L_K(\psi_2)\). If a term depends on an unknown parameter, it cannot be cancelled this way; it may change those orderings or ratios. \(\square\)

This theorem justifies a likelihood kernel, not an incomplete full likelihood. Whether a term is “constant” must be judged against the parameters in the current problem. For example, a term involving only the observed \(y_i\) is constant in \(\beta\) and \(\phi\); a term involving \(y_i\) and \(\phi\) is not constant when \(\phi\) is unknown.

Worked Likelihoods

Worked Example: Bernoulli Responses With a Logit Link

Let \(Y_i\) be independent Bernoulli responses with success probabilities \(p_i\), and let \(\log\{p_i/(1-p_i)\}=x_i^\mathsf{T}\beta=\eta_i\). The Bernoulli mass function is \(p_i^{y_i}(1-p_i)^{1-y_i}\), for \(y_i\in\{0,1\}\). Since \(p_i=e^{\eta_i}/(1+e^{\eta_i})\), the contribution becomes

\log f_i(y_i) = y_i\eta_i-\log(1+e^{\eta_i}).

Take three design vectors \(x_1=(1,0)^\mathsf{T}\), \(x_2=(1,1)^\mathsf{T}\), \(x_3=(1,2)^\mathsf{T}\), responses \((1,0,1)\), and coefficients \(\beta=(\log 2,-\log 2)^\mathsf{T}\). Then the linear predictors are \(\log 2,0,-\log 2\), giving probabilities \(2/3,1/2,1/3\). The likelihood and log-likelihood are

L(\beta)=\frac23\left(1-\frac12\right)\frac13=\frac19, \qquad \ell(\beta)=\log(1/9)=-\log 9.

The same result follows term by term: the three log contributions are \(\log(2/3)\), \(-\log 2\), and \(-\log 3\); their sum is \(-\log 9\). There is no omitted factorial or dispersion term in this Bernoulli likelihood.

Worked Example: Poisson Regression and Factorial Terms

For independent \(Y_i\sim\operatorname{Poisson}(\mu_i)\), use the log link \(\eta_i=\log\mu_i=x_i^\mathsf{T}\beta\). The Poisson mass function gives the full contribution

\log f_i(y_i) = y_i\eta_i-e^{\eta_i}-\log(y_i!).

Consider \(y=(0,2,1)\), design vectors \(x_1=(1,1)^\mathsf{T}\), \(x_2=(1,0)^\mathsf{T}\), \(x_3=(1,-1)^\mathsf{T}\), and \(\beta=(0,\log 2)^\mathsf{T}\). The predictors are \((\log 2,0,-\log 2)\), so the fitted means are \((2,1,1/2)\). The full log-likelihood is

\begin{aligned} \ell(\beta) &= \{0\log 2-2-\log(0!)\} +\{2(0)-1-\log(2!)\}\\ &\quad+\{1(-\log 2)-\tfrac12-\log(1!)\}\\ &=-2-1-\log 2-\log 2-\tfrac12\\ &=-\tfrac72-2\log 2. \end{aligned}

Here \(0!=1\) and \(1!=1\), so their log terms are zero; \(2!=2\), so its log term is \(\log 2\). With the observed counts fixed, \(-\sum_i\log(y_i!)=-\log 2\) is constant in \(\beta\), and may be omitted from a regression likelihood kernel. It belongs in the full likelihood.

Worked Example: Grouped Binomial Counts

Suppose \(Y_i\sim\operatorname{Binomial}(m_i,p_i)\), independently, where the number of trials \(m_i\) is fixed and \(\operatorname{logit}(p_i)=\eta_i\). The probability mass function is \(\binom{m_i}{y_i}p_i^{y_i}(1-p_i)^{m_i-y_i}\). Substitution gives

\log f_i(y_i) = \log\binom{m_i}{y_i} +y_i\eta_i-m_i\log(1+e^{\eta_i}).

For one group, let \(m=3\), \(y=2\), and \(p=0.4\), so \(\eta=\log(0.4/0.6)=\log(2/3)\). The full probability is

\binom32(0.4)^2(0.6)=3(0.16)(0.6)=0.288, \qquad \log f=\log(0.288).

The parameter-free combinatorial contribution is \(\log\binom32=\log 3\); the parameter-dependent portion is \(2\log(2/3)-3\log(1+2/3)\). Since \(\log(1+2/3)=\log(5/3)\), adding the terms gives \(\log 3+2\log(2/3)-3\log(5/3)=\log(0.288)\). If the \(m_i\) and observed counts are fixed, the combinatorial terms can be omitted when estimating regression coefficients. They are still part of the full likelihood.

Worked Example: Gaussian Regression With Unknown Variance

Let \(Y_i\) be independent \(N(x_i^\mathsf{T}\beta,\sigma^2)\) responses. The normal density yields

\ell(\beta,\sigma^2) = -\frac n2\log(2\pi) -\frac n2\log(\sigma^2) -\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-x_i^\mathsf{T}\beta)^2.

For two observations \(y=(1,3)\) and an intercept-only mean \(\beta\), set \(\beta=2\) and \(\sigma^2=1\). The residuals are \(-1\) and \(1\), so their squared sum is \(2\). Therefore

\ell(2,1) = -\log(2\pi)-\frac12(2) = -\log(2\pi)-1.

The term \(-\log(2\pi)\) is constant in both \(\beta\) and \(\sigma^2\) and may be omitted from a kernel. By contrast, \(-\log(\sigma^2)\) depends on the unknown variance and must remain when estimating it. The residual term also depends on both parameters. Keeping only the residual sum of squares would not give the full likelihood for \((\beta,\sigma^2)\).

A Practical Construction and a Common Pitfall

A reliable construction separates the model substitution from the decision about constants. First write the observation-level mass function or density, then express its parameters using the GLM mean and regression predictor. Only after summing the log contributions should terms be classified relative to the unknown parameters of interest.

Write each response contribution.
Use the stated distribution, including its normalizing factor and support.

Insert the regression structure.
Replace the mean or canonical parameter using \(\eta_i=x_i^\mathsf{T}\beta\) and the specified link.

Multiply, then take logarithms.
Independence makes the joint likelihood a product and the log-likelihood a sum.

Check every unknown parameter.
Retain terms depending on any parameter being estimated; omit only terms independent of all of them when a kernel is sufficient.

Common mistake: “The factorial, combinatorial, or normalizing term does not involve \(\beta\), so it can always be dropped.” This reasoning is valid only for a likelihood kernel when the term is independent of every unknown parameter under consideration. In Gaussian regression, \(-n\log(\sigma^2)/2\) does not involve \(\beta\), but it does involve the unknown dispersion \(\sigma^2\), so it cannot be omitted from the joint likelihood for regression and variance. Likewise, an exponential-family term \(c(y_i,\phi;w_i)\) must be retained if it depends on an unknown \(\phi\).

Key takeaway: Construct the full log-likelihood first. A term is safely removable from a likelihood kernel only if it is constant in the entire unknown-parameter vector—not merely in the regression coefficients.

Check Your Understanding

For each question, distinguish the full likelihood from a likelihood kernel and name the parameters involved.

In a Poisson regression with fixed observed counts, which part of the log-likelihood can be omitted from a kernel for the regression coefficients, and why?
For grouped binomial data with fixed trial counts, identify the combinatorial term and state when it can be omitted.
In a Gaussian GLM with unknown variance, explain why a term that does not involve the regression coefficients may still need to be retained.
State the condition under which adding a term to a log-likelihood preserves its maximizers and likelihood ratios.
Why should parameter-dependent support restrictions be checked before treating a factor as a harmless constant?