Likelihood Geometry in Generalized Linear Models

From Regression Coefficients to a Likelihood

A generalized linear model (GLM) turns a vector of regression coefficients into a probability distribution for each response. It does so in stages: the coefficients determine linear predictors, the link function relates those predictors to response means, and the specified response family turns the means (and any dispersion parameters) into distributions. Understanding these stages clarifies both what the likelihood depends on and what is meant by the geometry of a GLM.

Throughout, regard the covariates as fixed and suppose there are $n$ independent observations. Write $y_i$ for the observed response and $x_i\in\mathbb{R}^p$ for its design vector. The regression coefficient vector is $\beta\in\mathbb{R}^p$. A dispersion parameter or other distributional parameters may also be present; these are distinct from $\beta$.

Definition: A generalized linear model specifies independent responses $Y_1,\ldots,Y_n$, a response distribution from a stated family, and a mean structure $$ \eta_i=x_i^{\mathsf T}\beta,\qquad g(\mu_i)=\eta_i,\qquad \mu_i=E[Y_i], $$ where $\eta_i$ is the linear predictor, $g$ is the link function, and $\mu_i=g^{-1}(\eta_i)$ when $g$ is invertible on the relevant mean domain. The likelihood is the joint probability mass or density of the observed responses, viewed as a function of the model parameters.

The link acts on the mean, not directly on an observed response. For example, the logit link is $g(\mu)=\log\{\mu/(1-\mu)\}$, so its inverse maps any real predictor to a mean between zero and one. A link need not be the canonical link for the chosen response family. The response distribution and link together determine how the regression coefficients shape the likelihood.

A common likelihood-based specification uses independent observations from a one-parameter exponential-dispersion family. In one standard parameterization, its density or mass function has the form

f(y_i;\theta_i,\phi) = \exp\left\{ \frac{y_i\theta_i-b(\theta_i)}{a(\phi)} +c(y_i,\phi) \right\}, \qquad \mu_i=b'(\theta_i), \qquad \operatorname{Var}(Y_i)=a(\phi)b''(\theta_i).

Here $\theta_i$ is the distribution's natural parameter and $\phi$ is a dispersion parameter; $a$, $b$, and $c$ are specified by the family. This representation is useful, but the essential GLM structure is the response family together with the predictor and link. Not every GLM uses an unknown dispersion: for Bernoulli and Poisson models it is conventionally fixed.

The Independent-Observation Likelihood

Let $f_i(y_i;\beta,\phi)$ denote the mass function or density of observation $i$, after substituting its predictor and mean structure. Independence gives a product likelihood. If the response family has additional parameters, collect them with $\phi$ as nuisance parameters, say $\lambda$. Their inclusion does not change the definition of the regression predictor.

Result: For independent responses with densities or mass functions $f_i$ relative to their stated dominating measures, the likelihood is $$ L(\beta,\lambda;y) = \prod_{i=1}^n f_i(y_i;\beta,\lambda), \qquad \ell(\beta,\lambda;y) = \sum_{i=1}^n \log f_i(y_i;\beta,\lambda), $$ where each $f_i$ depends on $\beta$ through $\eta_i=x_i^{\mathsf T}\beta$ and $\mu_i=g^{-1}(\eta_i)$.

To verify the result, independence makes the joint distribution the product of the individual distributions. Consequently, its density or mass function at the observed vector $y=(y_1,\ldots,y_n)$ is $\prod_i f_i(y_i;\beta,\lambda)$. Taking the logarithm of this positive product gives $\log L=\sum_i\log f_i$, as stated. This factorization is why independent-observation GLMs can be described through a sequence of observation-level contributions.

Worked Example: A Bernoulli Logit Likelihood

Suppose $Y_1,Y_2$ are independent Bernoulli responses with observed values $y_1=y_2=1$. Use the logit link and design vectors $x_1=(1,0)^{\mathsf T}$, $x_2=(0,1)^{\mathsf T}$, with $\beta=(1,\log 3-1)^{\mathsf T}$. Find both means and the likelihood at this coefficient vector.

The predictors are $\eta_1=x_1^{\mathsf T}\beta=1$ and $\eta_2=x_2^{\mathsf T}\beta=\log 3-1=\log(3/e)$. The inverse logit is $g^{-1}(\eta)=e^\eta/(1+e^\eta)$, so

\mu_1=\frac{e}{1+e}\approx 0.7311, \qquad \mu_2=\frac{3/e}{1+3/e}=\frac{3}{e+3}\approx 0.5246.

For a Bernoulli observation with value one, the likelihood contribution is its success probability. Thus

L(\beta;y) = \mu_1\mu_2 = \frac{e}{1+e}\frac{3}{e+3} \approx (0.7311)(0.5246) \approx 0.3835.

The two coefficients determine the predictors and therefore the success probabilities. There is no free dispersion parameter in this Bernoulli model.

Likelihood Geometry: What Varies With the Coefficients?

For fixed design vectors and a fixed response family, the model can be viewed as a map from parameter values to joint distributions. First, $\beta$ maps to the predictor vector $X\beta$, where $X$ has rows $x_i^{\mathsf T}$. Next, the inverse link maps those predictors to the mean vector $(\mu_1,\ldots,\mu_n)$. Finally, the response-family specification maps the means, together with any dispersion or nuisance parameters, to the individual distributions and their joint likelihood.

The set of distributions obtained by varying $\beta$ is the regression model inside the larger collection of distributions allowed by the response family. The predictors are linear in $\beta$, but the means need not be: a nonlinear inverse link generally bends the map from coefficients to means. The resulting family of distributions is what is meant here by the likelihood geometry of a GLM. This description does not assert that a maximum exists or that the likelihood has any particular curvature.

Worked Example: A Poisson Log-Link Model

Let three independent responses have Poisson distributions with log link. Take $x_1=(1,0)^{\mathsf T}$, $x_2=(1,1)^{\mathsf T}$, $x_3=(1,2)^{\mathsf T}$, $\beta=(0,\log 2)^{\mathsf T}$, and observed values $(y_1,y_2,y_3)=(2,1,0)$. Calculate the predictors, means, and likelihood.

For the log link, $\log(\mu_i)=\eta_i$, so $\mu_i=e^{\eta_i}$. The predictors and means are

(\eta_1,\eta_2,\eta_3)=(0,\log 2,2\log 2), \qquad (\mu_1,\mu_2,\mu_3)=(1,2,4).

A Poisson response with mean $\mu_i$ has mass $e^{-\mu_i}\mu_i^{y_i}/y_i!$. Substituting the three observed values gives

\begin{aligned} L(\beta;y) &= \left(e^{-1}\frac{1^2}{2!}\right) \left(e^{-2}\frac{2^1}{1!}\right) \left(e^{-4}\frac{4^0}{0!}\right)\\ &= \left(\frac{e^{-1}}{2}\right)(2e^{-2})(e^{-4}) =e^{-7} \approx 0.0009119. \end{aligned}

The coefficients determine all three means through the design and link. The Poisson model fixes its dispersion conventionally, so there is no additional free dispersion parameter in this likelihood.

Proposition: Suppose $\beta$ ranges over $\mathbb{R}^p$, the link is one-to-one on the mean domain, and, for fixed nuisance parameters, the response distribution at each observation is uniquely determined by its mean. Then the regression coefficient is identifiable from the joint distribution if and only if the design matrix $X$ has column rank $p$.

To prove the forward implication needed for identifiability, suppose two coefficient vectors $\beta$ and $\tilde{\beta}$ give the same joint distribution. Their marginal distributions at each observation are therefore equal. By the assumption that a mean uniquely determines each distribution, their mean vectors are equal. Since the link is one-to-one, equal means imply equal predictors, so $X\beta=X\tilde{\beta}$. If $X$ has column rank $p$, its null space contains only zero; hence $X(\beta-\tilde{\beta})=0$ implies $\beta=\tilde{\beta}$.

Conversely, if $X$ does not have column rank $p$, there is a nonzero vector $v$ such that $Xv=0$. For any $\beta$ and any real $t$, $X(\beta+tv)=X\beta+tXv=X\beta$. Thus $\beta$ and $\beta+tv$ give the same predictors, means, and distributions, although the coefficient vectors differ. The regression coefficient is not identifiable. $\square$

This result concerns identifiability with the nuisance parameters held fixed. It does not say that an estimate exists, or that estimating nuisance parameters jointly cannot introduce further complications. It isolates a basic geometric feature: redundant design columns create directions in coefficient space that leave the entire model distribution unchanged.

Worked Example: Gaussian Means and a Separate Dispersion Parameter

Let $Y_1,Y_2$ be independent normal responses with means $x_i^{\mathsf T}\beta$ and common variance $\sigma^2$. Take $x_1=(1,0)^{\mathsf T}$, $x_2=(1,1)^{\mathsf T}$, $\beta=(1,2)^{\mathsf T}$, observed values $y_1=1,y_2=3$, and $\sigma^2=4$. Write the likelihood at these parameter values.

The identity link gives $\mu_i=\eta_i$. Here $\eta_1=1$ and $\eta_2=3$, so each observed response equals its specified mean. The normal density with variance $4$ is

f(y_i;\mu_i,4) = \frac{1}{\sqrt{8\pi}} \exp\left\{-\frac{(y_i-\mu_i)^2}{8}\right\}.

Each residual is zero, making each exponential factor equal to one. Independence therefore gives

L(\beta,\sigma^2;y) = \left(\frac{1}{\sqrt{8\pi}}\right)^2 = \frac{1}{8\pi} \approx 0.03979.

The vector $\beta$ controls the two means; $\sigma^2$ controls the response spread around those means. If the inferential target is $\beta$, the variance is a nuisance parameter, even though it is part of the likelihood and may itself be unknown. Calling it a nuisance parameter does not mean it can be omitted from the probability model.

Regression, Dispersion, and Other Nuisance Parameters

The distinction among parameter roles is easiest to keep straight by asking what feature of the distribution each parameter controls. Regression coefficients enter the systematic mean structure through $x_i^{\mathsf T}\beta$. A dispersion parameter typically governs variability around that structure. Additional nuisance parameters may control shape or other distributional features. Which parameters are nuisance depends on the inferential target, not on whether they appear in the likelihood.

Parameter role	What it controls in the model	Illustration
Regression coefficient $\beta$	Predictors and, through the link, the observation means	Logit regression changes Bernoulli success probabilities
Dispersion parameter $\phi$	Often the variance scale, in addition to the mean structure	Gaussian variance $\sigma^2$
Other nuisance parameter	A distributional feature not itself the target of inference	A shape parameter in a response-family specification

A common mistake is to say that a parameter is “not in the model” because it is not a regression coefficient. That is wrong: an unknown dispersion parameter still affects the response distribution and therefore the likelihood. The correction is to distinguish the mean model, governed by the predictor, link, and $\beta$, from the full likelihood model, which also includes dispersion and any other unknown distributional parameters.

A second pitfall is applying the link to each observed $y_i$ and treating $g(y_i)$ as the linear predictor. The model instead specifies $g(E[Y_i])=x_i^{\mathsf T}\beta$. In general, $g(E[Y_i])$ is not the same quantity as $g(y_i)$; the observed value enters the likelihood after the distribution has been specified.

Key takeaway: A GLM likelihood is built by mapping coefficients to predictors, predictors through a link to means, and means plus dispersion or nuisance parameters to response distributions. Keeping these parameter roles separate makes the model's structure and identifiability easier to assess.

Check Your Understanding

Use the definitions and examples above to check that you can move between a GLM's parameters and its likelihood.

For independent observations, why is the joint likelihood a product of observation-level likelihood contributions?
In a GLM, what does the link function act on, and how is that quantity related to the linear predictor?
Explain why an unknown Gaussian variance can be a nuisance parameter while still belonging to the full likelihood.
What property of the design matrix ensures that different coefficient vectors cannot produce the same predictor vector?
In the Poisson example, identify which quantities were determined by $\beta$ and which distributional feature was fixed by the response family.

Parameter role	What it controls in the model	Illustration
Regression coefficient \(\beta\)	Predictors and, through the link, the observation means	Logit regression changes Bernoulli success probabilities
Dispersion parameter \(\phi\)	Often the variance scale, in addition to the mean structure	Gaussian variance \(\sigma^2\)
Other nuisance parameter	A distributional feature not itself the target of inference	A shape parameter in a response-family specification

Likelihood Geometry in Generalized Linear Models

What You'll Learn