Regularity Conditions for GLM Maximum Likelihood

Why Regularity Conditions Matter

A likelihood can be written down without being smooth, identifiable, or well behaved near the parameter value that generated the data. Large-sample likelihood arguments need conditions that prevent these problems. In a generalized linear model (GLM), the relevant checks concern the response distribution, the link and mean functions, the parameter space, the design, and—when it is estimated—the dispersion parameter.

These conditions are not a claim that every GLM must satisfy one universal checklist. They are a practical set of sufficient conditions for the standard likelihood calculations used in regular large-sample theory. They also clarify what must be checked rather than assumed.

A GLM Likelihood and Its Parameter

For independent responses \(Y_1,\ldots,Y_n\), a common exponential-dispersion representation is

f_i(y_i;\beta,\phi) = \exp\left\{ \frac{y_i\theta_i-b(\theta_i)}{a_i(\phi)} +c(y_i,\phi) \right\}.

Here \(\theta_i\) is the natural parameter, \(\mu_i=E[Y_i]=b'(\theta_i)\), and \(\operatorname{Var}(Y_i)=a_i(\phi)b''(\theta_i)\). The GLM relates the mean to a linear predictor, usually \(g(\mu_i)=\eta_i=x_i^\mathsf{T}\beta\), possibly with a known offset. The coefficient vector is \(\beta\); \(\phi\) is a dispersion parameter when the response family includes one. For some families, such as Bernoulli and Poisson, dispersion is fixed rather than estimated.

Write \(\gamma\) for the full parameter vector, containing \(\beta\) and any unknown dispersion parameter. A regular likelihood analysis is local to a true parameter value \(\gamma_0\). For fixed-design models, the design vectors are treated as fixed. For random-design models, conditions are imposed on the joint distribution of covariates and responses, or on the conditional response model given the covariates.

Definition: A regular parameter value \(\gamma_0\) lies in an open part of the parameter space where the likelihood is defined and differentiable in a neighborhood of \(\gamma_0\). Regularity conditions also require a parameter-independent response support, sufficient integrability of likelihood derivatives, and locally informative parameterization. These are sufficient conditions, not a definition of every possible regular model.

The Main Conditions and Their Meaning

The following conditions are commonly checked before applying standard large-sample likelihood results.

Independent sampling and a defined likelihood. The likelihood is the product of the specified response densities or mass functions, conditional on any fixed design. The assumed distribution must describe the sampling regime being analyzed.
Parameter-independent support. The set of possible response values does not change with \(\gamma\). For example, the support of a Bernoulli response is \(\{0,1\}\) for every probability strictly between zero and one. When the support changes with the parameter, differentiating a likelihood integral can require additional boundary terms.
Differentiability and domination. The log density is sufficiently differentiable in a neighborhood of \(\gamma_0\). To interchange differentiation and integration, the density derivatives must be dominated locally by integrable functions, or another valid differentiation-under-the-integral condition must hold. Merely having a pointwise derivative is not enough.
Interior parameter. The true coefficient vector is finite and lies in the interior of its allowed parameter space. An unknown dispersion is also strictly inside its allowed range; for example, \(\phi_0>0\) for a variance parameter. Boundary values can produce nonstandard likelihood behavior.
Identifiability and information. Distinct parameter values should not generate the same response distribution, at least in the neighborhood relevant to inference. In addition, the Fisher information for the parameter being estimated should be nonsingular. Global identifiability and nonsingular local information are related but distinct checks.
Finite moments of derivatives. Score components should have finite second moments, and the likelihood derivatives used in a particular result must be integrable. Asymptotic arguments also need control of sums of scores and Hessians, commonly through finite moments and a suitable law of large numbers or central limit theorem.

For fixed designs whose size grows, a typical information condition is that the smallest eigenvalue of the information matrix increases at the appropriate rate, rather than remaining bounded. For random designs, a typical condition is a finite, positive-definite expected information matrix, together with moments sufficient for the relevant limit theorems. The exact scaling depends on the sampling regime; the essential point here is that information must not vanish in a parameter direction.

Two Likelihood Facts Under Regularity Conditions

The first result explains why the score has mean zero. Let \(f(y;\gamma)\) be a density or mass function with support not depending on \(\gamma\). Suppose that, in a neighborhood of \(\gamma_0\), differentiation under the integral or sum is valid, \(\int f(y;\gamma)\,dy=1\), and the score \(s(Y;\gamma)=\partial \log f(Y;\gamma)/\partial\gamma\) is integrable. For discrete responses, replace the integral by a sum.

Result: Under these conditions, \(E_{\gamma_0}[s(Y;\gamma_0)]=0\).

Proof. Since \(f\) integrates to one for every \(\gamma\), differentiation at \(\gamma_0\) gives

0 = \left.\frac{\partial}{\partial\gamma} \int f(y;\gamma)\,dy\right|_{\gamma=\gamma_0} = \int \left.\frac{\partial f(y;\gamma)}{\partial\gamma}\right|_{\gamma=\gamma_0}dy.

Where \(f(y;\gamma_0)>0\), the derivative satisfies \(\partial f/\partial\gamma=f\,\partial\log f/\partial\gamma\). Thus the last integral equals \(E_{\gamma_0}[s(Y;\gamma_0)]\), proving the result. The support and domination conditions justify the differentiation and the integral identity. \(\square\)

A second useful fact links score variability to the curvature of the log likelihood. Suppose the same differentiation conditions hold through second order, the second derivatives are integrable, and the score has finite second moments. For a vector parameter, write \(H(Y;\gamma)=\partial^2\log f(Y;\gamma)/\partial\gamma\partial\gamma^\mathsf{T}\).

Result: At \(\gamma_0\), the Fisher information can be written in either form \(I(\gamma_0)=E_{\gamma_0}[ss^\mathsf{T}]=-E_{\gamma_0}[H]\).

Proof. Differentiate the normalization identity twice. The \((j,k)\) entry gives

0 = \int \frac{\partial^2 f}{\partial\gamma_j\partial\gamma_k}\,dy = \int f\left[ \frac{\partial^2\log f}{\partial\gamma_j\partial\gamma_k} + \frac{\partial\log f}{\partial\gamma_j} \frac{\partial\log f}{\partial\gamma_k} \right]dy,

where all terms are evaluated at \(\gamma_0\). The identity follows entry by entry. The finite-moment and domination assumptions ensure these expectations exist and the differentiation is valid. \(\square\)

What the Conditions Look Like in GLMs

For a canonical-link GLM with known dispersion, \(\theta_i=x_i^\mathsf{T}\beta\), and the log likelihood, apart from terms not involving \(\beta\), is

\ell(\beta) = \sum_{i=1}^n \frac{y_i x_i^\mathsf{T}\beta-b(x_i^\mathsf{T}\beta)}{a_i(\phi)}.

If \(b\) is twice differentiable on an open interval containing every true natural parameter, the coefficient score and expected information are

U(\beta) = \sum_{i=1}^n \frac{x_i(y_i-\mu_i)}{a_i(\phi)}, \qquad I(\beta) = \sum_{i=1}^n \frac{b''(\theta_i)}{a_i(\phi)}x_ix_i^\mathsf{T}.

These expressions follow by differentiating the displayed log likelihood and using \(b'(\theta_i)=\mu_i\). The variance relation gives the information expression as the score covariance; the preceding result gives the equivalent expected negative Hessian. In this canonical case, finite coefficient-score second moments follow if the displayed information entries are finite. For a noncanonical link, differentiability of the link and inverse mean map is also needed, and derivatives of the resulting score must have the required moments.

Worked Example: Regularity Checks for Logistic Regression

Let \(Y_i\) be Bernoulli with \(P(Y_i=1)=p_i\), and let \(\operatorname{logit}(p_i)=x_i^\mathsf{T}\beta\). Take two fixed design vectors \(x_1=(1,0)^\mathsf{T}\), \(x_2=(1,1)^\mathsf{T}\), and evaluate the conditions at \(\beta_0=(0,\log 3)^\mathsf{T}\).

The resulting probabilities are \(p_1=1/2\) and \(p_2=3/4\), so both are strictly between zero and one. The response support is \(\{0,1\}\), independent of \(\beta\). The log likelihood is a smooth function of every finite \(\beta\), and \(\beta_0\) is an interior point of \(\mathbb{R}^2\). Bernoulli scores are bounded for these fixed design vectors, so their second moments exist.

For logistic regression, the information is \(\sum_i p_i(1-p_i)x_ix_i^\mathsf{T}\). Here,

I(\beta_0) = \begin{pmatrix} 1/4+3/16 & 3/16\\ 3/16 & 3/16 \end{pmatrix} = \begin{pmatrix} 7/16 & 3/16\\ 3/16 & 3/16 \end{pmatrix}.

Its determinant is \((21-9)/256=3/64>0\), and its first diagonal entry is positive. Therefore it is positive definite. This checks local information for these two coefficients at this design and parameter value. It does not, by itself, establish every global or large-sample property of a fitted model.

Worked Example: Poisson Regression With a Log Link

Suppose independent responses have Poisson means \(\mu_i=\exp(x_i^\mathsf{T}\beta)\), with design vectors \(x_1=(1,0)^\mathsf{T}\), \(x_2=(1,1)^\mathsf{T}\), and \(x_3=(1,2)^\mathsf{T}\). At \(\beta_0=(\log 2,\log 1.5)^\mathsf{T}\), the means are \(2,3,\) and \(4.5\).

The Poisson support is the nonnegative integers for every finite positive mean. The log link and the mean function are smooth at these parameter values, and the coefficient parameter space is open. Each response has finite moments of every order. The canonical natural parameters are the finite numbers \(\log 2,\log 3,\log 4.5\), all in the open domain \(\mathbb{R}\) of the Poisson cumulant function.

The information is \(\sum_i\mu_i x_ix_i^\mathsf{T}\). Its entries are \(\sum_i\mu_i=9.5\), \(\sum_i\mu_i x_{i,2}=12\), and \(\sum_i\mu_i x_{i,2}^2=21\), so

I(\beta_0)= \begin{pmatrix} 9.5 & 12\\ 12 & 21 \end{pmatrix}, \qquad \det I(\beta_0)=9.5(21)-12^2=55.5>0.

The matrix is positive definite because its first diagonal entry and determinant are positive. Thus the two coefficient directions have nonzero local information in this example.

Worked Example: Unknown Dispersion in Gaussian Regression

Let \(Y_i\) be independent normal responses with mean \(x_i^\mathsf{T}\beta\) and variance \(\phi\), where \(\phi>0\) is unknown. Use three fixed design vectors \(x_1=(1,0)^\mathsf{T}\), \(x_2=(1,1)^\mathsf{T}\), and \(x_3=(1,2)^\mathsf{T}\). Consider \(\beta_0=(1,2)^\mathsf{T}\) and \(\phi_0=4\).

The response support is \(\mathbb{R}\) for all parameters, and the normal density is smooth in \((\beta,\phi)\) for \(\phi>0\). The true dispersion \(4\) is interior to \((0,\infty)\). The normal distribution has finite moments of every order, including those needed for the score and Hessian. The coefficient information is \(X^\mathsf{T}X/\phi_0\), where

X^\mathsf{T}X= \begin{pmatrix}3&3\\3&5\end{pmatrix}, \qquad \det(X^\mathsf{T}X)=15-9=6>0.

Consequently the coefficient information is nonsingular. The dispersion score is \(U_\phi=-3/(2\phi)+\sum_{i=1}^3(Y_i-x_i^\mathsf{T}\beta)^2/(2\phi^2)\). At the true parameter its expectation is \(-3/8+12/32=0\), since the expected sum of squared residuals is \(3\phi_0=12\). This example checks that dispersion also needs a valid interior domain and finite derivative moments; it is not enough to check conditions for \(\beta\) alone.

Interpreting the Checks and Avoiding a Common Mistake

The examples illustrate two separate questions. First, can the likelihood and its derivatives be manipulated validly near the true parameter? This is where support, differentiability, interiority, and moment assumptions enter. Second, does the model contain enough information to distinguish nearby parameter values? This is where identifiability and nonsingular information enter. A smooth likelihood can still have no information in a coefficient direction.

Common mistake: “The log likelihood has a second derivative, so the usual information-based large-sample results apply.” Pointwise differentiability alone does not justify differentiating an integral, guarantee finite score variance, or ensure nonsingular information. The correction is to check local domination or another valid interchange condition, the necessary derivative moments, an interior true parameter, and positive information in the directions of interest. These checks are prerequisites, not proofs of consistency or asymptotic normality.

For GLM coefficients, inspect the design and the model’s variance weights together: a full-rank design is not enough if the relevant weights vanish or information fails to grow. For an unknown dispersion, additionally check that its true value is interior and that the joint likelihood derivatives involving dispersion are integrable. Finally, these regularity conditions do not guarantee that every finite-sample likelihood has a finite maximizer; existence and separation are separate issues.

Key takeaway: Regular likelihood inference requires a valid local calculus, an interior true parameter, finite derivative moments, and informative parameter directions. A GLM’s familiar likelihood formula does not automatically supply these properties.

Check Your Understanding

Use the conditions in this tutorial to explain what must be checked in each case.

Why is parameter-independent support useful when differentiating the integral of a density?
What is the difference between global identifiability and nonsingular Fisher information?
For a canonical-link GLM, which features of the design and variance function appear in the expected information matrix?
Why does an unknown positive dispersion parameter require an interiority check?
Can a pointwise second derivative of the log likelihood alone justify the usual information identity? Explain.