Regularity Conditions for Likelihood Inference

Why Regularity Conditions Matter

Likelihood calculations often involve differentiating a log-likelihood, taking expectations, or treating information as a nonsingular matrix. Those steps need conditions: if the support changes with the parameter, for example, differentiating the density inside its support can miss parameter-dependent boundary effects. Regularity conditions make the local calculations meaningful and identify the assumptions behind standard likelihood theory.

In an exponential family, some of these conditions are particularly transparent. An open natural parameter space provides room to vary the parameter locally; an interior point gives strong moment and smoothness properties; and a minimal representation rules out redundant natural parameters. These properties do not, by themselves, guarantee that a maximum likelihood estimate exists for every sample.

A Regular Exponential-Family Setup

Consider a dominated exponential family with density or mass function \(f_\eta(x)=h(x)\exp\{\eta^{\mathsf T}T(x)-A(\eta)\}\). Here \(\eta\in\mathbb R^d\) is the natural parameter, \(T(x)\in\mathbb R^d\) is the sufficient statistic, \(h\) is a carrier function, and \(A\) is the log-partition function. The natural parameter space consists of those \(\eta\) for which the normalizing integral is finite. The regular setting of interest has a natural parameter space containing an open neighborhood of the true value \(\eta_0\).

Definition: A useful regularity package for local likelihood inference requires: (1) the true parameter is an interior point of an open parameter space; (2) the model has a common, parameter-independent support under a common dominating measure; (3) the model is identifiable; (4) the log-likelihood is sufficiently smooth and the moments needed for its derivatives are finite; and (5) the Fisher information is finite and nonsingular at the true parameter. In a full exponential family, an interior natural parameter and a minimal representation provide key parts of this package.

“Common support” means there is a fixed set \(S\) such that, apart from sets of dominating-measure zero, the observations that can occur do not depend on the parameter. A density may be zero on a fixed part of the sample space, as determined by \(h\); the concern is a support boundary that moves with the parameter. Identifiability means distinct parameter values correspond to distinct probability distributions. Minimality is the natural-coordinate version of this idea: no nonzero linear combination of the components of \(T\) is constant almost everywhere on the carrier support.

Interior Natural Parameters Give Smoothness and Moments

The log-partition function is defined by normalizing the exponential tilt: \(e^{A(\eta)}=\int h(x)e^{\eta^{\mathsf T}T(x)}\,d\mu(x)\). At an interior point of the natural parameter space, this integral remains finite in a neighborhood. That neighborhood supplies an integrable bound for derivatives of the integrand, so differentiation under the integral is justified rather than assumed.

Theorem: If \(\eta_0\) lies in the interior of the natural parameter space, then \(A(\eta)\) is finite and infinitely differentiable in a neighborhood of \(\eta_0\). In particular, every polynomial moment of \(T(X)\) is finite there. The first and second derivatives of \(A\) are the mean vector and covariance matrix of \(T(X)\), respectively.

Proof. Choose \(\delta>0\) small enough that every corner \(\eta_0+\delta s\), where \(s\in\{-1,1\}^d\), belongs to the natural parameter space. For \(t\in\mathbb R^d\), polynomial factors in \(\|t\|\) can be bounded by a constant times a finite sum of exponentials of the form \(\exp\{\delta s^{\mathsf T}t\}\). Thus, for each fixed derivative order \(k\), there is a finite constant \(C_k\) such that, for \(\eta\) sufficiently close to \(\eta_0\),

\|T(x)\|^k h(x)e^{\eta^{\mathsf T}T(x)} \leq C_k h(x)\sum_{s\in\{-1,1\}^d} e^{(\eta_0+\delta s)^{\mathsf T}T(x)}.

Each term on the right is integrable by the definition of the natural parameter space. The dominated convergence theorem therefore permits differentiation under the integral, to every finite order, on a sufficiently small neighborhood of \(\eta_0\). Taking the logarithm preserves smoothness because the normalizing integral is positive. Differentiating once gives \(E_\eta[T(X)]\); differentiating twice gives the covariance matrix \(\operatorname{Cov}_\eta(T(X))\). The same domination argument gives finite polynomial moments. \(\square\)

This theorem supplies smoothness and moment control in natural coordinates. It does not establish every regularity condition in every model: support and identifiability still need to be checked, and a curved parameterization can introduce additional rank conditions.

Identifiability and Nonsingular Information

In natural coordinates, the Fisher information is the covariance matrix of \(T(X)\), equivalently the Hessian of \(A\). Its nonsingularity has a direct interpretation: there must be no direction in parameter space along which the sufficient statistic is effectively constant.

Result: In a full exponential family at an interior natural parameter, the information matrix is positive definite if and only if no nonzero vector \(a\in\mathbb R^d\) makes \(a^{\mathsf T}T(X)\) constant almost surely. Minimality therefore implies nonsingular information.

Proof. For every vector \(a\),

a^{\mathsf T}\operatorname{Cov}_\eta(T(X))a =\operatorname{Var}_\eta(a^{\mathsf T}T(X)).

A variance is nonnegative, and it is zero exactly when its random variable is constant almost surely. Hence the quadratic form is strictly positive for every nonzero \(a\) exactly when no such nonzero linear combination is constant. That is positive definiteness of the information matrix, and the stated minimality condition excludes precisely those constant combinations. \(\square\)

For a curved model with natural parameter \(\eta=g(\theta)\), let \(J(\theta)\) be the Jacobian matrix of \(g\). The information in the \(\theta\) coordinates is \(J(\theta)^{\mathsf T}I(\eta)J(\theta)\), by the chain rule. Thus, even if the full natural-coordinate information is positive definite, the curved model needs \(J(\theta)\) to have full column rank for its information to be nonsingular. Global identifiability also requires that distinct allowed \(\theta\) values do not represent the same distribution.

Worked Checks in Familiar Families

Worked Example: Bernoulli Model at an Interior Parameter

Let \(X\sim\operatorname{Bernoulli}(p)\), with \(0<p<1\). In natural coordinates, \(\eta=\log\{p/(1-p)\}\), so \(\eta\in\mathbb R\), and \(A(\eta)=\log(1+e^\eta)\). The support is the fixed set \(\{0,1\}\), and each \(\eta\) determines a unique \(p=e^\eta/(1+e^\eta)\), so the model is identifiable.

The derivatives are \(A'(\eta)=e^\eta/(1+e^\eta)=p\) and \(A''(\eta)=e^\eta/(1+e^\eta)^2=p(1-p)\). For every finite \(\eta\), \(0<p<1\), so the information is finite and strictly positive. The sufficient statistic \(T(X)=X\) is not constant almost surely, which also verifies minimality in this one-parameter representation.

For \(n\) independent observations with \(r\) successes, the likelihood in \(p\) is proportional to \(p^r(1-p)^{n-r}\). If \(r=n\), it increases toward its supremum as \(p\) tends to 1, but 1 is outside the parameter space. This sample has no MLE in the stated open model, even though the model satisfies the local regularity conditions at every true \(p\in(0,1)\). Regularity is not a guarantee of finite-sample MLE existence.

Worked Example: Poisson Model and Moment Finiteness

Let \(X\sim\operatorname{Poisson}(\lambda)\), where \(\lambda>0\). Its natural parameter is \(\eta=\log\lambda\in\mathbb R\), its statistic is \(T(X)=X\), and \(A(\eta)=e^\eta\). The support \(\{0,1,2,\ldots\}\) is fixed, and \(\eta\mapsto\lambda=e^\eta\) is one-to-one.

At any finite \(\eta\), the derivatives \(A'(\eta)=e^\eta=\lambda\) and \(A''(\eta)=e^\eta=\lambda\) are finite, and the information is \(\lambda>0\). For instance, if \(\lambda=2\), then \(\eta=\log 2\), \(A'(\eta)=2\), and \(A''(\eta)=2\). The statistic \(X\) is not almost surely constant because both \(P(X=0)=e^{-2}\) and \(P(X=1)=2e^{-2}\) are positive. The representation is therefore minimal and its information is nonsingular at this parameter value.

Worked Example: A Uniform Model with Moving Support

Let \(X_1,\ldots,X_n\) be independent with \(X_i\sim\operatorname{Uniform}(0,\theta)\), \(\theta>0\). Use the density representative \(f_\theta(x)=1/\theta\) for \(0\leq x\leq\theta\), and zero otherwise. For observations \(x_1,\ldots,x_n\), the likelihood is

L(\theta)=\theta^{-n}\mathbf 1\{\theta\geq x_{(n)}\}, \qquad x_{(n)}=\max_i x_i.

For example, take \(n=3\) and observations \(0.4,0.7,0.9\). When \(\theta<0.9\), the likelihood is zero. When \(\theta\geq0.9\), it is \(\theta^{-3}\), which decreases as \(\theta\) increases. Therefore its maximum is attained at \(\hat\theta=0.9\), with likelihood \(0.9^{-3}=1/0.729\approx1.372\).

This model does not have parameter-independent support: its support is \([0,\theta]\). The indicator in the likelihood changes at a parameter-dependent boundary, so smoothness conditions for ordinary likelihood arguments do not apply. The inclusive endpoint convention above makes the displayed boundary maximum valid for this density representative. Replacing the density by a version that is zero at \(x=\theta\) changes the pointwise likelihood at the observed boundary and can make the maximum unattained; this illustrates why likelihood calculations at support boundaries require care.

What to Check—and a Common Mistake

A regularity check is a sequence of distinct questions, not a single label attached to a model. For a likelihood argument, identify the parameterization being used, verify that the true parameter is interior, check the support, and examine identifiability and information rank. In an exponential family, interiority of the natural parameter gives strong control over the log-partition function and moments, but it does not repair moving support or redundant coordinates.

Common mistake: “The density is smooth inside its support, so the model is regular.” This misses the support condition. A parameter-dependent indicator can contribute boundary behavior that is invisible to differentiation inside the support.
Correction: Check whether the support itself changes with the parameter before relying on ordinary likelihood derivatives.
Common mistake: “Nonsingular information means the MLE always exists.” The Bernoulli sample with all successes disproves this for the open parameter space \(0<p<1\).
Correction: Treat local regularity and finite-sample existence as separate properties. Existence also depends on the observed sample and on the parameter space’s boundaries.

These conditions matter because they clarify which local calculations can be trusted and where special treatment is needed. In particular, smoothness and finite moments justify differentiating the exponential-family normalizer, while identifiability and nonsingular information prevent distinct local parameter directions from producing indistinguishable distributions. The precise consequences for score expectations and information identities require additional calculations.

Check Your Understanding

For each question, state which condition is involved and explain your reasoning.

Why does an interior natural parameter provide the local integrable bounds needed to differentiate the log-partition function?
In the Bernoulli model, what happens to the information as \(p\) approaches 0 or 1, and why are those endpoints excluded from the stated natural parameter space?
How does a constant nonzero linear combination of the sufficient statistics show that information is singular?
Why does the uniform model violate the common-support condition even though its density is simple within its support?
Can a regular model have a sample for which the MLE does not exist in the parameter space? Give the Bernoulli example and explain.