Existence and Uniqueness of Maximum Likelihood Estimates

Why Existence and Uniqueness Need Separate Checks

A likelihood can have a score equation with a solution, but that alone does not say whether the solution is unique. Nor does the shape of a likelihood guarantee that its maximum occurs at a finite parameter value: the likelihood may increase toward the edge of the natural parameter space. In an exponential family, these questions can be organized around two objects: the curvature of the log-partition function and the observed sufficient statistic.

The central distinction is between uniqueness, which follows from strict concavity under minimality, and existence, which requires the observed statistic to match the model mean at some finite natural parameter. The condition is exact for the regular minimal family described below; membership in the interior of a convex support is an equivalent shortcut only under additional assumptions.

Likelihood in Canonical Form

Let \(X_1,\ldots,X_n\) be independent and identically distributed observations from a \(d\)-dimensional canonical exponential family with density or mass function, relative to a fixed base measure,

Here \(\eta\in\mathbb{R}^d\) is the natural parameter, \(T(X)\in\mathbb{R}^d\) is the sufficient statistic for one observation, \(A(\eta)\) is the log-partition function, and \(\Omega\) is the natural parameter space. Assume \(\Omega\) is open and convex, and \(A(\eta)\) is finite on \(\Omega\). Write

Apart from terms not involving \(\eta\), the sample log-likelihood is

The usual regularity for differentiating the normalizing integral is assumed: around each \(\eta\in\Omega\), the defining integral for \(A\) remains finite in a neighborhood. This permits differentiation under the integral and yields the familiar moment identities below.

Log-Partition Derivatives and Strict Concavity

The log-partition function is not merely a normalizing constant. Its first derivative gives the mean of the sufficient statistic, and its second derivative gives its covariance matrix. These identities make the curvature of the likelihood directly interpretable.

Proof. Let \(Z(\eta)\) be the integral of \(h(x)\exp\{\eta^T T(x)\}\) against the base measure, so \(A(\eta)=\log Z(\eta)\). The neighborhood finiteness assumption justifies differentiating this integral: locally, factors that are polynomial in \(T(x)\) can be bounded by a finite sum of exponential integrands with natural parameters in that neighborhood. Thus

Differentiating once more gives

For any \(a\in\mathbb{R}^d\), \(a^T\nabla^2A(\eta)a=\operatorname{Var}_\eta(a^TT(X))\). If this variance is zero, then \(a^TT(X)\) is constant almost surely. Minimality says this can happen only when \(a=0\). Hence the Hessian is positive definite. Finally, \(\nabla^2\ell(\eta)=-n\nabla^2 A(\eta)\), which is negative definite, so \(\ell\) is strictly concave on the convex set \(\Omega\). \(\square\)

The Exact Criterion for a Finite, Unique MLE

Proof. Differentiating the sample log-likelihood gives

If a finite maximum is attained at \(\hat\eta\in\Omega\), the openness of \(\Omega\) means that \(\hat\eta\) is an interior point of the parameter domain. A differentiable function at an interior maximum has zero gradient, so \(\nabla A(\hat\eta)=\bar T\). This proves necessity.

Conversely, suppose that \(\nabla A(\hat\eta)=\bar T\) for some \(\hat\eta\in\Omega\). Then \(\nabla\ell(\hat\eta)=0\). Concavity implies that a stationary point is a global maximum: for any \(\eta\in\Omega\), concavity gives \(\ell(\eta)\leq\ell(\hat\eta)+\nabla\ell(\hat\eta)^T(\eta-\hat\eta)=\ell(\hat\eta)\). Strict concavity, proved above, makes this maximum unique. Therefore the MLE exists, is finite, and is unique. \(\square\)

The set \(\{\nabla A(\eta):\eta\in\Omega\}\) is the family’s mean-parameter space. It is the precise target for the observed average. In regular models it lies inside the relative interior of the convex support of \(T(X)\), the closed convex hull of values that \(T(X)\) can take. But interior convex-support membership is not, without further conditions such as steepness, a universal substitute for membership in the mean-parameter space.

Worked Examples

What Convexity Does—and Does Not—Guarantee

Strict concavity means there cannot be two distinct finite maximizers in a minimal representation. It does not by itself ensure that a maximizer exists inside \(\Omega\). For existence, the observed average must be achieved by the gradient of \(A\) at a parameter in the allowed natural parameter space.

A tempting shortcut is to check only whether the observed average lies in the interior of the convex support of \(T(X)\). This is useful when additional assumptions ensure that the gradient map covers that interior, as in many steep full exponential families. Without those assumptions, the safe test is the exact one: solve \(\nabla A(\eta)=\bar T\) and verify that a solution lies in \(\Omega\).

Another important qualification is minimality. If a representation contains redundant statistics, the likelihood can be flat along some parameter directions. A maximum may then exist without its parameter value being unique. Removing redundant directions, or working in a minimal representation, is what makes the strict-concavity conclusion apply.

Common Mistake

• Wrong reasoning: “The likelihood is concave, so it has a unique MLE.” Concavity alone allows a flat set of maxima, and it does not guarantee that the supremum is attained at a finite parameter.
• Correction: Establish strict concavity using minimality to get uniqueness if a maximum exists. Then check existence separately by asking whether \(\bar T\) equals \(\nabla A(\eta)\) for some \(\eta\in\Omega\).