Existence and Boundary Behavior of Maximum-Likelihood Estimates

Why the Location of the Sample Mean Matters

For an exponential family, the likelihood depends on the observations through a sufficient statistic. In many finite-support models, maximizing the likelihood amounts to asking whether the observed average of that statistic can be matched by the model at a finite parameter value. Convex geometry makes the answer visible: an average in the interior of the convex support gives a finite estimate when the full natural parameter space is available, while a boundary average does not.

The qualification about the parameter space matters. If the natural parameter is restricted, an interior average need not be attainable within that restriction. We will distinguish the unrestricted geometric result from the exact criterion that applies under parameter restrictions.

Finite-Support Exponential Families and Their Likelihoods

Definition: Consider a finite-support exponential family for a statistic $T\in\mathbb{R}^d$, with possible values $t_1,\ldots,t_m$ and positive base weights $q_1,\ldots,q_m$. Its natural-parameter representation is $$ P_\eta(T=t_j)=q_j\exp\{\eta^\mathsf{T}t_j-A(\eta)\},\qquad A(\eta)=\log\left(\sum_{j=1}^m q_j e^{\eta^\mathsf{T}t_j}\right). $$ The convex support is $C=\operatorname{conv}\{t_1,\ldots,t_m\}$. For $n$ independent observations, write $\overline{T}=n^{-1}\sum_{i=1}^n T_i$. A finite maximum-likelihood estimate is a maximizing natural parameter $\hat\eta$ that belongs to the parameter space and has finite coordinates.

Assume throughout that the support points have full affine dimension $d$. For observed values $T_1,\ldots,T_n$, terms in the log-likelihood that do not depend on $\eta$ can be omitted, leaving

\ell(\eta)=n\eta^\mathsf{T}\overline{T}-nA(\eta)+\text{constant}.

Because the support is finite, differentiating the finite sum defining $A$ gives

\nabla A(\eta)= \frac{\sum_{j=1}^m q_j e^{\eta^\mathsf{T}t_j}t_j} {\sum_{j=1}^m q_j e^{\eta^\mathsf{T}t_j}}.

This is a weighted average of all the support points, with every weight strictly positive at finite $\eta$. The likelihood score is $n(\overline{T}-\nabla A(\eta))$, so any finite interior maximizer must match the observed average: $\nabla A(\hat\eta)=\overline{T}$.

Worked Example: A Bernoulli Sample With an Interior Mean

Suppose $X_1,\ldots,X_5$ are Bernoulli observations and the data contain three successes. In natural-parameter form, $T=X$, the support is $\{0,1\}$, $q_0=q_1=1$, and $A(\eta)=\log(1+e^\eta)$. The observed mean is $\overline{T}=3/5$.

The score equation is $$ 0=5\left(\frac35-\frac{e^\eta}{1+e^\eta}\right). $$ Solving gives $3/5=e^\eta/(1+e^\eta)$, hence $3+3e^\eta=5e^\eta$, $e^\eta=3/2$, and $\hat\eta=\log(3/2)$, a finite value. The convex support is $[0,1]$, and $3/5$ is in its interior.

The Geometric Criterion

Theorem: Suppose the support is finite and has full affine dimension $d$, all base weights are positive, and the natural parameter space is the full space $\mathbb{R}^d$. Then a finite maximum-likelihood estimate exists if and only if $\overline{T}\in\operatorname{ri}(C)$, the relative interior of the convex support. Under the full-dimensionality assumption, this is the ordinary interior. The estimate is unique.

Proof. First, for any finite $\eta$, the gradient $\nabla A(\eta)$ is a convex combination of all support points with strictly positive weights. It lies in the interior of $C$: if it were on the boundary, a supporting hyperplane would contain the weighted average on one side and the whole convex support on the other. Since every weight is positive, equality at the average would force every support point onto that hyperplane, contradicting full affine dimension. Thus a score-matching finite estimate can exist only if $\overline{T}\in\operatorname{ri}(C)$.

For the converse, suppose $\overline{T}$ is in the interior. Choose $\delta>0$ such that the ball of radius $\delta$ around $\overline{T}$ lies in $C$. For every unit vector $u$, this implies $\max_j u^\mathsf{T}(t_j-\overline{T})\geq\delta$. Consider $g(\eta)=A(\eta)-\eta^\mathsf{T}\overline{T}$. If $q_{\min}=\min_j q_j>0$, then, writing $\eta=ru$ for $r>0$, $$ g(ru)=\log\sum_j q_j e^{r u^\mathsf{T}(t_j-\overline{T})} \geq r\delta+\log q_{\min}. $$ Consequently $g(\eta)$ tends to infinity as $\|\eta\|$ tends to infinity. It is continuous, so it attains a minimum on $\mathbb{R}^d$: outside a sufficiently large ball its values exceed $g(0)$, and on the closed ball it attains a minimum by compactness. At a minimizer, $\nabla g=0$, giving $\nabla A(\hat\eta)=\overline{T}$. The log-likelihood is therefore maximized there.

Finally, along any nonzero direction $v$, the second derivative of $A(\eta+sv)$ with respect to $s$ is the variance of $v^\mathsf{T}T$ under the positive weights proportional to $q_j e^{(\eta+sv)^\mathsf{T}t_j}$. Full affine dimension ensures that $v^\mathsf{T}t_j$ is not constant over the support, so this variance is positive. Thus $A$ is strictly convex, the log-likelihood is strictly concave, and its maximizer is unique. $\square$

Worked Example: All Bernoulli Observations Are Successes

Now suppose $n=4$ and all four observations equal 1. The mean is $\overline{T}=1$, on the boundary of $[0,1]$. The log-likelihood, apart from constants, is $$ \ell(\eta)=4\eta-4\log(1+e^\eta). $$ Its derivative is $4/(1+e^\eta)>0$ for every finite $\eta$, so the likelihood is strictly increasing and has no finite maximizer. As $\eta$ tends to infinity, the success probability tends to 1 and $\ell(\eta)$ tends to 0, its supremum. The boundary mean is not the gradient of $A$ at any finite $\eta$.

Boundary Means and Parameter Restrictions

For finite support, a boundary average has a further geometric interpretation. A supporting hyperplane at a boundary point $\overline{T}$ has some nonzero normal $u$ such that $u^\mathsf{T}t_j\leq b=u^\mathsf{T}\overline{T}$ for every support point. Since $\overline{T}$ is the average of the observed statistics, each observed statistic must lie on that hyperplane: the average of numbers no greater than $b$ equals $b$ only if every number equals $b$. Thus the observations lie in a face of the convex support. In the full natural parameter space, moving the parameter in the direction $u$ increases mass on that face relative to support points outside it. The theorem shows that no finite parameter can be a maximizer.

Worked Example: A Multinomial Sample on a Face

Consider three categories with probabilities proportional to $e^{\eta_1}$, $e^{\eta_2}$, and $1$. The sufficient-statistic support points are $(1,0)$, $(0,1)$, and $(0,0)$, whose convex hull is a triangle. In three observations, suppose the category counts are $(2,1,0)$. Then $\overline{T}=(2/3,1/3)$, on the edge joining $(1,0)$ to $(0,1)$.

For category probabilities $p_1,p_2,p_3$, the likelihood is proportional to $p_1^2p_2$. Its supremum over the probability simplex occurs at $(p_1,p_2,p_3)=(2/3,1/3,0)$, but finite $(\eta_1,\eta_2)$ always gives $p_3>0$, so this point is unavailable. To see the limiting behavior, take $\eta_1=s+\log 2$ and $\eta_2=s$. Then the probabilities are $$ (p_1,p_2,p_3)=\left(\frac{2e^s}{3e^s+1},\frac{e^s}{3e^s+1},\frac{1}{3e^s+1}\right) \longrightarrow (2/3,1/3,0). $$ The likelihood approaches its supremum but no finite natural parameter attains it.

The full-space theorem should not be applied unchanged when the natural parameter is restricted. For an open convex restriction $\Theta\subseteq\mathbb{R}^d$, the exact criterion is

Criterion under restriction: For a finite-support family with full-dimensional support and an open convex natural parameter space $\Theta$, a finite maximum-likelihood estimate exists if and only if $\overline{T}\in\nabla A(\Theta)$. If this condition holds, the estimate is unique.

Indeed, a maximizer in the open set $\Theta$ must have zero score, so the condition is necessary. Conversely, if $\nabla A(\eta)=\overline{T}$ for some $\eta\in\Theta$, concavity of the log-likelihood makes $\eta$ a global maximizer on $\Theta$. Strict concavity gives uniqueness. Relative-interior membership alone does not ensure that the matching parameter belongs to a restricted space.

Worked Example: An Interior Mean Outside the Restricted Gradient Image

For a Bernoulli model, restrict the natural parameter to $\Theta=(0,1)$, and observe two successes in five trials. The mean $2/5$ is interior to the convex support $[0,1]$. But for every $\eta\in(0,1)$, the fitted success probability $e^\eta/(1+e^\eta)$ exceeds $1/2$, so $2/5\notin\nabla A(\Theta)$.

The log-likelihood derivative is $5(2/5-e^\eta/(1+e^\eta))<0$ throughout $(0,1)$. It decreases as $\eta$ increases, and its supremum is approached as $\eta$ decreases to 0, which is excluded. Hence there is no MLE in this restricted parameter space, despite the mean being in the interior of the convex support.

Common Pitfall and Practical Interpretation

A frequent error is to reason, “The observed mean is interior to the convex support, so a finite MLE exists.” That conclusion requires the full natural parameter space (or, under restrictions, that the mean lies in the image $\nabla A(\Theta)$). Interior membership guarantees a solution in $\mathbb{R}^d$ for the finite-support, full-dimensional family; it does not guarantee that the solution survives an arbitrary restriction.

The distinction also clarifies what “no finite estimate” means. The likelihood can have a finite supremum even when no finite natural parameter attains it. A boundary probability vector may describe the limiting fit, as in the multinomial example, without corresponding to any finite natural parameter. When diagnosing existence, identify the parameter space and check the score-matching condition there, rather than treating boundary limits as finite estimates.

Check Your Understanding

Use the convex support and the stated parameter space in each answer.

Why is $\nabla A(\eta)$ in the relative interior of the convex support for finite $\eta$ in the full-dimensional finite-support model?
For a full natural parameter space, what condition on $\overline{T}$ is necessary and sufficient for a finite MLE?
In a Bernoulli sample with all failures, what happens to the likelihood as the natural parameter tends to negative infinity?
Why does an interior sample mean fail to guarantee an MLE when the natural parameter space is restricted?
In the multinomial example with counts $(2,1,0)$, which face contains the observed mean, and why can no finite natural parameter attain the limiting probability vector?