Why Boundary Observations Matter
The likelihood equations for a regular exponential family ask the fitted mean of a sufficient statistic to equal its observed sample mean. But not every observed mean can be matched by a distribution with a finite natural parameter. When the observed mean lies on the boundary of the statistic's convex support, likelihood can keep improving as the natural parameter moves without bound. The likelihood may have a supremum, yet no finite parameter at which that supremum is attained.
This issue appears in ordinary discrete models, not only in complicated multivariate problems. A sample with no successes in a Bernoulli model, or a category with zero observations in a multinomial model, can force an estimate to an edge of the probability simplex. The convex-support perspective gives a geometric way to identify such cases and to separate an estimate on an extended boundary from a finite maximum likelihood estimate.
Convex Support and the Finite-Support Setting
Consider an independent sample \(X_1,\ldots,X_n\) from a canonical exponential family. For this tutorial, first take a finite set of possible sufficient-statistic values \(z_1,\ldots,z_m\) that affinely spans \(\mathbb{R}^d\). Suppose the probability of value \(z_j\) is \[ p_\eta(z_j)=h_j\exp\{\eta^\top z_j-A(\eta)\}, \qquad h_j>0,\quad \eta\in\mathbb{R}^d, \] where \(A(\eta)=\log\sum_{j=1}^m h_j\exp(\eta^\top z_j)\). The affine-spanning assumption is the minimality condition in this representation: no nonzero direction of \(\eta\) leaves all probability ratios unchanged.
The log-likelihood, with terms not depending on \(\eta\) omitted, is \[ \ell(\eta)=n\{\eta^\top\bar{T}-A(\eta)\}. \] Its score equation is \(\bar{T}=\nabla A(\eta)=E_\eta[T(X)]\). The important geometric fact is that every finite \(\eta\) assigns strictly positive probability to every support point. Its fitted mean is therefore a strictly positive weighted average of all the \(z_j\), which lies in the interior of their convex hull. A boundary observed mean cannot solve this equation at a finite \(\eta\).
Why the Interior-Boundary Criterion Holds
A complete argument makes both directions of the criterion clear. Write \(w_j=h_j\), and let \(w_{\min}=\min_j w_j>0\). The negative log-likelihood, apart from a positive factor and an additive constant, is the convex function \[ g(\eta)=A(\eta)-\eta^\top\bar{T}. \] A minimizer of \(g\) is exactly a maximizer of the log-likelihood.
First suppose \(\bar{T}\) is in the interior of \(C\). There is an \(\epsilon>0\) such that the ball of radius \(\epsilon\) around \(\bar{T}\) is contained in \(C\). For every unit vector \(u\), the point \(\bar{T}+\epsilon u\) belongs to \(C\); a linear functional on a convex hull reaches its maximum at one of its generating points. Hence \[ \max_j u^\top z_j\ \geq\ u^\top\bar{T}+\epsilon. \] For \(\eta=ru\), \(r>0\), the log-partition function satisfies \[ A(ru)=\log\sum_j w_j e^{r u^\top z_j} \ \geq\ r\max_j u^\top z_j+\log w_{\min}. \] It follows that \(g(ru)\geq r\epsilon+\log w_{\min}\). Thus \(g(\eta)\) tends to infinity as \(\|\eta\|\) tends to infinity. A continuous function with this property attains a global minimum: outside a sufficiently large ball its value exceeds \(g(0)\), and on the closed ball it attains a minimum by compactness. The resulting \(\eta\) is finite.
The Hessian of \(A\) is \(\nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X))\). For any nonzero vector \(v\), \(v^\top\nabla^2 A(\eta)v=\operatorname{Var}_\eta(v^\top T(X))>0\): the affine-spanning assumption ensures that \(v^\top z_j\) is not constant across support points, and every point has positive probability. Therefore \(g\) is strictly convex, so its minimizer, and hence the MLE, is unique.
Now suppose \(\bar{T}\) is on the boundary of \(C\). A supporting-hyperplane theorem gives a nonzero vector \(u\) and a constant \(c\) such that \(u^\top z_j\leq c\) for every \(j\), while \(u^\top\bar{T}=c\). Along the ray \(\eta+su\), for \(s\geq0\), differentiate the log-likelihood: \[ \frac{d}{ds}\ell(\eta+su) =n\{u^\top\bar{T}-E_{\eta+su}[u^\top T(X)]\}. \] Every support point has positive probability at finite \(\eta+su\), and at least one support point has \(u^\top z_j<c\); thus the expectation is strictly less than \(c\). The derivative is strictly positive for every finite \(s\). The likelihood therefore increases along this ray and cannot attain a maximum at any finite \(\eta\). This proves the boundary direction and the theorem. \(\square\)
The theorem concerns finite natural parameters in the stated family. A probability model may be enlarged to include boundary distributions, such as a Bernoulli distribution with success probability zero. The likelihood can then attain its supremum in that enlarged model, even though no finite canonical parameter achieves it.
Worked Discrete Examples
Worked Example: A Bernoulli Sample with No Successes
Let \(X_i\sim\operatorname{Bernoulli}(p)\), independently, with \(0<p<1\). In canonical form, \(T(X_i)=X_i\), \(\eta=\log\{p/(1-p)\}\), and \(A(\eta)=\log(1+e^\eta)\). The convex support is \(C=[0,1]\). Suppose the observed sample is \(0,0,0,0\), so \(\bar{T}=0\), the boundary of \(C\).
The log-likelihood in \(p\) is \(\ell(p)=4\log(1-p)\). For \(0<p<1\), \[ \frac{d}{dp}\ell(p)=-\frac{4}{1-p}<0, \] so the likelihood increases as \(p\) decreases toward zero. Its supremum is \(1\), attained at \(p=0\) if that endpoint is added to the model, but \(p=0\) corresponds to \(\eta=-\infty\), not a finite canonical parameter. There is no finite MLE in the stated model. In contrast, if the sample contains \(k\) successes with \(0<k<4\), then \(\bar{T}=k/4\) is interior and the score equation gives \(\hat p=k/4\), a finite estimate.
Worked Example: A Zero Cell in a Multinomial Sample
Suppose each observation falls in one of three categories with probabilities \(p_1,p_2,p_3>0\), where \(p_1+p_2+p_3=1\). Use category 3 as the reference: \(T(X)=(I\{X=1\},I\{X=2\})\), \(\eta_j=\log(p_j/p_3)\) for \(j=1,2\). The convex support is the triangle with vertices \((0,0),(1,0),(0,1)\). Observe counts \((3,2,0)\) in a sample of size \(5\), giving \(\bar{T}=(3/5,2/5)\), on the edge joining \((1,0)\) and \((0,1)\).
For probabilities in the interior, the likelihood is proportional to \(p_1^3p_2^2p_3^0\). The usual multinomial maximization over the closed probability simplex gives \((\hat p_1,\hat p_2,\hat p_3)=(3/5,2/5,0)\). To see why there is no finite canonical MLE, let \(p_3=\delta>0\), \(p_1=(1-\delta)(3/5)\), and \(p_2=(1-\delta)(2/5)\), for \(0<\delta<1\). The likelihood is proportional to \[ \left(\frac{3}{5}\right)^3\left(\frac{2}{5}\right)^2(1-\delta)^5, \] which increases toward its supremum as \(\delta\) decreases to zero. No allowed \(\delta>0\) attains that supremum. The zero count places the observed statistic on a face of the convex support; the fitted boundary distribution is not represented by finite \(\eta_1,\eta_2\).
Worked Example: All-Zero Poisson Data
Let \(X_i\sim\operatorname{Poisson}(\lambda)\), independently, with \(\lambda>0\). The canonical statistic is \(T(X_i)=X_i\), the natural parameter is \(\eta=\log\lambda\), and the convex support is \([0,\infty)\). If all \(n\) observations are zero, then \(\bar{T}=0\), the boundary point of the convex support.
The likelihood as a function of \(\lambda\) is \(L(\lambda)=e^{-n\lambda}\), and its derivative is \(-n e^{-n\lambda}<0\). Thus the likelihood increases as \(\lambda\) decreases toward zero, with supremum \(1\). Since \(\lambda=0\) is excluded, the supremum is not attained; in canonical coordinates, \(\eta=\log\lambda\) tends to \(-\infty\). If instead the observed total is \(s=\sum_i X_i>0\), the log-likelihood is \(s\log\lambda-n\lambda\) plus a constant. Its derivative \(s/\lambda-n\) vanishes at \(\hat\lambda=s/n>0\), and the observed mean lies in the interior \((0,\infty)\). This positive finite estimate contrasts with the all-zero case.
What the Criterion Does—and Does Not—Say
The convex-support criterion is a diagnostic for existence, not a general recipe for solving every likelihood equation. When the observed mean is interior, the theorem guarantees a unique finite solution in this finite-support minimal family; the score equation then characterizes it. When the mean is on the boundary, the likelihood can approach its supremum by concentrating probability on the exposed face containing the data. The theorem says no finite parameter attains a maximum, but it does not require that the supremum correspond to a unique distribution on that face.
A common mistake is to say, “The score equation has a solution at the boundary, so the MLE exists.” In the Bernoulli all-zero case, substituting \(\hat p=0\) into the probability parameterization gives a legitimate distribution only if the model includes \(p=0\); its canonical parameter is not finite. The correction is to specify the parameter space and distinguish a finite MLE in the original family from a maximizer in an extended closure. Another mistake is to interpret a zero cell as a probability known to be exactly zero: the data make the likelihood supremum occur there, but do not establish that the true probability is zero.
Check Your Understanding
Use the convex support and parameter space to justify each answer.
- For a Bernoulli sample of size \(8\) with exactly \(4\) successes, where is the observed statistic relative to its convex support, and is the canonical MLE finite?
- In a three-category multinomial sample, what geometric feature of the observed count vector occurs when category 2 has count zero?
- Why does the likelihood for an all-zero Poisson sample have a supremum but no finite natural-parameter maximizer?
- Explain why a boundary probability estimate in the closed simplex need not be a finite canonical-parameter MLE.
- Which assumption in the finite-support theorem ensures strict convexity of the log-partition function in every nonzero direction?