Convexity and Geometry of the Log-Partition Function

Curvature and the Geometry of Means

The log-partition function does more than normalize an exponential family. Its first derivatives give the expected canonical statistics, while its second derivatives measure their covariance. This connects the shape of a scalar-valued function to the variation of the statistics under the family: covariance can never be negative, so the log-partition function bends in a consistent direction.

That observation also raises two geometric questions. When is the bending strict in every parameter direction? And where, in the space of possible statistic values, can the gradient of the log-partition function lie? The answers depend on whether the canonical statistics vary independently up to additive constants, and on the convex support of their possible values.

Definition: Consider a full, finite-dimensional exponential family with density \(p_\theta(x)=\exp\{\theta^\mathsf{T}T(x)-A(\theta)\}\) relative to a measure \(\mu\), where \(T=(T_1,\ldots,T_d)^\mathsf{T}\) is measurable and \(A(\theta)=\log\int \exp\{\theta^\mathsf{T}T(x)\}\,d\mu(x)\). The natural parameter space is the set of \(\theta\) for which this integral is finite. We work on its nonempty interior, denoted by \(\Theta^\circ\), which is an open convex set.

Assume the integral is positive on \(\Theta^\circ\). The density is then positive \(\mu\)-almost everywhere wherever the carrier measure is supported. On this open parameter space, local exponential integrability permits differentiation under the integral: in a neighborhood of any \(\theta\), polynomial factors in the coordinates of \(T\) are dominated by finite sums of exponentials with nearby natural parameters. Consequently, \(A\) is twice differentiable there. The established gradient and Hessian identities are

\nabla A(\theta)=E_\theta[T], \qquad \nabla^2 A(\theta)=\operatorname{Cov}_\theta(T).

Here the covariance matrix has entries \(\operatorname{Cov}_\theta(T_i,T_j)\). For any vector \(v\in\mathbb R^d\), its quadratic form is \(v^\mathsf{T}\nabla^2 A(\theta)v=\operatorname{Var}_\theta(v^\mathsf{T}T)\), which is nonnegative.

Why the Log-Partition Function Is Convex

A twice-differentiable function on an open convex set is convex if its second derivative along every line segment is nonnegative. The covariance identity gives exactly that condition.

Theorem: On \(\Theta^\circ\), \(A\) is convex. It is strictly convex if and only if there is no nonzero \(v\in\mathbb R^d\) for which \(v^\mathsf{T}T\) is constant \(\mu\)-almost everywhere. Equivalently, the functions \(1,T_1,\ldots,T_d\) are linearly independent up to \(\mu\)-almost-everywhere equality.

Proof. Take any \(\theta_0,\theta_1\in\Theta^\circ\), set \(v=\theta_1-\theta_0\), and define \(g(t)=A(\theta_0+tv)\) for \(0\leq t\leq 1\). Convexity of \(\Theta^\circ\) ensures the segment lies in the domain. The chain rule and covariance identity give

g''(t)=v^\mathsf{T}\nabla^2A(\theta_0+tv)v =\operatorname{Var}_{\theta_0+tv}(v^\mathsf{T}T)\geq 0.

Thus \(g\) is convex on every segment, so \(A\) is convex. If no nonzero linear combination \(v^\mathsf{T}T\) is constant almost everywhere, then for every nonzero \(v\) its variance is positive at every parameter value. The displayed second derivative is then positive along every nonconstant segment, making \(A\) strictly convex.

Conversely, suppose \(v\ne 0\) and \(v^\mathsf{T}T=c\) \(\mu\)-almost everywhere for some constant \(c\). For any \(\theta\) and any \(s\) for which \(\theta+sv\) is in the domain, the normalizing integral satisfies

\int e^{(\theta+sv)^\mathsf{T}T}\,d\mu =e^{sc}\int e^{\theta^\mathsf{T}T}\,d\mu, \qquad A(\theta+sv)=A(\theta)+sc.

Since \(\Theta^\circ\) is open, it contains a nonconstant short segment in direction \(v\); \(A\) is affine on that segment and therefore not strictly convex. Finally, a relation \(a_0+\sum_{j=1}^d a_jT_j=0\) almost everywhere with some \(a_j\ne 0\) is precisely a constant linear combination of the coordinates of \(T\). This proves the equivalence and the theorem. \(\square\)

Worked Example: Strict Curvature for Three Statistic Values

Let the sample space have three points with counting measure, and let the statistic \(T\) take values \(-1,0,2\), once each. The natural parameter is \(\theta\in\mathbb R\), and \(A(\theta)=\log(e^{-\theta}+1+e^{2\theta})\). At \(\theta=0\), each point has probability \(1/3\). The mean, which is the slope of \(A\), is

A'(0)=E_0[T]=\frac{-1+0+2}{3}=\frac13.

The second derivative is the variance. At zero,

A''(0)=\operatorname{Var}_0(T) =E_0[T^2]-E_0[T]^2 =\frac{1+0+4}{3}-\left(\frac13\right)^2 =\frac53-\frac19 =\frac{14}{9}>0.

More generally, at every finite \(\theta\), all three points have positive probability, and \(T\) is not constant on them. Its variance is therefore positive, so \(A''(\theta)>0\) everywhere and \(A\) is strictly convex.

Strict Convexity and Independent Directions

For a vector \(v\), the curvature in direction \(v\) is the variance of the single statistic \(v^\mathsf{T}T\). A direction has zero curvature exactly when that statistic does not vary under the family. Because the tilted density is positive relative to the carrier measure, “constant with probability one under a finite parameter” is equivalent here to “constant \(\mu\)-almost everywhere.” Thus the strictness criterion is about affine relations among the canonical statistics, not merely whether each coordinate varies on its own.

For example, two statistics may each vary but still have zero curvature in some combined direction if one is an affine function of the other. The Hessian then has a null direction. This does not contradict convexity: a positive semidefinite Hessian is enough for convexity, while positive definiteness in every nonzero direction gives strict convexity.

Worked Example: A Varying Statistic with a Flat Direction

Let \(X\in\{0,1\}\), with counting measure, and use the two-dimensional statistic \(T(X)=(X,2X)^\mathsf{T}\). Writing \(\theta=(\theta_1,\theta_2)^\mathsf{T}\), the log-partition function is

A(\theta)=\log\{1+\exp(\theta_1+2\theta_2)\}.

Set \(p=\exp(\theta_1+2\theta_2)/\{1+\exp(\theta_1+2\theta_2)\}\), so \(P_\theta(X=1)=p\). Since \(T=X(1,2)^\mathsf{T}\), its covariance matrix is

\nabla^2 A(\theta) =p(1-p) \begin{pmatrix} 1&2\\ 2&4 \end{pmatrix}.

This matrix is positive semidefinite but singular. In fact, for \(v=(2,-1)^\mathsf{T}\), \(v^\mathsf{T}T=2X-2X=0\) at both sample points, so the directional variance is zero. Correspondingly, moving from \(\theta\) to \(\theta+sv\) leaves \(\theta_1+2\theta_2\), and hence \(A\), unchanged. The function is convex but not strictly convex on \(\mathbb R^2\).

Slopes as Attainable Mean Vectors

The gradient has a direct statistical interpretation: \(\nabla A(\theta)\) is the mean vector of \(T\) under parameter \(\theta\). Its location is constrained by the possible values of \(T\), even before the family’s particular exponential weighting is considered.

Definition: The convex support \(C\) of \(T\) is the smallest closed convex subset of \(\mathbb R^d\) that contains \(T(x)\) for \(\mu\)-almost every \(x\). Its relative interior, \(\operatorname{relint}(C)\), is its interior within its affine hull; this allows \(C\) to lie in a lower-dimensional affine subspace.

Theorem: For every \(\theta\in\Theta^\circ\), the slope \(\nabla A(\theta)=E_\theta[T]\) belongs to \(\operatorname{relint}(C)\). Thus every finite-parameter mean vector is inside the relative interior of the convex support.

Proof. The expectation exists because the parameter lies in the interior of the natural parameter space. First, \(m=E_\theta[T]\) belongs to \(C\). If it did not, the separation theorem for a closed convex set would give a linear functional that strictly separates \(m\) from \(C\). But \(T\in C\) almost surely, so taking expectations of that functional would contradict the strict separation.

If \(C\) has dimension zero, it consists of a single point and the result follows. Otherwise, suppose \(m\) lies on the relative boundary of \(C\). A supporting-hyperplane theorem within the affine hull of \(C\) then supplies a nonconstant linear functional \(x\mapsto q^\mathsf{T}x\) on that affine hull such that \(q^\mathsf{T}(x-m)\leq 0\) for every \(x\in C\). In particular, \(q^\mathsf{T}(T-m)\leq 0\) almost surely. Its expectation is \(q^\mathsf{T}(E_\theta[T]-m)=0\). A nonpositive integrable random variable with mean zero equals zero almost surely, so \(T\) lies almost surely in the supporting hyperplane \(q^\mathsf{T}(x-m)=0\). That would place all of \(C\) in a proper hyperplane of its own affine hull, contradicting the definition of the affine hull. Therefore \(m\in\operatorname{relint}(C)\). \(\square\)

This result locates every finite-parameter slope, but it does not by itself say that every point of \(\operatorname{relint}(C)\) is attained by some finite \(\theta\). The exact set of attainable means can depend on the family and its natural parameter space. Boundary points, in particular, may arise only as limits when parameters move toward the edge of the domain or diverge.

Worked Example: Slopes Fill the Interior of a Triangle

Let \(T\) take the three values \((0,0)\), \((1,0)\), and \((0,1)\), each with counting measure. For \(\theta=(\alpha,\beta)^\mathsf{T}\),

A(\alpha,\beta)=\log(1+e^\alpha+e^\beta).

Define \(p_0=1/(1+e^\alpha+e^\beta)\), \(p_1=e^\alpha/(1+e^\alpha+e^\beta)\), and \(p_2=e^\beta/(1+e^\alpha+e^\beta)\). These are positive and sum to one. Differentiating gives

\nabla A(\alpha,\beta)=(p_1,p_2)^\mathsf{T}.

The convex support is the triangle with vertices \((0,0),(1,0),(0,1)\). Its relative interior consists of points \((m_1,m_2)\) with \(m_1>0\), \(m_2>0\), and \(m_1+m_2<1\). The gradient always satisfies these inequalities because \(p_1,p_2,p_0>0\). Conversely, for any point satisfying them, set \(p_0=1-m_1-m_2\), \(p_1=m_1\), and \(p_2=m_2\). The finite parameters

\alpha=\log\left(\frac{p_1}{p_0}\right), \qquad \beta=\log\left(\frac{p_2}{p_0}\right)

produce exactly those probabilities and therefore that mean vector. In this example, every point in the relative interior is an attainable slope; no boundary point is attained at finite \((\alpha,\beta)\).

The Hessian confirms strict curvature. It is the covariance matrix of \(T\):

\nabla^2A= \begin{pmatrix} p_1(1-p_1)&-p_1p_2\\ -p_1p_2&p_2(1-p_2) \end{pmatrix}, \qquad \det(\nabla^2A)=p_0p_1p_2>0.

Its first diagonal entry is positive and its determinant is positive, so the two-dimensional Hessian is positive definite for every finite parameter pair.

Common Pitfall and Practical Interpretation

A common mistake is to argue, “The Hessian is a covariance matrix, so it is positive definite and the log-partition function is strictly convex.” The first part only establishes positive semidefiniteness: a covariance matrix can be singular when some linear combination of the statistics is constant. The correction is to check every direction \(v\ne 0\), or equivalently to check that no nonzero linear combination \(v^\mathsf{T}T\) is constant almost everywhere.

The geometric interpretation is useful in estimation and parameterization. The gradient maps natural parameters to mean vectors that the model can realize, and the Hessian describes how those means change locally as the natural parameter changes. Strict convexity means there is curvature in every parameter direction; a flat direction signals an affine relation among the statistics. The convex support, meanwhile, provides a geometric constraint on possible means, while its boundary indicates limits that may require parameter sequences rather than a finite parameter value.

Check Your Understanding

Use the covariance-Hessian identity and the geometry of the statistic values to answer these questions.

Why does a positive semidefinite Hessian imply convexity, but not necessarily strict convexity?
What condition on \(v^\mathsf{T}T\) makes the Hessian’s quadratic form vanish in direction \(v\)?
If the convex support is a line segment in \(\mathbb R^2\), in what sense should the relative interior be understood?
For the three-point triangle example, why can no finite parameter pair produce a mean on an edge of the triangle?
Does the theorem that finite-parameter means lie in the relative interior prove that every relative-interior point is attainable for every exponential family? Explain.