Why Mean Parameters and Convex Duality Belong Together
An exponential family can describe the same distribution in two useful ways. Its natural parameter controls how the sufficient statistic is tilted, while its mean parameter records the expected value of that statistic. The log-partition function connects the two: its gradient gives the mean. Convex duality then turns this connection into a direct expression for the likelihood maximized over natural parameters.
This relationship is useful even before carrying out a likelihood calculation. It tells us which mean values a model can represent, gives a formula for the supremum of the likelihood kernel, and explains why boundary observations require care. The conjugate formula itself does not guarantee that a supremum is attained at a finite parameter.
Exponential-Family Setup and Mean Parameters
Assume the integral defining \(A(\eta)\) is finite on an open set \(\Theta\). To differentiate it at a particular \(\eta\), assume also that for each coordinate \(j\), a neighborhood of \(\eta\) is available in the natural parameter space with enough exponential-moment control to dominate the difference quotients. A sufficient local condition is that \(\eta\pm 2\delta e_j\in\Theta\) for some \(\delta>0\), where \(e_j\) is the \(j\)th coordinate vector. This condition permits the factor \(|T_j(X)|\) arising in differentiation to be bounded by a constant times the sum of the two corresponding exponential tilts. Differentiation under the integral is then justified by dominated convergence.
Proof. Write \(Z(\eta)=\exp\{A(\eta)\}\). For each coordinate \(j\), the stated domination justifies differentiating the integral defining \(Z\): $$ \frac{\partial Z(\eta)}{\partial\eta_j} =\int T_j(x)h(x)\exp\{\eta^{\mathsf T}T(x)\}\,d\nu(x). $$ Since \(A(\eta)=\log Z(\eta)\), division by \(Z(\eta)\) gives $$ \frac{\partial A(\eta)}{\partial\eta_j} =\frac{\int T_j(x)h(x)\exp\{\eta^{\mathsf T}T(x)\}\,d\nu(x)} {\int h(x)\exp\{\eta^{\mathsf T}T(x)\}\,d\nu(x)} =E_\eta[T_j(X)]. $$ This holds for each coordinate, proving the identity. \(\square\)
Thus the mean-parameter space attained by the family is the image \(\{\nabla A(\eta):\eta\in\Theta\}\). That image need not include every point on the boundary of the convex set of possible statistic values. The distinction matters when interpreting a conjugate at a boundary point.
Worked Example: Bernoulli Mean and Its Convex Conjugate
Let \(X\) be Bernoulli with success probability \(p\). Its canonical representation has \(T(X)=X\), natural parameter \(\eta=\log\{p/(1-p)\}\), and $$ A(\eta)=\log(1+e^\eta). $$ Differentiating gives $$ A'(\eta)=\frac{e^\eta}{1+e^\eta}=p. $$ The natural parameter therefore maps to the success probability, which is the mean of \(X\).
For a scalar argument \(m\), define the convex conjugate $$ A^*(m)=\sup_{\eta\in\mathbb{R}}\{\eta m-A(\eta)\}. $$ For \(0<m<1\), the derivative of the expression being maximized is \(m-e^\eta/(1+e^\eta)\). Setting it to zero gives \(e^\eta=m/(1-m)\), or \(\eta=\log\{m/(1-m)\}\). Substitution yields $$ A^*(m)=m\log m+(1-m)\log(1-m). $$ At \(m=0\), the supremum is \(\sup_\eta-\log(1+e^\eta)=0\), approached as \(\eta\to-\infty\). At \(m=1\), it is \(\sup_\eta\{-\log(1+e^{-\eta})\}=0\), approached as \(\eta\to+\infty\). Hence the same formula extends to the endpoints using \(0\log 0=0\). If \(m<0\) or \(m>1\), the supremum is \(+\infty\).
For \(n\) Bernoulli observations with observed average \(\bar X=3/5\), the log-likelihood per observation, omitting no parameter-dependent terms, is \(\eta(3/5)-A(\eta)\). Its supremum is $$ A^*(3/5)=\frac35\log\frac35+\frac25\log\frac25. $$ The maximizing parameter is \(\eta=\log(3/2)\), which corresponds to mean \(3/5\).
The Convex Conjugate and the Likelihood Kernel
For a convex function \(A\) on \(\Theta\), its convex conjugate is the extended-real-valued function $$ A^*(\mu)=\sup_{\eta\in\Theta}\{\eta^{\mathsf T}\mu-A(\eta)\}. $$ The value is allowed to be \(+\infty\). The log-partition function is convex: if \(\eta_1,\eta_2\in\Theta\) and \(0<\lambda<1\), Hölder's inequality applied to the normalizing integral gives $$ \exp A(\lambda\eta_1+(1-\lambda)\eta_2) \leq \exp\{\lambda A(\eta_1)+(1-\lambda)A(\eta_2)\}. $$ Taking logarithms proves convexity.
The conjugate has a precise relationship to the mean parameter. The result below states the equality at a mean generated by a natural parameter; it does not require that every possible \(\mu\) be attainable.
Proof. For a differentiable convex function, the tangent plane at \(\eta\) lies below the function: for every \(\lambda\in\Theta\), $$ A(\lambda)\geq A(\eta)+\nabla A(\eta)^{\mathsf T}(\lambda-\eta). $$ Set \(\mu=\nabla A(\eta)\) and rearrange: $$ \lambda^{\mathsf T}\mu-A(\lambda)\leq\eta^{\mathsf T}\mu-A(\eta). $$ The right-hand side is therefore an upper bound for every term in the supremum defining \(A^*(\mu)\). Taking \(\lambda=\eta\) achieves equality, so the supremum equals \(\eta^{\mathsf T}\mu-A(\eta)\). \(\square\)
For independent observations \(X_1,\ldots,X_n\), let \(\bar T=n^{-1}\sum_{i=1}^n T(X_i)\). The log-likelihood is $$ \ell(\eta)=\sum_{i=1}^n\log h(X_i)+n\{\eta^{\mathsf T}\bar T-A(\eta)\}. $$ Consequently, $$ \sup_{\eta\in\Theta}\ell(\eta) =\sum_{i=1}^n\log h(X_i)+nA^*(\bar T). $$ This is a supremum identity: it remains valid when the supremum is not achieved by a finite \(\eta\). It also shows why the conjugate is the maximized likelihood kernel as a function of the observed average sufficient statistic.
There is a related expression when the model is parameterized by a candidate mean \(\mu\). Suppose \(\mu=\nabla A(\eta_\mu)\) and such an \(\eta_\mu\) is specified. The theorem gives \(A^*(\mu)=\eta_\mu^{\mathsf T}\mu-A(\eta_\mu)\), so the parameter-dependent part of the log-likelihood at that candidate is $$ n\{\eta_\mu^{\mathsf T}\bar T-A(\eta_\mu)\} =n\{A^*(\mu)+\eta_\mu^{\mathsf T}(\bar T-\mu)\}. $$ The likelihood in mean coordinates therefore includes both the conjugate term and a term measuring the discrepancy between the observed statistic and the candidate mean.
Worked Example: Poisson Mean and a Boundary Observation
For a Poisson random variable with rate \(\lambda>0\), take \(T(X)=X\), \(\eta=\log\lambda\), and \(A(\eta)=e^\eta\). Then \(A'(\eta)=e^\eta=\lambda\). For \(m>0\), maximizing \(\eta m-e^\eta\) gives \(e^\eta=m\), so $$ A^*(m)=m\log m-m. $$ At \(m=0\), the expression is \(-e^\eta\), whose supremum is \(0\) as \(\eta\to-\infty\); this agrees with the limit of \(m\log m-m\) as \(m\downarrow0\). For \(m<0\), taking \(\eta\to-\infty\) makes \(\eta m-e^\eta\to+\infty\), so \(A^*(m)=+\infty\).
Suppose three observations are \(1,2,3\). Their average is \(\bar X=2\). The likelihood kernel per observation is \(2\eta-e^\eta\), maximized where \(2-e^\eta=0\), that is, at \(\eta=\log2\). Its value is \(2\log2-2\), exactly \(A^*(2)\). If all three observations were zero, then \(\bar X=0\); the supremum of the kernel \(-e^\eta\) would be \(0\), but no finite \(\eta\) achieves it. The conjugate still gives the correct supremum.
Worked Example: Gaussian Location with Known Variance
Let \(X\sim N(\mu,\sigma^2)\), with known \(\sigma^2>0\) and unknown location \(\mu\). In the canonical representation, \(T(X)=X\), \(\eta=\mu/\sigma^2\), and \(A(\eta)=\sigma^2\eta^2/2\); the remaining terms belong to the base density. Thus \(A'(\eta)=\sigma^2\eta=\mu\).
For a candidate mean \(m\), maximize \(\eta m-\sigma^2\eta^2/2\). Completing the square gives $$ \eta m-\frac{\sigma^2\eta^2}{2} =\frac{m^2}{2\sigma^2}-\frac{\sigma^2}{2}\left(\eta-\frac{m}{\sigma^2}\right)^2. $$ The maximum is attained at \(\eta=m/\sigma^2\), and \(A^*(m)=m^2/(2\sigma^2)\).
For observations \(1,2,3\) and \(\sigma^2=4\), \(\bar X=2\). The maximizing natural parameter is \(2/4=1/2\), and the log-likelihood kernel per observation at this value is $$ \frac12(2)-\frac{4(1/2)^2}{2}=1-\frac12=\frac12. $$ The conjugate gives the same value: \(A^*(2)=2^2/(2\cdot4)=1/2\). At a candidate mean \(m=1\), the natural parameter is \(1/4\), and the kernel is \((1/4)(2)-4(1/4)^2/2=1/2-1/8=3/8\). The candidate-mean formula gives the same result: \(A^*(1)+(1/4)(2-1)=1/8+1/4=3/8\).
Interpretation and a Common Pitfall
The gradient identity gives a coordinate conversion: natural parameters specify the exponential tilt, and their gradient means specify the expected sufficient statistics. Convex conjugacy then summarizes how large the likelihood kernel can be for a given observed average statistic. In applications, this is a useful way to recognize the structure of a likelihood calculation and to keep track of which values belong to the model's mean-parameter space.
Common mistake: It is tempting to write the likelihood in mean coordinates as simply \(-nA^*(\mu)\), or to assume that every finite mean has a corresponding finite natural parameter. Neither statement is generally correct. The maximized likelihood kernel at observed \(\bar T\) is \(nA^*(\bar T)\), while at a candidate mean \(\mu\) it is \(n\{A^*(\mu)+\eta_\mu^{\mathsf T}(\bar T-\mu)\}\). Also, a boundary mean can yield a finite conjugate value even when the supremum is approached only as the natural parameter tends to infinity in magnitude. Always specify the domain and distinguish a supremum from an attained maximum.
Check Your Understanding
Use the definitions and calculations above to check each claim.
- For a canonical exponential family, what is the mean parameter in terms of the log-partition function, and what condition justifies differentiating the normalizing integral?
- State the definition of \(A^*(\mu)\). If \(\mu=\nabla A(\eta)\), why does the supremum equal \(\eta^{\mathsf T}\mu-A(\eta)\)?
- For a Poisson family, calculate \(A^*(3)\) and identify the natural parameter at which the supremum is attained.
- Explain why the conjugate formula at a boundary mean need not correspond to any finite natural parameter.
- Write the parameter-dependent log-likelihood in candidate mean coordinates, and explain why it is not generally just \(nA^*(\mu)\).