Why Mean Parameters Have a Geometric Domain
In an exponential family, the canonical parameter controls the distribution through a linear tilt of a statistic. The expectation of that statistic is often easier to interpret, but it cannot take arbitrary values: its possible values are constrained by the statistic’s support. Convex geometry describes those constraints, while the log-partition function connects the two parameterizations.
We use a \(d\)-dimensional canonical exponential family with density, relative to a fixed measure \(\nu\), \(f_\eta(x)=\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}\), where \(T(x)\in\mathbb R^d\) and \(A(\eta)=\log Z(\eta)\), with \(Z(\eta)=\int \exp\{\eta^\mathsf{T}T(x)\}\,d\nu(x)\). The natural parameter domain is \(D=\{\eta:Z(\eta)<\infty\}\); its interior is denoted by \(\Theta\). Throughout, assume \(\Theta\) is nonempty and open. The family is minimal when no nonzero vector \(a\in\mathbb R^d\) has \(a^\mathsf{T}T(x)\) constant \(\nu\)-almost everywhere. Equivalently, the convex support defined below is full-dimensional.
The tilted distribution has a strictly positive density relative to \(\nu\), so it has the same null sets as the carrier measure. Thus changing \(\eta\) changes the probabilities within the support, but does not change the convex support \(C\). Minimality rules out an affine constraint on \(T\), such as all values lying on a line in a two-dimensional statistic space.
The Gradient Gives the Mean Parameter
The identity \(\nabla A(\eta)=E_\eta[T(X)]\) follows by differentiating the normalizing integral. The local domination in this calculation matters: finiteness of the derivative integrand at just one parameter value would not, by itself, justify differentiating under the integral.
To justify the result, fix a coordinate \(j\). Since \(\Theta\) is open, choose \(\delta>0\) small enough that \(\eta+\delta e_j\) and \(\eta-\delta e_j\) both belong to \(\Theta\), where \(e_j\) is the \(j\)th coordinate vector. For \(|s|\leq\delta/2\) and every real \(t\), \(|t|e^{st}\leq K_\delta(e^{\delta t}+e^{-\delta t})\) for a finite constant \(K_\delta\): indeed, \(|t|e^{-\delta|t|/2}\) is bounded. With \(t=T_j(x)\), this gives \[ |T_j(x)|e^{(\eta+s e_j)^\mathsf{T}T(x)} \leq K_\delta\left\{ e^{(\eta+\delta e_j)^\mathsf{T}T(x)} +e^{(\eta-\delta e_j)^\mathsf{T}T(x)} \right\}. \] The right side is integrable with respect to \(\nu\), independently of \(s\), because both neighboring parameters have finite normalizers. The mean-value theorem bounds the difference quotients of the integrand by the same integrable function. Dominated convergence therefore permits differentiation under the integral: \[ \frac{\partial Z(\eta)}{\partial\eta_j} =\int T_j(x)e^{\eta^\mathsf{T}T(x)}\,d\nu(x). \] Dividing by \(Z(\eta)\) gives \(\partial A(\eta)/\partial\eta_j=E_\eta[T_j(X)]\). Applying this argument to each coordinate proves the result. \(\square\)
Worked Example: Bernoulli Mean and Canonical Parameter
Let \(X\in\{0,1\}\), with counting measure as \(\nu\), and take \(T(X)=X\). The normalizer and log-partition function are \(Z(\eta)=1+e^\eta\) and \(A(\eta)=\log(1+e^\eta)\), for every \(\eta\in\mathbb R\).
Differentiating gives \(\mu(\eta)=A'(\eta)=e^\eta/(1+e^\eta)\). For \(\eta=\log 3\), the mean is \(3/(1+3)=0.75\). Conversely, solving \(0.75=e^\eta/(1+e^\eta)\) gives \(e^\eta=0.75/0.25=3\), so \(\eta=\log 3\). The convex support is \(C=[0,1]\), and the means attained at finite canonical parameters are exactly its interior, \((0,1)\).
Convex Support and the Image of the Mean Map
For a full-dimensional convex support, every mean at a finite interior canonical parameter lies in \(\operatorname{int}C\). First, an expectation of a random vector taking values in \(C\) lies in \(C\). If that expectation were on the boundary, the supporting-hyperplane theorem for closed convex sets would give a nonzero \(a\) and a constant \(b\) such that \(a^\mathsf{T}t\leq b\) on \(C\), while \(a^\mathsf{T}\mu=b\). Then \(b-a^\mathsf{T}T(X)\) is nonnegative and has expectation zero, so it is zero almost surely. This puts the essential range in a hyperplane, contradicting full dimensionality.
Minimality also gives a one-to-one map from canonical parameters to means. The argument uses strict convexity of \(A\). For distinct \(\eta_0,\eta_1\in\Theta\) and \(0<t<1\), Hölder’s inequality gives \(Z((1-t)\eta_0+t\eta_1)\leq Z(\eta_0)^{1-t}Z(\eta_1)^t\). Equality in Hölder would require \(e^{\eta_0^\mathsf{T}T}\) and \(e^{\eta_1^\mathsf{T}T}\) to be proportional \(\nu\)-almost everywhere. That would make \((\eta_1-\eta_0)^\mathsf{T}T\) constant almost everywhere, contrary to minimality. Taking logarithms shows that \(A\) is strictly convex.
For injectivity, suppose \(\eta_0\ne\eta_1\). A differentiable strictly convex function satisfies the strict supporting inequalities \(A(\eta_1)>A(\eta_0)+\nabla A(\eta_0)^\mathsf{T}(\eta_1-\eta_0)\) and \(A(\eta_0)>A(\eta_1)+\nabla A(\eta_1)^\mathsf{T}(\eta_0-\eta_1)\). Adding them yields \((\nabla A(\eta_1)-\nabla A(\eta_0))^\mathsf{T}(\eta_1-\eta_0)>0\). The gradients cannot be equal. Combined with the preceding boundary argument, this proves the first claim.
For the bijection claim, boundedness of \(T\) and finiteness of \(\nu\) imply \(Z(\eta)<\infty\) for every \(\eta\), so \(\Theta=\mathbb R^d\). Fix any \(\mu\in\operatorname{int}C\). Because \(\mu\) is interior, there are finitely many points \(t_1,\ldots,t_m\) in the essential range and \(r>0\) such that \(\max_i u^\mathsf{T}(t_i-\mu)\geq r\) for every unit vector \(u\). Choose \(\epsilon\) with \(0<\epsilon<r\). For each \(i\), the event \(B_i=\{x:\|T(x)-t_i\|<\epsilon\}\) has positive measure; write \(c_i=\nu(B_i)>0\). For any \(\eta\), integration over \(B_i\) gives \(Z(\eta)\geq c_i\exp\{\eta^\mathsf{T}t_i-\epsilon\|\eta\|\}\). Consequently, for \(g_\mu(\eta)=A(\eta)-\eta^\mathsf{T}\mu\), \[ g_\mu(\eta)\geq (r-\epsilon)\|\eta\|+\min_i\log c_i. \] Thus \(g_\mu(\eta)\to\infty\) as \(\|\eta\|\to\infty\). It is continuous, so it attains a global minimum at some \(\eta_\mu\in\mathbb R^d\). At this unconstrained minimum its gradient is zero: \(\nabla g_\mu(\eta_\mu)=\nabla A(\eta_\mu)-\mu=0\). Hence \(\nabla A(\eta_\mu)=\mu\). Every interior point is attained, and injectivity makes its preimage unique. \(\square\)
Worked Example: A Three-Category Multinomial Family
Let one observation have categories \(0,1,2\). Use \(T(0)=(0,0)^\mathsf{T}\), \(T(1)=(1,0)^\mathsf{T}\), and \(T(2)=(0,1)^\mathsf{T}\). Then \(A(\eta)=\log(1+e^{\eta_1}+e^{\eta_2})\), and \(\nabla A(\eta)=(p_1,p_2)^\mathsf{T}\), where \(p_1=e^{\eta_1}/(1+e^{\eta_1}+e^{\eta_2})\) and \(p_2=e^{\eta_2}/(1+e^{\eta_1}+e^{\eta_2})\).
At \(\eta=(\log 2,0)^\mathsf{T}\), the normalizing denominator is \(1+2+1=4\), so the mean is \((p_1,p_2)=(1/2,1/4)\); the remaining category probability is \(p_0=1/4\). For any interior point \((p_1,p_2)\) of the triangle \(p_1>0,\ p_2>0,\ p_1+p_2<1\), the inverse is \(\eta_1=\log(p_1/p_0)\), \(\eta_2=\log(p_2/p_0)\), where \(p_0=1-p_1-p_2\). Substituting the calculated probabilities recovers \((\log 2,0)\). The convex support is the triangle with vertices \((0,0),(1,0),(0,1)\); its interior, not its edges, corresponds to finite canonical parameters.
Worked Example: Normal Mean with Known Variance
Suppose \(X\sim N(\mu,\sigma^2)\), where \(\sigma^2>0\) is fixed. Relative to the base measure with density \(N(0,\sigma^2)\), the density ratio is \(\exp\{\eta X-\sigma^2\eta^2/2\}\) when \(\eta=\mu/\sigma^2\). Thus \(T(X)=X\) and \(A(\eta)=\sigma^2\eta^2/2\).
The mean map is \(\nabla A(\eta)=\sigma^2\eta\), whose inverse is \(\eta=\mu/\sigma^2\). For \(\sigma^2=4\) and \(\eta=1/2\), the mean is \(4(1/2)=2\), and the inverse gives \(2/4=1/2\). The convex support is all of \(\mathbb R\), so its interior is all of \(\mathbb R\). This example has an unbounded statistic, but its map is still onto; boundedness in the theorem above is a sufficient condition, not a necessary one.
When the Interior Is Not Fully Reached
Minimality ensures uniqueness of the canonical parameter for any mean that is attained. It does not, by itself, guarantee that every point in the convex-support interior is attained. For an unbounded statistic, or when the log-partition function remains finite at a boundary of its natural parameter domain, additional conditions are needed for full coverage. In particular, the bounded-statistic theorem above should not be silently extended to all regular families.
For a concrete illustration, let \(T\) take values \(n=1,2,\ldots\) with base masses proportional to \(e^{-\sqrt n}\). The normalizer is finite at \(\eta=0\), and both \(\sum_n e^{-\sqrt n}\) and \(\sum_n n e^{-\sqrt n}\) are finite. For \(\eta>0\), the terms \(e^{\eta n-\sqrt n}\) do not even tend to zero, so the normalizer diverges. The open natural parameter space is therefore \((-\infty,0)\). Its convex support is \([1,\infty)\), with interior \((1,\infty)\). As \(\eta\) increases toward zero from below, the mean tends to the finite base-distribution mean \(m_0\); hence the means at parameters in \((-\infty,0)\) stay below \(m_0\). Interior values larger than \(m_0\) are not attained. This is a limitation on coverage, not on uniqueness among the means that are attained.
Check Your Understanding
Use the definitions and conditions in this tutorial to answer the following.
- What does minimality rule out, and how does that property support strict convexity of the log-partition function?
- Why is a local integrable bound needed when differentiating the normalizing integral? Which neighboring parameter values provide one?
- For the three-category example, what is the convex support, and which category probabilities correspond to its interior?
- Does injectivity alone imply that every point in the convex-support interior is attained? Explain the distinction.
- In the discrete boundary example, why does finiteness of the normalizer and mean at \(\eta=0\) fail to establish coverage by parameters in the open natural parameter space?