Natural Parameter Spaces from Integrability

Why Integrability Determines the Parameter Space

A canonical exponential-family expression has the form $h(x)\exp\{\eta\cdot T(x)\}$. Writing down this expression does not guarantee that it can be normalized into a probability distribution: its integral or sum may be infinite. The natural parameter space consists precisely of the parameter values for which the normalizing quantity is finite and positive. Its shape depends on the carrier, the statistic, and the support—not just on the algebraic appearance of the exponential.

Here $x$ lies in a measurable sample space, $\mu$ is a base measure, $h$ is a nonnegative measurable carrier function, and $T(x)=(T_1(x),\ldots,T_d(x))$ is a measurable statistic with finite coordinates wherever the carrier contributes. The natural parameter $\eta$ lies in $\mathbb R^d$, and $\eta\cdot T(x)=\sum_{j=1}^d\eta_jT_j(x)$. In a discrete model, integration with respect to $\mu$ is a sum.

Definition: The normalizing integral for the representation $h(x)\exp\{\eta\cdot T(x)\}$ is $$ Z(\eta)=\int h(x)\exp\{\eta\cdot T(x)\}\,\mu(dx). $$ The natural parameter space is $$ \mathcal N=\{\eta\in\mathbb R^d:0<Z(\eta)<\infty\}. $$ Only parameters in $\mathcal N$ yield a finite, positive normalizer and hence a probability density or mass function proportional to this expression.

The positivity condition is usually automatic when the carrier is positive on a set of positive base measure: the exponential factor is strictly positive there. Stating it explicitly makes the definition valid even for a degenerate carrier. The representation matters as well: as emphasized in the earlier material on carrier measures and equivalent representations, the integral must be computed using the chosen carrier and base measure together.

A Reliable Integrability Check

For each proposed $\eta$, examine $h(x)\exp\{\eta\cdot T(x)\}$ on the full support. A useful calculation often separates the support into regions: infinite tails for unbounded supports, neighborhoods of endpoints where the carrier or statistic is singular, and an interior region where the integrand is bounded. For sums, test whether the terms are summable; for integrals, test local integrability and tail behavior.

A finite expression obtained by formal algebra is not enough. In particular, completing a square may simplify an integrand, but the resulting integral still has to converge. Likewise, checking only the interior of a continuous support misses possible divergence at its endpoints.

Worked Example: A Geometric Kernel on the Nonnegative Integers

Let the support be $x=0,1,2,\ldots$, with counting measure, $h(x)=1$, and $T(x)=x$. Find the natural parameter space.

The normalizing sum is a geometric series: $$ Z(\eta)=\sum_{x=0}^{\infty}\exp(\eta x) =\sum_{x=0}^{\infty}(e^\eta)^x. $$ A geometric series with nonnegative ratio $r$ converges exactly when $r<1$. Here $r=e^\eta$, so convergence holds exactly when $e^\eta<1$, or $\eta<0$. At $\eta=0$, every summand equals $1$, and the sum diverges; for $\eta>0$, the terms do not even tend to zero. Therefore $$ \mathcal N=(-\infty,0). $$ For $\eta<0$, the sum is $Z(\eta)=1/(1-e^\eta)$, which is finite and positive.

Worked Example: A Continuous Exponential Kernel

Take support $x>0$, Lebesgue measure, $h(x)=1$, and $T(x)=x$. The normalizing integral is $$ Z(\eta)=\int_0^\infty e^{\eta x}\,dx. $$ If $\eta<0$, direct integration gives $$ Z(\eta)=\left[\frac{e^{\eta x}}{\eta}\right]_{0}^{\infty} =-\frac{1}{\eta}, $$ which is finite and positive. If $\eta=0$, the integral is the area of an interval of infinite length, so it diverges. If $\eta>0$, the integrand grows without bound as $x$ increases, and the integral diverges. Thus the natural parameter space is again $(-\infty,0)$, but the convergence test is now an integral over a continuous, one-sided support.

Worked Example: A Two-Statistic Gaussian-Shaped Kernel

Let the support be the whole real line, with Lebesgue measure, $h(x)=1$, and $T(x)=(x,x^2)$. For $\eta=(\eta_1,\eta_2)$, $$ Z(\eta_1,\eta_2)=\int_{-\infty}^{\infty} \exp(\eta_1x+\eta_2x^2)\,dx. $$ If $\eta_2<0$, put $a=-\eta_2>0$. Completing the square gives $$ -a x^2+\eta_1x =-a\left(x-\frac{\eta_1}{2a}\right)^2 +\frac{\eta_1^2}{4a}. $$ Consequently, $$ Z(\eta_1,\eta_2) =\exp\left(\frac{\eta_1^2}{4a}\right) \int_{-\infty}^{\infty} \exp\left[-a\left(x-\frac{\eta_1}{2a}\right)^2\right]dx =\sqrt{\frac{\pi}{a}}\exp\left(\frac{\eta_1^2}{4a}\right), $$ finite and positive for every real $\eta_1$ when $\eta_2<0$.

If $\eta_2>0$, the quadratic term makes the exponent tend to $+\infty$ in both tails, so the integral diverges. If $\eta_2=0$, the integrand is $e^{\eta_1x}$. When $\eta_1>0$ it has an infinite integral on the positive tail; when $\eta_1<0$ it has an infinite integral on the negative tail; and when $\eta_1=0$ it is identically $1$. Thus every point with $\eta_2=0$ also gives divergence. The exact natural parameter space is $$ \mathcal N=\{(\eta_1,\eta_2)\in\mathbb R^2:\eta_2<0\}. $$

Worked Example: Endpoint Singularities on a Bounded Support

On $0<x<1$, take Lebesgue measure, $h(x)=1$, and $T(x)=(\log x,\log(1-x))$. The kernel becomes $x^{\eta_1}(1-x)^{\eta_2}$, so $$ Z(\eta_1,\eta_2)=\int_0^1x^{\eta_1}(1-x)^{\eta_2}\,dx. $$ Near $x=0$, the factor $(1-x)^{\eta_2}$ is bounded above and below by positive constants on a sufficiently short interval, for any fixed real $\eta_2$. Thus integrability there is equivalent to that of $\int_0^\epsilon x^{\eta_1}\,dx$. For a real $a$, this power integral is finite exactly when $a>-1$: if $a\ne-1$, its antiderivative is $x^{a+1}/(a+1)$, and if $a=-1$, it is $\log x$, which diverges at zero. Hence the condition at zero is $\eta_1>-1$.

At $x=1$, the factor $x^{\eta_1}$ is bounded above and below by positive constants near the endpoint. Applying the same power-integral test to $(1-x)^{\eta_2}$ gives the condition $\eta_2>-1$. On any closed interval strictly inside $(0,1)$, both factors are continuous and bounded, so there is no further restriction. Therefore $$ \mathcal N=\{(\eta_1,\eta_2):\eta_1>-1,\ \eta_2>-1\}. $$ Both inequalities are strict: equality at either boundary produces a logarithmic divergence.

Convexity of the Natural Parameter Space

Although integrability may be difficult to test at an individual parameter value, the set of all admissible parameters has a useful general structure. The result below requires no differentiability of $Z$.

Theorem: For a fixed nonnegative measurable carrier $h$, base measure $\mu$, and measurable statistic $T$, the natural parameter space $\mathcal N=\{\eta:0<Z(\eta)<\infty\}$ is convex. That is, if $\eta^{(0)},\eta^{(1)}\in\mathcal N$, then $(1-t)\eta^{(0)}+t\eta^{(1)}\in\mathcal N$ for every $0\le t\le1$.

To prove the claim for $0<t<1$, define $g_j(x)=h(x)\exp\{\eta^{(j)}\cdot T(x)\}$ for $j=0,1$. At points where $h(x)>0$, $$ h(x)\exp\{[(1-t)\eta^{(0)}+t\eta^{(1)}]\cdot T(x)\} =g_0(x)^{1-t}g_1(x)^t. $$ Where $h(x)=0$, the left side is zero and does not affect the integral. Hölder's inequality, with exponents $1/(1-t)$ and $1/t$, therefore gives $$ Z((1-t)\eta^{(0)}+t\eta^{(1)}) \le \left(\int g_0\,d\mu\right)^{1-t} \left(\int g_1\,d\mu\right)^t =Z(\eta^{(0)})^{1-t}Z(\eta^{(1)})^t<\infty. $$ The integral is also positive: both endpoint integrals are positive, so the set on which $h>0$ has positive $\mu$-measure; the exponential factor is positive there for every finite parameter. The cases $t=0$ and $t=1$ are the assumed endpoints. Thus every point on the segment belongs to $\mathcal N$, proving convexity. $\square$

Boundary Points and a Common Pitfall

Convexity does not imply that the natural parameter space is open, closed, or that its boundary is always excluded. Boundary membership must be decided by the original sum or integral. For example, on $x=0,1,\ldots$, let $h(x)=1/(x+1)^2$ and $T(x)=x$. At $\eta=0$, the normalizing sum is $\sum_{x=0}^{\infty}1/(x+1)^2$, which is finite. For $\eta<0$, each term is no larger than its value at zero, so the sum remains finite. For $\eta>0$, its terms $e^{\eta x}/(x+1)^2$ do not tend to zero and the sum diverges. Hence this representation has $\mathcal N=(-\infty,0]$: its finite boundary point is included.

Common mistake: “The natural parameter space is the set where the kernel looks well behaved, so a boundary point can be accepted or rejected by analogy with nearby values.” That reasoning is unreliable. A kernel can be finite at every individual support point while its sum diverges, as in the geometric example at $\eta=0$; conversely, a boundary sum can converge, as in the last example. The correction is to test the normalizing integral or sum at the boundary itself, including whether discrete terms fail to approach zero or continuous endpoint integrals diverge.

Key takeaway: Find the natural parameter space by solving $0<Z(\eta)<\infty$ for the chosen carrier representation. Check all tails and support boundaries, and do not infer endpoint inclusion from the behavior at interior parameter values.

Check Your Understanding

For each question, identify the convergence test that determines the answer.

For support $x=0,1,\ldots$, $h(x)=1$, and $T(x)=2x$, what is the natural parameter space?
For the kernel $e^{\eta x^2}$ on the whole real line with Lebesgue measure, determine the natural parameter space and justify the boundary case.
For $Z(\eta)=\int_0^1x^\eta\,dx$, which values of $\eta$ give a finite normalizing integral?
Why does Hölder's inequality imply that a line segment between two admissible natural parameters is also admissible?
Can a natural parameter space contain a finite boundary point? Give the criterion that decides this.