Natural Parameter Spaces and Regularity

From Canonical Form to a Parameter Domain

In canonical exponential-family form, the parameter enters through an exponential tilt of a fixed carrier measure. But not every numerical value of the natural parameter necessarily produces a finite normalizing constant. The set of values that do is the natural parameter space; its shape determines whether small changes in the parameter remain admissible.

Let $T(x)=(T_1(x),\ldots,T_d(x))^{\mathsf T}$ be a measurable, finite-valued statistic, and let $\nu$ be a fixed, nonzero, sigma-finite measure on the sample space. For $\eta\in\mathbb R^d$, define the normalizing integral

Z(\eta)=\int \exp\{\eta^{\mathsf T}T(x)\}\,\nu(dx).

The measure $\nu$ includes the fixed carrier factor; for example, if the canonical form is written with carrier $h(x)$ relative to a measure $\mu$, then $\nu(dx)=h(x)\mu(dx)$. Because the integrand is positive and $\nu$ is nonzero, $Z(\eta)>0$, though it may be infinite.

Definition: The natural parameter space is $$ \mathcal N=\{\eta\in\mathbb R^d: Z(\eta)<\infty\}. $$ For $\eta\in\mathcal N$, the log-normalizer is $A(\eta)=\log Z(\eta)$, and the corresponding normalized distribution has density $$ f_\eta(x)=\exp\{\eta^{\mathsf T}T(x)-A(\eta)\} $$ with respect to $\nu$. The interior $\mathcal N^\circ$ consists of points having an open neighborhood contained in $\mathcal N$; the boundary $\partial\mathcal N$ consists of points every neighborhood of which intersects both $\mathcal N$ and its complement.

Finiteness is the key condition in the definition. A point on the boundary can belong to $\mathcal N$ if its normalizing integral is finite, or fail to belong if the integral diverges there. Thus “boundary point” and “inadmissible parameter” are not synonymous.

Convexity of the Natural Parameter Space

The domain of finiteness has a useful geometric property that follows directly from the integral definition.

Theorem: $\mathcal N$ is convex. In other words, if $\eta_0,\eta_1\in\mathcal N$ and $0\leq\lambda\leq1$, then $(1-\lambda)\eta_0+\lambda\eta_1\in\mathcal N$.

Proof. The cases $\lambda=0$ and $\lambda=1$ are immediate. For $0<\lambda<1$, write $\eta_\lambda=(1-\lambda)\eta_0+\lambda\eta_1$. Pointwise,

\exp\{\eta_\lambda^{\mathsf T}T(x)\} = \bigl(\exp\{\eta_0^{\mathsf T}T(x)\}\bigr)^{1-\lambda} \bigl(\exp\{\eta_1^{\mathsf T}T(x)\}\bigr)^\lambda.

Hölder’s inequality, with exponents $1/(1-\lambda)$ and $1/\lambda$, gives

Z(\eta_\lambda) \leq Z(\eta_0)^{1-\lambda}Z(\eta_1)^\lambda <\infty.

Therefore $\eta_\lambda\in\mathcal N$, proving convexity. $\square$

Convexity does not imply that $\mathcal N$ is open or closed. It may include some boundary points and exclude others; it can even have empty interior. These distinctions matter when using local calculus in a likelihood argument.

Worked Examples: Open and Boundary-Inclusive Domains

Worked Example: Bernoulli Natural Parameter Space

Let $x\in\{0,1\}$, $T(x)=x$, and give both outcomes unit carrier weight. Then

$Z(\eta)=\sum_{x=0}^1 e^{\eta x}=1+e^\eta.$

For every real $\eta$, $1+e^\eta$ is finite and positive. Hence $\mathcal N=\mathbb R$, which is open, and every finite natural parameter is an interior point. There are no finite boundary points of this domain.

Worked Example: A Finite Boundary Point

Take the sample space $\{1,2,\ldots\}$, counting measure, $T(n)=n$, and carrier weights $\nu(\{n\})=1/n^2$. The normalizer is

$Z(\eta)=\sum_{n=1}^{\infty} e^{\eta n}/n^2.$

If $\eta<0$, then $e^{\eta n}\leq1$, so $Z(\eta)\leq\sum_{n=1}^{\infty}1/n^2<\infty$. At $\eta=0$, the normalizer is exactly the convergent series $\sum_{n=1}^{\infty}1/n^2$. If $\eta>0$, the terms $e^{\eta n}/n^2$ do not tend to zero, so the series diverges. Thus $\mathcal N=(-\infty,0]$, with interior $(-\infty,0)$ and included boundary point $0$. The distribution at $\eta=0$ is valid after normalization, even though that parameter is not an interior point.

These examples separate two questions: whether a distribution can be normalized at a parameter, and whether that parameter has a neighborhood of normalizable distributions. Finiteness answers the first; membership in $\mathcal N^\circ$ answers the second.

Why Interior Points Support Local Smoothness

An open parameter domain permits small perturbations in every coordinate while keeping the parameter admissible. In an exponential family, an interior point also gives enough room around the parameter to control the normalizing integral and its derivatives. The following result states this precisely for finite-dimensional $T$.

Theorem: If $\eta_0\in\mathcal N^\circ$, then $Z$ and $A=\log Z$ are infinitely differentiable in a neighborhood of $\eta_0$.

Proof. Since $\eta_0$ is interior, choose $a>0$ small enough that the closed coordinate cube $\{\eta:|\eta_j-\eta_{0j}|\leq3a,\ j=1,\ldots,d\}$ lies in $\mathcal N$. In particular, every corner $\eta_0+2as$, for $s\in\{-1,1\}^d$, belongs to $\mathcal N$.

For a multi-index $\alpha=(\alpha_1,\ldots,\alpha_d)$ of nonnegative integers, formal differentiation of the integrand gives $T(x)^\alpha e^{\eta^{\mathsf T}T(x)}$, where $T(x)^\alpha=\prod_jT_j(x)^{\alpha_j}$. We now give an integrable bound that justifies these derivatives. For $|\eta_j-\eta_{0j}|\leq a$,

e^{\eta^{\mathsf T}T(x)} \leq e^{\eta_0^{\mathsf T}T(x)}e^{a\|T(x)\|_1}.

If $k=\sum_j\alpha_j$, then $|T(x)^\alpha|\leq\|T(x)\|_1^k$. There is a finite constant $C_{k,a}$ such that $y^k\leq C_{k,a}e^{ay}$ for every $y\geq0$. Consequently,

|T(x)^\alpha|e^{\eta^{\mathsf T}T(x)} \leq C_{k,a}e^{\eta_0^{\mathsf T}T(x)}e^{2a\|T(x)\|_1}.

For every $u\in\mathbb R^d$, $\exp\{2a\|u\|_1\}\leq\sum_{s\in\{-1,1\}^d}\exp\{2as^{\mathsf T}u\}$: one of the sign vectors attains $s^{\mathsf T}u=\|u\|_1$, and its term is included in the sum. Therefore the last bound is at most

C_{k,a}\sum_{s\in\{-1,1\}^d} \exp\{(\eta_0+2as)^{\mathsf T}T(x)\}.

This is integrable with respect to $\nu$, because its integral is the finite sum $C_{k,a}\sum_s Z(\eta_0+2as)$, and all the corner parameters lie in $\mathcal N$. The bound holds uniformly in the smaller cube $|\eta_j-\eta_{0j}|\leq a$. The dominated convergence theorem therefore justifies differentiation under the integral for each coordinate. Applying the same argument to each derivative integrand, for every multi-index $\alpha$, proves that all orders of partial derivatives exist and are continuous there. Thus $Z$ is infinitely differentiable near $\eta_0$. Since $Z(\eta)>0$, composition with the smooth logarithm shows that $A=\log Z$ is also infinitely differentiable. $\square$

The bound uses the corners at radius $2a$, not a smaller radius: the extra room controls both changes in $\eta$ and the polynomial factors introduced by differentiation. This local smoothness is one reason likelihood arguments commonly use an open natural parameter domain.

When the Natural Parameter Space Has Empty Interior

Worked Example: A Natural Parameter Space Consisting of One Point

Let the sample space be the integers, with counting measure, $T(k)=k$, and carrier weights $\nu(\{k\})=e^{-\sqrt{|k|}}$. At $\eta=0$,

$Z(0)=\sum_{k\in\mathbb Z}e^{-\sqrt{|k|}}<\infty.$

To check convergence, group positive integers with $m^2\leq k<(m+1)^2$. There are $2m+1$ integers in this group, and each summand is at most $e^{-m}$. The resulting bound $\sum_{m\geq1}(2m+1)e^{-m}$ converges; the negative integers give the same bound, and the $k=0$ term is finite.

If $\eta>0$, the terms for positive $k$ are $e^{\eta k-\sqrt{k}}$, which do not tend to zero because $\eta k-\sqrt{k}\to\infty$. If $\eta<0$, the terms for negative $k$ are $e^{|\eta||k|-\sqrt{|k|}}$, which also do not tend to zero. Thus $Z(\eta)=\infty$ for every $\eta\neq0$, and $\mathcal N=\{0\}$. This space has empty interior in $\mathbb R$. A normalized distribution exists at its only parameter, but there is no open neighborhood of admissible natural parameters.

This example shows why one should not assume that a finite normalizer at a single point automatically gives local regularity. The open-neighborhood condition is a substantive feature of the model, not a consequence of normalization at the parameter under study.

Openness and Regular Likelihood Arguments

When the parameter space is open, an interior parameter can be perturbed a small distance in any direction and remain in the model. This supports local Taylor expansions and unconstrained derivative arguments: a local optimum in an open domain cannot be blocked merely because a small perturbation is inadmissible. At a boundary point, some directions may leave the parameter space, so the same two-sided local reasoning need not apply.

For this reason, the regular version of a natural exponential family is commonly taken to have an open natural parameter domain, often $\mathcal N^\circ$ or an open subset of it. The distinction is particularly important when a boundary point has a finite normalizer: it defines a legitimate distribution, but it does not thereby acquire the local neighborhood required for interior-point arguments.

Openness alone is not a complete guarantee of regular likelihood theory. Other conditions can matter, including identifiability, behavior of the support, and suitable integrability. The result proved above establishes local smoothness of the normalizer under the stated exponential-family construction; it does not assert every condition needed for every likelihood theorem.

Common Mistake

Wrong reasoning: “The normalizing integral is finite at this parameter, so the parameter is an interior point and ordinary local likelihood arguments apply.”
Correction: Finiteness establishes membership in $\mathcal N$, not in $\mathcal N^\circ$. Check whether an entire neighborhood is contained in the natural parameter space. The series example with $\mathcal N=(-\infty,0]$ has a valid distribution at $0$, but $0$ is a boundary point.

Key takeaway: Define the natural parameter space by finiteness of the normalizing integral. Its convexity follows from Hölder’s inequality, but it need not be open. Interior points provide room for local perturbations and, in finite-dimensional exponential families, local smoothness of the normalizer; finite boundary points do not become interior merely by being normalizable.

Check Your Understanding

Use the integral definition of the natural parameter space to answer the following.

Why does Hölder’s inequality imply that the natural parameter space is convex?
In the model with $Z(\eta)=\sum_{n=1}^{\infty}e^{\eta n}/n^2$, why is $0$ in the natural parameter space but not in its interior?
What additional property, beyond a finite normalizer at $\eta_0$, is needed for $\eta_0$ to be an interior point?
In the smoothness proof, what role does the integrable bound using the corner parameters play?
Does an open natural parameter domain by itself guarantee every regularity condition needed for likelihood inference? Explain.