Natural-Parameter Spaces and Finite Normalizers

Why the Natural-Parameter Domain Matters

An exponential-family expression is a probability model only when its normalizing sum or integral is finite. The exponential term can be finite at every individual outcome and still have an infinite total over an unbounded support. The set of natural parameters for which normalization succeeds is therefore part of the model, not a technical detail to be filled in afterward.

The previous material introduced the log-partition function as the logarithm of a normalizing quantity. Here the question is more basic: for which values of the natural parameter does that quantity exist as a finite number? Finite-support examples give one answer; an integral over an unbounded support can give another.

Definition: Let $\nu$ be a nonzero positive measure on an outcome space, and let $T$ be a measurable real-valued function. For a scalar natural parameter $\eta$, define the normalizing quantity $$ Z(\eta)=\int \exp\{\eta T(y)\}\,d\nu(y). $$ The natural-parameter space is the set $\mathcal{H}=\{\eta\in\mathbb{R}:0<Z(\eta)<\infty\}$. For a discrete outcome space, the integral is a sum. The corresponding log-partition function is $A(\eta)=\log Z(\eta)$ on $\mathcal{H}$.

The measure $\nu$ is held fixed as $\eta$ varies. It can encode a sum over possible outcomes or an integral with a fixed weighting. Since the integrand is positive and $\nu$ is nonzero, $Z(\eta)$ cannot be zero; its possible failure here is that it is infinite. If $Z(\eta)$ is finite, dividing the kernel by $Z(\eta)$ gives total probability one.

A finite endpoint belongs to $\mathcal{H}$ precisely when the normalizer is finite there. Thus a domain may include one or both of its finite endpoints, exclude them, or have no finite endpoint. The fact that an exponential is finite at each outcome does not settle this question: the whole sum or integral must be checked.

Two General Facts About the Domain

The shape of the domain is constrained even before its exact endpoints are calculated. The first result says it is convex, which in one dimension means that it is an interval, possibly with either endpoint included.

Theorem: Suppose $\nu$ is a nonzero positive measure and $T$ is measurable. If $\eta_1,\eta_2\in\mathcal{H}$, then every $\eta=\lambda\eta_1+(1-\lambda)\eta_2$, with $0<\lambda<1$, also belongs to $\mathcal{H}$. Hence $\mathcal{H}$ is convex.

Proof. The integrand at the intermediate parameter factors as

\exp\{\eta T(y)\} = \bigl(\exp\{\eta_1T(y)\}\bigr)^\lambda \bigl(\exp\{\eta_2T(y)\}\bigr)^{1-\lambda}.

Both factors are measurable and nonnegative. Hölder's inequality, with exponents $1/\lambda$ and $1/(1-\lambda)$, gives

Z(\eta) \leq \left(\int \exp\{\eta_1T(y)\}\,d\nu(y)\right)^\lambda \left(\int \exp\{\eta_2T(y)\}\,d\nu(y)\right)^{1-\lambda} = Z(\eta_1)^\lambda Z(\eta_2)^{1-\lambda}<\infty.

The integral is also strictly positive: its integrand is strictly positive everywhere, and $\nu$ is nonzero. Thus $0<Z(\eta)<\infty$, proving that $\eta\in\mathcal{H}$. The cases $\lambda=0$ and $\lambda=1$ are the assumed endpoints themselves. $\square$

A second simple fact explains why finite-outcome models often have unrestricted natural parameters.

Proposition: If $\nu$ is concentrated on finitely many outcomes, assigns finite mass to each, and has positive total mass, then $Z(\eta)<\infty$ for every real $\eta$, provided $T$ is finite at those outcomes.

Proof. Write the support as $y_1,\ldots,y_m$, with masses $w_j\geq 0$. Then

Z(\eta)=\sum_{j=1}^{m} w_j\exp\{\eta T(y_j)\}.

For any finite real $\eta$, every $T(y_j)$ is finite, so each exponential and each summand is finite. A sum of finitely many finite nonnegative terms is finite. At least one mass is positive, and its exponential term is positive, so the sum is positive as well. Therefore every real $\eta$ belongs to $\mathcal{H}$. $\square$

Worked Examples: Sums and Integrals

Worked Example: Bernoulli Outcomes

Let $Y\in\{0,1\}$, with $T(y)=y$ and unit mass at each of the two outcomes. Find the natural-parameter space and log-partition function.

The normalizer is a two-term sum:

Z(\eta)=\sum_{y\in\{0,1\}}\exp(\eta y) =\exp(0)+\exp(\eta) =1+\exp(\eta).

For every finite real $\eta$, $\exp(\eta)$ is positive and finite, so $1+\exp(\eta)$ is also positive and finite. Consequently, $\mathcal{H}=\mathbb{R}$, and

A(\eta)=\log\bigl(1+\exp(\eta)\bigr).

The normalized probability of $Y=1$ is $\exp(\eta)/(1+\exp(\eta))$, which is strictly between zero and one for finite $\eta$. Probabilities zero and one are limiting cases as $\eta$ tends to negative or positive infinity; they are not attained at finite natural parameters.

Worked Example: Poisson Counts

For counts $y=0,1,2,\ldots$, take $T(y)=y$ and weight $1/y!$. Determine when the normalizing sum is finite.

For a finite real $\eta$, put $x=\exp(\eta)$, so $x>0$. The power series for the exponential function gives

Z(\eta) =\sum_{y=0}^{\infty}\frac{\exp(\eta y)}{y!} =\sum_{y=0}^{\infty}\frac{x^y}{y!} =\exp(x) =\exp\{\exp(\eta)\}.

This is finite and positive for every $x>0$, hence for every finite real $\eta$. Thus $\mathcal{H}=\mathbb{R}$, with $A(\eta)=\exp(\eta)$. For example, at $\eta=0$, the sum is $\sum_{y=0}^\infty 1/y!=\exp(1)$, not an infinite quantity. As with Bernoulli, a boundary distribution with zero mean is approached as $\eta$ tends to negative infinity; it does not correspond to a finite endpoint of $\mathcal{H}$.

Worked Example: Gamma Kernel with Fixed Shape

Fix a shape parameter $\alpha>0$. On $y>0$, use $T(y)=y$ and the weight $y^{\alpha-1}$ with respect to ordinary integration. Find the full natural-parameter space, including the behavior at its finite boundary.

The normalizer is

Z(\eta)=\int_0^\infty \exp(\eta y)y^{\alpha-1}\,dy.

First suppose $\eta<0$. The substitution $t=(-\eta)y$ gives $y=t/(-\eta)$ and $dy=dt/(-\eta)$, so

Z(\eta) =(-\eta)^{-\alpha}\int_0^\infty e^{-t}t^{\alpha-1}\,dt =\frac{\Gamma(\alpha)}{(-\eta)^\alpha}<\infty.

The last integral is the defining gamma integral, which is finite for $\alpha>0$. At the boundary $\eta=0$,

Z(0)=\int_0^\infty y^{\alpha-1}\,dy=\infty,

because its integral over $[1,\infty)$ diverges when $\alpha>0$. If $\eta>0$, the logarithm of the integrand is $\eta y+(\alpha-1)\log y$, which tends to positive infinity as $y$ tends to infinity. The integrand is therefore at least one for all sufficiently large $y$, so its integral over that infinite tail diverges. Combining the three cases gives

\mathcal{H}=(-\infty,0), \qquad A(\eta)=\log\Gamma(\alpha)-\alpha\log(-\eta).

For a gamma distribution with rate $\beta>0$, the natural parameter in this representation is $\eta=-\beta$. Its restriction to negative values is exactly what makes the integral finite; the boundary value zero is not allowed.

A Practical Boundary Check

The examples show why the support and the weights in the normalizing expression matter. A finite sum of positive terms remains finite for every finite parameter. By contrast, an integral over an unbounded range can diverge when the exponential term stops decreasing fast enough to control the rest of the integrand. In the gamma example, the integrand is pointwise finite at $\eta=0$, but its integral is infinite.

Write the normalizer.
Use the sum or integral for the model, including its support and fixed weights.

Check interior parameter values.
Use a series identity for a sum when available, or a substitution and known integral for a continuous model.

Test each finite boundary separately.
Substitute the boundary value into the original normalizer. Do not infer finiteness from values strictly inside the domain.

State the domain and its endpoint status.
Specify which values are included, and give the log-partition function only where the normalizer is finite.

A common mistake is to argue, “The exponential is finite for every outcome, so the normalizer is finite for every parameter.” Pointwise finiteness does not imply that an infinite sum or integral is finite. For example, at the gamma boundary $\eta=0$, every value $y^{\alpha-1}$ at a fixed $y>0$ is finite, but its integral over $(0,\infty)$ diverges. The correction is to evaluate the whole sum or integral, including its tails, at each candidate parameter value.

The domain also clarifies what a parameterization can represent. Bernoulli and Poisson models have all real finite natural parameters, while the fixed-shape gamma kernel permits only negative ones. Values approached only as $\eta$ tends to infinity or negative infinity are limits, not finite members of the natural-parameter space.

Key takeaway: A natural parameter is admissible exactly when its normalizing sum or integral is positive and finite. The domain is convex, finite-support models have domain $\mathbb{R}$, and an unbounded-support model may exclude a finite boundary where its normalizer diverges.

Check Your Understanding

For each question, justify your answer using the normalizing sum or integral.

For the Bernoulli normalizer $1+\exp(\eta)$, does $\eta=100$ belong to the natural-parameter space? Explain.
Why does finiteness of each individual summand not by itself prove that an infinite normalizing sum is finite?
For the fixed-shape gamma kernel, what happens to the normalizer at $\eta=0$, and why?
State the convexity conclusion for the natural-parameter space and the inequality used to prove it.
In the Poisson example, is the zero-mean limiting case represented by a finite value of $\eta$? Explain.

Natural-Parameter Spaces and Finite Normalizers

What You'll Learn