Boundary Parameters and Limiting Distributions

Why Boundary Limits Need Care

An exponential-family distribution can change substantially as its natural parameter approaches the edge of the parameter region. On finite support, increasing a natural parameter can concentrate probability on states where the canonical statistic is largest. On infinite support, a boundary can instead produce a perfectly valid distribution—or make the normalizing constant diverge so that probability escapes every fixed bounded set. The parameter limit alone does not tell us which behavior occurs.

We will distinguish two questions: does the limiting parameter have a finite normalizer, and do the corresponding probability distributions converge to a probability distribution? These questions are related, but their answers require checking the normalizer and the probability mass, not merely substituting a boundary value into a formula.

Definition: For a base measure $\nu$, carrier function $h$, and canonical statistic $T$, write the unnormalized weight as $h(x)\exp\{\eta^T T(x)\}$. Its normalizer is $$ Z(\eta)=\int h(x)\exp\{\eta^T T(x)\}\,d\nu(x). $$ The natural parameter set is the set of finite parameter values $\eta$ for which $Z(\eta)<\infty$. Its interior is the set of points having an open neighborhood contained in that set. A parameter on the boundary belongs to the natural parameter set only if its own normalizer is finite.

When $0<Z(\eta)<\infty$, the normalized law has density or mass function $f_\eta(x)=h(x)\exp\{\eta^T T(x)\}/Z(\eta)$ with respect to $\nu$. A boundary value with finite normalizer therefore defines an ordinary probability law. A parameter tending to infinity is not itself a finite natural parameter; nevertheless, the associated sequence of laws may converge to a probability distribution.

Limits on Finite Support

Suppose the support consists of finitely many states $x_1,\ldots,x_m$, each with positive carrier weight $h(x_j)$. For any finite natural parameter, the normalizer is a finite sum. Along a path $\eta(t)=\eta_0+tv$, where $t$ tends to positive infinity, the factor involving $t$ is $\exp\{t v^T T(x_j)\}$. States maximizing $v^T T(x_j)$ dominate this sum. If several states tie for the maximum, their relative probabilities need not vanish.

Theorem: Let $S$ be a finite support, with $h(x)>0$ for every $x\in S$, and let $T(x)$ be a finite-dimensional statistic. For fixed $\eta_0$ and $v$, define $M=\{x\in S:v^T T(x)=\max_{y\in S}v^T T(y)\}$. As $t\to\infty$, the law with parameter $\eta_0+tv$ converges to the distribution supported on $M$ with probabilities proportional to $h(x)\exp\{\eta_0^T T(x)\}$ for $x\in M$.

Proof. Put $c=\max_{y\in S}v^T T(y)$ and $b(x)=h(x)\exp\{\eta_0^T T(x)\}$. After dividing both numerator and denominator by $\exp(tc)$, the probability at $x$ is $$ P_{\eta_0+tv}(X=x)= \frac{b(x)\exp\{t(v^T T(x)-c)\}} {\sum_{y\in S}b(y)\exp\{t(v^T T(y)-c)\}}. $$ For $x\in M$, the exponential factor is $1$. For $x\notin M$, $v^T T(x)-c<0$, so that factor tends to zero. Because the sum has finitely many terms, its limit is $\sum_{y\in M}b(y)>0$. Thus the probabilities converge to $b(x)/\sum_{y\in M}b(y)$ on $M$, and to zero off $M$. The limiting probabilities sum to one, proving distributional convergence. $\square$

Worked Example: Bernoulli Probabilities at Both Infinite Boundaries

Let $X\in\{0,1\}$, with canonical statistic $T(X)=X$ and carrier weights equal to one. Its mass function is $$ P_\eta(X=x)=\frac{\exp(\eta x)}{1+\exp(\eta)},\qquad x\in\{0,1\},\quad \eta\in\mathbb R. $$ The normalizer is $Z(\eta)=1+\exp(\eta)$, finite for every finite $\eta$.

As $\eta\to\infty$, divide numerator and denominator by $\exp(\eta)$. Then $P_\eta(X=1)=\exp(\eta)/(1+\exp(\eta))\to1$ and $P_\eta(X=0)=1/(1+\exp(\eta))\to0$. The limit is the point mass at $1$, the state maximizing $T(x)=x$.

As $\eta\to-\infty$, $\exp(\eta)\to0$, so $P_\eta(X=0)\to1$ and $P_\eta(X=1)\to0$. The limit is the point mass at $0$, now the state minimizing $T$. Both are valid limiting distributions, even though neither limit corresponds to a finite natural parameter.

Worked Example: Tied Maximizers Retain Their Relative Weights

Consider the three states $a,b,c$ with carrier weights $h(a)=1$, $h(b)=2$, and $h(c)=1$, and statistic values $T(a)=0$, $T(b)=T(c)=1$. The scalar natural parameter $\eta$ gives normalizer $$ Z(\eta)=1+3\exp(\eta). $$ As $\eta\to\infty$, the state $a$ has negligible weight relative to the two states whose statistic value is maximal. Specifically, $P_\eta(a)=1/(1+3\exp(\eta))\to0$, while $P_\eta(b)=2\exp(\eta)/(1+3\exp(\eta))\to2/3$ and $P_\eta(c)=\exp(\eta)/(1+3\exp(\eta))\to1/3$.

The limit is not a point mass: it is supported on both maximizers. Their limiting probability ratio is $2:1$, inherited from their carrier weights. Keeping only the maximizing statistic value would miss this relative weighting.

For finite support, every finite parameter has a finite normalizer, so there is no finite boundary separating a region of finite normalizers from one of infinite normalizers. The important boundary behavior in these examples occurs as the parameter tends to positive or negative infinity. Infinite support can behave differently even at a finite boundary value.

Finite Boundary Values: Check the Normalizer

A finite boundary can belong to the natural parameter set. If the normalizer remains finite there, it defines a valid boundary distribution. To establish convergence to it, however, one should also verify that the normalizers and probabilities converge. A useful way to do this is to find an integrable dominating function for the unnormalized weights.

Worked Example: A Finite Boundary Law on the Nonnegative Integers

Let $X\in\{0,1,2,\ldots\}$, with $h(k)=(k+1)^{-2}$ and $T(k)=k$. The normalizer is $$ Z(\eta)=\sum_{k=0}^{\infty}\frac{\exp(\eta k)}{(k+1)^2}. $$ If $\eta\leq0$, every summand is at most $(k+1)^{-2}$, whose sum is $\pi^2/6$; hence $Z(\eta)<\infty$. If $\eta>0$, the summands $\exp(\eta k)/(k+1)^2$ do not tend to zero, so the series diverges. Therefore the natural parameter set is $\eta\leq0$, and its interior is $\eta<0$. In particular, $\eta=0$ is included in the natural parameter set.

At the boundary, $Z(0)=\sum_{k=0}^{\infty}(k+1)^{-2}=\pi^2/6$, so the boundary law is $$ P_0(X=k)=\frac{6}{\pi^2(k+1)^2},\qquad k=0,1,2,\ldots. $$ Now let $\eta\uparrow0$ through values $\eta\leq0$. For each fixed $k$, $\exp(\eta k)/(k+1)^2\to1/(k+1)^2$, and $$ 0\leq\frac{\exp(\eta k)}{(k+1)^2}\leq\frac{1}{(k+1)^2}. $$ The right-hand side is summable. The dominated convergence theorem gives $Z(\eta)\to Z(0)=\pi^2/6$, and therefore $P_\eta(X=k)\to P_0(X=k)$ for every $k$.

This pointwise convergence is also convergence in total variation. Indeed, for probability mass functions $p_\eta$ and $p_0$, $$ \sum_{k=0}^{\infty}|p_\eta(k)-p_0(k)| =2-2\sum_{k=0}^{\infty}\min\{p_\eta(k),p_0(k)\}. $$ For each $k$, the minimum tends to $p_0(k)$, and it is bounded above by the summable sequence $p_0(k)$. Dominated convergence makes the sum of minima tend to $1$, so the displayed sum tends to zero. Thus this finite boundary value yields a valid probability law and the family converges to it in total variation.

When the Normalizer Diverges

At another finite boundary, the unnormalized weights may fail to be summable. Divergence means that substituting the boundary value does not give a probability law by normalization. It does not, by itself, identify the behavior of the normalized distributions; that behavior must be examined separately.

Worked Example: A Geometric Family Whose Mass Escapes

Let $X\in\{0,1,2,\ldots\}$, $h(k)=1$, and $T(k)=k$. For $\eta<0$, set $r=\exp(\eta)$, so $0<r<1$. The geometric series gives $$ Z(\eta)=\sum_{k=0}^{\infty}\exp(\eta k)=\frac{1}{1-\exp(\eta)}. $$ For $\eta=0$, the series is $\sum_{k=0}^{\infty}1$, which diverges; for $\eta>0$, its terms do not tend to zero. The natural parameter set is therefore $\eta<0$, and the normalized probabilities are $P_\eta(X=k)=(1-r)r^k$.

As $\eta\uparrow0$, $r\uparrow1$, and for any fixed nonnegative integer $m$, $$ P_\eta(X\leq m)=\sum_{k=0}^{m}(1-r)r^k=1-r^{m+1}\longrightarrow0. $$ Thus every fixed finite set $\{0,\ldots,m\}$ receives probability tending to zero. No probability distribution on the nonnegative integers can be the limit: any such distribution must assign probability approaching one to $\{0,\ldots,m\}$ as $m$ increases, whereas for each fixed $m$ the limiting probability here is zero. Equivalently, the sequence is not tight; its mass escapes to arbitrarily large values. The divergent normalizer at $\eta=0$ does not produce a boundary probability law.

Common Pitfalls and a Practical Check

A common mistake is to call the natural parameter set only the interior of the region where the normalizer is finite. Under the definition used here, it is the full set of finite parameters with finite normalizer. The example with weights $(k+1)^{-2}\exp(\eta k)$ has natural parameter set $\eta\leq0$, even though its interior is $\eta<0$. Its boundary point $\eta=0$ is included because its normalizer is $\pi^2/6$, not because it is an “extended” parameter outside the natural parameter set.

A second mistake is to infer convergence of probability laws solely from convergence of unnormalized weights, or solely from a diverging normalizer. For a finite support and an infinite parameter limit, compare the statistic values and retain all tied maximizers. For a finite boundary on infinite support, check summability at the boundary and justify convergence, for example with a valid dominating function. If the normalizer diverges, inspect probabilities of fixed sets to determine whether mass remains tight or escapes.

Locate the parameter boundary.
Determine the full set of finite $\eta$ for which $Z(\eta)<\infty$; do not replace it automatically by its interior.

Evaluate the boundary normalizer.
If it is finite, normalize at that value. If it diverges, there is no probability law obtained by direct normalization there.

Analyze the sequence of laws.
On finite support, identify maximizing states along an infinite-parameter path. On infinite support, check convergence of probabilities and whether mass escapes.

Key takeaway: Boundary behavior is determined by both the normalizer and the normalized probabilities. Finite-support exponential tilts concentrate on maximizing states, with ties retaining their relative carrier weights. At a finite boundary, a finite normalizer can define a genuine boundary law; a divergent normalizer cannot, and may signal mass escaping to infinity.

Check Your Understanding

Answer each question by checking the normalizer and, where needed, the limiting probabilities.

For a finite-support family with $\eta(t)=\eta_0+tv$, which states can have positive probability in the limit as $t\to\infty$?
In the three-state example, why does the limiting probability of $b$ equal $2/3$ rather than $1/2$?
For $Z(\eta)=\sum_{k\geq0}\exp(\eta k)/(k+1)^2$, state the natural parameter set and its interior. Is $\eta=0$ included, and why?
For the geometric family, compute $P_\eta(X\leq m)$ and explain what its limit says about tightness.
Why does convergence of unnormalized weights at each fixed state not, by itself, establish convergence to a probability distribution?