Minimal Exponential-Family Representations

Why Minimality Matters

An exponential-family density can contain several sufficient-statistic components even when some of them carry exactly the same information. In that case, the natural parameter vector has more coordinates than the probability distributions require. The resulting representation is redundant: distinct parameter vectors can describe the same distribution. Minimality provides a precise test for this phenomenon and clarifies when natural parameters are identifiable.

Consider a family with densities, relative to a fixed measure \(\nu\), of the form

f_\eta(x)=h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}, \qquad \eta\in\mathcal H\subseteq\mathbb R^k,

where \(T=(T_1,\ldots,T_k)^\mathsf{T}\) is the vector of sufficient-statistic components, \(h\) is a nonnegative carrier function, and \(A\) is the log-normalizing function. Assume \(A(\eta)\) is finite for each \(\eta\in\mathcal H\) and that these expressions define probability densities. The set \(S=\{x:h(x)>0\}\) is the carrier support: outside \(S\), every density in this representation is zero.

Definition: The representation is minimal if there do not exist \(a\in\mathbb R^k\), \(a\ne0\), and \(b\in\mathbb R\) such that \(a^\mathsf{T}T(x)=b\) for \(\nu\)-almost every \(x\in S\). Equivalently, no nonzero linear combination of the sufficient-statistic components is almost surely constant under the family. Since every \(f_\eta\) is positive on \(S\) wherever \(h>0\), the relation holds \(\nu\)-almost everywhere on \(S\) exactly when it holds almost surely under any member of the family.

The condition concerns the statistic components as functions on the family’s support, not whether a particular observed data set happens to make two computed values equal. A relation that holds only at some observations, but not almost everywhere under the model, does not violate minimality.

Detecting and Removing Redundancy

If a nonzero vector \(a\) satisfies \(a^\mathsf{T}T(x)=b\) on the carrier support, one statistic component can be expressed using the others. Choose an index \(q\) for which \(a_q\ne0\). Then, on \(S\),

T_q(x)=\frac{b}{a_q}-\sum_{j\ne q}\frac{a_j}{a_q}T_j(x).

Consequently, the exponent involving the statistics can be rewritten as

\eta^\mathsf{T}T(x) = \frac{\eta_q b}{a_q} +\sum_{j\ne q}\left(\eta_j-\frac{\eta_q a_j}{a_q}\right)T_j(x).

The first term is constant in \(x\). It can be absorbed into the normalizing function, leaving a representation using only the other \(k-1\) statistics. This algebra shows why a linear relation signals an unnecessary coordinate. It does not, by itself, say that every possible restricted parameterization of the same model must have nonidentifiable parameters; that distinction will matter below.

Worked Example: A Redundant Bernoulli Representation

Let \(X\in\{0,1\}\), and use the statistic vector \(T(X)=(X,2X+3)^\mathsf{T}\), with natural parameter \(\eta=(\alpha,\beta)^\mathsf{T}\). Counting measure is the reference measure and \(h(x)=1\) on \(\{0,1\}\). The normalizing function is obtained by summing the two unnormalized probabilities:

\begin{aligned} \exp\{A(\alpha,\beta)\} &=\exp(3\beta)+\exp(\alpha+5\beta),\\ A(\alpha,\beta) &=3\beta+\log\{1+\exp(\alpha+2\beta)\}. \end{aligned}

The statistic components satisfy \(T_2(X)-2T_1(X)=3\) for both possible values of \(X\). Thus the representation is not minimal. Its probability of \(X=1\) is

P_{\alpha,\beta}(X=1) = \frac{\exp(\alpha+5\beta)} {\exp(3\beta)+\exp(\alpha+5\beta)} = \frac{\exp(\alpha+2\beta)}{1+\exp(\alpha+2\beta)}.

Only the combination \(\alpha+2\beta\) affects the distribution. For every real \(t\), replacing \((\alpha,\beta)\) by \((\alpha-2t,\beta+t)\) leaves that combination unchanged, so it leaves the Bernoulli distribution unchanged. For example, \((0,0)\) and \((-2,1)\) both give \(P(X=1)=1/2\). The two-dimensional natural parameter is therefore not identifiable in this representation.

Minimality and Identifiability

A parameterization is identifiable if distinct parameter values give distinct probability distributions. For a full-dimensional exponential family, minimality is exactly the condition that makes the natural parameter identifiable. Here “full-dimensional” means that the natural parameter space contains a nonempty open subset of \(\mathbb R^k\); the usual regular natural parameter space has this property.

Theorem: Suppose the natural parameter space \(\mathcal H\) contains a nonempty open subset of \(\mathbb R^k\). The map \(\eta\mapsto f_\eta\) is injective (up to equality \(\nu\)-almost everywhere) if and only if the representation is minimal.

Proof. First suppose the representation is minimal and \(f_\eta=f_{\eta'}\) \(\nu\)-almost everywhere for two parameters in \(\mathcal H\). On \(S\), the densities are positive wherever \(h>0\), so taking their logarithms gives

(\eta-\eta')^\mathsf{T}T(x)=A(\eta)-A(\eta') \quad\text{for \(\nu\)-almost every }x\in S.

If \(\eta\ne\eta'\), this is a nonzero linear combination of the statistic components that is constant on \(S\), contradicting minimality. Hence \(\eta=\eta'\), proving injectivity.

Conversely, suppose the representation is not minimal. Then there are \(a\ne0\) and \(b\) such that \(a^\mathsf{T}T(x)=b\) for \(\nu\)-almost every \(x\in S\). Since \(\mathcal H\) contains an open subset, choose \(\eta\) in that subset. For sufficiently small nonzero \(t\), \(\eta+ta\) also belongs to it. The normalizing integral at \(\eta+ta\) is

\begin{aligned} \exp\{A(\eta+ta)\} &=\int_S h(x)\exp\{(\eta+ta)^\mathsf{T}T(x)\}\,d\nu(x)\\ &=\exp(tb)\int_S h(x)\exp\{\eta^\mathsf{T}T(x)\}\,d\nu(x), \end{aligned}

so \(A(\eta+ta)=A(\eta)+tb\). On \(S\), the exponents in the two densities therefore agree after normalization, and the densities are equal \(\nu\)-almost everywhere. Outside \(S\), both densities are zero because \(h(x)=0\). Thus \(\eta\) and \(\eta+ta\) are distinct parameters giving the same distribution, so the parameterization is not injective. This proves both directions. \(\square\)

The treatment of \(h=0\) is important in this argument. A density ratio is defined only where the denominator is positive, so it cannot be used at points outside \(S\). The proof instead compares the densities on \(S\), where their exponential forms apply, and notes separately that both are zero outside \(S\). For a sample, if any observation has \(h(x_i)=0\), the likelihood is zero for every parameter. If all observations lie in \(S\) and satisfy the relation, the likelihoods agree by direct substitution. Thus the conclusion about equal distributions does not depend on taking a ratio at points where both densities vanish.

Examples of Minimal and Nonminimal Statistics

Worked Example: A Minimal Normal Representation

Suppose \(X\sim N(\mu,\sigma^2)\), where the variance \(\sigma^2>0\) is known and \(\mu\in\mathbb R\). Rewriting the density gives

f_\eta(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{x^2}{2\sigma^2}\right\} \exp\left\{\eta x-\frac{\sigma^2\eta^2}{2}\right\}, \qquad \eta=\frac{\mu}{\sigma^2}.

Thus \(T(X)=X\), \(h(x)=(2\pi\sigma^2)^{-1/2}\exp\{-x^2/(2\sigma^2)\}\), and \(A(\eta)=\sigma^2\eta^2/2\). To check minimality, suppose \(aX=b\) almost surely for some scalar \(a\). If \(a\ne0\), then \(X=b/a\) almost surely, which would imply \(\operatorname{Var}(X)=0\). But \(\operatorname{Var}(X)=\sigma^2>0\), a contradiction. Hence no nonzero coefficient gives a constant statistic, and this representation is minimal. Its parameter \(\eta\), and therefore \(\mu=\sigma^2\eta\), is identifiable.

Worked Example: Redundancy in a Three-Category Model

Let \(X\in\{1,2,3\}\), with \(T(X)=(I\{X=1\},I\{X=2\},I\{X=3\})^\mathsf{T}\), where \(I\{\cdot\}\) is an indicator. With counting measure and \(h(x)=1\), the normalizer is

A(\eta)=\log\{\exp(\eta_1)+\exp(\eta_2)+\exp(\eta_3)\}.

For every possible observation, \(T_1(X)+T_2(X)+T_3(X)=1\). The representation is nonminimal, with \(a=(1,1,1)^\mathsf{T}\) and \(b=1\). Indeed, adding the same constant \(c\) to all three coordinates changes \(A\) to \(A+c\), while it changes \(\eta^\mathsf{T}T(X)\) to \(\eta^\mathsf{T}T(X)+c\). The density is unchanged.

A two-coordinate representation uses \(T^*(X)=(I\{X=1\},I\{X=2\})^\mathsf{T}\) and natural parameters \((\eta_1-\eta_3,\eta_2-\eta_3)\). For example,

P(X=1) = \frac{\exp(\eta_1-\eta_3)} {1+\exp(\eta_1-\eta_3)+\exp(\eta_2-\eta_3)}, \qquad P(X=3) = \frac{1} {1+\exp(\eta_1-\eta_3)+\exp(\eta_2-\eta_3)}.

These probabilities depend only on two parameter contrasts, not on the common shift. The third indicator is unnecessary because the three indicators always sum to one.

When the Parameter Set Is Restricted

Minimality is a property of the statistic representation on its support. Identifiability also depends on which parameter values are allowed. The theorem’s open-set condition ensures that a redundant direction can actually be followed to nearby distinct parameter values. Without that condition, a nonminimal representation can still be identifiable on a restricted parameter set.

Worked Example: A Nonminimal Statistic With an Identifiable Restriction

Return to the Bernoulli representation \(T(X)=(X,2X+3)^\mathsf{T}\), but restrict the parameter to \((\alpha,\beta)=(s,0)\) with \(s\in\mathbb R\). The statistic vector is still nonminimal, because \(T_2-2T_1=3\) remains true. Yet the success probability on this restricted parameter set is

P_s(X=1)=\frac{\exp(s)}{1+\exp(s)}.

If two restricted parameter values \(s_1\) and \(s_2\) give the same success probability \(p\), then \(e^{s_1}=p/(1-p)=e^{s_2}\), so \(s_1=s_2\). Thus the restricted one-dimensional parameterization is identifiable, even though the two-component statistic representation is nonminimal. There is no contradiction: the restriction does not allow movement along the redundant direction \((-2,1)^\mathsf{T}\).

Common Mistakes and Practical Use

Checking only the observed sample. Two statistic values may coincide for one data set without any almost-sure linear relation in the model. Minimality is assessed over the carrier support, up to measure-zero sets.
Confusing a redundant representation with a redundant model. The same family of distributions may have both a redundant parameterization and a smaller, identifiable parameterization. Minimality concerns the representation being used.
Assuming nonminimality always forces nonidentifiability. The equivalence requires a parameter space with room to move in the redundant direction, as supplied by the full-dimensional open-set condition. Restricted parameter spaces require a separate identifiability check.
Dividing by a density where it is zero. Compare exponential forms only on \(S=\{h>0\}\). Outside \(S\), both densities vanish, so equality there is direct rather than a density-ratio calculation.

Key takeaway: A canonical exponential-family representation is minimal exactly when no nonzero linear combination of its sufficient-statistic components is constant almost surely on the carrier support. With a full-dimensional open natural parameter space, this is also exactly the condition for the natural parameter to be identifiable.

Check Your Understanding

Use the support and parameter-space conditions carefully in each answer.

State the minimality condition for a statistic vector \(T\) in a \(k\)-parameter exponential-family representation.
Why must the density-equality argument be handled separately on \(S=\{h>0\}\) and outside \(S\)?
For the three-category indicator representation, identify a nonzero vector \(a\) and constant \(b\) that demonstrate nonminimality.
Explain why a nonminimal representation with an open natural parameter space cannot be identifiable.
Can a nonminimal representation be identifiable after restricting its parameter set? Give the reason using the Bernoulli example.