Minimality and Identifiability of Canonical Statistics

Why Minimality Matters

An exponential-family representation describes a distribution through a vector of canonical statistics and a vector of natural parameters. But the chosen statistics may contain redundant information. If one statistic is an affine combination of the others on the support of the model, different natural-parameter vectors can produce exactly the same distribution. The representation then has more coordinates than the distributions require.

Minimality is the precise condition that rules out this kind of redundancy. It depends not just on the formulas used for the statistics, but on their values on the support of the family. This distinction matters: a relation that holds throughout the real line may fail on a restricted support, and a relation that fails on the real line may hold on a small discrete support.

Definition on the Carrier Support

Consider a family with densities, relative to a common sigma-finite measure \(\nu\), of the form

f_\eta(x)=h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\},\qquad \eta\in\Theta.

Here \(T=(T_1,\ldots,T_k)^\mathsf{T}\) is a measurable vector of real-valued statistics, \(h\) is a fixed carrier density, and \(A(\eta)\) is finite on the parameter set \(\Theta\). Assume \(h>0\) on the common support \(S=\{x:h(x)>0\}\), up to \(\nu\)-null sets. Relations among statistics are assessed on \(S\), up to sets of \(\nu\)-measure zero.

Definition: The representation is minimal if the only vector \(a\in\mathbb{R}^k\) and constant \(c\in\mathbb{R}\) for which \(a^\mathsf{T}T(x)=c\) for \(\nu\)-almost every \(x\in S\) are \(a=0\) and \(c=0\). Equivalently, no nonzero linear combination of the canonical statistics is constant almost everywhere on the carrier support.

The constant is essential in this definition. For example, a statistic that is identically 1 on the support is redundant: changing its natural parameter only changes the exponent by a constant, which is absorbed by the normalizing term \(A(\eta)\). Minimality rules out both constant statistics and affine relations among multiple statistics.

How a Relation Creates Redundancy

Suppose there is a nonzero vector \(a\) such that \(a^\mathsf{T}T(x)=c\) almost everywhere on \(S\). If both \(\eta\) and \(\eta+ta\) are allowed parameter values, then their unnormalized densities differ only by the constant factor \(\exp(tc)\). The normalizer changes by precisely the same factor on the log scale:

\begin{aligned} \int h(x)\exp\{(\eta+ta)^\mathsf{T}T(x)\}\,d\nu(x) &=\exp(tc)\int h(x)\exp\{\eta^\mathsf{T}T(x)\}\,d\nu(x),\\ A(\eta+ta)&=A(\eta)+tc. \end{aligned}

Consequently, \(f_{\eta+ta}(x)=f_\eta(x)\) almost everywhere. The two parameter vectors are different when \(t\ne 0\) and \(a\ne 0\), but they describe the same distribution. The relation therefore identifies a direction in parameter space that the distribution cannot distinguish.

Worked Example: Redundant Statistics for a Bernoulli Support

Let \(x\in\{0,1\}\), let \(\nu\) be counting measure, and take \(h(x)=1\). Consider the two statistics \(T_1(x)=x\) and \(T_2(x)=2x\).

Check the relation. For every point in the support, \(2T_1(x)-T_2(x)=2x-2x=0\). Thus \(a=(2,-1)^\mathsf{T}\ne 0\) gives \(a^\mathsf{T}T(x)=0\), so this representation is not minimal.

See the parameter redundancy. Its unnormalized mass is \(\exp\{\eta_1x+2\eta_2x\}=\exp\{(\eta_1+2\eta_2)x\}\). Summing over \(x=0,1\) gives \(A(\eta)=\log(1+\exp(\eta_1+2\eta_2))\). The distribution depends on \(\eta_1,\eta_2\) only through their combination \(\eta_1+2\eta_2\).

For instance, \((\eta_1,\eta_2)=(0,0)\) and \((2,-1)\) both give \(\eta_1+2\eta_2=0\). Both assign mass \(1/2\) to each support point. Their parameter vectors differ, but their distributions are identical.

Minimality and Identifiability

A parameterization is identifiable if no two distinct parameter values correspond to the same distribution. For exponential families, the affine-relation definition gives a direct criterion. The forward implication below does not require the parameter set to be open. To obtain the converse, we need to ensure that a redundant direction can actually be followed while remaining in the family.

Theorem: For a family in the form above with a common carrier and \(h>0\) on its support, minimality implies that \(\eta\mapsto f_\eta\) is one-to-one. Conversely, if the natural parameter space contains a nonempty open subset of \(\mathbb{R}^k\), then nonminimality implies that the parameterization is not one-to-one there. Thus, for a full exponential family with open natural parameter space, minimality and identifiability are equivalent.

Proof. Suppose first that \(f_\eta=f_{\eta'}\) almost everywhere for two allowed parameter values. Since the carrier is common and positive on \(S\), taking the ratio of the densities and then logarithms gives

(\eta-\eta')^\mathsf{T}T(x)=A(\eta)-A(\eta') \quad\text{for \(\nu\)-almost every }x\in S.

If the representation is minimal, the definition forces \(\eta-\eta'=0\). Therefore \(\eta=\eta'\), proving identifiability.

For the converse, nonminimality supplies \(a\ne 0\) and \(c\) such that \(a^\mathsf{T}T=c\) almost everywhere on \(S\). Choose an interior point \(\eta\) of the open natural parameter space. Since it is open, there is a ball around \(\eta\) contained in the space; hence \(\eta+ta\) is in that ball for all sufficiently small \(|t|\). Choose such a \(t\ne 0\). The calculation above shows \(A(\eta+ta)=A(\eta)+tc\), so the two densities are equal almost everywhere even though \(\eta+ta\ne\eta\). The parameterization is not one-to-one. \(\square\)

Worked Example: A Minimal Family on Three Support Points

Let the support be \(\{0,1,2\}\), with counting measure and \(h(x)=1\). Set \(T_1(x)=x\) and \(T_2(x)=x^2\). To test minimality, suppose \(a_1x+a_2x^2=c\) at all three support points.

At \(x=0\), the equation gives \(c=0\). At \(x=1\), it gives \(a_1+a_2=0\). At \(x=2\), it gives \(2a_1+4a_2=0\), or \(a_1+2a_2=0\). Subtracting the equation \(a_1+a_2=0\) yields \(a_2=0\), and then \(a_1=0\). There is no nontrivial affine relation, so the representation is minimal.

Identifiability can also be checked directly. Write \(p_j=P(X=j)\). The normalizer cancels in probability ratios, giving \(u=\log(p_1/p_0)=\eta_1+\eta_2\) and \(v=\log(p_2/p_0)=2\eta_1+4\eta_2\). Solving, \(\eta_2=(v-2u)/2\) and \(\eta_1=u-\eta_2\). Thus the probabilities determine both natural parameters uniquely, in agreement with the theorem.

The Log-Partition Hessian as a Diagnostic

When the natural parameter lies in an open region where differentiation under the integral is valid, the Hessian of \(A\) is the covariance matrix of the canonical statistics. This gives a useful local diagnostic of redundancy. The precise connection is that a fixed linear combination has zero variance exactly when that combination is constant almost everywhere on the support.

Result: At an interior parameter value where the log-partition derivatives exist, \(\nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X))\). If the representation is minimal, this covariance matrix is positive definite at every such \(\eta\). If it is nonminimal, the matrix is singular wherever it is defined.

Justification. The Hessian-covariance identity follows by differentiating the log normalizing integral in the natural parameter; these derivatives are justified on an open neighborhood where the required exponential moments are finite. For any vector \(a\),

a^\mathsf{T}\nabla^2A(\eta)a =\operatorname{Var}_\eta\{a^\mathsf{T}T(X)\}.

A random variable has variance zero if and only if it equals a constant almost surely. The density \(f_\eta\) is positive wherever \(h>0\), so its probability measure and \(\nu\) have the same null sets on \(S\). Therefore, if the variance in the display is zero, \(a^\mathsf{T}T\) is constant \(\nu\)-almost everywhere on \(S\). Under minimality this forces \(a=0\), so the quadratic form is positive for every nonzero \(a\), which is positive definiteness. Conversely, a nonzero affine relation gives a nonzero \(a\) for which that variance is zero, making the Hessian singular. \(\square\)

This interpretation connects a global feature of the representation—whether any affine relation holds on the support—to curvature at each regular parameter value. A zero-variance direction carries no random variation in the corresponding combination of statistics, and the distribution cannot use that direction to distinguish natural parameters.

Worked Example: A Redundant Representation on the Real Line

Take \(S=\mathbb{R}\), Lebesgue measure, and \(h(x)=1\). Use \(T_1(x)=x\), \(T_2(x)=x^2\), and \(T_3(x)=2x+x^2\). The integral defining the normalizer is finite when \(\eta_2+\eta_3<0\), since that is the coefficient of \(x^2\) in the exponent and must be negative for an integrable Gaussian-shaped kernel.

The statistics satisfy \(T_3-2T_1-T_2=0\) for every real \(x\). Thus \(a=(-2,-1,1)^\mathsf{T}\) is a nonzero redundancy direction. Choose \(\eta=(0,-1,0)\); it satisfies \(\eta_2+\eta_3=-1<0\). Moving by \(a\) gives \(\eta'=(-2,-2,1)\), for which \(\eta'_2+\eta'_3=-1<0\) as well.

At \(\eta\), the variable part of the exponent is \(-x^2\). At \(\eta'\), it is \(-2x-2x^2+(2x+x^2)=-x^2\). The unnormalized densities, normalizers, and hence distributions are identical. The representation has three canonical statistics, but its exponent effectively has only two independently varying combinations.

Common Pitfall: Check the Actual Support

A tempting but incorrect test is to decide minimality from algebraic relationships between the formulas for \(T_1,\ldots,T_k\) without checking where \(X\) can occur. A relation only matters if it holds almost everywhere on the carrier support.

Wrong reasoning: “The statistic \(x^2\) is not a linear function of \(x\) on \(\mathbb{R}\), so the pair \((x,x^2)\) must be minimal.”
Correction: On support \(\{0,1\}\), \(x^2=x\) at every support point, so the two statistics are redundant there. On support \(\{0,1,2\}\), the calculation in the worked example shows that \((x,x^2)\) is minimal.

A related caution concerns the parameter set. A nonminimal representation makes distinct natural parameters produce the same distribution whenever the parameter space allows movement along a redundancy direction. A restricted parameter subset might contain no such pair, so identifiability on that subset alone does not establish minimality of the statistics.

Key takeaway: To test minimality, ask whether a nonzero linear combination of the canonical statistics is constant almost everywhere on the carrier support. Such a relation creates a redundant parameter direction; its absence guarantees identifiability, and in regular open families it is equivalent to identifiability.

Check Your Understanding

Use the support and carrier measure in each question when assessing affine relations.

For support \(\{0,1\}\), are \(T_1(x)=x\) and \(T_2(x)=x^2\) minimal? Exhibit a relation or explain why none exists.
Suppose \(a^\mathsf{T}T(x)=c\) almost everywhere on the carrier support. Why does shifting \(\eta\) to \(\eta+ta\) leave the distribution unchanged when both parameter values are allowed?
For support \(\{0,1,2\}\), verify whether \(T_1(x)=x\) and \(T_2(x)=x^2\) have a nontrivial affine relation.
What does a zero eigenvalue of the log-partition Hessian indicate about a linear combination of canonical statistics?
Why does the converse from nonminimality to nonidentifiability require a condition on the parameter space?