Curved Exponential Families as Constrained Surfaces

Why a Constraint Changes the Geometry

A regular exponential family can have several natural parameters, even when the scientific model has only one free parameter. A restriction linking those natural parameters traces out a lower-dimensional subset of the family. If that subset bends through natural-parameter space rather than lying in an affine subspace, the restricted model is called a curved exponential family.

This is more than a change of coordinates. The parent family allows its natural parameters to vary independently, whereas the restricted model permits only combinations satisfying the constraint. That distinction affects how scores and information are calculated: changes in the scientific parameter move along the constraint surface, not in arbitrary directions in the parent space.

Definition: Let a minimal regular exponential family have densities \( f_\eta(x)=\exp\{\eta^\mathsf{T}T(x)-A(\eta)\} \) with respect to a common carrier measure, where \(\eta\) ranges over an open natural parameter space \(H\subset\mathbb{R}^k\). A curved exponential family is a restriction with natural parameter \(\eta=h(\theta)\), where \(\theta\) ranges over an open subset \(\Theta\subset\mathbb{R}^q\), \(q<k\), and \(h(\Theta)\subset H\). For a regular curved parameterization, \(h\) is at least continuously differentiable, one-to-one, and has derivative of rank \(q\). The image is curved when it is locally a nonlinear \(q\)-dimensional surface rather than an affine subset. Locally, such a surface can be expressed by \(k-q\) smooth constraints with linearly independent gradients.

The minimality assumption removes redundant natural coordinates from the parent representation. The open set \(H\) matters too: every parameter value used in the restriction must remain inside the parent family’s natural parameter space. A constraint that touches the edge of \(H\) raises additional questions about boundary behavior; those are distinct from the geometry considered here.

Normal Coordinates and Natural-Parameter Constraints

For \(X\sim N(\mu,\sigma^2)\), with \(\sigma^2>0\), the density can be written in canonical form using the statistic \(T(x)=(x,x^2)^\mathsf{T}\):

f(x)=\exp\left\{\eta_1x+\eta_2x^2-A(\eta_1,\eta_2)\right\},\qquad \eta_1=\frac{\mu}{\sigma^2},\quad \eta_2=-\frac{1}{2\sigma^2}.

The natural parameter space is \(H=\{(\eta_1,\eta_2):\eta_1\in\mathbb{R},\ \eta_2<0\}\). Completing the square in the normalizing integral gives \(A(\eta_1,\eta_2)=\frac12\log\{\pi/(-\eta_2)\}-\eta_1^2/(4\eta_2)\). The inverse relations are \(\mu=-\eta_1/(2\eta_2)\) and \(\sigma^2=-1/(2\eta_2)\). Thus, a scientific restriction on \((\mu,\sigma^2)\) becomes a constraint on \((\eta_1,\eta_2)\).

Worked Example: Normal Variance Equal to Squared Mean

Consider \(X\sim N(\mu,\mu^2)\) with \(\mu>0\). The parent normal family has two natural parameters, but this model has only one free parameter. Substituting \(\sigma^2=\mu^2\) gives

\eta_1=\frac{1}{\mu},\qquad \eta_2=-\frac{1}{2\mu^2}.

Eliminating \(\mu\), we obtain \(\eta_2=-\eta_1^2/2\), with \(\eta_1>0\). The model is therefore the portion of a parabola in \(H\) satisfying that inequality. Its parameterization \(h(\mu)=(1/\mu,-1/(2\mu^2))^\mathsf{T}\) has derivative \(h'(\mu)=(-1/\mu^2,1/\mu^3)^\mathsf{T}\), which is nonzero for every \(\mu>0\). It is a regular one-dimensional curved family inside a two-dimensional parent family.

The constraint is genuinely nonlinear: the slope of the parabola changes with \(\eta_1\). In the original coordinates, the restriction is a relationship between mean and variance; in natural coordinates it is a curved path.

Worked Example: A Different Normal Constraint

Now let \(X\sim N(\mu,1/\mu)\), where \(\mu>0\). This is a valid normal model because its variance is positive. The natural parameters are

\eta_1=\frac{\mu}{1/\mu}=\mu^2,\qquad \eta_2=-\frac{1}{2(1/\mu)}=-\frac{\mu}{2}.

Since \(\mu=-2\eta_2\), eliminating \(\mu\) gives \(\eta_1=4\eta_2^2\), with \(\eta_2<0\). The derivative of \(h(\mu)=(\mu^2,-\mu/2)^\mathsf{T}\) is \((2\mu,-1/2)^\mathsf{T}\), which is nonzero throughout the parameter space. Hence this is another regular curved family. Its parabola opens in a different direction from the one in the preceding example; the shape depends on the chosen constraint, not just on the fact that the parent distribution is normal.

Worked Example: An Affine Restriction Is Not Curved

Suppose \(X\sim N(\mu,\sigma_0^2)\), where the positive variance \(\sigma_0^2\) is fixed and \(\mu\in\mathbb{R}\). Then

\eta_1=\frac{\mu}{\sigma_0^2},\qquad \eta_2=-\frac{1}{2\sigma_0^2}.

As \(\mu\) varies, \(\eta_2\) stays fixed and \(\eta_1\) ranges over the real line. The image is a straight horizontal line in natural-parameter space, not a nonlinear curve. This is a lower-dimensional subfamily of the two-parameter normal family, but it is an affine restriction rather than a curved one. The distinction is geometric: “fewer free parameters” alone does not mean “curved.”

Constraints, Tangents, and Dimension

A curve or surface can be described either by a parameterization \(h(\theta)\) or by equations imposed directly on \(\eta\). In the two-dimensional normal examples, one equation defines a one-dimensional curve. More generally, if \(g:H\to\mathbb{R}^{k-q}\) is continuously differentiable, \(g(\eta)=0\), and the derivative of \(g\) has rank \(k-q\) on the constraint set, then the implicit function theorem gives a local surface of dimension \(q\). Its tangent directions \(v\) satisfy \(Dg(\eta)v=0\): moving infinitesimally along the surface preserves the constraints to first order.

For a parameterization, the columns of the \(k\)-by-\(q\) derivative matrix \(Dh(\theta)\) give the tangent directions. They show which combinations of parent natural parameters can change when \(\theta\) changes. This is the relevant geometry for inference within the restricted family.

Theorem: Suppose the parent family is a minimal regular exponential family on an open natural parameter space \(H\), with log-partition function \(A\) twice differentiable there. Let \(X_1,\ldots,X_n\) be independent observations from the restricted model with \(\eta=h(\theta)\), where \(h:\Theta\to H\) is continuously differentiable and has rank \(q\). Write \(T_i=T(X_i)\), \(\bar T=n^{-1}\sum_{i=1}^nT_i\), and \(J(\theta)=Dh(\theta)\). The score and Fisher information for \(\theta\) are \(U_\theta= nJ(\theta)^\mathsf{T}\{\bar T-\nabla A(h(\theta))\}\) and \(I_\theta=nJ(\theta)^\mathsf{T}\nabla^2 A(h(\theta))J(\theta)\).

Proof. The log-likelihood, omitting any term independent of \(\theta\), is \(\ell(\theta)=h(\theta)^\mathsf{T}\sum_{i=1}^nT_i-nA(h(\theta))\). Applying the chain rule to each term gives

\nabla_\theta\ell(\theta) =J(\theta)^\mathsf{T}\sum_{i=1}^nT_i -nJ(\theta)^\mathsf{T}\nabla A(h(\theta)) =nJ(\theta)^\mathsf{T}\{\bar T-\nabla A(h(\theta))\}.

In a regular exponential family, \(\nabla A(\eta)=E_\eta[T(X)]\) and \(\nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X))\). The score therefore has expectation zero. Since the observations are independent, the covariance matrix of \(\sum_i T_i\) is \(n\nabla^2 A(h(\theta))\). Multiplying on the left by \(J^\mathsf{T}\) and on the right by \(J\) gives the covariance matrix of the score, which is the Fisher information. This proves both formulas. \(\square\)

The formula shows why information in the parent family cannot simply be read as information for the restricted parameter: it must be projected through the tangent directions \(J(\theta)\). For a regular minimal parent family, \(\nabla^2 A\) is positive definite. If \(J\) has full column rank, then for every nonzero \(a\in\mathbb{R}^q\), \(J a\ne 0\), so \(a^\mathsf{T}I_\theta a=n(Ja)^\mathsf{T}\nabla^2 A(Ja)>0\). Thus the restricted information is positive definite under these assumptions.

Why the Curvature Matters

A curved restriction can couple quantities that vary independently in the parent model. A sample may produce a parent-family estimate outside the constrained surface; that does not make the restricted model invalid. It means the best-fitting parameter must be found subject to the constraint. The score formula expresses this directly: only tangent components of the parent natural-parameter score contribute to changes in \(\theta\).

Curvature also means the tangent directions can change with \(\theta\). The Fisher information formula accounts for this locally at each parameter value. It describes the first-order information along the surface; it does not say that the surface itself is flat or that a single fixed linear reparameterization will remove the constraint.

Common Mistakes

Calling every restriction curved. A fixed-variance normal model lies on a straight line in the parent natural-parameter space. Lower dimension is necessary here, but nonlinearity distinguishes a curved family from an affine subfamily.
Confusing natural parameters with mean and variance. For a normal distribution, \(\eta_1\) is not the mean and \(\eta_2\) is not the variance. Use \(\eta_1=\mu/\sigma^2\) and \(\eta_2=-1/(2\sigma^2)\) before translating a constraint.
Using arbitrary parent directions for restricted inference. The allowed local changes are the columns of \(Dh(\theta)\). The information for \(\theta\) is the parent covariance of the canonical statistic projected through those directions.
Ignoring the domain. A proposed parameterization must map into \(H\). For the normal family, every point used must satisfy \(\eta_2<0\), equivalently a finite, positive variance.

Key takeaway: A curved exponential family is a lower-dimensional, nonlinear restriction of a regular parent family in natural-parameter space. Derive its constraint by translating the scientific parameter relationship into canonical coordinates, and analyze its local behavior through the tangent matrix \(Dh(\theta)\).

Check Your Understanding

Use the definitions and calculations in this tutorial to answer the following.

For \(X\sim N(\mu,\mu^2)\) with \(\mu>0\), state the natural-parameter constraint and explain why the parameterization is regular.
Why is a normal model with fixed positive variance a lower-dimensional subfamily but not a curved one?
For a smooth restriction \(\eta=h(\theta)\), what does the matrix \(Dh(\theta)\) represent geometrically?
In the score formula, why does the parent-family score get multiplied by \(J(\theta)^\mathsf{T}\)?
For \(X\sim N(\mu,1/\mu)\), \(\mu>0\), verify that the natural parameters satisfy \(\eta_1=4\eta_2^2\).