Why Curved Exponential Families Arise
A full exponential-family model can describe several aspects of a distribution through separate natural parameters. Scientific knowledge may link those aspects, restricting the possible parameter combinations to a lower-dimensional set. For example, a model might specify that a normal distribution’s variance is determined by its mean. The resulting model remains inside the normal exponential family, but its natural parameters can no longer vary independently.
The key distinction is between the dimension of the full family and the dimension of the restricted model. A smooth restriction produces a curved exponential family when it traces a lower-dimensional, non-affine subset of the full family’s natural-parameter space. This geometric description is about the allowed parameter values; it does not by itself assert anything about the existence or behavior of a maximum likelihood estimate.
Definition and Normal Canonical Form
The rank condition ensures that the \(d\) model coordinates give \(d\) locally distinct directions in the full natural-parameter space. It excludes a parameterization that changes dimension locally because one or more coordinates have no independent effect. Requiring the image not to lie in an affine plane distinguishes genuine curvature from a lower-dimensional restriction that is simply linear in natural coordinates. In examples below, the parameterization is also one-to-one, so different values of \(\theta\) correspond to different distributions.
For a normal random variable \(X\sim N(\mu,\sigma^2)\), with \(\sigma^2>0\), the density can be written as $$ f(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left\{-\frac{(x-\mu)^2}{2\sigma^2}\right\}. $$ Expanding the square gives $$ f(x)=\exp\left\{\frac{\mu}{\sigma^2}x-\frac{1}{2\sigma^2}x^2-\frac{\mu^2}{2\sigma^2}-\frac{1}{2}\log(2\pi\sigma^2)\right\}. $$ Thus \(T(x)=(x,x^2)^\mathsf{T}\), and the natural parameters are \(\eta_1=\mu/\sigma^2\) and \(\eta_2=-1/(2\sigma^2)\). The natural-parameter space is \(\eta_2<0\). In these coordinates, the normal log-partition function is $$ A(\eta)=-\frac{\eta_1^2}{4\eta_2}+\frac{1}{2}\log\left(\frac{-\pi}{\eta_2}\right). $$ The logarithm is defined because \(\eta_2<0\).
Worked Example: Mean Equals the Standard Deviation
Suppose \(X\sim N(\theta,\theta^2)\), where \(\theta>0\). Both the mean and the variance depend on one parameter, so this model has dimension one, while the full normal family has dimension two.
Substituting \(\mu=\theta\) and \(\sigma^2=\theta^2\) into the natural parameters gives $$ \eta_1(\theta)=\frac{\theta}{\theta^2}=\frac{1}{\theta}, \qquad \eta_2(\theta)=-\frac{1}{2\theta^2}. $$ Consequently, \(\eta_2=-\eta_1^2/2\), with \(\eta_1>0\). The allowed natural parameters lie on a parabola in the two-dimensional natural-parameter space.
To check regularity, differentiate the parameterization: $$ \eta'(\theta)=\left(-\frac{1}{\theta^2},\frac{1}{\theta^3}\right)^\mathsf{T}. $$ This vector is nonzero for every \(\theta>0\), so the Jacobian has rank one. The map is one-to-one because \(\eta_1=1/\theta\) determines \(\theta\). The image is not a line: its points satisfy the nonlinear relation \(\eta_2=-\eta_1^2/2\) for every \(\eta_1>0\). This is therefore a genuinely curved, one-dimensional subfamily of the two-dimensional normal family.
What Restriction Does to the Likelihood
For independent observations \(X_1,\ldots,X_n\) from a full exponential family, the likelihood under the restricted model is obtained by substituting \(\eta(\theta)\) into the full-family likelihood. Writing \(T_\Sigma=\sum_{i=1}^n T(X_i)\), this gives $$ L(\theta)=\left\{\prod_{i=1}^n h(X_i)\right\} \exp\left\{\eta(\theta)^\mathsf{T}T_\Sigma-nA\bigl(\eta(\theta)\bigr)\right\}. $$ The restriction changes which natural-parameter values are allowed, not the form of the density for any allowed value.
Proof. The joint density under the submodel factors as $$ f_\theta(x_1,\ldots,x_n)= \left\{\prod_{i=1}^n h(x_i)\right\} \exp\left\{\eta(\theta)^\mathsf{T}\sum_{i=1}^n T(x_i)-nA\bigl(\eta(\theta)\bigr)\right\}. $$ The first factor depends on the data but not on \(\theta\). The second depends on the data only through \(\sum_i T(x_i)\), and on the parameter through \(\theta\). By the factorization criterion for sufficiency, this statistic is sufficient in the restricted model as well. The statistic’s dimension need not equal the model’s dimension: a two-component statistic can remain sufficient when \(\theta\) is one-dimensional. \(\square\)
Worked Example: A Different Linked Normal Model
Now let \(X\sim N(\theta,1+\theta^2)\), where \(\theta\in\mathbb R\). This is again a one-dimensional restriction of the two-dimensional normal family, but with a different relationship between mean and variance. Its natural parameterization is $$ \eta_1(\theta)=\frac{\theta}{1+\theta^2}, \qquad \eta_2(\theta)=-\frac{1}{2(1+\theta^2)}. $$
The derivative is $$ \eta'(\theta)=\left(\frac{1-\theta^2}{(1+\theta^2)^2}, \frac{\theta}{(1+\theta^2)^2}\right)^\mathsf{T}. $$ The two entries cannot both be zero: the second is zero only at \(\theta=0\), where the first equals \(1\). Thus the Jacobian has rank one everywhere. To check that the parameterization is one-to-one, note that \(\eta_2\) determines \(\theta^2\), while the sign of \(\eta_1\) determines the sign of \(\theta\) when \(\theta\ne 0\); \(\eta_1=0\) occurs only at \(\theta=0\).
The image is not a line. In particular, it contains the three points \((0,-1/2)\), \((1/2,-1/4)\), and \((-1/2,-1/4)\), corresponding to \(\theta=0,1,-1\). The first point is not on the horizontal line through the latter two, and the three points are not collinear. This model is therefore genuinely curved.
Worked Likelihood Under a Curved Restriction
Worked Example: Two Observations from \(N(\theta,\theta^2)\)
Take independent observations \(x_1=1\) and \(x_2=3\) from \(N(\theta,\theta^2)\), with \(\theta>0\). The direct likelihood is $$ L(\theta)=\frac{1}{2\pi\theta^2} \exp\left\{-\frac{(1-\theta)^2+(3-\theta)^2}{2\theta^2}\right\}. $$ For example, at \(\theta=2\), the squared deviations sum to \((1-2)^2+(3-2)^2=2\). Hence $$ L(2)=\frac{1}{8\pi}\exp\left(-\frac{2}{8}\right) =\frac{e^{-1/4}}{8\pi}. $$
The same result follows from the canonical form. Here \(\sum_i x_i=4\), \(\sum_i x_i^2=10\), and at \(\theta=2\), \(\eta_1=1/2\), \(\eta_2=-1/8\). The log-partition function is \(A=\tfrac12+\tfrac12\log(8\pi)\), so $$ \log L(2)=\frac{1}{2}(4)-\frac{1}{8}(10)-2\left\{\frac{1}{2}+\frac{1}{2}\log(8\pi)\right\} =-\frac{1}{4}-\log(8\pi). $$ Exponentiating yields \(L(2)=e^{-1/4}/(8\pi)\), matching the direct calculation. This checks both the normal canonical representation and the substitution of the curved parameterization.
Why the Distinction Matters
A constraint in the original parameters does not necessarily create a genuinely curved family in natural coordinates. For example, the normal subfamily \(X\sim N(\theta,1)\) has \(\eta_1=\theta\) and \(\eta_2=-1/2\). It is one-dimensional inside the two-dimensional normal family, but its natural parameters lie on a straight line. It is a lower-dimensional exponential-family submodel, yet it is not curved under the definition above.
The natural-parameter map is therefore essential: inspect the restriction after expressing the model in the full family’s natural coordinates. This matters because likelihood calculations take place on the restricted set of allowed parameters. The full-family representation supplies a convenient common framework, while the restriction determines which combinations of its natural parameters are possible.
Common Pitfall
- Wrong reasoning: “There is one parameter but two natural parameters, so the model must be curved.” A dimension reduction alone does not establish curvature; a straight line is also lower-dimensional.
- Correction: Write the natural parameters as functions of the model parameter, verify the required smoothness and rank, and check whether the image is non-affine. For \(N(\theta,1)\), the image is a line; for \(N(\theta,\theta^2)\), it lies on the parabola \(\eta_2=-\eta_1^2/2\).
Check Your Understanding
Answer each question by identifying the relevant natural-parameter restriction.
- For \(X\sim N(\theta,\theta^2)\), \(\theta>0\), write both natural parameters as functions of \(\theta\) and state their relationship.
- Why does a nonzero derivative of the natural-parameter map verify the rank condition for a one-dimensional submodel?
- Explain why \(N(\theta,1)\) is lower-dimensional but not genuinely curved within the full normal family.
- For independent observations from a restricted exponential family, what data summary appears in the likelihood’s parameter-dependent term?
- In the model \(N(\theta,1+\theta^2)\), how do the natural parameters distinguish positive and negative values of \(\theta\)?