Tangent Spaces and Likelihood Geometry

Introduction

A curved exponential-family model restricts a vector of natural parameters to a lower-dimensional surface. At a particular parameter value, the tangent space describes the changes the model can make to first order. Fisher information supplies the geometry for measuring those changes: it determines which tangent direction best approximates a proposed movement in the full natural-parameter space.

This geometry clarifies the role of the score. A score measures the first-order change in log likelihood along a direction. Only its component along the model’s tangent space can produce a first-order improvement within the curved model. The component normal to that space may still matter at second order if the curve itself bends.

Natural-Parameter Coordinates and the Tangent Space

Consider a regular $k$-parameter exponential family with density or mass function $f(x;\eta)=h(x)\exp\{\eta^T T(x)-A(\eta)\}$, where $\eta$ lies in an open natural parameter space $\mathcal H\subset\mathbb R^k$. Assume $A$ is twice continuously differentiable. A $q$-dimensional curved subfamily, with $q\leq k$, is specified by a smooth map $\eta=\eta(\theta)$, where $\theta$ belongs to an open subset of $\mathbb R^q$. Write $J(\theta)$ for the $k\times q$ Jacobian matrix of this map.

Definition: At $\theta_0$, the tangent space of the curved family in natural-parameter coordinates is the column space of $J_0=J(\theta_0)$: $$ \mathcal T_{\eta_0}=\{J_0a:a\in\mathbb R^q\},\qquad \eta_0=\eta(\theta_0). $$ If $J_0$ has full column rank, this space has dimension $q$. It contains the first-order natural-parameter changes available to the model near $\eta_0$.

For $n$ independent observations, let $S_n=\sum_{i=1}^n T(X_i)$. The log likelihood, up to a term not depending on $\eta$, is $\ell(\eta)=\eta^T S_n-nA(\eta)$. Its natural-coordinate score and Fisher information are $U_\eta=S_n-n\nabla A(\eta)$ and $I_\eta=nV(\eta)$, where $V(\eta)=\nabla^2A(\eta)=\operatorname{Cov}_\eta(T(X))$. We assume $V(\eta_0)$ is positive definite, as holds for a regular minimal representation. The curved score and information follow by differentiating through the map $\eta(\theta)$.

Result: At $\theta_0$, the score and Fisher information for $\theta$ are $$ U_\theta=J_0^T U_\eta,\qquad I_\theta=nJ_0^TV_0J_0,\qquad V_0=V(\eta_0). $$ The matrix $V_0$ defines the Fisher information inner product in natural coordinates: $\langle u,v\rangle_{V_0}=u^TV_0v$.

The score formula follows from the chain rule: the derivative of $\ell(\eta(\theta))$ is $J_0^T U_\eta$. The information formula follows by taking the covariance of this score, using $\operatorname{Cov}(U_\eta)=nV_0$. When $J_0$ has full column rank and $V_0$ is positive definite, $J_0^TV_0J_0$ is positive definite: for nonzero $a$, $a^TJ_0^TV_0J_0a=(J_0a)^TV_0(J_0a)>0$.

Local Likelihood and Information Projection

For a small natural-parameter displacement $h$, Taylor expansion of $A$ at $\eta_0$ gives

\ell(\eta_0+h)-\ell(\eta_0) =U_\eta^T h-\frac{n}{2}h^TV_0h+o(\|h\|^2).

Here $n$ is fixed as $h\to0$, and the remainder follows from twice continuous differentiability of $A$. The first term is the score’s directional gain; the second is the local information penalty for moving away from $\eta_0$. Restricting to tangent displacements $h=J_0a$ gives a quadratic approximation in the model’s available directions.

Theorem: Suppose $V_0$ is positive definite and $J_0$ has full column rank. The unconstrained maximizer of the local quadratic approximation is $d=V_0^{-1}U_\eta/n$. Its information-metric projection onto $\mathcal T_{\eta_0}$ is $J_0\hat a$, where $$ \hat a=(J_0^TV_0J_0)^{-1}J_0^TV_0d =(nJ_0^TV_0J_0)^{-1}J_0^TU_\eta. $$ Thus $J_0\hat a$ is the maximizing tangent displacement in the local quadratic approximation.

To prove the result, let $Q(h)=U_\eta^Th-\frac n2h^TV_0h$, the quadratic part of the log-likelihood change. Its gradient is $U_\eta-nV_0h$, so its unique unconstrained maximizer is $d=V_0^{-1}U_\eta/n$. On the tangent space, set $h=J_0a$. Differentiating $Q(J_0a)$ with respect to $a$ gives $J_0^TU_\eta-nJ_0^TV_0J_0a$. Setting this equal to zero yields the displayed $\hat a$; positive definiteness of $J_0^TV_0J_0$ ensures it is the unique maximizer there.

The projection claim follows from the same equation: $J_0^TV_0(d-J_0\hat a)=0$. The residual is therefore $V_0$-orthogonal to every tangent vector. Moreover, for any $a$, $$ (d-J_0a)^TV_0(d-J_0a) =(d-J_0\hat a)^TV_0(d-J_0\hat a) +(J_0\hat a-J_0a)^TV_0(J_0\hat a-J_0a). $$ The cross term vanishes by that orthogonality. The second term is nonnegative, so $J_0\hat a$ is the closest tangent displacement in the Fisher information metric. $\square$

Worked Examples

Worked Example: Projecting a Score in a Curved Bernoulli Model

In each of $n=10$ independent pairs, let $X_1$ and $X_2$ be independent Bernoulli variables with natural parameters $\eta_1$ and $\eta_2$. Their joint cumulant function is $A(\eta)=\log(1+e^{\eta_1})+\log(1+e^{\eta_2})$. Restrict the model to $\eta(t)=(t,2t)$, and evaluate local geometry at $t=0$. Suppose the observed totals are $\sum X_1=8$ and $\sum X_2=6$.

At $t=0$, both success probabilities are $1/2$, so the natural-coordinate score is $U_\eta=(8-10/2,6-10/2)^T=(3,1)^T$. The Jacobian is $J=(1,2)^T$, and $V_0=\frac14 I_2$. Thus the curved score is $J^TU_\eta=3+2=5$, and the curved information is $10J^TV_0J=10(\frac14+\frac44)=12.5$. The local maximizing change in $t$ is $5/12.5=0.4$.

In the full natural space, the local step is $d=V_0^{-1}U_\eta/10=(4/10)(3,1)^T=(1.2,0.4)^T$. Its information projection onto the tangent line has coefficient $$ \frac{J^TV_0d}{J^TV_0J} =\frac{\frac14(1.2)+\frac12(0.4)}{1.25} =\frac{0.5}{1.25}=0.4. $$ The projected step is therefore $0.4(1,2)^T=(0.4,0.8)^T$, the same movement implied by the curved score and information. The full-space step is not tangent, but its information projection gives the local likelihood step available to this model.

Worked Example: A Poisson Score with a Negative Tangent Direction

Let $X_1,X_2$ be independent Poisson variables with means $e^{\eta_1},e^{\eta_2}$, restricted to $\eta(t)=(t,2t)$. At $t=0$, both means are 1. For one observed pair $(X_1,X_2)=(2,0)$, the natural score is $U_\eta=(2-1,0-1)^T=(1,-1)^T$, and $V_0=I_2$.

The tangent vector is $J=(1,2)^T$. Consequently, the curved score is $J^TU_\eta=1-2=-1$, and the curved information is $J^TV_0J=1+4=5$. The local quadratic approximation therefore gives a change in $t$ of $-1/5=-0.2$. The full-space local step is $d=(1,-1)^T$; its information projection coefficient is $(J^Td)/(J^TJ)=(1-2)/5=-0.2$. The projected natural-coordinate movement is $(-0.2,-0.4)^T$. The negative sign matters: the local score favors decreasing $t$, even though the first coordinate alone has a positive score.

Worked Example: Zero Tangent Score but Nonzero Curvature Effect

Use the paired Bernoulli model from the first example, but restrict it to $\eta(t)=(t,t^2)$. At $t=0$, observe four pairs with totals $\sum X_1=2$ and $\sum X_2=3$. Then $U_\eta=(2-2,3-2)^T=(0,1)^T$, $J(0)=(1,0)^T$, and $V_0=\frac14 I_2$. The tangent score is $J(0)^TU_\eta=0$, while the curved information is $4J(0)^TV_0J(0)=1$.

A zero score means the first derivative of the log likelihood along the curve is zero; it does not settle whether $t=0$ is a local maximum. To check the second derivative, the natural-coordinate Hessian of the log likelihood is $-nV_0=-I_2$ at zero. Also, $\eta''(0)=(0,2)^T$. Differentiating the log likelihood twice along the curve gives $$ \left.\frac{d^2}{dt^2}\ell(\eta(t))\right|_{t=0} =J(0)^T(-I_2)J(0)+\eta''(0)^TU_\eta =-1+2=1. $$ Thus $\ell(\eta(t))-\ell(\eta(0))=\frac12t^2+o(t^2)$, which is positive for sufficiently small nonzero $t$. The point is not a local maximum. The normal score, invisible to the first-order tangent score, contributes through the curve’s second derivative.

How to Read the Geometry

The raw natural-coordinate score $U_\eta$ is a gradient in the chosen coordinates, while $V_0^{-1}U_\eta$ is the corresponding information-scaled direction. The curved score $J_0^TU_\eta$ records directional derivatives along the tangent coordinates. Information projection then selects the tangent movement that best matches the full-space local step under the metric $V_0$. This is a local description; it does not, by itself, establish that a global maximum exists or that an exact maximizer has been found.

Common Mistake

Mistake: Treating a zero curved score as proof that the parameter value maximizes the likelihood. Correction: A zero tangent score says only that every first-order tangent derivative vanishes. The curve’s second derivative and the score component normal to the tangent space can affect second-order behavior, as the parabola example shows.
Mistake: Projecting with ordinary Euclidean distance regardless of the model. Correction: The local likelihood uses Fisher information, so the relevant distance is $(u-v)^TV_0(u-v)$. Euclidean projection is appropriate only in special cases, such as $V_0$ being a scalar multiple of the identity.

Key takeaway: The tangent space describes the curved model’s first-order directions; the Fisher information metric determines their local likelihood cost. The score projected through those directions gives the local constrained likelihood step, while curvature can still influence higher-order behavior.

Check Your Understanding

Try these questions before reviewing the worked examples.

For a smooth map $\eta(\theta)$, how is its tangent space at $\theta_0$ defined?
Why is the curved score $J_0^TU_\eta$, rather than the full vector $U_\eta$?
What inner product defines the information projection in natural coordinates?
In the Poisson example, why can a positive score in the first natural coordinate coexist with a negative curved score?
Why does a zero tangent score not necessarily imply a local maximum along a curved path?