Scores and Information on Curved Subfamilies

From a Full Exponential Family to a Curved Subfamily

In a full exponential family, the natural parameter can vary freely over an open subset of a Euclidean space. A curved subfamily restricts that parameter to values generated by a lower-dimensional parameter. The likelihood still comes from the full family, but its derivatives must now account for how the natural parameter changes with the subfamily parameter.

The chain rule gives the subfamily score. Applying it a second time gives the observed information, which has an extra term involving the curvature of the parameterization. By contrast, the expected information has a particularly simple form: it is the full-family information transformed by the Jacobian.

Definition: Suppose one observation has density or mass function \(f(x;\eta)=h(x)\exp\{\eta^{T}T(x)-A(\eta)\}\), where the natural parameter \(\eta\) lies in an open set \(\mathcal H\subset\mathbb R^q\). A \(p\)-dimensional curved subfamily is specified by a twice continuously differentiable map \(\eta:\Theta\to\mathcal H\), where \(\Theta\) is open in \(\mathbb R^p\) and \(p<q\). The subfamily has density \(f(x;\eta(\theta))\). Its Jacobian is the \(q\times p\) matrix \(J(\theta)=D\eta(\theta)\).

For \(n\) independent observations \(X_1,\ldots,X_n\), write \(T_n=\sum_{i=1}^n T(X_i)\). The log-likelihood in full natural coordinates, up to terms that do not depend on \(\eta\), is \(\ell(\eta)=\eta^T T_n-nA(\eta)\). Its full-family score and expected information are \(U_\eta=T_n-n\nabla A(\eta)\) and \(I_\eta=n\nabla^2 A(\eta)\), respectively. We assume throughout that the true parameter is in the interior of the natural parameter space and that \(A\) has the derivatives used below. In a regular exponential family, \(\nabla A(\eta)=E_\eta[T(X)]\) and \(\nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X))\).

The Subfamily Score

Let \(\ell_c(\theta)=\ell(\eta(\theta))\) denote the log-likelihood after restriction to the curved subfamily. The gradient \(\nabla_\theta\ell_c\) is a \(p\)-vector, while \(U_\eta\) is a \(q\)-vector. The Jacobian converts between them.

Theorem: Under the smoothness conditions above, the subfamily score is \(\displaystyle U_\theta(\theta)=\nabla_\theta\ell_c(\theta)=J(\theta)^T U_\eta(\eta(\theta))\). If the model is identifiable locally and \(J\) has full column rank, the parameterization has \(p\) independent first-order directions at that point; the score formula itself does not require full column rank.

To prove the formula, write the full-family score in coordinates as \(U_{\eta,a}=\partial\ell/\partial\eta_a\), for \(a=1,\ldots,q\). For each subfamily coordinate \(\theta_r\), the multivariate chain rule gives \(\displaystyle \frac{\partial\ell_c}{\partial\theta_r}=\sum_{a=1}^q\frac{\partial\ell}{\partial\eta_a}\frac{\partial\eta_a}{\partial\theta_r}\). These \(p\) component equations are exactly \(U_\theta=J^T U_\eta\). Thus the subfamily score is the full score combined in the directions in which the parameterization can change. \(\square\)

Worked Example: A Bernoulli Curve and Its Score

For one observation, let \(X_1\) and \(X_2\) be independent Bernoulli variables with natural parameters \(\eta_1=\theta\) and \(\eta_2=\theta^2\). Their joint log-likelihood, omitting terms independent of \(\theta\), is \(\ell_c(\theta)=x_1\theta-\log(1+e^\theta)+x_2\theta^2-\log(1+e^{\theta^2})\). Take \(\theta=0.5\) and observed values \((x_1,x_2)=(1,0)\).

The Jacobian is \(J=(1,2\theta)^T\), so at \(\theta=0.5\), \(J=(1,1)^T\). The full-family score is \(U_\eta=(x_1-p_1,x_2-p_2)^T\), where \(p_a=e^{\eta_a}/(1+e^{\eta_a})\). Here \(p_1=0.622459\) and \(p_2=0.562177\), giving \(U_\eta=(0.377541,-0.562177)^T\). Therefore \(\displaystyle U_\theta=J^TU_\eta=0.377541-0.562177=-0.184636.\)

The score can also be checked by differentiating the displayed log-likelihood directly: its derivative is \(x_1-p_1+2\theta(x_2-p_2)\), which at the stated values is \(0.377541-0.562177=-0.184636\).

Expected Information on the Curved Subfamily

The expected information measures the variance of the score under the model. In the full exponential family, \(E_\eta[U_\eta]=0\) and \(\operatorname{Var}_\eta(U_\eta)=I_\eta\), where \(I_\eta=n\nabla^2 A(\eta)\). These identities follow from \(U_\eta=T_n-n\nabla A(\eta)\) and \(\operatorname{Var}(T_n)=n\operatorname{Cov}(T(X))\).

Theorem: The expected Fisher information for \(\theta\) in the curved subfamily is \(\displaystyle I_\theta(\theta)=E_\theta[U_\theta U_\theta^T]=J(\theta)^T I_\eta(\eta(\theta))J(\theta)=nJ(\theta)^T\nabla^2A(\eta(\theta))J(\theta).\)

For the proof, use the score formula and treat \(J\) as fixed at the parameter value at which the expectation is taken: \(\displaystyle E_\theta[U_\theta U_\theta^T]=E_\theta[J^TU_\eta U_\eta^TJ]=J^TE_\eta[U_\eta U_\eta^T]J=J^TI_\eta J.\) The mean of \(U_\eta\) is zero, so its second moment matrix is its covariance matrix, namely \(I_\eta\). This proves the formula. \(\square\)

The result is a \(p\times p\) information matrix, not the \(q\times q\) full-family information matrix. It describes information for changes in \(\theta\), after those changes have been mapped into natural-parameter changes by \(J\). If \(J\) has full column rank and \(I_\eta\) is positive definite, then \(I_\theta\) is positive definite: for any nonzero \(v\in\mathbb R^p\), \(Jv\ne0\), and \(v^TI_\theta v=(Jv)^TI_\eta(Jv)>0\). If either condition fails, the subfamily information may be singular.

Worked Example: Induced Information in a Poisson Curve

Let \(X_1,X_2\) be independent Poisson variables with means \(e^{\eta_1}\) and \(e^{\eta_2}\), and restrict to \(\eta_1=\theta,\ \eta_2=\theta^2\). For this full family, \(A(\eta)=e^{\eta_1}+e^{\eta_2}\), so the full-family information for one pair is \(\displaystyle I_\eta=\begin{pmatrix}e^{\eta_1}&0\\0&e^{\eta_2}\end{pmatrix}.\)

At \(\theta=1\), \(J=(1,2)^T\), \(\eta=(1,1)^T\), and \(I_\eta=\operatorname{diag}(e,e)\). The curved-subfamily information is the scalar \(\displaystyle I_\theta=J^TI_\eta J=e+4e=5e\approx13.5914.\)

The full-family answer is instead a \(2\times2\) matrix, \(\operatorname{diag}(e,e)\). It is not the information for the one-dimensional parameter \(\theta\); the Jacobian-based calculation combines the two natural-coordinate contributions in the proportions \(1\) and \(2\).

Observed Information and the Curvature Term

Expected information depends on the covariance of the full score and requires only the first derivative \(J\). Observed information is the negative Hessian of the realized log-likelihood, so the second derivative of \(\eta(\theta)\) also matters. This distinction is important: the observed information at a particular sample need not equal the expected information.

Result: For \(r,s\in\{1,\ldots,p\}\), the observed information on the subfamily is \(\displaystyle \mathcal J_{\theta,rs}=-\frac{\partial^2\ell_c}{\partial\theta_r\partial\theta_s}=(J^T\mathcal J_\eta J)_{rs}-\sum_{a=1}^q U_{\eta,a}\frac{\partial^2\eta_a}{\partial\theta_r\partial\theta_s},\) where \(\mathcal J_\eta=-\nabla_\eta^2\ell\) is the full-family observed information.

To derive this, first differentiate \(\partial\ell_c/\partial\theta_r=\sum_a U_{\eta,a}\,\partial\eta_a/\partial\theta_r\) with respect to \(\theta_s\). The product and chain rules give \(\displaystyle \frac{\partial^2\ell_c}{\partial\theta_s\partial\theta_r}=\sum_{a,b}\frac{\partial^2\ell}{\partial\eta_a\partial\eta_b}\frac{\partial\eta_b}{\partial\theta_s}\frac{\partial\eta_a}{\partial\theta_r}+\sum_a U_{\eta,a}\frac{\partial^2\eta_a}{\partial\theta_s\partial\theta_r}.\) Negating both sides yields the stated expression. In a canonical exponential family, \(\mathcal J_\eta=n\nabla^2A(\eta)=I_\eta\), but the second term still depends on the observed score. Its expectation is zero because \(E_\eta[U_\eta]=0\); taking expectations therefore recovers the expected-information formula. \(\square\)

Worked Example: Observed Versus Expected Information in the Same Poisson Curve

Continue with the Poisson model \(\eta=(\theta,\theta^2)\), and take \(\theta=1\) with observed counts \((x_1,x_2)=(2,1)\). The full-family score is \(U_\eta=(x_1-e^{\eta_1},x_2-e^{\eta_2})^T\), so \(U_\eta=(2-e,1-e)^T\). The curved score is \(\displaystyle U_\theta=(1,2)U_\eta=(2-e)+2(1-e)=4-3e\approx-4.1548.\)

The full-family observed information is \(\operatorname{diag}(e,e)\), and the second derivative of \(\eta_2=\theta^2\) is \(2\). Thus the observed information is \(\displaystyle \mathcal J_\theta=(1,2)\begin{pmatrix}e&0\\0&e\end{pmatrix}\begin{pmatrix}1\\2\end{pmatrix}-2(1-e)=5e-2(1-e)=7e-2\approx17.028.\)

The expected information at the same parameter value is \(5e\approx13.591\). The difference is \(2e-2\), which equals the sample-dependent curvature contribution \(-2U_{\eta,2}\). This confirms both that the extra term is not generally zero for observed data and that its expectation is zero.

Worked Example: A Curved Mean Model for Normal Data

Let \(X_1\) and \(X_2\) be independent \(N(\eta_1,1)\) and \(N(\eta_2,1)\) observations, with \(\eta_1=\theta\) and \(\eta_2=\theta^2\). This is a canonical exponential family with \(A(\eta)=(\eta_1^2+\eta_2^2)/2\), so the full-family information is the \(2\times2\) identity matrix. At \(\theta=1\), \(J=(1,2)^T\), and hence the expected information is \(\displaystyle I_\theta=J^TJ=1^2+2^2=5.\)

For observed values \((x_1,x_2)=(2,0)\), the full-family score is \(U_\eta=(x_1-\eta_1,x_2-\eta_2)^T=(1,-1)^T\), giving curved score \(U_\theta=1-2=-1\). The full-family observed information is also the identity. Since \(\eta_2''(\theta)=2\), the observed information is \(\displaystyle \mathcal J_\theta=5-2(-1)=7.\)

Directly, the log-likelihood apart from constants is \(-\{(2-\theta)^2+(0-\theta^2)^2\}/2\). Its derivative at \(1\) is \(-1\), and its second derivative there is \(-7\), confirming the score and observed information.

Common Pitfall and Practical Use

A common mistake is to report \(I_\eta\) as the information for a curved parameter, or to use only \(J^T\mathcal J_\eta J\) as the observed information. The first confuses a \(q\)-dimensional parameter with the \(p\)-dimensional model actually being fitted. The second drops the second-derivative term and is generally wrong for the observed Hessian. The shorter Jacobian formula is correct for expected information; for observed information, include the curvature term unless it vanishes at the point in question.

In practice, first calculate the full-family score and information in natural coordinates, then apply \(J^T\) for the subfamily score and \(J^TI_\eta J\) for expected information. For an observed Hessian, differentiate the composed log-likelihood twice or use the full chain-rule expression. These calculations support likelihood-based inference while keeping clear which parameter space each information matrix describes.

Check Your Understanding

Use the definitions and derivations above to check each step of your reasoning.

For a smooth map \(\eta(\theta)\) from \(\mathbb R^p\) into \(\mathbb R^q\), what are the dimensions of \(J\), \(U_\eta\), and \(U_\theta\)?
Why does the expected information on the subfamily equal \(J^TI_\eta J\)? State the score property used in the argument.
When is \(J^TI_\eta J\) positive definite, and what can happen if the Jacobian is rank deficient?
In the Poisson example with \(\eta=(\theta,\theta^2)\), identify the observed-information term that is absent from the expected-information formula.
Explain why the full-family information matrix is not, by itself, the information matrix for a lower-dimensional curved parameter.