Likelihood Scores as Moment-Matching Equations

Why a Likelihood Score Matches Moments

In an exponential family, the natural parameter controls the distribution through a statistic of the data. The likelihood score therefore compares two quantities: what the sample actually records in that statistic, and what the model expects to record. When the score is zero in natural coordinates, these quantities match.

This statement is an algebraic property of the canonical likelihood, not a guarantee that a finite solution exists for every sample. To make the claim precise, first derive how the log-partition function encodes the expected sufficient statistic.

Setup and the Log-Partition Gradient

Definition: A regular $d$-parameter exponential family in natural coordinates has densities or mass functions, relative to a fixed measure, of the form $$ f_\eta(x)=h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\},\qquad \eta\in\Theta\subseteq\mathbb{R}^d. $$ Here $T(x)\in\mathbb{R}^d$ is the sufficient statistic, $h(x)$ is a nonnegative carrier function, $\Theta$ is an open natural parameter space, and $A(\eta)$ is the log-partition function that normalizes the density. In particular, $$ A(\eta)=\log\int h(x)\exp\{\eta^\mathsf{T}T(x)\}\,\nu(dx), $$ where $\nu$ is the fixed underlying measure, and the integral is finite for $\eta\in\Theta$.

For the next identity, the fixed support and open natural parameter space matter. They allow differentiation of the normalizing integral locally. Specifically, around any $\eta\in\Theta$, choose a sufficiently small closed box contained in $\Theta$. Exponentials at its finitely many corners provide integrable bounds for the integrand and its first derivatives throughout a smaller box. Differentiation under the integral is therefore justified by dominated convergence.

Theorem: For a regular exponential family as defined above, the expected sufficient statistic is the gradient of the log-partition function: $$ \nabla A(\eta)=E_\eta[T(X)]. $$

Proof. Define $Z(\eta)=\exp\{A(\eta)\}$. For coordinate $j$, differentiation under the integral gives $$ \frac{\partial Z(\eta)}{\partial\eta_j} = \int T_j(x)h(x)\exp\{\eta^\mathsf{T}T(x)\}\,\nu(dx). $$ Since $A(\eta)=\log Z(\eta)$, divide by $Z(\eta)$: $$ \frac{\partial A(\eta)}{\partial\eta_j} = \int T_j(x)\frac{h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}}{1}\,\nu(dx) = E_\eta[T_j(X)]. $$ This holds for every coordinate $j$, proving the vector identity. $\square$

The Sample Score Equation

Let $X_1,\ldots,X_n$ be independent observations from this same family, with common natural parameter $\eta$. The likelihood is a product, and its log is a sum. Terms involving $h(X_i)$ do not depend on $\eta$, so the log-likelihood and its gradient are

\ell(\eta) = \sum_{i=1}^n\log h(X_i) +\eta^\mathsf{T}\sum_{i=1}^nT(X_i)-nA(\eta), \qquad \nabla\ell(\eta) = \sum_{i=1}^nT(X_i)-n\nabla A(\eta).

Theorem: At any interior natural parameter value $\eta$, the sample likelihood score is $$ \nabla\ell(\eta) = \sum_{i=1}^nT(X_i)-nE_\eta[T(X)]. $$ Consequently, a finite interior zero of the score satisfies $$ \frac{1}{n}\sum_{i=1}^nT(X_i)=E_\eta[T(X)]. $$

Proof. Independence gives the product likelihood $$ L(\eta)=\prod_{i=1}^n h(X_i)\exp\{\eta^\mathsf{T}T(X_i)-A(\eta)\}. $$ Taking logarithms produces the stated expression for $\ell(\eta)$. Differentiating it gives $\nabla\ell(\eta)=\sum_iT(X_i)-n\nabla A(\eta)$. Substituting the proved identity $\nabla A(\eta)=E_\eta[T(X)]$ yields the score formula. The score equals zero exactly when the displayed sample average equals the model expectation. $\square$

The equation is vector-valued: each component of the observed statistic average must match the corresponding component of its model expectation. The aggregate $\sum_iT(X_i)$ is sufficient in this representation, so the likelihood score depends on the observations through that aggregate.

Worked Moment-Matching Equations

Worked Example: A Three-Category Model

Suppose each observation takes one of three categories. Use category 3 as the baseline, with $T(X)=(I\{X=1\},I\{X=2\})^\mathsf{T}$. The canonical model has $$ A(\eta)=\log(1+e^{\eta_1}+e^{\eta_2}). $$ In a sample of 10 observations, suppose the category counts are $4,3,3$. Find the natural parameter at which the score is zero.

The observed average statistic is $(4/10,3/10)^\mathsf{T}=(0.4,0.3)^\mathsf{T}$. Differentiating $A$ gives $$ \nabla A(\eta)= \left( \frac{e^{\eta_1}}{1+e^{\eta_1}+e^{\eta_2}}, \frac{e^{\eta_2}}{1+e^{\eta_1}+e^{\eta_2}} \right)^\mathsf{T}. $$ Set these model probabilities equal to $0.4$ and $0.3$. The remaining category has probability $0.3$, so the ratios to its probability give $$ e^{\eta_1}=\frac{0.4}{0.3}=\frac{4}{3}, \qquad e^{\eta_2}=\frac{0.3}{0.3}=1. $$ Thus $\eta_1=\log(4/3)$ and $\eta_2=0$. To check, the denominator is $1+4/3+1=10/3$, giving probabilities $0.4,0.3,0.3$. The score components, observed count minus 10 times model probability, are $4-4=0$ and $3-3=0$.

Worked Example: An Exponential Sample

Let $X_i$ have an exponential distribution with rate $\lambda>0$, independently for $i=1,\ldots,n$. In natural coordinates, $\eta=-\lambda<0$, $T(X)=X$, and $A(\eta)=-\log(-\eta)$. For observations $1,2,3,4$, find the score root.

The observed statistic sum is $10$, and the sample average is $10/4=2.5$. The model expectation is $$ E_\eta[X]=A'(\eta)=-\frac{1}{\eta}. $$ Moment matching requires $-1/\eta=2.5$, so $\eta=-0.4$, corresponding to $\lambda=0.4$. Directly, the log-likelihood, up to terms not depending on $\eta$, is $10\eta-4A(\eta)$; its derivative is $$ 10-4A'(\eta)=10+\frac{4}{\eta}. $$ At $\eta=-0.4$, this equals $10-10=0$, as required.

Worked Example: A Geometric Sample

Consider the geometric distribution on $0,1,2,\ldots$, written in canonical form as $f_\eta(k)=(1-e^\eta)e^{\eta k}$, where $\eta<0$. Here $T(X)=X$ and $A(\eta)=-\log(1-e^\eta)$. For observations $0,1,1,2,3$, solve the score equation.

The sum is $7$, so the observed average is $7/5$. Differentiating the log-partition function gives $$ E_\eta[X]=A'(\eta)=\frac{e^\eta}{1-e^\eta}. $$ Set this equal to $7/5$. Writing $r=e^\eta$, the equation becomes $r/(1-r)=7/5$, hence $5r=7-7r$, $12r=7$, and $r=7/12$. Therefore $\eta=\log(7/12)<0$, which is in the natural parameter space. Checking the score gives $7-5(7/5)=0$.

Interpreting the Equation—and Its Limits

Moment matching gives a concise interpretation of the likelihood score in a full exponential family with natural parameter $\eta$. It also explains why sufficient statistics are central: these are precisely the sample summaries that appear in the score, and their expectations determine the matching equations.

A common mistake is to conclude that matching equations always have a finite solution, or that every score root automatically proves a global maximum. The score calculation establishes only the equation for an interior stationary point. For example, in the three-category model, if every observation falls in category 1, the observed average statistic is $(1,0)^\mathsf{T}$. A finite natural parameter gives all three category probabilities strictly positive, so it cannot make the first probability equal to 1. There is no finite parameter solving the matching equations in this case. Existence and global behavior require additional analysis.

Another important qualification concerns parameterization. The displayed equation is for the natural-coordinate score. If a model is instead parameterized by $\theta$ with $\eta=\eta(\theta)$, the chain rule gives $$ \nabla_\theta\ell(\theta) = J(\theta)^\mathsf{T} \left\{\sum_{i=1}^nT(X_i)-nE_{\eta(\theta)}[T(X)]\right\}, $$ where $J(\theta)$ is the Jacobian matrix of $\eta(\theta)$. A zero of this transformed score need not force every component of the bracketed vector to be zero. The direct moment-matching equivalence applies when the natural-coordinate score itself is set to zero.

Key takeaway: For an independent sample from a regular exponential family with a common natural parameter, the natural-coordinate score is the observed sufficient-statistic sum minus $n$ times its model expectation. A finite interior score root therefore matches the observed and expected sufficient-statistic averages.

Check Your Understanding

Use the definitions and score identity above to answer these questions.

Why does differentiating the log-partition function require conditions that justify differentiating under the integral?
For a two-component sufficient statistic, how many moment-matching equations arise from setting the natural-coordinate score to zero?
In the exponential-sample example, what observed average is matched to the model expectation?
Why can the three-category sample with all observations in category 1 have no finite score root?
What role does the Jacobian play when the model is expressed in a parameterization other than natural coordinates?