Anatomy of an Estimating Equation

What an Estimating Equation Says

Many estimators can be described by an equation involving the data and an unknown parameter: choose a value of the parameter that makes a specified function of the observations equal to zero. The equation may come from a likelihood score, a theoretical moment, or another scientific condition. Its form is simple, but interpreting it requires separating the equation defined by the model from the equation calculated using a particular sample.

Let $X_1,\ldots,X_n$ be observations, and let $\theta$ belong to a parameter space $\Theta$. For a $p$-dimensional parameter, an estimating function $\psi(x,\theta)$ takes values in $\mathbb{R}^p$. The sample estimating equation asks for a value of $\theta$ that makes the sum of the observed estimating functions zero.

Definition: The sample estimating function is $$ \Psi_n(\theta)=\sum_{i=1}^n \psi(X_i,\theta). $$ A sample root is a value $\hat{\theta}\in\Theta$ satisfying $\Psi_n(\hat{\theta})=0$. When the observations share a distribution indexed by $\theta_0$, the population estimating function is $$ \Psi(\theta)=E_{\theta_0}[\psi(X,\theta)], $$ provided the expectation exists. A population root is a value $\theta\in\Theta$ such that $\Psi(\theta)=0$. The parameter value the equation is intended to identify is often denoted $\theta_0$; it is a population root when $\Psi(\theta_0)=0$.

The expectation in the population equation is taken under the data-generating distribution, indexed here by $\theta_0$, while the argument $\theta$ is the value being assessed. These need not be the same value. The sample equation uses observed data; the population equation describes what its left-hand side averages to under a specified distribution.

It is also common to use the sample average $n^{-1}\Psi_n(\theta)$ instead of the sum. Since $n$ is positive, these versions have exactly the same roots. For independent, identically distributed observations, the expectation of the sample average is the population estimating function whenever the expectation exists:

E_{\theta_0}\!\left[\frac{1}{n}\Psi_n(\theta)\right] =\frac{1}{n}\sum_{i=1}^n E_{\theta_0}[\psi(X_i,\theta)] =E_{\theta_0}[\psi(X,\theta)] =\Psi(\theta).

This identity does not say that the sample equation equals the population equation for a realized data set. It says that its average, across repeated samples, has the population value.

Roots, Targets, and Scaling

An estimating equation is a rule for producing candidate parameter values, not a guarantee that a candidate exists or is unique. There may be no root in $\Theta$, several roots, or a root at which the equation is poorly behaved. Even when a root exists, further assumptions are needed before calling it a reliable estimator of the intended target.

Result: For any fixed sample size $n$, the equations $\sum_{i=1}^n\psi(X_i,\theta)=0$ and $n^{-1}\sum_{i=1}^n\psi(X_i,\theta)=0$ have exactly the same roots. More generally, multiplying an estimating function by a fixed nonzero scalar does not change its roots.

Proof. For any $\theta$, multiplication by the nonzero scalar $1/n$ preserves whether a vector is zero:

\frac{1}{n}\Psi_n(\theta)=0 \quad\Longleftrightarrow\quad \Psi_n(\theta)=0.

The same argument applies to multiplication by any nonzero scalar. Thus choosing a sum or an average changes the scale of the equation, not its root set. $\square$

For vector equations, $\psi$ has one component for each parameter dimension, and all components must vanish together. A root is not obtained by setting just one component to zero while ignoring the others.

Likelihood Scores as Estimating Equations

Suppose a single observation has density or mass function $f(x;\theta)$. Where $f(x;\theta)>0$ and the derivative exists, the score for one observation is the derivative of its log likelihood with respect to the parameter. For independent observations, the sample score is the derivative of the sample log-likelihood. Setting that score equal to zero produces a sample estimating equation.

Definition: For a scalar parameter, the one-observation score is $$ s(x;\theta)=\frac{\partial}{\partial\theta}\log f(x;\theta). $$ For independent observations, the sample score is $$ \frac{\partial}{\partial\theta}\ell(\theta) =\sum_{i=1}^n s(X_i;\theta). $$ The score equation sets this sum equal to zero. For a vector parameter, the score is the vector of partial derivatives of the log-likelihood.

A score equation is one important kind of estimating equation, but not every estimating equation is derived from a likelihood. Also, solving the score equation supplies a stationary point under suitable conditions; it does not by itself establish that the point maximizes the likelihood.

Result: Suppose $\ell(\theta)$ is differentiable on an open parameter space $\Theta\subseteq\mathbb{R}^p$. If $\hat{\theta}$ is a local maximum of $\ell$ at an interior point of $\Theta$, then its score is zero: $$ \nabla\ell(\hat{\theta})=0. $$ Proof. For any vector $v\in\mathbb{R}^p$, openness ensures that $\hat{\theta}+tv$ remains in $\Theta$ for all sufficiently small positive and negative $t$. Define $g(t)=\ell(\hat{\theta}+tv)$. Since $\hat{\theta}$ is a local maximum, $g$ has a local maximum at $t=0$. Differentiability implies $g'(0)=0$. By the chain rule, $$ g'(0)=v^\mathsf{T}\nabla\ell(\hat{\theta}). $$ This holds for every $v$. Taking $v$ to be each coordinate vector shows that every component of $\nabla\ell(\hat{\theta})$ is zero. $\square$

The result is a necessary condition, not a sufficient one. It applies to an interior maximum; a maximum at a boundary need not have zero score, and a zero score can occur at a minimum or another stationary point.

Worked Example: The Score Equation for a Normal Mean

Let $X_1,\ldots,X_n$ be independent observations from $N(\mu,4)$, with known variance $4$ and unknown mean $\mu\in\mathbb{R}$. Suppose the observed data are $2,3,7$. Derive the sample and population score equations and find their roots.

For one observation, the log density, apart from its constant term, is $- (x-\mu)^2/8$. Differentiating with respect to $\mu$ gives $$ s(x;\mu)=\frac{x-\mu}{4}. $$

The sample score equation is therefore $$ \sum_{i=1}^3\frac{X_i-\mu}{4}=0. $$ For the observed values, this becomes $(2-\mu+3-\mu+7-\mu)/4=0$. Multiplying by $4$ and solving gives $12-3\mu=0$, so the sample root is $\hat{\mu}=4$.

If the population distribution is $N(\mu_0,4)$, then $E_{\mu_0}[X]=\mu_0$, and the population score function is $$ E_{\mu_0}[s(X;\mu)]=\frac{\mu_0-\mu}{4}. $$ Its root is $\mu=\mu_0$. Thus the population score equation targets the true mean, while the sample root is the observed sample mean. In this example the score has one root, but the general necessary-condition result alone would not establish that every score root is a maximum.

Moment Equations for General Parameters

A moment equation starts with a relationship between a population quantity and the parameter. If a model implies $E_\theta[g(X)]=m(\theta)$, one can compare the observed sample average of $g(X)$ with its model-implied value. The resulting estimating function is $\psi(x,\theta)=g(x)-m(\theta)$, when the expectation exists.

Its population counterpart is $E_{\theta_0}[g(X)]-m(\theta)$. If the model's moment relation holds at $\theta_0$, then $\theta_0$ is a population root. A sample root solves the corresponding equation using the data. This construction makes explicit which feature of the distribution is being used to learn about the parameter.

Worked Example: A Moment Equation for an Exponential Rate

Let $X_1,\ldots,X_n$ be independent exponential observations with rate $\lambda>0$, so $E_\lambda[X]=1/\lambda$. The observed values are $0.5,1,1.5$. Use the first-moment equation to estimate $\lambda$.

Take $g(x)=x$ and $m(\lambda)=1/\lambda$. The sample estimating function, using the average form, is $$ \frac{1}{3}\sum_{i=1}^3X_i-\frac{1}{\lambda}. $$ The observed sample mean is $(0.5+1+1.5)/3=3/3=1$. The sample equation is $1-1/\lambda=0$. Since $\lambda>0$, solving gives the unique root $\hat{\lambda}=1$.

Under a population distribution with rate $\lambda_0$, the population equation is $1/\lambda_0-1/\lambda=0$. Both rates are positive, so this equation has the unique root $\lambda=\lambda_0$. The sample root and population root play different roles: the first depends on the observed values, and the second describes the target implied by the model.

Worked Example: A Nonlinear Moment Equation for a Poisson Parameter

Suppose $X_1,\ldots,X_n$ are independent Poisson observations with parameter $\lambda>0$, and the observed data are $0,1,3$. Use the second raw moment to form a moment equation and solve it.

For a Poisson random variable, $E_\lambda[X]=\lambda$ and $\operatorname{Var}_\lambda(X)=\lambda$. Since $E[X^2]=\operatorname{Var}(X)+(E[X])^2$, the model gives $E_\lambda[X^2]=\lambda+\lambda^2$. Set the sample average of $X_i^2$ equal to this theoretical moment: $$ \frac{1}{3}\sum_{i=1}^3X_i^2-(\lambda+\lambda^2)=0. $$

The observed average of the squares is $(0^2+1^2+3^2)/3=10/3$. Thus $\lambda^2+\lambda-10/3=0$. The quadratic formula gives $$ \lambda=\frac{-1\pm\sqrt{1+40/3}}{2} =\frac{-1\pm\sqrt{43/3}}{2}. $$

Because $\lambda$ must be positive, only the plus sign is admissible: $\hat{\lambda}=(-1+\sqrt{43/3})/2$, approximately $1.393$. The other algebraic root is negative and lies outside the parameter space. This example shows why solving the equation and checking the parameter space are both necessary.

Common Pitfalls and Why the Distinctions Matter

The equation's components carry different meanings: $\psi$ specifies the contribution of one observation, $\Psi_n$ aggregates observed contributions, and $\Psi$ averages a contribution under a population distribution. Keeping those roles separate prevents several common errors.

Confusing a sample root with a population root. A value solving the equation for observed data need not equal the root of the population equation in that particular sample. One is data-dependent; the other is defined through a distributional expectation.
Assuming every estimating equation comes from likelihood. A score equation is derived from a log-likelihood, while a moment equation is built from a population moment relation. Other constructions are possible too.
Treating every score root as a maximum. A zero derivative is a stationarity condition. It does not distinguish a maximum from a minimum or another stationary point, and boundary maxima require separate consideration.
Ignoring the parameter space. An algebraic solution outside $\Theta$, such as a negative rate, is not an admissible root for the model's parameter.
Assuming a root exists or is unique. Either property must be checked for the equation and parameter space at hand; it is not part of the definition.

Estimating equations provide a common language for methods that otherwise look different. The score equation for a mean and the moment equation for a rate both ask for roots, but their ingredients and justifications differ. Later analysis of consistency and limiting distributions depends on making these definitions precise first.

Check Your Understanding

Use the definitions and examples above to answer the following questions.

For an estimating function $\psi(X,\theta)$, what distinguishes the sample estimating function from the population estimating function?
Why do the sample sum equation and sample average equation have the same roots?
For independent $N(\mu,4)$ observations, write the one-observation score for $\mu$ and state the population score root.
In the exponential example, why is the positive solution to the moment equation the only admissible root?
Why does a zero likelihood score at an interior point not, by itself, prove that the point is a likelihood maximum?