Taylor Linearization of Sample Estimating Equations

Why Linearize an Estimating Equation?

An estimator defined as a root of a sample equation is often difficult to analyze directly. A useful step is to compare that equation at the estimator with its value at the population root. Differentiability makes this comparison precise: the change in the equation is a Jacobian times the change in the parameter, plus a remainder. When the remainder is small, this gives a first-order representation of the estimator’s error in terms of the sample estimating function at the true parameter.

The central task here is the linearization itself. It is distinct from showing that the estimating function has a limiting distribution: the expansion identifies the term whose distribution may later be studied, without asserting a normal limit.

Population and Sample Equations

Definition: Let $X_1,\ldots,X_n$ be observations, and let $\psi(x,\theta)$ be a $p$-dimensional estimating function for a parameter $\theta\in\Theta\subseteq\mathbb{R}^p$. The sample estimating function is $$ \Psi_n(\theta)=\frac{1}{n}\sum_{i=1}^n\psi(X_i,\theta). $$ The population estimating function is $m(\theta)=E[\psi(X,\theta)]$, when this expectation exists. A parameter $\theta_0$ satisfying $m(\theta_0)=0$ is a population root. A sample root $\hat{\theta}$ satisfies $\Psi_n(\hat{\theta})=0$.

In the main result, the observations are independent and identically distributed, $\theta_0$ is an interior point of $\Theta$, and $\hat{\theta}$ is a measurable sample root that is consistent: $\hat{\theta}\xrightarrow{p}\theta_0$. A population root need not be unique for the algebra below, but identification and regularity conditions are commonly used to establish consistency.

Assume $\psi(x,\theta)$ is differentiable with respect to $\theta$ in a neighborhood of $\theta_0$. Write $D_\theta\psi(x,\theta)$ for its $p\times p$ Jacobian matrix, with entry $(j,k)$ equal to the derivative of component $j$ with respect to parameter coordinate $k$. Then $$ D_\theta\Psi_n(\theta)=\frac{1}{n}\sum_{i=1}^nD_\theta\psi(X_i,\theta). $$ We use the Euclidean norm for vectors and its induced operator norm for matrices.

A candidate population Jacobian is $$ J=E[D_\theta\psi(X,\theta_0)]. $$ It is not automatically valid to identify $J$ with the derivative of $m(\theta)$. That identification requires conditions allowing differentiation under the expectation, such as an integrable bound on the derivatives in a neighborhood of $\theta_0$. For the expansion below, what matters directly is that the sample Jacobian near $\theta_0$ approximates a fixed matrix $J$.

The Taylor Identity and Remainder

Let $\delta=\hat{\theta}-\theta_0$. For each realized sample, consider the one-dimensional path $t\mapsto\Psi_n(\theta_0+t\delta)$, where $0\leq t\leq1$. If the segment lies in the differentiability neighborhood, the fundamental theorem of calculus gives an exact identity.

Result: If $\Psi_n$ is continuously differentiable along the segment from $\theta_0$ to $\hat{\theta}$, then $$ \Psi_n(\hat{\theta})=\Psi_n(\theta_0)+ \left\{\int_0^1D_\theta\Psi_n(\theta_0+t\delta)\,dt\right\}\delta. $$ This is the multivariate integral form of the first-order Taylor expansion. The matrix inside braces is the average Jacobian along the segment.

To verify the identity, apply the scalar fundamental theorem of calculus to each component of the vector-valued path. Its derivative is $D_\theta\Psi_n(\theta_0+t\delta)\delta$. Integrating from zero to one yields the displayed expression. In particular, the expansion does not presume that the Jacobian is constant along the path.

Subtract a fixed matrix $J$ from the average Jacobian and define the remainder $$ R_n=\left\{\int_0^1[D_\theta\Psi_n(\theta_0+t\delta)-J]\,dt\right\}\delta. $$ Then the identity becomes $$ \Psi_n(\hat{\theta})=\Psi_n(\theta_0)+J\delta+R_n. $$ Using the operator norm and the triangle inequality gives the useful bound $$ \|R_n\|\leq \left\{\sup_{0\leq t\leq1} \|D_\theta\Psi_n(\theta_0+t\delta)-J\|\right\}\|\delta\|. $$ Thus it is the Jacobian’s variation from $J$ along the entire segment—not just its value at one endpoint—that controls the remainder.

A second-derivative bound gives a more explicit, quadratic version. Suppose $\Psi_n$ is twice continuously differentiable on the segment, and its second derivative, viewed as a bilinear map, has norm at most $K_n$ there. Taylor’s formula with integral remainder gives $$ \Psi_n(\hat{\theta})=\Psi_n(\theta_0)+D_\theta\Psi_n(\theta_0)\delta+\widetilde R_n, \qquad \|\widetilde R_n\|\leq\frac{K_n}{2}\|\delta\|^2. $$ Indeed, the remainder is the integral from zero to one of $(1-t)D^2\Psi_n(\theta_0+t\delta)[\delta,\delta]$. Taking norms bounds it by $K_n\|\delta\|^2\int_0^1(1-t)\,dt=K_n\|\delta\|^2/2$. This version uses the Jacobian at $\theta_0$; the earlier version instead compares the Jacobian along the path directly with $J$.

First-Order Representation of a Consistent Root

The next result turns the exact expansion into a stochastic approximation. Assume the observations are independent and identically distributed, $\hat{\theta}\xrightarrow{p}\theta_0$, and $\Psi_n(\hat{\theta})=0$. Assume also that $J$ is nonsingular and that the sample Jacobian is uniformly close to a continuous matrix-valued function $A(\theta)$ on some neighborhood $U$ of $\theta_0$: $$ \sup_{\theta\in U}\|D_\theta\Psi_n(\theta)-A(\theta)\|\xrightarrow{p}0, \qquad A(\theta_0)=J. $$ This is a sufficient uniform-convergence condition; it can be established under suitable regularity conditions on the derivatives. By continuity of $A$ and consistency, the Jacobian along the random segment from $\theta_0$ to $\hat{\theta}$ converges uniformly in probability to $J$.

Theorem: Under these assumptions, the sample root has the representation $$ \hat{\theta}-\theta_0=-J^{-1}\Psi_n(\theta_0)+r_n, \qquad \|r_n\|\leq o_p(1)\|\Psi_n(\theta_0)\|. $$ Equivalently, the error made by replacing the estimator’s deviation with $-J^{-1}\Psi_n(\theta_0)$ is of smaller stochastic order than $\|\Psi_n(\theta_0)\|$, in the bound stated.

Proof. Consistency implies that the segment from $\theta_0$ to $\hat{\theta}$ lies in $U$ with probability tending to one. Define $$ \eta_n=\sup_{0\leq t\leq1} \|D_\theta\Psi_n(\theta_0+t\delta)-J\|. $$ The uniform convergence assumption, continuity of $A$ at $\theta_0$, and $\delta\xrightarrow{p}0$ imply $\eta_n\xrightarrow{p}0$. The Taylor identity and the sample-root equation give $$ 0=\Psi_n(\theta_0)+J\delta+R_n, \qquad \|R_n\|\leq\eta_n\|\delta\|. $$ Because $J$ is nonsingular, $\|J\delta\|\geq \|\delta\|/\|J^{-1}\|$. The equation therefore implies $$ \frac{\|\delta\|}{\|J^{-1}\|} \leq \|\Psi_n(\theta_0)\|+\eta_n\|\delta\|. $$ On the event $\eta_n\leq1/(2\|J^{-1}\|)$, whose probability tends to one, rearrangement yields $$ \|\delta\|\leq2\|J^{-1}\|\|\Psi_n(\theta_0)\|. $$ Finally, rearrange the root equation: $$ \delta=-J^{-1}\Psi_n(\theta_0)-J^{-1}R_n. $$ Setting $r_n=-J^{-1}R_n$ and applying the two bounds gives $$ \|r_n\|\leq2\|J^{-1}\|^2\eta_n\|\Psi_n(\theta_0)\|. $$ Since $\eta_n\xrightarrow{p}0$, this proves the claimed representation. $\square$

The result does not say that $\hat{\theta}-\theta_0$ is normal, or even specify its rate. Those conclusions require additional information about $\Psi_n(\theta_0)$. The representation isolates the estimating function at the population root as the quantity whose behavior determines the estimator’s first-order behavior.

Worked Examples

Worked Example: A Linear Mean Equation

Let $X_1,\ldots,X_n$ be independent observations from $N(\mu,1)$, and define $\psi(X_i,\theta)=X_i-\theta$. The population root is $\theta_0=\mu$, since $E[X_i-\mu]=0$. The sample equation is $$ \Psi_n(\theta)=\bar X-\theta=0, $$ so $\hat{\theta}=\bar X$. The derivative is $D_\theta\Psi_n(\theta)=-1$ for every $\theta$, and hence $J=-1$.

For a realized sample $1,2,4$, $n=3$, $\bar X=7/3$. Take $\mu=2$, so $\Psi_n(\theta_0)=7/3-2=1/3$, and $\hat{\theta}-\theta_0=7/3-2=1/3$. The linear representation gives $$ -J^{-1}\Psi_n(\theta_0)=-(-1)^{-1}\frac13=\frac13. $$ It is exact: the estimating function is linear, so the Taylor remainder is zero.

Worked Example: A Nonlinear Scalar Equation

Suppose $X_i$ are independent exponential observations with rate $\lambda=1$, so $E[X_i]=1$. For $\theta>0$, use $\psi(X_i,\theta)=1/\theta-X_i$. The population equation is $1/\theta-1=0$, with root $\theta_0=1$. Its derivative is $-1/\theta^2$, giving $J=-1$ at the root. The sample equation has solution $\hat{\theta}=1/\bar X$.

Consider a realized sample with $\bar X=1.2$. Then $\Psi_n(1)=1-1.2=-0.2$, while $\hat{\theta}=1/1.2=5/6$ and $\delta=5/6-1=-1/6$. The first-order approximation is $$ -J^{-1}\Psi_n(1)=-(-1)^{-1}(-0.2)=-0.2. $$ It is close to, but not equal to, the actual deviation $-1/6$. To check the remainder, for this sample $$ \Psi_n(1+\delta)=-0.2-\delta+\frac{\delta^2}{1+\delta}. $$ Thus $R_n=\delta^2/(1+\delta)=(1/36)/(5/6)=1/30$, and $-0.2+1/6+1/30=0$, as required by the root equation. The nonlinear term accounts exactly for the difference between the approximation and the actual deviation.

Worked Example: A Two-Parameter Equation

Let $X_i$ and $Z_i$ be independent random variables, each taking $1$ and $-1$ with probability $1/2$. Consider $$ \psi((X_i,Z_i),\theta)= \begin{pmatrix} X_i-\theta_1\\ Z_i-\theta_2-2\theta_1^2 \end{pmatrix}. $$ Both population means are zero, so $\theta_0=(0,0)^\mathsf{T}$ is a population root. The population Jacobian at that root is $J=-I_2$, where $I_2$ is the $2\times2$ identity matrix.

Take $n=20$. Suppose the observed $X_i$'s include eleven $1$'s and nine $-1$'s, giving $\bar X=(11-9)/20=0.1$. Suppose the $Z_i$'s include twelve $1$'s and eight $-1$'s, giving $\bar Z=(12-8)/20=0.2$. The sample equation is solved first by $\hat{\theta}_1=\bar X=0.1$, and then by $$ \hat{\theta}_2=\bar Z-2\hat{\theta}_1^2=0.2-2(0.1)^2=0.18. $$ Thus $\delta=(0.1,0.18)^\mathsf{T}$, while $\Psi_n(\theta_0)=(0.1,0.2)^\mathsf{T}$. The first-order approximation is $$ -J^{-1}\Psi_n(\theta_0)= -(-I_2)^{-1} \begin{pmatrix}0.1\\0.2\end{pmatrix} = \begin{pmatrix}0.1\\0.2\end{pmatrix}. $$ The actual remainder is $\delta-(-J^{-1}\Psi_n(\theta_0))=(0,-0.02)^\mathsf{T}$. Direct substitution verifies the expansion: $\Psi_n(\theta_0)+J\delta=(0,0.02)^\mathsf{T}$, so adding this remainder gives zero. The example shows how curvature in one parameter coordinate can produce a nonzero vector remainder.

Common Pitfall and Practical Importance

A frequent mistake is to write $0=\Psi_n(\theta_0)+J(\hat{\theta}-\theta_0)$ immediately, on the grounds that the derivative has a population limit. This omits the remainder and skips the conditions that make it small. The sample Jacobian is random, may depend on $\theta$, and must be controlled along the segment between the root and the estimator. Consistency places that segment near $\theta_0$; uniform derivative convergence and continuity then justify approximating its Jacobian by $J$. Neither consistency alone nor pointwise convergence of the derivative at $\theta_0$ supplies this control.

A second check is the sign. The equation is $0=\Psi_n(\theta_0)+J(\hat{\theta}-\theta_0)+R_n$, so solving for the parameter deviation introduces a minus sign: $-J^{-1}\Psi_n(\theta_0)$. Linearization matters because it separates the local sensitivity of the equation, represented by $J$, from the sample fluctuation at the population root. It is a foundation for later distributional and variance calculations, but those require additional assumptions.

Check Your Understanding

Use the definitions and expansion above to answer the following questions.

In the integral Taylor identity, why is the Jacobian averaged along the segment from $\theta_0$ to $\hat{\theta}$, rather than evaluated automatically at just one endpoint?
For the scalar equation $\Psi_n(\theta)=\bar X-\theta^3$ with a positive population root $\theta_0$, what is the derivative at the root, and what condition must it satisfy for the inverse-Jacobian representation to apply?
What role does consistency play in controlling the Jacobian along the segment?
For a twice differentiable sample estimating function with second-derivative bound $K_n$, what is the stated bound on the Taylor remainder?
Why does the first-order representation by itself not imply that the estimator has an asymptotically normal distribution?