Why Roots of Sample Equations Need Conditions
A Z-estimator is defined by solving an estimating equation. It is tempting to argue that if the sample equation approaches a population equation, then its root must approach the population root. That conclusion does not follow from pointwise convergence alone: the estimator selects its argument using the data, so the equation must be controlled uniformly over the relevant parameter values. The population equation must also distinguish its target root from values far away, and a numerical or approximate solution must have a sufficiently small residual.
This tutorial makes those requirements precise. The central result applies both when an estimator solves the sample equation exactly and when it only approximately solves it. It is a consistency result: it establishes convergence in probability, not a convergence rate or a limiting distribution.
Use a fixed norm on each finite-dimensional space; for definiteness, the Euclidean norm may be used. Uniform convergence in probability on \(\Theta\) means \(\sup_{\theta\in\Theta}\|\psi_n(\theta)-\psi(\theta)\|\xrightarrow{p}0\). Assume this supremum is measurable. When \(\Theta\) is not compact, it is common to establish the same control on a compact set to which the estimator belongs with probability tending to one.
A Consistency Theorem for Approximate Roots
The theorem uses a separation condition: outside every fixed neighborhood of \(\theta_0\), the population equation stays bounded away from zero. This is stronger than merely saying that \(\theta_0\) is its unique root. On a compact parameter space, continuity of \(\psi\) and uniqueness of the root imply the separation condition.
Proof. Fix \(\varepsilon>0\) and suppose the set of parameter values at least \(\varepsilon\) from \(\theta_0\) is nonempty. If \(\|\hat{\theta}_n-\theta_0\|\geq\varepsilon\), then \(\|\psi(\hat{\theta}_n)\|\geq\Delta_\varepsilon\). The triangle inequality gives \[ \|\psi(\hat{\theta}_n)\| \leq \|\psi(\hat{\theta}_n)-\psi_n(\hat{\theta}_n)\| +\|\psi_n(\hat{\theta}_n)\| \leq \sup_{\theta\in\Theta}\|\psi_n(\theta)-\psi(\theta)\| +\|\psi_n(\hat{\theta}_n)\|. \] Consequently, \[ P(\|\hat{\theta}_n-\theta_0\|\geq\varepsilon) \leq P\!\left( \sup_{\theta\in\Theta}\|\psi_n(\theta)-\psi(\theta)\| +\|\psi_n(\hat{\theta}_n)\|\geq\Delta_\varepsilon \right) \longrightarrow 0. \] The last convergence follows because both nonnegative terms on the left inside the probability converge to zero in probability. If the set is empty, the probability in question is zero. Thus \(\hat{\theta}_n\xrightarrow{p}\theta_0\). An exact root has residual identically zero, so it meets the residual condition. \(\square\)
Worked Example: A Normal Location Equation on a Compact Set
Let \(X_1,\ldots,X_n\) be independent and identically distributed as \(N(1,1)\). Take \(\Theta=[-5,5]\), and define the sample and population estimating functions by \[ \psi_n(\theta)=\frac{1}{n}\sum_{i=1}^n(X_i-\theta)=\bar X-\theta, \qquad \psi(\theta)=E[X_i-\theta]=1-\theta. \] The target is \(\theta_0=1\).
Uniform convergence. For every \(\theta\in\Theta\), \[ |\psi_n(\theta)-\psi(\theta)| =|\bar X-\theta-(1-\theta)| =|\bar X-1|. \] Therefore the supremum over \(\Theta\) is exactly \(|\bar X-1|\), which converges to zero in probability by the law of large numbers.
Identification. For any \(\theta\), \(|\psi(\theta)|=|1-\theta|=|\theta-\theta_0|\). Hence \(\Delta_\varepsilon=\varepsilon\) whenever the set in its definition is nonempty.
The unconstrained sample root is \(\bar X\), but it need not lie in \(\Theta\) for every finite sample. Define the constrained estimate \(\hat{\theta}_n=\Pi_{[-5,5]}(\bar X)\), the projection of \(\bar X\) onto the interval. Its residual is \[ |\psi_n(\hat{\theta}_n)|=|\bar X-\Pi_{[-5,5]}(\bar X)|. \] Since \(\bar X\xrightarrow{p}1\) and \(1\) is in the interior of \([-5,5]\), the probability that \(\bar X\) lies outside the interval tends to zero. On the event that it lies inside, the residual is zero. Thus the residual converges to zero in probability, and the theorem gives \(\hat{\theta}_n\xrightarrow{p}1\).
Worked Examples with Nonlinear and Vector Equations
Worked Example: A Bernoulli Score with an Approximate Root
Let \(Y_1,\ldots,Y_n\) be independent Bernoulli observations with \(P(Y_i=1)=0.6\). For \(\Theta=[-2,2]\), consider the intercept score \[ \psi_n(\theta)=\bar Y-\frac{e^\theta}{1+e^\theta}, \qquad \psi(\theta)=0.6-\frac{e^\theta}{1+e^\theta}. \] The population root is \(\theta_0=\log(0.6/0.4)=\log(1.5)\), which lies in \([-2,2]\).
Uniform convergence. The difference between the sample and population functions is \(\bar Y-0.6\), independent of \(\theta\). Hence \[ \sup_{\theta\in\Theta}|\psi_n(\theta)-\psi(\theta)| =|\bar Y-0.6|\xrightarrow{p}0 \] by the law of large numbers.
Separation. Write \(g(\theta)=e^\theta/(1+e^\theta)\). Its derivative is \(g'(\theta)=e^\theta/(1+e^\theta)^2\). On \([-2,2]\), this derivative is at least \(m=e^2/(1+e^2)^2>0\): the derivative is symmetric about zero and decreases as \(|\theta|\) increases. By the mean value theorem, \[ |\psi(\theta)| =|g(\theta_0)-g(\theta)| \geq m|\theta-\theta_0|. \] Thus \(\Delta_\varepsilon\geq m\varepsilon\) whenever the separated set is nonempty.
To define an approximate root even if the empirical proportion is outside the range \(g([-2,2])\), let \(C=[g(-2),g(2)]\), let \(\tilde p\) be the projection of \(\bar Y\) onto \(C\), and set \(\hat{\theta}_n=\log(\tilde p/(1-\tilde p))\). Then \(\hat{\theta}_n\in[-2,2]\) and \[ |\psi_n(\hat{\theta}_n)|=|\bar Y-\tilde p| \leq|\bar Y-0.6|\xrightarrow{p}0, \] because \(0.6=g(\theta_0)\in C\), and projection onto an interval cannot increase distance to a point in that interval. Uniform convergence, separation, and the theorem imply \(\hat{\theta}_n\xrightarrow{p}\log(1.5)\).
Worked Example: A Two-Component Moment Equation
Let \(W_i\) be independent with \(P(W_i=1)=P(W_i=-1)=1/2\), and let \(\epsilon_i\) be independent \(N(0,1)\) variables independent of the \(W_i\). Set \(Y_i=1+2W_i+\epsilon_i\), and write \(Z_i=(1,W_i)^\mathsf{T}\). For \(\theta\in\Theta=[-5,5]^2\), define \[ \psi_n(\theta)=\frac{1}{n}\sum_{i=1}^n Z_i(Y_i-Z_i^\mathsf{T}\theta). \]
The target is \(\theta_0=(1,2)^\mathsf{T}\). Since \(Y_i=Z_i^\mathsf{T}\theta_0+\epsilon_i\), \(E[Z_i\epsilon_i]=0\), and \[ E[Z_iZ_i^\mathsf{T}] = \begin{pmatrix}1&E[W_i]\\E[W_i]&E[W_i^2]\end{pmatrix} = \begin{pmatrix}1&0\\0&1\end{pmatrix}, \] the population function is \(\psi(\theta)=\theta_0-\theta\). It has a unique root, with \(\|\psi(\theta)\|=\|\theta-\theta_0\|\), so \(\Delta_\varepsilon=\varepsilon\).
For uniform convergence, put \(Q_n=n^{-1}\sum_i Z_iZ_i^\mathsf{T}\). Direct subtraction yields \[ \psi_n(\theta)-\psi(\theta) = \frac{1}{n}\sum_{i=1}^n Z_i\epsilon_i -(Q_n-I)(\theta-\theta_0). \] Consequently, \[ \sup_{\theta\in\Theta}\|\psi_n(\theta)-\psi(\theta)\| \leq \left\|\frac{1}{n}\sum_{i=1}^n Z_i\epsilon_i\right\| +\|Q_n-I\|\sup_{\theta\in\Theta}\|\theta-\theta_0\|. \] The first term converges to zero by the law of large numbers, since \(E[Z_i\epsilon_i]=0\) and its components have finite means. The entries of \(Q_n\) converge to those of \(I\) by the law of large numbers; the final supremum is finite because \(\Theta\) is compact. Thus the score converges uniformly in probability.
For each sample, choose a minimizer \(\hat{\theta}_n\) of the continuous function \(\|\psi_n(\theta)\|\) over the compact set \(\Theta\). Such a minimizer exists. Since \(\theta_0\in\Theta\), \[ \|\psi_n(\hat{\theta}_n)\| \leq\|\psi_n(\theta_0)\| =\left\|\frac{1}{n}\sum_{i=1}^n Z_i\epsilon_i\right\| \xrightarrow{p}0. \] The minimizer is therefore an approximate root. Applying the theorem gives \(\hat{\theta}_n\xrightarrow{p}(1,2)^\mathsf{T}\).
Localization When the Parameter Space Is Not Compact
Compactness is a convenient sufficient condition, not an essential feature of the argument. Suppose the estimator belongs to a compact set \(K\subseteq\Theta\) with probability tending to one. If uniform convergence holds on \(K\), \(\theta_0\in K\), and the population function is separated from zero on \(K\) outside every neighborhood of \(\theta_0\), the same proof applies on the event \(\hat{\theta}_n\in K\). The probability of its complement tends to zero and therefore does not affect convergence in probability.
This localization condition matters because uniform convergence on each fixed compact set does not itself prevent an estimator from wandering outside those sets. In a noncompact problem, the analyst must separately control where the estimator lies, or establish suitable global uniform convergence and global separation.
Common Mistake and Practical Importance
- Wrong reasoning: “The sample equation converges pointwise to the population equation, so any sample root converges to the population root.” Pointwise convergence controls the equation at each fixed parameter value, not necessarily at a random value selected from the data.
- Correction: Establish uniform convergence over the estimator’s relevant parameter region, identify the target through separation from zero, and verify that the estimator’s sample-equation residual converges to zero.
- Another pitfall: Unique identification does not guarantee a sample root exists for every data set. Approximate roots allow for numerical solutions or samples with no exact root, but their residual must still be shown to vanish in probability.
These conditions clarify the logic behind consistency proofs for estimating equations. Uniform convergence transfers the sample equation’s behavior to the population equation even at a data-dependent estimate. Separation then converts a small population residual into proximity to the target. This argument is distinct from deriving a rate or a normal approximation; those require additional conditions.
Check Your Understanding
Use the theorem’s three ingredients to assess each situation.
- Why does uniform convergence provide more control than pointwise convergence when the argument of the estimating function is \(\hat{\theta}_n\)?
- In the theorem, what role does the positive quantity \(\Delta_\varepsilon\) play in the consistency proof?
- For an exact root, what condition on the sample-equation residual holds automatically?
- In the Bernoulli example, why is projecting \(\bar Y\) onto \(g([-2,2])\) useful when defining an estimator on every sample?
- If the parameter space is noncompact, name one additional condition that can make the compact-space argument applicable.