Iterated GMM And Weight Updating

Why Update the Weight More Than Once?

In generalized method of moments, a parameter estimate is chosen to make sample moments small, with their relative importance determined by a weight matrix. Efficient GMM uses the inverse covariance matrix of the moments. In practice that covariance is unknown, so it is estimated using a preliminary parameter estimate. Two-step feasible GMM performs this estimation once and then minimizes the criterion using the resulting fixed weight.

Iterated GMM repeats that operation: minimize using a weight matrix, estimate the moment covariance at the resulting parameter value, update the weight, and minimize again. This can move the estimate and the weight toward a mutually stable pair. Repetition is an algorithm, however, not a guarantee of convergence or of better finite-sample performance.

Definition: Let \(\bar g_n(\theta)=n^{-1}\sum_{i=1}^n g_i(\theta)\) be a vector of sample moments, and let \(\hat S_n(\theta)\) estimate their covariance matrix (or long-run covariance matrix when dependence requires it). Iterated GMM alternates between minimizing \(\bar g_n(\theta)'\hat W_n^{(r)}\bar g_n(\theta)\) over \(\theta\) and updating \(\hat W_n^{(r+1)}=\hat S_n(\theta^{(r+1)})^{-1}\), when that inverse exists.

The Iteration and Its Stopping Rule

A starting estimate \(\theta^{(0)}\) is often obtained using a simple positive-definite weight, such as the identity matrix. Alternatively, one can start from an initial weight \(W^{(0)}\). At iteration \(r\), the weight is held fixed while the parameter criterion is minimized. The covariance is then estimated at the new parameter estimate, and its inverse becomes the weight for the next minimization.

\theta^{(r+1)} =\arg\min_{\theta}\ \bar g_n(\theta)'\hat W_n^{(r)}\bar g_n(\theta), \qquad \hat S_n^{(r+1)}=\hat S_n(\theta^{(r+1)}), \qquad \hat W_n^{(r+1)}=(\hat S_n^{(r+1)})^{-1}.

One can stop when successive estimates and weights change by less than chosen numerical tolerances. For example, a parameter-based criterion is \(\|\theta^{(r+1)}-\theta^{(r)}\|<\varepsilon_\theta\); a weight-based criterion can also be checked. These are computational stopping rules, not proofs that a unique fixed point exists. The minimization at each iteration must be solved adequately, and the estimated covariance must be usable: it should be positive definite, or at least invertible on the relevant moment space. If it is singular or nearly singular, direct inversion can be unstable.

A useful distinction is that the weight is fixed during each minimization and updated between minimizations. This defines iterated GMM. It is not the same as minimizing a criterion in which the weight is recalculated as a function of each candidate \(\theta\) inside that minimization; that is a different updating method.

Worked Example: One Update in a Linear GMM Problem

Suppose a hypothetical overidentified model has two sample moments \(\bar g_n(\theta)=(\theta-1,\theta-3)'\). Start with \(W^{(0)}=I\). The starting estimate minimizes the sum of squared moments:

Q_0(\theta)=(\theta-1)^2+(\theta-3)^2, \qquad \frac{dQ_0}{d\theta}=2(\theta-1)+2(\theta-3)=0.

Solving gives \(\theta^{(0)}=2\). For this numerical illustration, suppose the plug-in estimate of the moment covariance at any candidate \(\theta\) is \(\hat S_n(\theta)=\operatorname{diag}(1,1+\theta^2/4)\). At \(\theta^{(0)}=2\), this gives \(\hat S_n^{(0)}=\operatorname{diag}(1,2)\), so \(W^{(0)}_{\text{updated}}=\operatorname{diag}(1,1/2)\).

Hold this updated weight fixed and minimize:

Q_1(\theta)=(\theta-1)^2+\frac12(\theta-3)^2, \qquad \frac{dQ_1}{d\theta}=2(\theta-1)+(\theta-3)=0.

Thus \(3\theta-5=0\), and the next estimate is \(\theta^{(1)}=5/3\approx1.667\). This one update illustrates the mechanics: the two moments were initially equally weighted, while the updated covariance assigns less weight to the second moment because its estimated variance is larger.

Why Convergence Is a Separate Question

In a linear moment model \(\bar g_n(\theta)=\hat a-\hat G\theta\), with a weight \(W\) fixed during minimization, the first-order condition is \(\hat G'W(\hat a-\hat G\theta)=0\). If \(\hat G'W\hat G\) is invertible, the minimizer is \((\hat G'W\hat G)^{-1}\hat G'W\hat a\). Iteration therefore creates a map: the current estimate determines a covariance and weight, and that weight determines the next estimate. Convergence requires this map to settle down; invertibility of each weight alone does not establish convergence.

The following example makes that map explicit. It also shows why successive estimates can move in alternating directions even when they eventually settle near a stable value.

Worked Example: Following a Fixed-Point Iteration

Continue with the hypothetical moments and covariance formula above. With a diagonal weight \(W=\operatorname{diag}(1,w)\), minimizing \((\theta-1)^2+w(\theta-3)^2\) gives \(\theta=(1+3w)/(1+w)\), by setting its derivative \(2(\theta-1)+2w(\theta-3)\) to zero. If the current estimate is \(t\), the updated second weight is \(w=1/(1+t^2/4)\). Therefore the next estimate is

f(t)=\frac{1+3/(1+t^2/4)}{1+1/(1+t^2/4)} =\frac{16+t^2}{8+t^2}.

Starting from \(t=2\), the first update is \(f(2)=20/12=1.6667\). At \(t=5/3\), the second covariance diagonal element is \(1+(5/3)^2/4=61/36\approx1.6944\), and its inverse is \(36/61\approx0.5902\). The next estimate is \((1+3(36/61))/(1+36/61)=169/97\approx1.7423\). Further updates move back toward approximately \(1.728\); the sequence alternates around its limiting value.

A fixed point satisfies \(t=f(t)\), or \(t^3-t^2+8t-16=0\), whose relevant root is approximately \(1.728\). To check local stability, differentiate the update map: \(f'(t)=-16t/(8+t^2)^2\). At \(t\approx1.728\), \(|f'(t)|\approx0.229<1\). Thus a small deviation from this fixed point is reduced locally by the iteration, with the negative derivative accounting for the alternating movement. This local calculation supports convergence near the fixed point; it does not rule out other behavior from every possible starting value.

For nonlinear GMM, the same logic applies but the minimization may have multiple local minima, and the update map may be more complicated. Numerical convergence should therefore be assessed rather than assumed. In particular, examine the path of estimates and weights, verify that the moment covariance remains well-conditioned, and consider whether different reasonable starting values lead to materially different results.

Large-Sample Inference After Iteration

Under correct population moment restrictions, identification, regularity conditions for GMM asymptotics, and consistent covariance estimation, iterated GMM that converges to the efficient weight targets the same efficient population solution as two-step feasible GMM. The key reason is that a consistent preliminary estimate makes the estimated covariance converge to the population moment covariance at the true parameter. Further consistent updates do not change that probability limit. Under standard conditions, both procedures have the same first-order asymptotic distribution when they use the same consistent covariance estimator.

This equivalence is an asymptotic statement. In a finite sample, the two-step estimate uses a covariance evaluated at its preliminary estimate; iterated GMM uses later covariance estimates. Their numerical estimates and reported standard errors can consequently differ. Extra updates do not automatically remove finite-sample bias, ensure better coverage, or improve numerical stability.

Theorem: For a correctly specified, identified GMM model with a consistent estimator of the moment covariance and the usual differentiability and central-limit conditions, a converged iterated estimator using \(\hat W\to_p S^{-1}\) has asymptotic covariance \((G'S^{-1}G)^{-1}\) for \(\sqrt n(\hat\theta-\theta_0)\). Here \(G\) is the population derivative of the moment vector at \(\theta_0\), and \(S\) is its covariance. With a general fixed limiting weight \(W\), the corresponding sandwich covariance is \((G'WG)^{-1}G'WSWG(G'WG)^{-1}\).

In practice, inference uses sample analogues. For a general weight, estimate the sandwich covariance using the estimated Jacobian \(\hat G\), estimated covariance \(\hat S_n\), and the weight used for the final GMM estimate. If the final weight consistently estimates \(\hat S_n^{-1}\), the efficient-form estimate is \((\hat G'\hat W\hat G)^{-1}\). This is the estimated covariance of \(\sqrt n(\hat\theta-\theta_0)\); divide it by \(n\) to obtain an estimated covariance for \(\hat\theta\) itself.

Worked Example: Standard Error From the Final Weight

Use the converged estimate \(\hat\theta\approx1.728\) in the same hypothetical example, where \(\hat G=(1,1)'\) and \(\hat S_n=\operatorname{diag}(1,1+\hat\theta^2/4)\). Substitution gives \(1+1.728^2/4=1.746496\), so the covariance estimate is approximately \(\operatorname{diag}(1,1.746)\). Its inverse is approximately \(\hat W=\operatorname{diag}(1,0.5726)\).

For the efficient-weight formula, the estimated variance of \(\sqrt n(\hat\theta-\theta_0)\) is

(\hat G'\hat W\hat G)^{-1} =(1+0.5726)^{-1} \approx0.6359.

If the hypothetical sample size is \(n=400\), the estimated variance of \(\hat\theta\) is \(0.6359/400\approx0.001590\). Its standard error is therefore \(\sqrt{0.001590}\approx0.03987\) in the units of \(\theta\). This calculation is conditional on the stated covariance and Jacobian estimates and the regular asymptotic approximation; it does not mean the estimate is exact in this sample.

Common Pitfalls and Practical Interpretation

Calling each update a new minimization with a changing weight. In iterated GMM, the weight is fixed inside each minimization and updated afterward. Recomputing it for every candidate value within one minimization defines a different procedure.
Assuming that more iterations must improve the estimate. Iteration may converge slowly, fail to converge, or reach different solutions from different starting values. Check the numerical path and the conditioning of the covariance estimate.
Using the efficient variance formula with an arbitrary weight. If the final weight is not an estimate of \(S^{-1}\), the general sandwich expression is needed; \((\hat G'\hat W\hat G)^{-1}\) alone is not generally valid.
Interpreting asymptotic equivalence as finite-sample identity. Two-step and iterated GMM can differ in finite samples even when they have the same first-order asymptotic distribution under correct specification and regularity conditions.

The practical value of iterated GMM is that it can make the parameter and estimated weight mutually stable rather than basing the final weight only on a preliminary estimate. Its justification rests on the model, the quality of the covariance estimate, and actual convergence—not simply on the number of times the algorithm is run.

Check Your Understanding

Answer each question using the definitions and calculations in this tutorial.

In iterated GMM, when is the weight matrix held fixed, and when is it updated?
Why does the existence of an inverse covariance estimate at every iteration not by itself prove that the algorithm converges?
For a converged estimator with a general limiting weight \(W\), which covariance formula applies, and what simplifies when \(W=S^{-1}\)?
Why can two-step feasible GMM and iterated GMM have different finite-sample estimates but the same first-order asymptotic distribution?
In the fixed-point example, what does \(|f'(t)|<1\) near the fixed point tell us, and what does it not establish?