Why Many Moments Can Become a Problem
Adding moment conditions can improve identification or use information that a smaller set leaves out. But more conditions also mean more covariance quantities must be estimated and, for efficient GMM, a larger matrix must be inverted. When the number of moments is large relative to the sample size, or when the moments are nearly redundant, that inversion can be unstable. Small estimation errors in the covariance matrix can then cause large changes in the weights, the estimate, and its reported uncertainty.
The challenge is not simply that there are “too many” moments according to one universal numerical cutoff. The relevant issues include the number of moments relative to the sample size, the accuracy and conditioning of the estimated covariance matrix, how strongly the moments identify the parameters, and how the moments were chosen. A method that helps in one setting can harm another if it discards identifying information or treats noisy weights as known.
The familiar fixed-number-of-moments theory provides useful reference points: the population moments should identify \(\theta\), and the covariance matrix of the moments should be well behaved. In a growing-dimension problem, those properties must hold uniformly as \(q_n\) changes. For example, identification requires that the derivative matrix \(D_n=E[\partial g_i(\theta_0)/\partial\theta']\), together with the weighting matrix, provide information in every parameter direction. Having many moments does not guarantee this: additional moments may be redundant, weakly informative, or very noisy.
Where Instability Enters
Let \(\Omega_n\) denote the covariance matrix of the moment vector at the true parameter, under conditions where such a covariance is appropriate for the sampling design. Efficient GMM uses a weight related to \(\Omega_n^{-1}\). In feasible GMM, \(\Omega_n\) is estimated, often by a sample covariance matrix or a long-run covariance estimator. As \(q_n\) grows, the estimate has many entries, and those entries can be noisy. If the estimated matrix is singular, it has no ordinary inverse; if it is nearly singular, inversion can magnify its estimation error.
The issue is particularly clear when some directions in the moment vector have very small variance. Such directions correspond to small eigenvalues of \(\Omega_n\). Inverting the matrix gives these directions large weights. That may be appropriate if the direction is genuinely measured precisely, but it is risky when its small estimated variance is mostly sampling noise. Near-duplicate moments can create the same problem: a contrast between them may have little variance and receive an excessive estimated weight.
Worked Example: A Nearly Singular Weight Matrix
Suppose a hypothetical analysis estimates the covariance matrix of two moment conditions as \(S=\begin{pmatrix}1&0.99\\0.99&1\end{pmatrix}\). The two moments are strongly correlated, so examine the eigenvalues before using \(S^{-1}\).
The eigenvectors proportional to \((1,1)'\) and \((1,-1)'\) have eigenvalues \(1.99\) and \(0.01\), respectively. Thus the covariance is much smaller in the difference direction. The condition number, the ratio of the largest to smallest eigenvalue for this positive definite matrix, is \(1.99/0.01=199\).
Direct inversion gives \(S^{-1}=\frac{1}{1-0.99^2}\begin{pmatrix}1&-0.99\\-0.99&1\end{pmatrix}\), where \(1-0.99^2=0.0199\). The diagonal entries are about \(50.25\), and the off-diagonal entries are about \(-49.75\). In the difference direction, the inverse eigenvalue is \(1/0.01=100\). A small estimated discrepancy between the two moments can therefore have a very large effect on the GMM criterion.
For illustration, ridge regularization with \(\lambda=0.09\) replaces \(S\) by \(S+\lambda I\), whose eigenvalues are \(2.08\) and \(0.10\). Its condition number is \(2.08/0.10=20.8\), and the inverse eigenvalue in the difference direction is now \(10\), rather than \(100\). The regularized matrix is better conditioned, but the calculation alone does not establish that \(\lambda=0.09\) is suitable for any particular application.
A Short Justification: Why Inversion Magnifies Error
Write the estimated covariance as \(\hat\Omega_n=\Omega_n+E_n\), where \(E_n\) is its estimation error. If \(\Omega_n\) is invertible and \(\|\Omega_n^{-1}E_n\|<1\) in a compatible matrix norm, factor the estimate as \(\hat\Omega_n=\Omega_n(I+\Omega_n^{-1}E_n)\). The inverse is then \(\hat\Omega_n^{-1}=(I+\Omega_n^{-1}E_n)^{-1}\Omega_n^{-1}\).
For a matrix \(A\) with \(\|A\|<1\), the geometric expansion \((I+A)^{-1}=I-A+A^2-\cdots\) converges. Taking \(A=\Omega_n^{-1}E_n\) shows that the inverse depends on the error through \(\Omega_n^{-1}E_n\), not just through \(E_n\). In particular, the first-order change is approximately \(-\Omega_n^{-1}E_n\Omega_n^{-1}\). If \(\Omega_n\) has a small eigenvalue, its inverse has a large eigenvalue, and even modest covariance-estimation error can produce a large change in the estimated weight. This is why a small raw error in a covariance matrix is not, by itself, evidence that its inverse is reliable.
Regularization and Dimension Control
Regularization modifies the weighting problem to limit the influence of unstable directions. Ridge weighting is a common example: use \(W_{n,\lambda}=(\hat\Omega_n+\lambda I)^{-1}\), where \(\lambda>0\) is a tuning parameter and \(I\) is the identity matrix of the same dimension. If \(\hat\Omega_n\) is positive semidefinite, adding \(\lambda I\) makes it positive definite. An eigenvalue \(s_j\) becomes \(s_j+\lambda\), so its inverse weight falls from \(1/s_j\) to \(1/(s_j+\lambda)\).
This stabilization has a cost. It no longer uses the unregularized efficient weight, and it can change the finite-sample GMM estimate. If the moments are correctly specified and the chosen weighting matrix remains positive definite, a consistent estimator may still be possible under suitable identification and asymptotic conditions. But the usual fixed-dimension formula cannot simply be presumed to apply when \(q_n\) grows or when \(\lambda\) changes with \(n\). The behavior of the regularization and the covariance estimate must be included in the justification for inference.
Other approaches control dimension more directly. A researcher may pre-specify a smaller set of substantively justified moments, truncate a growing basis to a manageable number of terms, or reduce the moment vector to stable directions using a justified dimension-reduction rule. The key check is whether the retained moments still identify the parameters. Dropping a direction solely because it is numerically inconvenient can remove the only information identifying a parameter. Conversely, retaining many nearly redundant moments does not create independent identifying information.
Worked Example: A Sample Covariance That Cannot Be Inverted
Suppose a hypothetical study has \(n=4\) observations and \(q_n=4\) sample moment components. The usual centered sample covariance is formed from four centered observation vectors. Because their centered sum is zero, those four vectors are linearly dependent. Consequently the covariance matrix has rank at most \(n-1=3\), so a \(4\times4\) covariance matrix constructed this way must be singular and cannot have an ordinary inverse.
Suppose, for illustration, its eigenvalues are \(3\), \(1\), \(0.5\), and \(0\). Adding \(\lambda=0.1\) to the diagonal gives eigenvalues \(3.1\), \(1.1\), \(0.6\), and \(0.1\). The regularized matrix is invertible; its inverse eigenvalues are approximately \(0.3226\), \(0.9091\), \(1.667\), and \(10\). Regularization has made inversion possible, but the zero-eigenvalue direction still receives a relatively large weight of \(10\). A larger penalty or a defensible reduction in dimension might be considered, with the resulting estimator and inference assessed under the chosen method.
The rank calculation is not a general rule that every GMM covariance estimate is singular whenever \(q_n\ge n\): covariance estimators can be structured or regularized in other ways. It does show why an unstructured centered sample covariance cannot be inverted in this particular \(n=4,\ q_n=4\) example.
Worked Weighting With a Penalty
The effect of regularization on the point estimate can be seen directly in a scalar-parameter example. Suppose two hypothetical sample moments are \(g_1(\theta)=\theta-1.8\) and \(g_2(\theta)=\theta-2.4\), and the estimated covariance of the moment contributions is \(S=\operatorname{diag}(1,9)\). This example treats those quantities as already computed summaries; it does not assert that the two moment conditions are correct for any real study.
Worked Example: How Ridge Weighting Changes the Estimate
With \(\lambda=1\), the regularized weight matrix is \(W_\lambda=(S+I)^{-1}=\operatorname{diag}(1/2,1/10)\). The GMM criterion is therefore \(Q_\lambda(\theta)=\frac{1}{2}(\theta-1.8)^2+\frac{1}{10}(\theta-2.4)^2\).
Differentiate and set the derivative to zero: \(\frac{dQ_\lambda(\theta)}{d\theta}=(\theta-1.8)/1+(\theta-2.4)/5=0\). Equivalently, \((0.5+0.1)\hat\theta_\lambda=0.5(1.8)+0.1(2.4)=1.14\). Thus \(\hat\theta_\lambda=1.14/0.6=1.900\).
For comparison, using the unregularized weight \(S^{-1}=\operatorname{diag}(1,1/9)\) gives \(\hat\theta=(1.8+(2.4/9))/(1+1/9)=1.86\), to three significant figures. The penalty increases the relative weight on the second moment, because its inverse weight changes from \(1/9\) to \(1/10\), while the first changes from \(1\) to \(1/2\). The estimate moves from \(1.86\) to \(1.90\). Neither calculation establishes which estimate is preferable; that depends on the data-generating conditions, the covariance estimation, and the regularization choice.
Inference When the Number of Moments Grows
For fixed \(q\), the familiar GMM sandwich variance has the general form \((D'WD)^{-1}D'W\Omega WD(D'WD)^{-1}/n\), when the relevant matrices are well defined and the usual regularity conditions hold. It describes uncertainty in a parameter estimate using the sensitivity of the moments, their covariance, and the weighting matrix. With many moments, each part can be estimated imprecisely, and matrix dimensions change with \(n\). A fixed-\(q\) approximation is not automatically justified just because the same formula can be written down for each sample.
In particular, confidence intervals and Wald tests require a valid approximation to the estimator’s sampling distribution and a reliable variance estimate. If the estimated weights are unstable, uncertainty calculations based on them may also be unstable. Regularization can alter the estimator’s variance and bias properties, so inference must account for the chosen penalty or dimension-reduction procedure. The nominal coverage or test size should not be treated as guaranteed without conditions appropriate to the high-dimensional setting.
The same caution applies to the Hansen \(J\) test. Its usual chi-square reference distribution with degrees of freedom based on the number of overidentifying restrictions is a fixed-dimension result under its assumptions. When the number of restrictions grows, that reference distribution need not remain accurate. A large or small computed \(J\) statistic should therefore not be interpreted using the fixed-dimension calibration unless a result for the growing-moment setting supports it.
Practical Choices and Common Mistakes
A useful analysis separates three decisions: which moments are scientifically defensible, how many of them the available data can support, and how their covariance should be estimated and used. A dimension budget can be planned before fitting the model by comparing \(q_n\) with \(n\), checking for near-collinearity, and assessing the stability of the covariance estimate. The ratio \(q_n/n\) is a useful warning signal, not a universal pass-or-fail threshold. The appropriate growth rate depends on the model, dependence structure, estimator, and inferential goal.
The practical goal is not to maximize the raw count of moment conditions. It is to retain enough reliable information to identify the target, while keeping covariance estimation and inference under control. Report the number of moments, the sample size, the weighting or regularization method, and any dimension-reduction decisions. Sensitivity checks across defensible choices can reveal whether the substantive conclusion depends strongly on those decisions, although such checks do not replace a valid inferential argument.
Check Your Understanding
Answer each question using the definitions and reasoning developed above.
- Why can two nearly redundant moments produce a poorly conditioned estimated covariance matrix?
- In ridge weighting, what happens to the inverse weight associated with an eigenvalue \(s_j\) when \(s_j\) is replaced by \(s_j+\lambda\)?
- Why must a researcher check identification after reducing the number of moment conditions?
- For a centered sample covariance formed from \(n\) observation vectors, why is its rank at most \(n-1\)?
- Why is the fixed-dimension chi-square reference distribution for Hansen’s \(J\) test not automatically valid when the number of moment conditions grows?