Population Gram Matrices and Identification

Why the Population Gram Matrix Matters

In random-design regression, each observation brings its own regressor vector. The population distribution of those vectors determines which coefficient directions the data can distinguish. A coefficient may be uniquely defined by the population regression problem only if no nonzero linear combination of regressors vanishes almost surely. The matrix that records these second moments makes that condition precise.

Its eigenvalues also describe how strongly the population criterion responds to coefficient changes. A small eigenvalue means that some direction in coefficient space is weakly represented in the regressors: moving the coefficient in that direction changes fitted values only a little on average. This is the geometric reason that near-collinearity or rare regressor variation can produce imprecise estimates.

The Gram matrix is a second-moment matrix, not necessarily a covariance matrix. In particular, if \(X\) includes an intercept, then its entries include regressor means as well as covariances. The intercept is part of the design vector and must be included when assessing identification of the full coefficient vector.

Identification Through the Population Criterion

Suppose the response satisfies \(Y=X'\beta_0+u\), where \(\beta_0\in\mathbb{R}^p\), \(E[\|X\|^2]<\infty\), \(E[u^2]<\infty\), and \(E[Xu]=0\). The last condition is the population moment condition for random-design least squares. These assumptions ensure that the relevant second moments exist. For a candidate coefficient \(b\), define the population squared-error risk \(R(b)=E[(Y-X'b)^2]\).

Proof. Put \(d=b-\beta_0\). Since \(Y=X'\beta_0+u\), the residual at \(b\) is \(Y-X'b=u-X'd\), while the residual at \(\beta_0\) is \(u\). Expanding their squared difference gives

The cross term is zero because \(E[Xu]=0\). Since \(Q\) is positive semidefinite, the difference is nonnegative, so \(\beta_0\) is a minimizer. If \(Q\succ0\), then \(d'Qd>0\) whenever \(d\ne0\), giving a unique minimizer. If \(Q\) is not positive definite, its positive semidefiniteness implies that some nonzero \(d\) has \(d'Qd=0\). Then \(R(\beta_0+d)=R(\beta_0)\), so the minimizer is not unique.

For the equivalent condition, \(a'Qa=E[(a'X)^2]\). A nonnegative random variable has expectation zero if and only if it equals zero almost surely. Thus \(a'Qa=0\) exactly when \(a'X=0\) almost surely. This proves both directions of the equivalence. \(\square\)

The same matrix appears in the population normal equations. Taking expectations in \(Y=X'\beta_0+u\) after multiplying by \(X\) gives

When \(Q\succ0\), it is invertible, and these moment equations have the unique solution \(\beta_0=Q^{-1}E[XY]\). When \(Q\) is singular, there is a nonzero \(a\) with \(a'X=0\) almost surely. Replacing \(\beta_0\) by \(\beta_0+ta\), for any scalar \(t\), leaves \(X'\beta_0\) unchanged almost surely. The data-generating conditional mean therefore cannot distinguish those coefficient vectors.

The Smallest Eigenvalue and Weak Directions

Because \(Q\) is symmetric, its eigenvalues are real and nonnegative. Write the smallest and largest eigenvalues as \(\lambda_{\min}(Q)\) and \(\lambda_{\max}(Q)\). The Rayleigh quotient characterization gives, for every \(d\),

Thus \(Q\succ0\) exactly when \(\lambda_{\min}(Q)>0\). More than a yes-or-no identification condition, the smallest eigenvalue measures the weakest curvature of the population risk: it is the least increase in risk per squared unit of coefficient displacement, over all directions.

Proof. By the spectral theorem, the operator norm of the inverse of a positive-definite symmetric matrix is the largest absolute eigenvalue of its inverse. The eigenvalues of \(Q^{-1}\) are \(1/\lambda_j(Q)\), so \(\|Q^{-1}\|_{\mathrm{op}}=1/\lambda_{\min}(Q)\). Since \(b_1-b_2=Q^{-1}(m_1-m_2)\), the operator norm inequality yields

The bound is sharp: if \(m_1-m_2\) is a multiple of a unit eigenvector for \(\lambda_{\min}(Q)\), equality holds. \(\square\)

How the Gram Matrix Enters First-Order Precision

The curvature calculation has a direct precision interpretation. Let the score for one observation at the true coefficient be \(Xu\), with covariance matrix \(\Omega=E[XX'u^2]\), assumed finite. If a first-order approximation to an estimator has the form \(Q^{-1}\) times an average score, and that average has limiting covariance \(\Omega\), the corresponding coefficient covariance is

This statement describes covariance propagation through the inverse population moment map; it does not assert a central limit theorem by itself. For example, if the errors are homoskedastic in the sense that \(E[u^2\mid X]=\sigma^2\), then \(\Omega=\sigma^2Q\), and consequently \(V=\sigma^2Q^{-1}\). Along a unit eigenvector \(v\) of \(Q\) with eigenvalue \(\lambda\), the variance in that direction is \(\sigma^2/\lambda\). The weakest eigenvalue therefore gives the largest directional variance in this homoskedastic case.

Under heteroskedasticity, the smallest eigenvalue still bounds sensitivity, but it does not alone determine the full covariance. For instance,

This follows from the operator norm product inequality and \(\|Q^{-1}\|_{\mathrm{op}}=1/\lambda_{\min}(Q)\). The bound also shows why the variability of the score matters: a design with a given \(Q\) can have different precision under different error distributions.

Scaling, Interpretation, and a Common Pitfall

Eigenvalues depend on how regressors are measured. Replacing a regressor by one thousand times its original value changes the numerical entries and eigenvalues of \(Q\), and also changes the units of its coefficient. A small eigenvalue is therefore most informative when the regressor scales are meaningful or have been standardized deliberately. The underlying issue is weak variation in a coefficient direction relative to the chosen parameterization, not a universal numerical threshold.

A common mistake is to say, “The Gram matrix is invertible, so the coefficients are precisely estimated.” Invertibility establishes unique population identification, but a matrix can be invertible and still have a very small eigenvalue. As the nearly collinear example shows, inversion then magnifies perturbations in the weak direction. Conversely, a small eigenvalue does not mean nonidentification unless it is exactly zero. Precision also depends on the score variability \(\Omega\), especially when errors are heteroskedastic.

The population matrix describes the design distribution, not the realized sample matrix. Its properties explain the population identification target and the geometry relevant to first-order precision. Claims about how a sample matrix behaves as sample size grows require separate convergence arguments.