Why a Small Eigenvalue Matters
A regression design can contain a direction that is only weakly represented in the population. OLS may still be consistent and asymptotically normal, but estimates along that direction are less precise, and their normal approximation can take a long time to become accurate. The relevant quantity is not just the sample size: it is how much information the design supplies in its weakest direction.
This tutorial studies a sequence of random-design regressions in which the population design may become increasingly close to singular. The key distinction is between eventual asymptotic behavior and finite-sample behavior. A shrinking eigenvalue need not destroy consistency, but it can inflate variance and slow the accumulation of information needed for a useful normal approximation.
The eigenvalue describes population information per observation in a direction, not the total sample information. Roughly, \(n\lambda_n\) is the information accumulated in the weakest direction. If it grows, consistency can remain possible even as \(\lambda_n\) shrinks. If it grows slowly, however, the estimate in that direction can remain noisy and its finite-sample distribution can depart substantially from its limiting normal law.
Variance Amplification Along a Weak Direction
Consider the linear model \(Y_{ni}=X_{ni}'\beta+\varepsilon_{ni}\), with errors of conditional mean zero and conditional variance \(\sigma^2\). Let \(\widehat Q_n=n^{-1}\sum_{i=1}^n X_{ni}X_{ni}'\). On the event that the sample design has full column rank, OLS satisfies \(\hat\beta-\beta=\widehat Q_n^{-1}n^{-1}\sum_i X_{ni}\varepsilon_{ni}\). If the errors are conditionally independent with variance \(\sigma^2\), its conditional covariance is \(\sigma^2(n\widehat Q_n)^{-1}\). Thus a small eigenvalue of \(\widehat Q_n\) produces a large variance in its corresponding eigenvector direction.
To relate this to the population matrix, suppose the design concentrates relatively: for a deterministic sequence \(\eta_n\to0\), \(\|Q_n^{-1/2}(\widehat Q_n-Q_n)Q_n^{-1/2}\|_{\mathrm{op}}\leq\eta_n\) with probability tending to one. This is a stronger requirement than entrywise convergence when \(Q_n\) itself is becoming ill-conditioned. On this event, the sample Gram matrix is close to the population matrix after rescaling each direction by its population information.
Proof. Put \(A_n=Q_n^{-1/2}(\widehat Q_n-Q_n)Q_n^{-1/2}\). Then \(\widehat Q_n=Q_n^{1/2}(I+A_n)Q_n^{1/2}\). Since \(A_n\) is symmetric and \(\|A_n\|_{\mathrm{op}}\leq\eta_n<1\), all eigenvalues of \(I+A_n\) lie between \(1-\eta_n\) and \(1+\eta_n\). Consequently, the eigenvalues of \((I+A_n)^{-1}\) lie between \(1/(1+\eta_n)\) and \(1/(1-\eta_n)\). Also, \((n\widehat Q_n)^{-1}=n^{-1}Q_n^{-1/2}(I+A_n)^{-1}Q_n^{-1/2}\). Applying the eigenvalue bounds to \(Q_n^{-1/2}a\), and multiplying by \(\sigma^2\), gives the claimed inequalities. If \(Q_nv_n=\lambda_nv_n\) and \(\|v_n\|=1\), then \(v_n'Q_n^{-1}v_n=1/\lambda_n\), yielding the variance interpretation. \(\square\)
This result describes variance, not a normal approximation by itself. Approximate covariance does not ensure that a suitably scaled score is approximately Gaussian, and relative concentration may itself be difficult when the weakest direction receives little data. Those are separate requirements.
Worked Example: A Rare Binary Regressor
Let \(X_i=(1,W_i)'\), where \(W_i\) is Bernoulli with success probability \(p\), and suppose the model has an intercept and a slope. The population Gram matrix is \(Q=\begin{pmatrix}1&p\\p&p\end{pmatrix}\). Its determinant is \(p(1-p)\), and its trace is \(1+p\). Its smaller eigenvalue is \(\{1+p-\sqrt{(1-p)^2+4p^2}\}/2\), which is asymptotic to \(p\) as \(p\) tends to zero. The direction that distinguishes the two groups therefore has information of order \(p\) per observation.
Let \(K=\sum_i W_i\) be the number of observations in the rare group. On the event \(0<K<n\), the centered sum of squares for the regressor is \(D=\sum_i(W_i-\bar W)^2=K(n-K)/n>0\). The fitted slope is the rare-group mean minus the common-group mean. If errors are independent with variance \(\sigma^2\), then, conditional on the regressors, $$ \operatorname{Var}(\hat\beta_1-\beta_1\mid W_1,\ldots,W_n) =\sigma^2\left(\frac{1}{K}+\frac{1}{n-K}\right). $$ When \(p\) is small and \(K\) is near \(np\), the leading term is \(\sigma^2/(np)\). The nominal sample size \(n\) can therefore greatly overstate the information for the slope: its effective sample size is about \(np\).
Consistency and a Normal Limit Can Survive
A useful way to see the distinction between a shrinking eigenvalue and a breakdown of asymptotics is to let the rare-group probability change with \(n\). Suppose \(p_n\to0\) while \(np_n\to\infty\). Then the rare group becomes a smaller fraction of the sample, but its count still diverges. The slope can be consistent, and after scaling by \(\sqrt{np_n}\) it has a normal limit.
Proof. First, \(K_n\) has mean \(np_n\) and variance \(np_n(1-p_n)\). Chebyshev's inequality gives $$ P\left(\left|\frac{K_n}{np_n}-1\right|>\epsilon\right) \leq \frac{1-p_n}{\epsilon^2np_n}\longrightarrow0 $$ for every \(\epsilon>0\). Hence \(K_n/(np_n)\xrightarrow{p}1\), so \(K_n\to\infty\) in probability and \(K_n/n\xrightarrow{p}0\). Also \(P(K_n=0)=(1-p_n)^n\leq e^{-np_n}\to0\), and \(P(K_n=n)=p_n^n\to0\).
On \(0<K_n<n\), the group-mean representation gives $$ \sqrt{np_n}(\hat\beta_1-\beta_1) =\frac{\sqrt{np_n}}{K_n}\sum_{i:W_{ni}=1}\varepsilon_{ni} -\frac{\sqrt{np_n}}{n-K_n}\sum_{i:W_{ni}=0}\varepsilon_{ni}. $$ For the first term, conditional on \(K_n=k\), the treated errors are \(k\) independent copies of \(\varepsilon_{ni}\). The ordinary iid central limit theorem applies as \(k\to\infty\). Since \(K_n/(np_n)\xrightarrow{p}1\), the conditional central limit theorem and Slutsky's theorem give \(\sqrt{np_n}K_n^{-1}\sum_{W_{ni}=1}\varepsilon_{ni}\xrightarrow{d}N(0,\sigma^2)\).
The second term has conditional mean zero and conditional variance \(np_n\sigma^2/(n-K_n)\), which converges in probability to zero because \(K_n/n\xrightarrow{p}0\) and \(p_n\to0\). Conditional Chebyshev bounds, together with \(P(K_n=n)\to0\), imply that this term converges in probability to zero. The probability of a rank-deficient design also tends to zero, so assigning any fixed finite value to the estimator there does not change the limiting distribution. This proves the stated normal limit. Since \(\sqrt{np_n}\to\infty\), the limit also implies \(\hat\beta_1-\beta_1\xrightarrow{p}0\). \(\square\)
The diverging count \(np_n\), rather than \(n\) alone, is what supports this result. The proof also illustrates why negligible rank-deficient events must be addressed explicitly: the slope is defined by the OLS formula only when both groups occur.
Worked Example: A Normal Limit That Is Still Rough at \(n=100\)
Consider the preceding triangular-array setting at \(n=100\), \(p_n=0.1\), with errors taking the values \(-1\) and \(1\) with equal probabilities. Here \(np_n=10\), and the theorem describes the eventual behavior along a sequence such as \(p_n=n^{-1/2}\); this single sample size is only a finite-sample illustration.
The rare-group count has \(E[K_n]=10\) and \(\operatorname{Var}(K_n)=9\). The probability of having no rare-group observations is \(0.9^{100}\), about \(0.0000266\), but a typical rare group still contains only about ten observations. Conditional on \(K_n=10\), the leading, rare-group part of the scaled slope error is \(\sqrt{10}\,K_n^{-1}\sum_{W_{ni}=1}\varepsilon_{ni}=S_{10}/\sqrt{10}\), where \(S_{10}\) is a sum of ten independent Rademacher variables.
This quantity has only eleven possible values, spaced \(2/\sqrt{10}\) apart. In particular, \(P(S_{10}=0)=\binom{10}{5}/2^{10}=252/1024\), about \(0.246\). A continuous normal distribution has no point mass at zero. Thus, at this group count, the score contribution that drives the asymptotic limit is visibly discrete. The control-group contribution is also present in the exact slope, but its variance after this scaling is \(10\sigma^2/90\), compared with \(\sigma^2\) for the treated-group contribution. It is smaller, not absent. The example shows why a theorem asserting convergence does not guarantee a close normal approximation when effective information is only moderate.
Near-Collinearity and Which Coefficient Is Uncertain
A small eigenvalue need not arise from a rare category. It can also arise when two regressors are almost redundant. In that case, the weakly informed direction is a contrast between coefficients: changes in that contrast can leave fitted values nearly unchanged. The individual coefficients may then be much less precise than the fitted linear combinations that the data identify well.
Worked Example: Two Nearly Redundant Regressors
Let \(Z_i\) and \(V_i\) be independent, mean-zero, variance-one variables, and take the two regressors to be \(X_i=(Z_i,Z_i+\delta V_i)'\), with \(\delta>0\). Then $$ Q=E[X_iX_i'] =\begin{pmatrix}1&1\\1&1+\delta^2\end{pmatrix}. $$ Its determinant is \(\delta^2\) and its trace is \(2+\delta^2\). The smaller eigenvalue is \(\{2+\delta^2-\sqrt{4+\delta^4}\}/2\), asymptotic to \(\delta^2/2\) as \(\delta\to0\). The population design therefore becomes near-singular at a quadratic rate in \(\delta\).
For independent homoskedastic errors, the population covariance approximation for OLS is \(\sigma^2Q^{-1}/n\), where $$ Q^{-1}=\frac{1}{\delta^2} \begin{pmatrix}1+\delta^2&-1\\-1&1\end{pmatrix}. $$ Thus the variance of the second coefficient is approximately \(\sigma^2/(n\delta^2)\), and the variance of the coefficient difference \(\hat\beta_1-\hat\beta_2\) is approximately \(\sigma^2(4+\delta^2)/(n\delta^2)\). Both can be large when \(\delta\) is small. By contrast, the sum of the two coefficient errors has approximate variance \(\sigma^2/n\). The weak direction is principally a coefficient difference, while the sum is much better informed. These are population covariance approximations; their use in a particular sample also requires adequate concentration of the sample Gram matrix.
Common Pitfalls and Practical Interpretation
- Using \(n\) as the only measure of information. In a rare-group design, the slope is driven by the count in that group. If \(np_n\) is small, a large total sample may still provide little information about the slope.
- Equating consistency with a good normal approximation. The condition \(np_n\to\infty\) gives eventual consistency and normality in the example, but it says nothing about how large \(np_n\) must be for a particular finite-sample approximation to be accurate.
- Assuming a small eigenvalue always means inconsistency. The weak-direction variance can grow in per-observation units while total information \(n\lambda_n\) still diverges. Conversely, if that total information stays bounded, the consistency argument above no longer applies.
- Writing the slope identity on a rank-deficient sample. For binary \(W_i\), the formula with denominator \(D=\sum_i(W_i-\bar W)^2\) is valid only when \(D>0\), equivalently \(0<K_n<n\). On the complementary event the slope is not estimable. One must condition on full rank or define an arbitrary extension and verify that the event's probability vanishes before making an asymptotic claim.
- Reading an approximate covariance as a normality result. A covariance calculation concerns scale. A normal limit additionally requires a suitable central limit theorem for the score and control of the sample design.
Check Your Understanding
Use the distinction between population information, sample rank, and finite-sample approximation to answer these questions.
- In a rare-regressor design with success probability \(p_n\), why is \(np_n\) more informative for the slope than the total sample size \(n\)?
- For a sequence with \(p_n\to0\), what condition in the proved result ensures the rare-group count diverges and the slope is consistent?
- Why is the OLS slope formula with denominator \(D\) invalid when \(D=0\), and what must be checked before assigning an arbitrary value on those samples?
- In the near-collinearity example, which coefficient combination is poorly informed, and how does the small eigenvalue explain its large variance?
- Why does a proved asymptotic normal limit not by itself establish a reliable normal approximation at a particular sample size?