Central Limit Theorems for Regression Scores

Why Regression Score Sums Have Their Own Limit

In random-design regression, the basic random contribution to the least-squares estimating equation is the product of a regressor vector and its error. The asymptotic linear representation of OLS expresses coefficient fluctuations in terms of a normalized sum of these contributions. Before using that representation to study coefficients, one must know when the score sum itself has a normal limit.

This tutorial focuses on that vector central limit theorem and its assumptions. It does not require homoskedasticity, nor does it require the errors to be independent of the regressors. It does require the score vectors to be centered and to have finite covariance under the iid framework considered here.

Definition: Let \(x_i\in\mathbb{R}^p\) be a column vector of regressors and \(u_i\in\mathbb{R}\) the regression error for observation \(i\). The regression score vector (also called the OLS moment vector) is \(Z_i=x_i u_i\). Its population mean is \(E[Z_i]=E[x_i u_i]\). If this mean is zero and its second moments are finite, its covariance matrix is \(\Omega=E[Z_iZ_i']=E[u_i^2x_ix_i']\).

The name “score” here refers to the contribution to the OLS estimating equation; it need not be a likelihood score. For example, the sample normal equations contain the sum \(\sum_{i=1}^n x_i u_i\) when evaluated at the true regression coefficient. The task here is to establish the distributional behavior of that sum, not to derive a limit for the OLS estimator.

The Multivariate Central Limit Theorem for Scores

The random-design framework assumes that the observation pairs \((x_i,u_i)\) are iid. Consequently, the score vectors \(Z_i=x_i u_i\) are iid as well. A finite covariance matrix is the relevant moment condition: in terms of regressors and errors it is \(E[u_i^2\|x_i\|^2]<\infty\), where \(\|\cdot\|\) is the Euclidean norm.

Theorem: Suppose \(Z_1,Z_2,\ldots\) are iid vectors in \(\mathbb{R}^p\), \(E[Z_i]=0\), and \(E[\|Z_i\|^2]<\infty\). Let \(\Omega=E[Z_iZ_i']\). Then

\frac{1}{\sqrt{n}}\sum_{i=1}^n Z_i \xrightarrow{d} N_p(0,\Omega).

The matrix \(\Omega\) may be singular; in that case the limiting multivariate normal distribution is degenerate in some directions.

The theorem applies to regression scores by setting \(Z_i=x_i u_i\). Thus, the direct conditions are iid observations, \(E[x_i u_i]=0\), and \(E[u_i^2\|x_i\|^2]<\infty\). When these hold, \(\Omega=E[u_i^2x_ix_i']\). No assumption that \(u_i\) is independent of \(x_i\) is needed for this conclusion.

Check centering.
Verify that \(E[x_i u_i]=0\). This is a restriction on the population score, not merely a statement that the errors have mean zero.

Check the second moment.
Verify \(E[u_i^2\|x_i\|^2]<\infty\), which ensures that every score component has finite variance and every pair has finite covariance.

Compute the covariance.
Under centering, use \(\Omega=E[u_i^2x_ix_i']\); do not replace it by a homoskedastic formula unless homoskedasticity is part of the model.

Why the Score Limit Holds

The multivariate result follows from the scalar iid central limit theorem applied to every linear projection of the score vector. The projection argument is useful because it shows exactly which scalar variances must be finite.

Proof: Fix any nonzero vector \(a\in\mathbb{R}^p\). The scalar random variables \(a'Z_i\) are iid, have mean \(a'E[Z_i]=0\), and have variance

\operatorname{Var}(a'Z_i)=E[(a'Z_i)^2]=a'\Omega a<\infty.

The scalar iid central limit theorem therefore gives

\frac{1}{\sqrt{n}}\sum_{i=1}^n a'Z_i \xrightarrow{d} N(0,a'\Omega a).

The expression on the left is \(a'\bigl(n^{-1/2}\sum_{i=1}^n Z_i\bigr)\). The Cramér–Wold theorem states that convergence of these projections for every \(a\in\mathbb{R}^p\) to the corresponding projections of a random vector implies convergence in distribution of the vectors. The projections on the right are exactly those of \(N_p(0,\Omega)\), including when \(a'\Omega a=0\). Hence the claimed multivariate convergence follows. \(\square\)

A common sufficient route to centering is conditional mean-zero error. Suppose \(E[|u_i|]<\infty\), \(E[\|x_i u_i\|]<\infty\), and \(E[u_i\mid x_i]=0\) almost surely. The conditional-expectation identity then gives

E[x_i u_i]=E\!\left[E[x_i u_i\mid x_i]\right] =E\!\left[x_iE[u_i\mid x_i]\right]=0.

The integrability conditions matter: conditional expectations and the displayed iterated-expectation step must be well-defined. Conditional mean zero is sufficient, but stronger than necessary. The central limit theorem only requires the unconditional score mean \(E[x_i u_i]\) to vanish; some dependence between \(x_i\) and \(u_i\) can remain.

Worked Score Calculations

Worked Example: Homoskedastic Scores With a Binary Regressor

Let \(x_i=(1,W_i)'\), where \(P(W_i=1)=P(W_i=-1)=1/2\). Suppose \(u_i\sim N(0,4)\) independently of \(W_i\), and the pairs are iid across observations. Find the score mean and covariance.

Independence and \(E[u_i]=0\) give \(E[x_i u_i]=E[x_i]E[u_i]=0\). Since \(E[u_i^2]=4\) and \(W_i^2=1\),

\Omega =E[u_i^2x_ix_i'] =4E\!\left[ \begin{pmatrix} 1&W_i\\ W_i&W_i^2 \end{pmatrix} \right] =4\begin{pmatrix} 1&0\\ 0&1 \end{pmatrix} =\begin{pmatrix}4&0\\0&4\end{pmatrix}.

The off-diagonal entries are zero because \(E[W_i]=0\); the diagonal entries are four because the intercept regressor and \(W_i\) both have squared value one. Also, \(E[u_i^2\|x_i\|^2]=4(1+1)=8<\infty\). Therefore, \(n^{-1/2}\sum_{i=1}^n x_i u_i\xrightarrow{d}N_2(0,\Omega)\).

Worked Example: Heteroskedastic Scores With Conditional Centering

Let \(W_i\) be Bernoulli with \(P(W_i=1)=P(W_i=0)=1/2\), and set \(x_i=(1,W_i)'\). Conditional on \(W_i=0\), let \(u_i\sim N(0,1)\); conditional on \(W_i=1\), let \(u_i\sim N(0,4)\). Assume iid pairs across observations. Compute \(E[x_i u_i]\) and \(\Omega\).

Both conditional error means are zero, so \(E[u_i\mid W_i]=0\), and therefore \(E[x_i u_i]=0\). For the covariance, condition on the two possible values of \(W_i\). When \(W_i=0\), the conditional second moment of \(u_i\) is one and \(x_ix_i'=\begin{pmatrix}1&0\\0&0\end{pmatrix}\). When \(W_i=1\), the conditional second moment is four and \(x_ix_i'=\begin{pmatrix}1&1\\1&1\end{pmatrix}\). Each case has probability one half, giving

\Omega =\frac12\begin{pmatrix}1&0\\0&0\end{pmatrix} +\frac12\,4\begin{pmatrix}1&1\\1&1\end{pmatrix} =\begin{pmatrix}5/2&2\\2&2\end{pmatrix}.

The matrix is positive definite: its first diagonal entry is positive and its determinant is \((5/2)(2)-2^2=1>0\). The score second moment is finite because \(W_i\) has only two values and both conditional error variances are finite. Thus the multivariate central limit theorem applies despite the variance of the error changing with \(W_i\).

Worked Example: Mean Zero Without Finite Score Variance

Let the only regressor be an intercept, \(x_i=1\), and let the iid errors have the symmetric density \(f(u)=\frac34|u|^{-5/2}\) for \(|u|\geq1\), and zero otherwise. Check whether the stated finite-covariance score theorem applies.

The density integrates to one: \(2(\frac34)\int_1^\infty u^{-5/2}\,du =\frac32\cdot\frac23=1\). Symmetry and finite absolute first moment imply \(E[u_i]=0\), with

E[|u_i|] =2\left(\frac34\right)\int_1^\infty u^{-3/2}\,du =\frac32(2)=3.

The score is \(Z_i=u_i\), so it is centered. But its second moment is infinite:

E[Z_i^2] =2\left(\frac34\right)\int_1^\infty u^{-1/2}\,du =\infty.

Consequently, the finite-covariance theorem stated above does not apply. The error mean being zero does not supply the missing score variance condition; no conclusion from this theorem about a normal limit is justified.

Centering Is a Substantive Requirement

The mean-zero condition is not a technical detail that can be dropped from the theorem. If \(\mu=E[Z_i]\neq0\) and the score has finite second moments, then the sample sum decomposes as

\frac{1}{\sqrt{n}}\sum_{i=1}^n Z_i =\sqrt{n}\,\mu+\frac{1}{\sqrt{n}}\sum_{i=1}^n(Z_i-\mu).

The second term has a centered normal limit under the iid finite-covariance conditions, but the first term grows in magnitude with \(\sqrt n\). Thus the uncentered score sum does not have the centered finite normal limit in the theorem. A central limit theorem can instead be stated for the centered scores \(Z_i-\mu\), but that is a different normalization target from the score sum used in the zero-moment regression setting.

Common pitfall: “The errors have mean zero, so the regression scores have mean zero.” This reasoning is not generally valid: \(E[u_i]=0\) alone does not imply \(E[x_i u_i]=0\) when regressors and errors are dependent. The correction is to verify \(E[x_i u_i]=0\) directly, or to establish it using a valid condition such as \(E[u_i\mid x_i]=0\) together with integrability. Likewise, finite error variance alone does not ensure finite score covariance if the regressors are sufficiently heavy-tailed.

What the Score Limit Establishes

The covariance \(\Omega\) describes variation in the score vector itself. Its off-diagonal entries record how score components move together, and its diagonal entries are the componentwise score variances. Under heteroskedasticity, the expression \(E[u_i^2x_ix_i']\) automatically weights the regressor products by the conditional error variability; a constant-variance simplification is not warranted without an additional assumption.

The result supplies a distributional limit for a normalized sum of iid regression moments. It is one ingredient in the asymptotic linear representation of OLS, but it is not by itself a theorem about coefficient estimates: converting the score limit into an estimator limit also requires handling the empirical Gram matrix. Keeping these steps separate makes it easier to diagnose whether a failure comes from score centering, score moments, or the design matrix.

Check Your Understanding

Use the assumptions and calculations in this tutorial to answer the following questions.

For \(Z_i=x_i u_i\), write the mean-zero and finite-covariance conditions in terms of \(x_i\) and \(u_i\).
Why does \(E[u_i\mid x_i]=0\), with suitable integrability, imply \(E[x_i u_i]=0\)? Is conditional mean zero necessary for the multivariate central limit theorem stated here?
In the heteroskedastic worked example, identify which assumption permits the score central limit theorem even though the error variance depends on the regressor.
Explain why \(E[u_i]=0\) alone does not establish that \(E[x_i u_i]=0\).
For a score vector with nonzero mean \(\mu\), identify the term in its normalized sum that prevents the stated centered limit from applying.