Central Limit Conditions for Empirical Scores

Why Empirical Scores Need a Central Limit Condition

An estimating equation is built from an average of observation-level contributions. To understand the random fluctuation of that average, one asks whether its centered, suitably scaled version is approximately Gaussian. That conclusion is not automatic: it depends on the score contributions having controlled variability, and in arrays of non-identically distributed observations, on no small set of unusually large contributions dominating the sum.

Let \(X_1,\ldots,X_n\) be observations and let \(\psi(X_i,\theta)\) be a \(p\)-dimensional estimating function. At a fixed parameter value \(\theta_0\), write \(\psi_i=\psi(X_i,\theta_0)\). The central limit question concerns \(n^{-1/2}\sum_{i=1}^n\{\psi_i-E[\psi_i]\}\). When \(\theta_0\) is the target of an estimating equation, typically \(E[\psi_i]=0\); this tutorial focuses on the conditions for the sum itself to have a Gaussian limit, not on how that limit is used to analyze an estimator.

Definition: The covariance matrix of a random vector \(\psi\) with finite second moments is \(\operatorname{Cov}(\psi)=E[(\psi-E[\psi])(\psi-E[\psi])^\mathsf{T}]\). It is positive semidefinite and describes the variances and covariances of the vector's coordinates. A multivariate normal limit may have a singular covariance matrix; positive definiteness is not needed for the central limit conclusion.

The Iid Multivariate Central Limit Theorem

The most familiar setting assumes a fixed dimension \(p\) and independent, identically distributed score vectors. Centering is essential: an empirical score with a nonzero expectation has a deterministic term of order \(\sqrt n\), which generally does not have a centered Gaussian limit.

Theorem: Suppose \(\psi_1,\psi_2,\ldots\) are iid \(p\)-dimensional random vectors, \(E[\psi_1]=0\), and \(E[\|\psi_1\|^2]<\infty\), where \(\|\cdot\|\) is the Euclidean norm. Set \(\Omega=\operatorname{Cov}(\psi_1)\). Then

\frac{1}{\sqrt n}\sum_{i=1}^n \psi_i \xrightarrow{d} N_p(0,\Omega).

The matrix \(\Omega\) may be singular.

Here is the multivariate reduction that justifies the conclusion. For any fixed \(a\in\mathbb{R}^p\), the scalar variables \(a^\mathsf{T}\psi_i\) are iid, have mean zero, and have variance \(a^\mathsf{T}\Omega a<\infty\). The ordinary scalar iid central limit theorem therefore gives

a^\mathsf{T}\left(\frac{1}{\sqrt n}\sum_{i=1}^n\psi_i\right) \xrightarrow{d} N(0,a^\mathsf{T}\Omega a).

The Cramér–Wold theorem says that convergence of these scalar projections for every \(a\) to the projections of one multivariate distribution implies convergence to that distribution. The displayed scalar limits are exactly the projections of \(N_p(0,\Omega)\), proving the theorem. If \(a^\mathsf{T}\Omega a=0\), the projection is zero almost surely because it has mean zero and variance zero, so its limit is the corresponding degenerate normal distribution. This also covers singular \(\Omega\). \(\square\)

Worked Example: A Poisson Mean Score

Let \(X_i\) be iid Poisson random variables with mean \(\lambda>0\). For the estimating function \(\psi(X_i,\lambda)=X_i-\lambda\), compute the limit of the normalized empirical score.

The Poisson mean and variance are \(E[X_i]=\lambda\) and \(\operatorname{Var}(X_i)=\lambda\). Thus \(E[\psi_i]=0\), and \(\operatorname{Var}(\psi_i)=\lambda<\infty\). The iid theorem applies with \(p=1\) and \(\Omega=\lambda\):

\sqrt n\left(\frac{1}{n}\sum_{i=1}^n(X_i-\lambda)\right) =\frac{1}{\sqrt n}\sum_{i=1}^n(X_i-\lambda) \xrightarrow{d}N(0,\lambda).

The variance in the limit is \(\lambda\), not \(1\): this is the limit for the unstandardized score average. Dividing the left side by \(\sqrt{\lambda}\) would instead give a standard normal limit.

Independent Triangular Arrays and Lindeberg's Condition

Identical distributions are convenient but not necessary. In a triangular array, the distribution of an observation may vary with both its row \(n\) and its position \(i\). Write the centered \(p\)-vector in row \(n\) as \(\psi_{ni}\), \(1\leq i\leq n\). Assume that the vectors are independent within each row and have finite second moments. A covariance condition alone controls the total variance, but does not rule out a few large terms carrying a substantial part of it.

Theorem: Suppose, for each \(n\), \(\psi_{n1},\ldots,\psi_{nn}\) are independent \(p\)-dimensional vectors with \(E[\psi_{ni}]=0\) and finite second moments. Suppose

\frac{1}{n}\sum_{i=1}^n \operatorname{Cov}(\psi_{ni})\longrightarrow\Omega

entrywise for a finite positive semidefinite matrix \(\Omega\), and, for every \(\varepsilon>0\),

\frac{1}{n}\sum_{i=1}^n E\!\left[\|\psi_{ni}\|^2 \mathbf{1}\{\|\psi_{ni}\|>\varepsilon\sqrt n\}\right] \longrightarrow 0.

Then

\frac{1}{\sqrt n}\sum_{i=1}^n\psi_{ni} \xrightarrow{d}N_p(0,\Omega).

The second display is the multivariate Lindeberg condition at the scale relevant to the normalized sum.

To prove the result, fix \(a\in\mathbb{R}^p\). The scalar variables \(a^\mathsf{T}\psi_{ni}\) are independent and centered, and their total variance after normalization converges:

\frac{1}{n}\sum_{i=1}^n\operatorname{Var}(a^\mathsf{T}\psi_{ni}) = a^\mathsf{T}\left\{\frac{1}{n}\sum_{i=1}^n\operatorname{Cov}(\psi_{ni})\right\}a \longrightarrow a^\mathsf{T}\Omega a.

If \(a\neq 0\), the event \(\{|a^\mathsf{T}\psi_{ni}|>\varepsilon\sqrt n\}\) is contained in \(\{\|\psi_{ni}\|>\varepsilon\sqrt n/\|a\|\}\), and \((a^\mathsf{T}\psi_{ni})^2\leq \|a\|^2\|\psi_{ni}\|^2\). Consequently, the scalar Lindeberg sum is bounded above by

\frac{\|a\|^2}{n}\sum_{i=1}^n E\!\left[\|\psi_{ni}\|^2 \mathbf{1}\{\|\psi_{ni}\|>(\varepsilon/\|a\|)\sqrt n\}\right],

which tends to zero by the assumed vector condition. The scalar Lindeberg–Feller theorem now gives convergence of each projection to \(N(0,a^\mathsf{T}\Omega a)\). If \(a^\mathsf{T}\Omega a=0\), the normalized projection has variance tending to zero, so it converges to zero in probability by Chebyshev's inequality. For every \(a\), the limit is therefore the corresponding projection of \(N_p(0,\Omega)\). The Cramér–Wold theorem yields the claimed vector convergence. \(\square\)

Worked Example: A Bivariate Score with Correlated Coordinates

Let \(Z_i\) and \(W_i\) be independent standard normal variables, independently replicated across \(i\), and define \(\psi_i=(Z_i,Z_i+W_i)^\mathsf{T}\). Find the covariance matrix and the limit of the normalized sum.

Both coordinates have mean zero. The first variance is \(\operatorname{Var}(Z_i)=1\); the second is \(\operatorname{Var}(Z_i+W_i)=1+1=2\). Their covariance is \(\operatorname{Cov}(Z_i,Z_i+W_i)=1+0=1\). Hence

\Omega= \begin{pmatrix} 1&1\\ 1&2 \end{pmatrix}, \qquad \det(\Omega)=2-1=1>0.

The score vectors are iid and have finite second moments, so the multivariate theorem gives

\frac{1}{\sqrt n}\sum_{i=1}^n \begin{pmatrix}Z_i\\Z_i+W_i\end{pmatrix} \xrightarrow{d} N_2\!\left( \begin{pmatrix}0\\0\end{pmatrix}, \begin{pmatrix}1&1\\1&2\end{pmatrix} \right).

The nonzero off-diagonal entry matters: the limiting coordinates are correlated, with correlation \(1/\sqrt{1\cdot2}=1/\sqrt2\). Applying separate univariate central limit theorems would not by itself establish this joint covariance structure.

Moment Bounds That Verify Lindeberg

Directly checking tail expectations can be difficult. A higher-moment bound provides a useful sufficient condition. For any \(r>0\), on the event \(\{\|\psi_{ni}\|>\varepsilon\sqrt n\}\) we have \(\|\psi_{ni}\|^2\leq \|\psi_{ni}\|^{2+r}/(\varepsilon\sqrt n)^r\). Summing expectations gives

\frac{1}{n}\sum_{i=1}^n E\!\left[\|\psi_{ni}\|^2 \mathbf{1}\{\|\psi_{ni}\|>\varepsilon\sqrt n\}\right] \leq \frac{1}{\varepsilon^r n^{1+r/2}} \sum_{i=1}^n E[\|\psi_{ni}\|^{2+r}].

Thus the Lyapunov-type condition \(n^{-1-r/2}\sum_{i=1}^n E[\|\psi_{ni}\|^{2+r}]\to0\) implies the Lindeberg condition: the right side tends to zero for each fixed \(\varepsilon>0\). For iid vectors with \(E[\|\psi_1\|^2]<\infty\), Lindeberg holds without requiring a higher moment. Indeed, its left side is \(E[\|\psi_1\|^2\mathbf{1}\{\|\psi_1\|>\varepsilon\sqrt n\}]\), which tends to zero by dominated convergence, since the integrand tends pointwise to zero and is bounded by the integrable variable \(\|\psi_1\|^2\).

Worked Example: Verifying a Lyapunov Bound

Suppose the independent centered scalar scores in row \(n\) satisfy \(E[|\psi_{ni}|^3]\leq 6\) for every \(i\) and \(n\), and that \(n^{-1}\sum_i\operatorname{Var}(\psi_{ni})\to\sigma^2<\infty\). Verify a sufficient condition for a normal limit.

Take \(r=1\). The Lyapunov expression is bounded by

\frac{1}{n^{3/2}}\sum_{i=1}^n E[|\psi_{ni}|^3] \leq \frac{6n}{n^{3/2}} =\frac{6}{\sqrt n}\longrightarrow0.

The moment bound therefore implies Lindeberg, and the scalar case of the triangular-array theorem yields \(n^{-1/2}\sum_i\psi_{ni}\xrightarrow{d}N(0,\sigma^2)\). If \(\sigma^2=0\), this means convergence to the point mass at zero.

Why Variance Convergence Is Not Enough

The Lindeberg condition prevents large terms from contributing a non-negligible share of the normalized sum's variation. The following array has stable total variance but violates this safeguard.

Worked Example: Rare Large Scores Do Not Produce a Normal Limit

For each \(n\), let \(\psi_{n1},\ldots,\psi_{nn}\) be independent with probabilities \(P(\psi_{ni}=\sqrt n)=1/(2n)\), \(P(\psi_{ni}=-\sqrt n)=1/(2n)\), and \(P(\psi_{ni}=0)=1-1/n\). Each score is centered, and

\operatorname{Var}(\psi_{ni}) = n\left(\frac{1}{2n}+\frac{1}{2n}\right)=1, \qquad \frac{1}{n}\sum_{i=1}^n\operatorname{Var}(\psi_{ni})=1.

But for \(0<\varepsilon<1\), every nonzero value exceeds \(\varepsilon\sqrt n\). Therefore the Lindeberg expression equals

\frac{1}{n}\sum_{i=1}^n E[\psi_{ni}^2\mathbf{1}\{|\psi_{ni}|>\varepsilon\sqrt n\}] =\frac{1}{n}\sum_{i=1}^n 1=1,

so it does not tend to zero. In fact, \(n^{-1/2}\sum_i\psi_{ni}\) is a sum of \(n\) independent variables taking \(1\) and \(-1\) with probabilities \(1/(2n)\) each and zero otherwise. Its characteristic function at \(t\) is

\left(1-\frac{1}{n}+\frac{\cos t}{n}\right)^n \longrightarrow \exp(\cos t-1).

This is the characteristic function of the difference of two independent Poisson random variables, each with mean \(1/2\), not that of \(N(0,1)\). The variance remains one, but occasional order-one contributions survive after normalization. This verifies concretely why covariance convergence alone cannot ensure a Gaussian limit.

Common Mistakes and Practical Checks

Using an uncentered score. The theorem applies to mean-zero terms. If \(E[\psi_{ni}]\neq0\), first account for the mean; it cannot simply be ignored.
Checking only the diagonal variances. A multivariate conclusion needs convergence of the full covariance matrix, including cross-covariances.
Assuming finite variance settles every array case. Finite second moments for each individual term do not rule out rare, large terms across rows. Check Lindeberg or a valid sufficient moment bound.
Demanding a nonsingular covariance unnecessarily. A singular covariance still defines a valid, possibly degenerate multivariate normal limit.

In practice, first verify centering and independence assumptions, then calculate the limiting covariance, and finally justify that large scores are negligible at the \(\sqrt n\) scale. For iid scores, a finite second moment is enough. For non-identically distributed arrays, the Lindeberg condition or a Lyapunov bound makes the needed tail control explicit.

Check Your Understanding

Use the conditions and examples above to test your understanding.

For iid centered score vectors, what moment condition and covariance matrix determine the multivariate normal limit?
Why must a multivariate argument account for cross-covariances as well as coordinate variances?
State the multivariate Lindeberg condition for an independent triangular array of centered scores.
How does a uniform \(2+r\) moment bound imply a Lyapunov-type condition when \(r>0\)?
In the rare-large-score example, why does variance convergence hold even though the Lindeberg condition fails?