Vector Laws of Large Numbers and Central Limit Theorems

Why Vector Limits Matter

In multivariate likelihood problems, a statistic often has several components: category counts, empirical moments, or a vector of sufficient statistics. Its average may approach a fixed mean, while its random fluctuation around that mean has a joint normal limit. The word joint matters: the limiting covariance records how the components vary together, information that separate one-dimensional limits do not provide.

These results supply a basic probabilistic input to likelihood asymptotics. For an exponential family, the score is a centered sum of sufficient statistics. A vector law of large numbers describes the statistics’ average, and a vector central limit theorem describes the score’s normalized fluctuation. Later likelihood arguments use these limits together with expansions of the likelihood; the results here do not by themselves establish asymptotic normality of a maximum-likelihood estimator.

Vector Convergence and Moment Conditions

For a random vector \(Y=(Y_1,\ldots,Y_d)^T\) with finite second moments, write its mean as \(\mu=E[Y]\) and its covariance matrix as \(\Sigma=\operatorname{Cov}(Y)=E[(Y-\mu)(Y-\mu)^T]\). For independent, identically distributed (i.i.d.) copies \(Y_1,\ldots,Y_n\), let \(\bar Y_n=n^{-1}\sum_{i=1}^nY_i\). The two principal limits concern \(\bar Y_n\) and \(\sqrt n(\bar Y_n-\mu)\).

Proof. Set \(\Sigma=\operatorname{Cov}(Y_1)\). Independence gives \(E\|\bar Y_n-\mu\|^2=\operatorname{tr}(\Sigma)/n\): expand the squared norm of the sum, and the cross terms have expectation zero because distinct centered observations are independent. Markov’s inequality therefore yields \(P(\|\bar Y_n-\mu\|>\varepsilon)\leq \operatorname{tr}(\Sigma)/(n\varepsilon^2)\), which tends to zero for each \(\varepsilon>0\). \(\square\)

The finite-second-moment condition is a convenient sufficient condition for this weak law. A more general i.i.d. weak law holds under the weaker condition \(E\|Y_1\|<\infty\), but the proof above uses the finite covariance directly. For the central limit theorem, finite second moments are the standard condition used here.

Proof. For any fixed \(a\in\mathbb{R}^d\), the scalar variables \(a^T(Y_i-\mu)\) are i.i.d., have mean zero, and have variance \(a^T\Sigma a<\infty\). The univariate central limit theorem gives \(a^T\sqrt n(\bar Y_n-\mu)\xrightarrow{d}N(0,a^T\Sigma a)\). The characteristic function of the possibly singular multivariate normal \(N_d(0,\Sigma)\), evaluated at \(a\), is \(\exp(-a^T\Sigma a/2)\). Thus the limiting distribution of every linear combination agrees with that of this normal vector. The Cramér–Wold theorem then gives the stated vector convergence. \(\square\)

Neither theorem requires the coordinates of \(Y_i\) to be independent. Their dependence within each observation is captured by the off-diagonal entries of \(\Sigma\). Independence is required across observations in the theorems as stated.

Worked Examples

Sufficient Statistics and Likelihood Asymptotics

Consider a regular exponential family with density or mass function \(f_\eta(x)=h(x)\exp\{\eta^TT(x)-A(\eta)\}\), relative to a common dominating measure. Here \(T(x)\in\mathbb{R}^d\), the natural parameter \(\eta\) lies in an open parameter set, and \(A\) is the log-partition function. Assume differentiation under the integral is justified in a neighborhood of the parameter value under consideration; in particular, assume the required first and second moments are finite. In this setting,

For completeness, the first identity follows by differentiating the normalizing integral: its derivative in coordinate \(j\) is the integral of \(T_j(x)f_\eta(x)\), while differentiating \(\exp(A(\eta))\) gives \(\exp(A(\eta))\partial_j A(\eta)\). Dividing by the normalizing integral gives \(E_\eta[T_j(X)]=\partial_j A(\eta)\). Differentiating this expectation in coordinate \(k\) gives \(\operatorname{Cov}_\eta(T_j(X),T_k(X))\): differentiating the density contributes \(T_k(X)-\partial_k A(\eta)\). Thus the Hessian entries are the covariance entries.

For i.i.d. observations, the log-likelihood, apart from terms not depending on \(\eta\), is \(\ell_n(\eta)=n\{\eta^T\bar T_n-A(\eta)\}\), where \(\bar T_n=n^{-1}\sum_{i=1}^nT(X_i)\). Its score is \(\nabla\ell_n(\eta)=\sum_{i=1}^n\{T(X_i)-\nabla A(\eta)\}\), and the information for one observation is \(\nabla^2A(\eta)=\operatorname{Cov}_\eta(T(X))\). The vector central limit theorem therefore gives

This identifies the scale and covariance of score fluctuations, while the law of large numbers gives \(\bar T_n\xrightarrow{p}\nabla A(\eta)\). Together, these limits are foundational inputs to likelihood arguments. They do not alone prove that an estimator exists, is consistent, or has a particular limiting distribution; those conclusions require additional arguments and conditions.

Common Pitfalls and Interpretation

A frequent error is to say that \(\bar Y_n\) has limiting covariance \(\Sigma\). Its covariance is \(\Sigma/n\), which tends to zero. The matrix \(\Sigma\) instead describes the limiting covariance of the scaled fluctuation \(\sqrt n(\bar Y_n-\mu)\). A related error is to infer that coordinates of the limiting normal vector are independent merely because each coordinate has a normal limit. Independence requires zero covariance for a jointly normal vector; the off-diagonal entries must be checked.

The role of the vector theorem is especially important when statistics are dependent within observations. The multinomial indicators, for example, have negative covariance because an observation cannot be in two categories at once. Ignoring this dependence changes the limiting covariance matrix and can lead to incorrect standard errors or likelihood calculations.