The Multivariate Central Limit Theorem for IID Vectors

Why a Vector Limit Is Needed

Many studies record several measurements on each sampled unit. For example, a clinical study might measure systolic and diastolic blood pressure for each participant. The averages of the two measurements vary together because they were observed on the same people. A central limit theorem for each average separately describes the marginal behavior, but the joint limit also needs to describe that dependence.

The multivariate central limit theorem (CLT) provides this joint description. Under independence and identical distribution across sampled units, and a finite covariance matrix, the centered vector of sample averages has an asymptotic multivariate normal distribution. Its covariance matrix is the covariance matrix of one observation, not the covariance matrix of the sample average.

Vectors, Means, and Covariance Matrices

Definition: A random vector $ \mathbf{X}=(X_1,\ldots,X_p)^\mathsf{T} $ is a measurable random quantity taking values in $ \mathbb{R}^p $. If each coordinate has a finite second moment, its mean vector is $ \boldsymbol{\mu}=E[\mathbf{X}] $, and its covariance matrix is $$ \Sigma=E\left[(\mathbf{X}-\boldsymbol{\mu})(\mathbf{X}-\boldsymbol{\mu})^\mathsf{T}\right]. $$ The $ (j,k) $ entry is $ \operatorname{Cov}(X_j,X_k) $. A collection $ \mathbf{X}_1,\ldots,\mathbf{X}_n $ is iid if the vectors are independent and have the same distribution.

The sample-average vector is $ \bar{\mathbf{X}}=n^{-1}\sum_{i=1}^n\mathbf{X}_i $. The coordinates within each vector may be dependent; the iid assumption concerns the vectors across the sampling units. This distinction is essential in repeated-measures and paired-data settings.

A $p$-dimensional normal vector with mean $ \boldsymbol{\mu} $ and covariance matrix $ \Sigma $, written $N_p(\boldsymbol{\mu},\Sigma)$, is characterized by the characteristic function $$ \varphi_{\mathbf{Z}}(\mathbf{t})= \exp\left(i\mathbf{t}^\mathsf{T}\boldsymbol{\mu} -\frac{1}{2}\mathbf{t}^\mathsf{T}\Sigma\mathbf{t}\right), \qquad \mathbf{t}\in\mathbb{R}^p. $$ This specification determines both the marginal normal distributions and their dependence. The covariance matrix is positive semidefinite; when it is positive definite, the normal distribution has a density on $ \mathbb{R}^p $.

The Multivariate Central Limit Theorem

Theorem (Multivariate CLT for iid vectors): Let $ \mathbf{X}_1,\ldots,\mathbf{X}_n $ be iid random vectors in $ \mathbb{R}^p $, with $ E[\mathbf{X}_1]=\boldsymbol{\mu} $ and $ E[\|\mathbf{X}_1\|^2]<\infty $. Let $ \Sigma=\operatorname{Cov}(\mathbf{X}_1) $. Then $$ \sqrt{n}\,(\bar{\mathbf{X}}-\boldsymbol{\mu}) = \frac{1}{\sqrt{n}}\sum_{i=1}^n(\mathbf{X}_i-\boldsymbol{\mu}) \xrightarrow{d}N_p(\mathbf{0},\Sigma). $$

The scaling is $ \sqrt{n} $, as in the univariate Lindeberg–Levy CLT. In particular, the unscaled average converges to its mean under the law of large numbers, while fluctuations around that mean are typically of order $n^{-1/2}$. The finite-second-moment assumption ensures that each entry of $ \Sigma $ is finite. No independence between coordinates of the same observation is required.

The theorem permits a positive semidefinite covariance matrix, including one that is singular. The limit is then understood through its characteristic function; it may be concentrated on a lower-dimensional subspace. The consequences of singularity require separate care, so examples below use positive definite covariance matrices.

Two Useful Derivations

Before applying the limit, it helps to verify the finite-sample covariance and to see what the theorem says about any chosen linear contrast.

Result: For iid vectors with covariance matrix $ \Sigma $, $$ E[\bar{\mathbf{X}}]=\boldsymbol{\mu}, \qquad \operatorname{Cov}(\bar{\mathbf{X}})=\frac{\Sigma}{n}. $$ Proof: Linearity of expectation gives $E[\bar{\mathbf{X}}]=n^{-1}\sum_{i=1}^n E[\mathbf{X}_i]=\boldsymbol{\mu}$. For the covariance, expand the centered average: $$ \operatorname{Cov}(\bar{\mathbf{X}}) =\frac{1}{n^2}\sum_{i=1}^n\sum_{k=1}^n E\left[(\mathbf{X}_i-\boldsymbol{\mu})(\mathbf{X}_k-\boldsymbol{\mu})^\mathsf{T}\right]. $$ When $i\ne k$, independence and zero means make the expectation the zero matrix. Each of the $n$ diagonal terms is $\Sigma$. Thus the sum is $n\Sigma/n^2=\Sigma/n$. $\square$

This identity explains the normalization: multiplying the centered average by $ \sqrt{n} $ multiplies its covariance by $n$, leaving $ \Sigma $.

Result: For any fixed vector $ \mathbf{a}\in\mathbb{R}^p $, the scalar projection of the normalized average satisfies $$ \mathbf{a}^\mathsf{T}\sqrt{n}(\bar{\mathbf{X}}-\boldsymbol{\mu}) \xrightarrow{d}N\left(0,\mathbf{a}^\mathsf{T}\Sigma\mathbf{a}\right). $$ Proof: Define $Y_i=\mathbf{a}^\mathsf{T}\mathbf{X}_i$. These variables are iid, with mean $ \mathbf{a}^\mathsf{T}\boldsymbol{\mu} $. Their variance is $$ \operatorname{Var}(Y_i) =E\left[\{\mathbf{a}^\mathsf{T}(\mathbf{X}_i-\boldsymbol{\mu})\}^2\right] =\mathbf{a}^\mathsf{T}\Sigma\mathbf{a}. $$ This variance is finite because $ \mathbf{a} $ is fixed and $E[\|\mathbf{X}_i\|^2]<\infty$. Applying the univariate Lindeberg–Levy CLT to $Y_1,\ldots,Y_n$ gives the stated convergence. If the variance is zero, each centered $Y_i$ is zero almost surely, and the claimed limit is the point mass at zero. $\square$

This projection result gives a useful way to interpret the covariance matrix: it determines the asymptotic variance of every linear combination of the coordinates. The multivariate theorem asserts the joint normal limit, not merely a list of separate coordinate limits.

Worked Applications

Worked Example: Joint Limits for Two Clinical Measurements

Suppose each participant has a measurement vector $ \mathbf{X}_i=(S_i,D_i)^\mathsf{T} $, where $S_i$ is systolic pressure and $D_i$ is diastolic pressure. Assume the participant vectors are iid with $$ \boldsymbol{\mu}= \begin{pmatrix}120\\80\end{pmatrix}, \qquad \Sigma= \begin{pmatrix}144&36\\36&64\end{pmatrix}. $$ For a sample of $n$ participants, find the asymptotic distribution of the centered sample-average vector.

The determinant of $ \Sigma $ is $144(64)-36^2=9216-1296=7920>0$, and its diagonal entries are positive, so this covariance matrix is positive definite. The iid vector CLT therefore gives $$ \sqrt{n} \begin{pmatrix} \bar S-120\\ \bar D-80 \end{pmatrix} \xrightarrow{d} N_2\left( \begin{pmatrix}0\\0\end{pmatrix}, \begin{pmatrix}144&36\\36&64\end{pmatrix} \right). $$

For $n=100$, the corresponding large-sample covariance matrix for $ (\bar S,\bar D)^\mathsf{T} $ is $ \Sigma/100 $, namely $$ \begin{pmatrix}1.44&0.36\\0.36&0.64\end{pmatrix}. $$ The standard errors are $ \sqrt{1.44}=1.2 $ and $ \sqrt{0.64}=0.8 $. The correlation between the averages is $0.36/(1.2\times0.8)=0.375$, the same as the correlation between one participant's two measurements. Thus the joint limit retains the dependence induced by paired measurements.

Worked Example: A Contrast of Two Coordinates

Using the covariance matrix above, consider the estimated mean difference $ \bar S-\bar D $. Determine its limiting centered distribution after $ \sqrt n $ scaling.

The contrast uses $ \mathbf{a}=(1,-1)^\mathsf{T} $. Its population mean is $ \mathbf{a}^\mathsf{T}\boldsymbol{\mu}=120-80=40 $. Its variance for one paired observation is $$ \mathbf{a}^\mathsf{T}\Sigma\mathbf{a} =144+64-36-36=136. $$ The two negative terms arise because the covariance appears twice in the variance of a difference. Hence $$ \sqrt n\{(\bar S-\bar D)-40\}\xrightarrow{d}N(0,136). $$ For $n=100$, the approximate standard error of the sample difference is $ \sqrt{136/100}=\sqrt{1.36}\approx1.166 $. Treating systolic and diastolic averages as if they came from independent groups would omit the covariance terms and give variance $144+64=208$ for the scaled contrast. That is not the sampling variance for paired observations.

Worked Example: Whitening a Two-Dimensional Average

Let $ \Sigma $ be positive definite and suppose a matrix $ \Sigma^{-1/2} $ satisfies $ \Sigma^{-1/2}\Sigma\Sigma^{-1/2}=I_p $. Apply this transformation to the normalized average in the clinical example.

Define $$ \mathbf{Z}_n=\Sigma^{-1/2}\sqrt n(\bar{\mathbf{X}}-\boldsymbol{\mu}). $$ A fixed linear transformation of a multivariate normal vector is normal, with transformed mean and covariance. This follows directly from the characteristic function: for any fixed matrix $A$, the characteristic function of $A\mathbf{Z}$ at $ \mathbf{t} $ is $ \varphi_{\mathbf{Z}}(A^\mathsf{T}\mathbf{t}) $; substituting the normal characteristic function gives mean $A\boldsymbol{\mu}$ and covariance $A\Sigma A^\mathsf{T}$. Applying the multivariate CLT and this transformation yields $$ \mathbf{Z}_n\xrightarrow{d}N_p(\mathbf{0},I_p). $$ In two dimensions, this means the two whitened coordinates have asymptotically unit variances and zero covariance. Whitening is useful when a problem calls for a common standardized scale, but it requires a nonsingular covariance matrix.

Common Pitfall and Practical Interpretation

A common mistake is to establish a univariate normal limit for every coordinate and then assume that this alone proves the joint limit or determines the joint distribution. Marginal limits do not specify dependence: two normal coordinates could be independent, positively correlated, or negatively correlated while having the same marginal distributions. The correction is to retain the full covariance matrix and use the multivariate CLT for the vector of observations. In applications, form one vector per independent sampling unit, calculate its covariance matrix, and then apply the theorem to the sample-average vector.

The theorem matters whenever conclusions involve several sample averages at once, such as a difference of paired measurements, a weighted score, or a joint confidence procedure. The covariance entries are not decorative: they determine the variability of every linear combination, as the contrast calculation shows. The iid assumption is also at the vector level. Repeated measurements within a participant may be dependent, while different participants must supply independent, identically distributed vectors for this theorem to apply.

Key takeaway: For iid $p$-dimensional observations with finite second moments, $ \sqrt n(\bar{\mathbf{X}}-\boldsymbol{\mu}) $ has an asymptotic $N_p(\mathbf{0},\Sigma)$ distribution. Use the covariance matrix of one complete observation vector to preserve the dependence among its coordinates.

Check Your Understanding

Use the theorem and the derivations above to answer these questions.

For iid vectors with covariance matrix $ \Sigma $, what is $ \operatorname{Cov}(\bar{\mathbf{X}}) $, and why does the normalization use $ \sqrt n $?
Why does the iid-vector assumption not require the coordinates within one vector to be independent?
For a fixed contrast $ \mathbf{a}^\mathsf{T}\bar{\mathbf{X}} $, what is the asymptotic variance after centering and multiplying by $ \sqrt n $?
In a paired-measurement study, why can an independence calculation give the wrong variance for the difference of sample averages?
What property of $ \Sigma $ is needed to define the whitening transformation using $ \Sigma^{-1/2} $?