Covariance Matrices and Their Entries

Why Organize Covariances in a Matrix?

For a single random variable, variance summarizes its spread. For a random vector, the components can each vary and can also vary together. A covariance matrix puts these quantities in one organized object: the variances appear on the diagonal, while the covariances between distinct components appear off the diagonal. Its entries retain the component ordering, so the matrix records which pair of variables each number describes.

The construction requires finite second moments. This ensures that each variance and covariance used as an entry is finite. The matrix is a summary of second-order variation; it does not, by itself, state that the components are independent.

Definition and Entrywise Interpretation

Definition: Let $X=(X_1,\ldots,X_p)^T$ be a random vector whose components have finite second moments, $E[X_i^2]<\infty$ for $i=1,\ldots,p$. Write $\mu=E[X]=(E[X_1],\ldots,E[X_p])^T$. The covariance matrix of $X$ is the $p$ by $p$ matrix $$ \operatorname{Cov}(X)=E\!\left[(X-\mu)(X-\mu)^T\right]. $$ Its $(i,j)$ entry is $\operatorname{Cov}(X_i,X_j)=E[(X_i-E[X_i])(X_j-E[X_j])]$.

The expectation of a matrix is taken entry by entry. Thus, the $(i,j)$ entry of the outer product $(X-\mu)(X-\mu)^T$ is $(X_i-E[X_i])(X_j-E[X_j])$, and taking its expectation gives the covariance of the two corresponding components.

On the diagonal, $i=j$, so the entry is $\operatorname{Var}(X_i)$. Off the diagonal, the entry measures how the two components vary together: a positive covariance indicates that their centered values tend to have the same sign, and a negative covariance indicates that they tend to have opposite signs. A zero covariance says that this particular second-order measure of joint variation is zero; it does not generally establish independence.

Theorem: Under the finite-second-moment condition, $\operatorname{Cov}(X)$ is symmetric, has diagonal entries $\operatorname{Var}(X_i)$, and has off-diagonal entries $\operatorname{Cov}(X_i,X_j)$. Entrywise, it can also be calculated as $$ \operatorname{Cov}(X)_{ij}=E[X_iX_j]-E[X_i]E[X_j]. $$

Proof. By the definition of matrix expectation, the $(i,j)$ entry is $$ E[(X_i-E[X_i])(X_j-E[X_j])]. $$ Expanding the product and using linearity of expectation gives $$ E[X_iX_j]-E[X_i]E[X_j]-E[X_i]E[X_j]+E[X_i]E[X_j] =E[X_iX_j]-E[X_i]E[X_j]. $$ The expectations exist: finite second moments imply finite first moments, and $2|X_iX_j|\leq X_i^2+X_j^2$ implies $E[|X_iX_j|]<\infty$. When $i=j$, the entry is $E[(X_i-E[X_i])^2]=\operatorname{Var}(X_i)$. For any $i,j$, swapping the two centered factors does not change their product, so $\operatorname{Cov}(X_i,X_j)=\operatorname{Cov}(X_j,X_i)$. Hence the matrix is symmetric. $\square$

Calculating Entries From Joint Moments

A convenient calculation uses two kinds of quantities: the component means $E[X_i]$ and the raw joint moments $E[X_iX_j]$. First find the means, then find the second moments and cross-products, and finally subtract the product of means for each entry. For a discrete vector, each expectation is a probability-weighted sum over the joint outcomes.

Fix the component order.
Write $X=(X_1,\ldots,X_p)^T$. Use this same order for both rows and columns of the matrix.

Find the means.
Calculate $E[X_i]$ for each component $i$.

Find raw second and cross moments.
Calculate $E[X_iX_j]$ for the entries needed.

Center each joint moment.
Use $\operatorname{Cov}(X_i,X_j)=E[X_iX_j]-E[X_i]E[X_j]$. Check that the diagonal gives the component variances and that paired off-diagonal entries agree.

Worked Example: A Covariance Matrix From a Joint Probability Table

Let $(X,Y)$ take the values $(0,0),(1,0),(0,2)$ with probabilities $1/2,1/4,1/4$, respectively. Calculate its covariance matrix.

First calculate the means: $$ E[X]=0\cdot\frac12+1\cdot\frac14+0\cdot\frac14=\frac14,\qquad E[Y]=0\cdot\frac12+0\cdot\frac14+2\cdot\frac14=\frac12. $$ The raw second moments are $$ E[X^2]=0^2\cdot\frac12+1^2\cdot\frac14+0^2\cdot\frac14=\frac14, $$ $$ E[Y^2]=0^2\cdot\frac12+0^2\cdot\frac14+2^2\cdot\frac14=1, \qquad E[XY]=0. $$ Therefore, $$ \operatorname{Var}(X)=\frac14-\left(\frac14\right)^2=\frac{3}{16}, \qquad \operatorname{Var}(Y)=1-\left(\frac12\right)^2=\frac34, $$ and $$ \operatorname{Cov}(X,Y)=0-\frac14\cdot\frac12=-\frac18. $$ In the order $(X,Y)$, the answer is $$ \operatorname{Cov}\!\begin{pmatrix}X\\Y\end{pmatrix} = \begin{pmatrix} \frac{3}{16} & -\frac18\\ -\frac18 & \frac34 \end{pmatrix}. $$ The diagonal entries are the two variances, and the equal off-diagonal entries are the covariance of $X$ and $Y$.

Reading the Matrix and Its Order

For a vector with three components, the entry in row 2, column 3 is the covariance between the second and third components. The entry in row 3, column 2 describes the same pair in reverse order and has the same value. Therefore, one must state the component order when giving a covariance matrix; changing the order of the components changes where entries appear.

The diagonal entries use squared units. If $X_i$ is measured in hours, its variance has units of hours squared. An off-diagonal entry has the product of the units of its two components: the covariance of height in centimeters and weight in kilograms has units of centimeter-kilograms. This provides a useful check on interpretation and calculations.

Worked Example: A Component and Its Sum

Let $U$ and $V$ be independent Bernoulli random variables with $P(U=1)=P(V=1)=1/2$, and set $W=U+V$. Calculate the covariance matrix of $(U,W)^T$ using moments.

The means are $E[U]=1/2$ and $E[W]=E[U]+E[V]=1$. Since $U^2=U$, $E[U^2]=1/2$. Also, $$ E[W^2]=E[(U+V)^2] =E[U^2]+2E[UV]+E[V^2] =\frac12+2\left(\frac14\right)+\frac12 =\frac32. $$ Here independence gives $E[UV]=E[U]E[V]=1/4$. The cross moment is $$ E[UW]=E[U(U+V)]=E[U^2]+E[UV]=\frac12+\frac14=\frac34. $$ Thus, $$ \operatorname{Var}(U)=\frac12-\left(\frac12\right)^2=\frac14,\qquad \operatorname{Var}(W)=\frac32-1^2=\frac12, $$ and $$ \operatorname{Cov}(U,W)=\frac34-\frac12\cdot1=\frac14. $$ So, in the order $(U,W)$, $$ \operatorname{Cov}\!\begin{pmatrix}U\\W\end{pmatrix} = \begin{pmatrix} \frac14 & \frac14\\ \frac14 & \frac12 \end{pmatrix}. $$ Although $W$ includes $U$, the off-diagonal entry is computed by the same joint-moment formula as any other covariance.

A Three-Component Calculation

For larger vectors, the same entrywise procedure applies. It is often useful to compute one triangular half of the matrix and then use symmetry to fill in the other half. The following example displays each mean, raw second moment, and cross moment needed.

Worked Example: Building a Three-by-Three Matrix

Let $Z=(Z_1,Z_2,Z_3)^T$ take the values $(0,0,0),(1,1,1),(2,0,1)$, each with probability $1/3$. Its component means are $$ E[Z_1]=\frac{0+1+2}{3}=1,\qquad E[Z_2]=\frac{0+1+0}{3}=\frac13,\qquad E[Z_3]=\frac{0+1+1}{3}=\frac23. $$ The raw square moments are $$ E[Z_1^2]=\frac{0+1+4}{3}=\frac53,\quad E[Z_2^2]=\frac{0+1+0}{3}=\frac13,\quad E[Z_3^2]=\frac{0+1+1}{3}=\frac23. $$ The raw cross moments are $$ E[Z_1Z_2]=\frac{0+1+0}{3}=\frac13,\quad E[Z_1Z_3]=\frac{0+1+2}{3}=1,\quad E[Z_2Z_3]=\frac{0+1+0}{3}=\frac13. $$ The three diagonal entries are $$ \operatorname{Var}(Z_1)=\frac53-1^2=\frac23,\quad \operatorname{Var}(Z_2)=\frac13-\left(\frac13\right)^2=\frac29,\quad \operatorname{Var}(Z_3)=\frac23-\left(\frac23\right)^2=\frac29. $$ For the distinct pairs, $$ \operatorname{Cov}(Z_1,Z_2)=\frac13-1\cdot\frac13=0, $$ $$ \operatorname{Cov}(Z_1,Z_3)=1-1\cdot\frac23=\frac13,\qquad \operatorname{Cov}(Z_2,Z_3)=\frac13-\frac13\cdot\frac23=\frac19. $$ Putting these entries in the order $(Z_1,Z_2,Z_3)$ gives $$ \operatorname{Cov}(Z)= \begin{pmatrix} \frac23 & 0 & \frac13\\ 0 & \frac29 & \frac19\\ \frac13 & \frac19 & \frac29 \end{pmatrix}. $$ The repeated off-diagonal values show the required symmetry. Each entry can also be checked directly against its pair of component labels.

Common Mistakes and a Practical Check

Putting standard deviations on the diagonal. The diagonal entries are variances, not standard deviations. For example, if $\operatorname{Var}(X_i)=9$, the corresponding diagonal entry is 9, not 3.
Using a raw cross moment as a covariance. In general, $E[X_iX_j]$ is not the covariance. Subtract $E[X_i]E[X_j]$ unless the relevant means make that product zero.
Putting entries in inconsistent order. If the vector is ordered as $(X_1,X_2,X_3)$, rows and columns must follow that same order.
Assuming a zero off-diagonal entry proves independence. It establishes zero covariance only. Independence is a stronger property and does not follow in general.

A reliable final check is to verify that every diagonal entry agrees with a separately calculated variance, that the two entries for each component pair agree, and that each off-diagonal entry has the product units of its two components. These checks catch many indexing and centering errors.

Key takeaway: For a random vector $X$, the covariance matrix has entry $(i,j)$ equal to $E[X_iX_j]-E[X_i]E[X_j]$. Its diagonal records component variances, and its off-diagonal entries record pairwise covariances in the specified component order.

Check Your Understanding

Use the entrywise formula and keep the component order explicit.

For a vector $X=(X_1,X_2)^T$, what does the $(2,1)$ entry of $\operatorname{Cov}(X)$ represent?
If $E[A]=2$, $E[B]=-1$, and $E[AB]=5$, calculate $\operatorname{Cov}(A,B)$.
Why must a covariance matrix be symmetric?
If $X_1$ is measured in minutes and $X_2$ in dollars, what are the units of $\operatorname{Cov}(X_1,X_2)$?
What does a zero off-diagonal entry establish, and what does it not establish in general?