Means and Covariance from Joint Moments

From Joint Moments to a Vector Summary

For a random vector, a mean is not a single number: it collects the means of the component variables. Covariance is also organized as a matrix, with each entry describing a pair of components. These summaries can be calculated from the vector’s joint law, but their existence depends on the integrability of the relevant moments.

Let \(X=(X_1,\ldots,X_p)^\mathsf{T}\) be a random vector. The first joint moments are the component expectations \(E[X_i]\); the second joint moments include \(E[X_iX_j]\) for every pair \(i,j\). The diagonal second moments \(E[X_i^2]\) determine variances, while the off-diagonal moments \(E[X_iX_j]\) are needed to determine covariances.

Definition: If \(E[|X_i|]<\infty\) for every \(i\), the mean vector is \(\mu=E[X]=(E[X_1],\ldots,E[X_p])^\mathsf{T}\). If, in addition, \(E[X_i^2]<\infty\) for every \(i\), the covariance matrix is the \(p\times p\) matrix \(\Sigma\) with entries \(\Sigma_{ij}=\operatorname{Cov}(X_i,X_j)=E[(X_i-\mu_i)(X_j-\mu_j)]\).

The definition makes explicit that a covariance entry is a central joint moment: it is an expectation of a product of deviations from the component means. When the second moments are finite, it can also be found from raw joint moments:

\Sigma_{ij}=E[X_iX_j]-E[X_i]E[X_j].

This formula is often the most direct route from a joint density or mass function to the covariance matrix. It also shows why separate marginal means and variances are not enough to determine the entire matrix: off-diagonal entries require joint information.

Moment Conditions and the Raw-Moment Formula

A finite covariance matrix requires finite second moments of every component. This condition also guarantees the first moments exist. To see this, if \(E[X_i^2]<\infty\), then Cauchy–Schwarz gives \(E[|X_i|]\leq \sqrt{E[X_i^2]}<\infty\). Conversely, if all component means and variances are finite, then \(E[X_i^2]=\operatorname{Var}(X_i)+E[X_i]^2<\infty\). Thus finite component second moments are precisely the condition needed for a finite covariance matrix.

Under this condition, the cross-products are integrable as well. Cauchy–Schwarz yields \(E[|X_iX_j|]\leq \sqrt{E[X_i^2]E[X_j^2]}<\infty\). We may therefore expand the centered product and take expectations term by term.

Proposition: If \(E[X_i^2]<\infty\) for every component, then \(\Sigma_{ij}=E[X_iX_j]-\mu_i\mu_j\) for all \(i,j\). Moreover, \(\Sigma\) is symmetric and positive semidefinite.

Proof. The finiteness condition above ensures that each term in the following expansion has a finite expectation. Using \(E[X_i]=\mu_i\) and \(E[X_j]=\mu_j\),

\begin{aligned} \Sigma_{ij} &=E[(X_i-\mu_i)(X_j-\mu_j)]\\ &=E[X_iX_j]-\mu_jE[X_i]-\mu_iE[X_j]+\mu_i\mu_j\\ &=E[X_iX_j]-\mu_i\mu_j. \end{aligned}

Interchanging \(i\) and \(j\) leaves this expression unchanged, so \(\Sigma_{ij}=\Sigma_{ji}\) and \(\Sigma\) is symmetric. For any fixed \(a\in\mathbb{R}^p\), all terms below are integrable because \(X\) has finite component second moments. Then

\begin{aligned} a^\mathsf{T}\Sigma a &=\sum_{i=1}^p\sum_{j=1}^p a_i a_j\Sigma_{ij}\\ &=E\left[\sum_{i=1}^p\sum_{j=1}^p a_i a_j(X_i-\mu_i)(X_j-\mu_j)\right]\\ &=E\left[\left\{\sum_{i=1}^p a_i(X_i-\mu_i)\right\}^2\right]\geq 0. \end{aligned}

The last expectation is nonnegative because it is the expectation of a square. Hence \(\Sigma\) is positive semidefinite. \(\square\)

In particular, \(\Sigma_{ii}=\operatorname{Var}(X_i)=E[X_i^2]-\mu_i^2\). A covariance matrix therefore has variances on its diagonal, and the symmetry and positive-semidefinite conditions provide useful checks on calculations.

Calculating Moments From a Joint Law

For a vector with joint density \(f\) on a support \(S\subseteq\mathbb{R}^p\), expectations are integrals over the support. For a discrete vector with joint mass function \(p(x)\), they are sums over its possible values. In either case, the same moment calculations apply: first find each component mean, then the raw second moments, and finally subtract products of means.

Calculation rule: For an integrable function \(g\), \(E[g(X)]=\int_S g(x)f(x)\,dx\) in the density case, and \(E[g(X)]=\sum_x g(x)p(x)\) in the discrete case. Use \(g(x)=x_i\), \(x_i^2\), or \(x_ix_j\) to obtain the moments needed for \(\mu\) and \(\Sigma\).

Worked Example: A Joint Mass Function on Three Points

Suppose \((X_1,X_2)\) takes the values \((0,0)\), \((1,0)\), and \((0,2)\) with probabilities \(1/2\), \(1/4\), and \(1/4\), respectively. The probabilities are nonnegative and sum to \(1/2+1/4+1/4=1\).

The component means are

E[X_1]=0\left(\frac12\right)+1\left(\frac14\right)+0\left(\frac14\right)=\frac14, \qquad E[X_2]=0\left(\frac12\right)+0\left(\frac14\right)+2\left(\frac14\right)=\frac12.

The raw second moments are

E[X_1^2]=\frac14,\qquad E[X_2^2]=4\left(\frac14\right)=1,\qquad E[X_1X_2]=0.

Applying the raw-moment formula gives

\begin{aligned} \operatorname{Var}(X_1)&=\frac14-\left(\frac14\right)^2=\frac{3}{16},\\ \operatorname{Var}(X_2)&=1-\left(\frac12\right)^2=\frac34,\\ \operatorname{Cov}(X_1,X_2)&=0-\left(\frac14\right)\left(\frac12\right)=-\frac18. \end{aligned}

Thus

\mu= \begin{pmatrix} 1/4\\ 1/2 \end{pmatrix}, \qquad \Sigma= \begin{pmatrix} 3/16&-1/8\\ -1/8&3/4 \end{pmatrix}.

As a check on positive semidefiniteness, the determinant is \((3/16)(3/4)-(-1/8)^2=9/64-1/64=1/8>0\), and the diagonal entries are positive. The covariance matrix is symmetric, as required.

Worked Example: Integrating Over a Triangular Support

Let \((X,Y)\) have joint density \(f(x,y)=2\) on \(0<y<x<1\), and \(f(x,y)=0\) elsewhere. The support has area \(\int_0^1\int_0^x dy\,dx=1/2\), so the density integrates to \(2(1/2)=1\).

Integrate first with respect to \(y\) to find the mean of \(X\), and reverse the order of integration for the mean of \(Y\):

\begin{aligned} E[X]&=\int_0^1\int_0^x 2x\,dy\,dx =2\int_0^1x^2\,dx=\frac23,\\ E[Y]&=\int_0^1\int_y^1 2y\,dx\,dy =2\int_0^1y(1-y)\,dy=\frac13. \end{aligned}

The needed second joint moments follow from the same support limits:

\begin{aligned} E[X^2]&=2\int_0^1x^2\left(\int_0^xdy\right)dx =2\int_0^1x^3dx=\frac12,\\ E[Y^2]&=2\int_0^1y^2\left(\int_y^1dx\right)dy =2\int_0^1y^2(1-y)dy=\frac16,\\ E[XY]&=\int_0^1\int_0^x 2xy\,dy\,dx =\int_0^1x^3dx=\frac14. \end{aligned}

Therefore,

\operatorname{Var}(X)=\frac12-\left(\frac23\right)^2=\frac1{18}, \qquad \operatorname{Var}(Y)=\frac16-\left(\frac13\right)^2=\frac1{18}, \qquad \operatorname{Cov}(X,Y)=\frac14-\frac23\frac13=\frac1{36}.

The mean vector is \((2/3,1/3)^\mathsf{T}\), and the covariance matrix has diagonal entries \(1/18\) and off-diagonal entries \(1/36\). The nonrectangular support matters in every integral: using the unit square as the integration region would calculate moments under a different law.

What Happens When Second Moments Do Not Exist?

A mean vector can exist even when a finite covariance matrix does not. This distinction is important: having finite first moments is enough to define \(\mu\), but not enough to justify the covariance entries or the raw-moment formula. The following example shows why checking moment conditions must come before calculating a covariance matrix.

Worked Example: A Finite Mean but Infinite Second Moment

Let \(Z\) have density \(f(z)=\frac32 z^{-5/2}\) for \(z\geq1\), and define the two-component vector \(X=(Z,0)^\mathsf{T}\). The density is normalized because

\int_1^\infty \frac32 z^{-5/2}\,dz =\frac32\left[\frac{-2}{3}z^{-3/2}\right]_1^\infty=1.

Its first moment is finite:

E[Z]=\frac32\int_1^\infty z^{-3/2}\,dz =\frac32\left[-2z^{-1/2}\right]_1^\infty=3.

But the second moment diverges:

E[Z^2]=\frac32\int_1^\infty z^{-1/2}\,dz=\infty.

Thus \(E[X]=(3,0)^\mathsf{T}\) exists, but \(\operatorname{Var}(Z)\) is not finite and \(X\) has no finite covariance matrix. In particular, the expression \(E[Z^2]-E[Z]^2\) cannot be treated as a finite covariance entry.

Common Mistake: Using Marginal Moments Alone

A frequent error is to calculate the two component variances and assume that they determine the covariance matrix. They determine only its diagonal. For \(i\ne j\), the covariance depends on the joint raw moment \(E[X_iX_j]\), so the joint law—or equivalent information about that cross-moment—is necessary.

A related error is to call \(E[X_iX_j]\) the covariance. It is the raw joint second moment. The correction is to subtract the product of the means: \(\operatorname{Cov}(X_i,X_j)=E[X_iX_j]-E[X_i]E[X_j]\). Before applying this identity, check that the relevant second moments are finite; otherwise the displayed quantities may not be finite and the calculation is not justified.

The mean vector and covariance matrix provide a compact description of location and second-order variation. Their value depends on careful calculation from the joint law and on distinguishing finite first moments from finite second moments. Symmetry and positive semidefiniteness are useful checks, but they do not replace verifying the integrals or sums that define the entries.

Check Your Understanding

Use the definitions and moment conditions to answer these questions.

For a vector with finite component second moments, express \(\Sigma_{ij}\) using \(E[X_iX_j]\) and the component means.
What condition on the component second moments guarantees that the mean vector and every covariance entry are finite?
For a discrete vector, what joint probability information is needed to calculate an off-diagonal covariance?
Why must a covariance matrix be symmetric and positive semidefinite?
Can a mean vector exist when a finite covariance matrix does not? Explain the distinction using the moment conditions.