Covariance from Joint Distributions

Why the Joint Distribution Matters

The means of two random variables describe their centers separately; covariance describes how their values vary together. To calculate it, we need information about which pairs of values can occur and how likely those pairs are. That information is contained in the joint distribution, not in the two marginal distributions alone.

For a discrete pair, the joint distribution assigns probabilities to pairs of values. For a continuous pair with a joint density, it assigns probability through integration over regions in the plane. In either case, the calculation has two useful forms: subtract the product of the means from the expected product, or take the expectation of the product of the centered variables.

Definition: If \(X\) and \(Y\) have finite second moments, their covariance is

\operatorname{Cov}(X,Y) = E\big[(X-E[X])(Y-E[Y])\big].

The quantities \(E[X]\) and \(E[Y]\) are the means of the two variables. Covariance summarizes the expected product of their deviations from those means.

The finite-second-moment condition, \(E[X^2]<\infty\) and \(E[Y^2]<\infty\), ensures that the centered product has a finite expectation. Indeed, for real numbers \(a,b\), \(2|ab|\leq a^2+b^2\). Applying this with \(a=X-E[X]\) and \(b=Y-E[Y]\) shows that the absolute value of the centered product has finite expectation. Thus the definition is well-defined under the stated condition.

The Raw-Moment Identity

Calculations from a joint table or density are often most direct when expressed in terms of the means and the expected product \(E[XY]\). The two ways of writing covariance are equivalent.

Theorem: If \(E[X^2]<\infty\) and \(E[Y^2]<\infty\), then

\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y].

Proof. Write \(\mu_X=E[X]\) and \(\mu_Y=E[Y]\). The finite-second-moment condition ensures the terms in the following expansion are integrable: \(E[|XY|]<\infty\) follows from \(2|XY|\leq X^2+Y^2\), and \(E[|X|]\) and \(E[|Y|]\) are finite. Expand the product inside the definition and use linearity of expectation:

\begin{aligned} \operatorname{Cov}(X,Y) &=E[(X-\mu_X)(Y-\mu_Y)]\\ &=E[XY-\mu_YX-\mu_XY+\mu_X\mu_Y]\\ &=E[XY]-\mu_YE[X]-\mu_XE[Y]+\mu_X\mu_Y\\ &=E[XY]-\mu_X\mu_Y. \end{aligned}

Since \(\mu_X=E[X]\) and \(\mu_Y=E[Y]\), this is the claimed identity. \(\square\)

Covariance from a Joint Probability Table

Suppose \(X\) takes values \(x_i\), \(Y\) takes values \(y_j\), and their joint probability mass function is \(p_{ij}=P(X=x_i,Y=y_j)\). For a finite table, the means and expected product are obtained by summing across the joint probabilities:

E[X]=\sum_i\sum_j x_i p_{ij},\qquad E[Y]=\sum_i\sum_j y_j p_{ij},\qquad E[XY]=\sum_i\sum_j x_i y_j p_{ij}.

The same formulas apply to a countable support when the relevant expectations are finite. The marginal probabilities can be found by summing a row or column, but the expected product uses each pair \((x_i,y_j)\) and its joint probability.

Worked Example: Computing Covariance from a Probability Table

Suppose the joint probabilities for \(X\in\{0,1\}\) and \(Y\in\{0,2\}\) are as follows. The entries in the body of the table are joint probabilities.

\(X\backslash Y\)	\(Y=0\)	\(Y=2\)	Row total
\(X=0\)	\(1/2\)	\(1/4\)	\(3/4\)
\(X=1\)	\(1/4\)	\(0\)	\(1/4\)
Column total	\(3/4\)	\(1/4\)	\(1\)

First check that the probabilities are valid: all entries are nonnegative and their sum is \(1/2+1/4+1/4+0=1\). The row and column totals give the marginal probabilities. Therefore

E[X]=0\left(\frac34\right)+1\left(\frac14\right)=\frac14, \qquad E[Y]=0\left(\frac34\right)+2\left(\frac14\right)=\frac12.

Only the pair \((1,2)\) contributes a nonzero value to \(XY\), and that pair has probability \(0\). Hence

E[XY] =(0)(0)\frac12+(0)(2)\frac14+(1)(0)\frac14+(1)(2)(0) =0.

Using the raw-moment identity,

\operatorname{Cov}(X,Y) =E[XY]-E[X]E[Y] =0-\frac14\cdot\frac12 =-\frac18.

The negative value reflects the pattern in this table: when \(X=1\), \(Y=0\), while the positive value of \(Y\) occurs only when \(X=0\). The calculation uses the joint probabilities, including the zero probability of the pair \((1,2)\).

Covariance from a Joint Density

If \((X,Y)\) has joint density \(f_{X,Y}(x,y)\), then expectations are integrals over the support of that density. Under finite second moments, the corresponding formulas are

\begin{aligned} E[X]&=\iint x f_{X,Y}(x,y)\,dx\,dy,\\ E[Y]&=\iint y f_{X,Y}(x,y)\,dx\,dy,\\ E[XY]&=\iint xy f_{X,Y}(x,y)\,dx\,dy, \end{aligned}

where each integral is over the joint support. These are the expectation formulas for a function of a random vector: the quantity being averaged is respectively \(x\), \(y\), or \(xy\), weighted by the joint density. The covariance then follows by subtracting the product of the two means from \(E[XY]\).

Worked Example: A Joint Density on a Triangle

Let the joint density be \(f_{X,Y}(x,y)=2\) on the triangular region \(0<y<x<1\), and \(0\) elsewhere. It is a valid density because

\int_0^1\int_0^x 2\,dy\,dx =\int_0^1 2x\,dx =1.

Integrate over the support, first with respect to \(y\) from \(0\) to \(x\), then with respect to \(x\) from \(0\) to \(1\). The means are

\begin{aligned} E[X] &=\int_0^1\int_0^x 2x\,dy\,dx =\int_0^1 2x^2\,dx =\frac23,\\ E[Y] &=\int_0^1\int_0^x 2y\,dy\,dx =\int_0^1 x^2\,dx =\frac13. \end{aligned}

For the expected product,

E[XY] =\int_0^1\int_0^x 2xy\,dy\,dx =\int_0^1 x^3\,dx =\frac14.

Thus

\operatorname{Cov}(X,Y) =\frac14-\frac23\cdot\frac13 =\frac14-\frac29 =\frac{9-8}{36} =\frac1{36}.

The support condition \(Y<X\) constrains which pairs can occur, and the joint density accounts for that constraint throughout the integrals. The final covariance is positive, even though the support itself is triangular.

Worked Example: Uniform Density on a Different Triangle

Let \(f_{X,Y}(x,y)=2\) for \(x>0\), \(y>0\), and \(x+y<1\), and let it be zero elsewhere. The density integrates to one:

\int_0^1\int_0^{1-x}2\,dy\,dx =\int_0^1 2(1-x)\,dx =1.

For the mean of \(X\),

E[X] =\int_0^1\int_0^{1-x}2x\,dy\,dx =\int_0^1 2x(1-x)\,dx =2\left(\frac12-\frac13\right) =\frac13.

The density and support are unchanged when \(x\) and \(y\) are exchanged, so the same calculation gives \(E[Y]=1/3\). For the product,

\begin{aligned} E[XY] &=\int_0^1\int_0^{1-x}2xy\,dy\,dx\\ &=\int_0^1 x(1-x)^2\,dx\\ &=\int_0^1 (x-2x^2+x^3)\,dx\\ &=\frac12-\frac23+\frac14 =\frac1{12}. \end{aligned}

Consequently,

\operatorname{Cov}(X,Y) =\frac1{12}-\frac13\cdot\frac13 =\frac1{12}-\frac19 =-\frac1{36}.

Here the sum \(X+Y\) is constrained to be less than \(1\): larger values of one coordinate leave less room for the other. The negative covariance is consistent with this restriction.

Centered Deviations and a Calculation Check

The raw-moment identity is convenient for arithmetic, while the defining expression explains what covariance measures. In a discrete model, each possible pair contributes the product \((x_i-E[X])(y_j-E[Y])\), weighted by its joint probability. In a density model, the same centered product is integrated against the joint density.

For example, in the probability-table calculation, the centered form gives

\begin{aligned} E[(X-\tfrac14)(Y-\tfrac12)] &=\frac12\left(-\frac14\right)\left(-\frac12\right) +\frac14\left(-\frac14\right)\left(\frac32\right) +\frac14\left(\frac34\right)\left(-\frac12\right)\\ &=\frac1{16}-\frac3{32}-\frac3{32} =-\frac18, \end{aligned}

matching the raw-moment result. This is a useful check when a table is small; for larger tables, the raw-moment calculation is often shorter.

Common Mistakes

Using only the means. The means do not determine covariance. The expected product \(E[XY]\), or equivalently the joint distribution, is also needed.
Multiplying the means in place of averaging the product. In general, \(E[XY]\) and \(E[X]E[Y]\) are different quantities. Covariance is their difference, not either quantity by itself.
Ignoring the support of a density. In a triangular model, integrating over the whole unit square uses regions where the density is zero or invalid. Set the integration limits from the actual joint support.
Dropping zero-probability pairs without checking the table. A zero entry contributes nothing to \(E[XY]\), but it can be important to the pattern of possible pairs and to interpreting the result.

Key takeaway: With finite second moments, compute covariance as \(E[XY]-E[X]E[Y]\), using the joint probability table or density to find each expectation. Equivalently, it is the expected product of the variables' centered deviations.

Check Your Understanding

Use the joint distribution in each question, and show the expectations needed for your calculation.

For the discrete table in the first worked example, calculate \(E[(X-E[X])(Y-E[Y])]\) directly by summing over its four pairs.
For a joint density, what region determines the limits of integration when calculating \(E[XY]\)?
Suppose \(E[X]=2\), \(E[Y]=3\), and \(E[XY]=7\). Find \(\operatorname{Cov}(X,Y)\).
Why does knowing \(E[X]\) and \(E[Y]\) alone not determine \(\operatorname{Cov}(X,Y)\)?
State the moment condition used here to ensure covariance is well-defined, and explain how \(2|ab|\leq a^2+b^2\) helps.