Moments, Variance, and Covariance

Why Moments Measure More Than the Mean

The mean describes a distribution’s center, but it does not describe how widely its values vary around that center. Variance measures this spread by averaging squared deviations. Covariance extends the same idea to two random variables: it records whether their deviations from their respective means tend to have the same or opposite signs. These quantities are useful for describing distributions and, in particular, for finding the variance of a sum when its terms may be dependent.

Work on a probability space \((\Omega,\mathcal{F},\mathbb{P})\). A random variable \(X\) has a discrete law with probability mass function \(p_X(x)=\mathbb{P}(X=x)\), or a continuous law with density \(f_X\), as appropriate. The expectation formulas below assume the relevant sums or integrals exist; for variance and covariance we will require finite second moments.

For a discrete random variable, these expectations are probability-weighted sums. For a continuous random variable with density \(f_X\), they are integrals against that density. In particular, the variance can be calculated either by summing or integrating squared deviations from the mean, or by using the centering identity proved below.

The sign of covariance summarizes a pattern of joint variation, not the full joint distribution. In particular, the covariance is defined from the joint law of \(X\) and \(Y\), not from their marginal laws alone. When two variables are dependent, the covariance term can change the variance of their sum.

Centering Identities

Proof. Write \(\mu_X=\mathbb{E}[X]\) and \(\mu_Y=\mathbb{E}[Y]\). Expanding the square in the definition of variance and using linearity of expectation gives \(\mathbb{E}[(X-\mu_X)^2]=\mathbb{E}[X^2-2\mu_X X+\mu_X^2]=\mathbb{E}[X^2]-2\mu_X\mathbb{E}[X]+\mu_X^2=\mathbb{E}[X^2]-\mu_X^2\). Expanding the product in the definition of covariance gives \(\mathbb{E}[(X-\mu_X)(Y-\mu_Y)]=\mathbb{E}[XY]-\mu_X\mathbb{E}[Y]-\mu_Y\mathbb{E}[X]+\mu_X\mu_Y=\mathbb{E}[XY]-\mu_X\mu_Y\). All terms are integrable under the finite-second-moment assumptions; in particular, the product \(XY\) is integrable by the Cauchy–Schwarz inequality. Finally, the centered version of \(aX+bY\) is \(a(X-\mu_X)+b(Y-\mu_Y)\). Squaring this expression, taking expectations, and applying the definitions of variance and covariance yields the stated formula. \(\square\)

For \(a=b=1\), the sum rule is \(\operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y)\). It requires finite second moments, but it does not require independence. If the covariance is positive, the variance of the sum exceeds the sum of the individual variances; if it is negative, it is smaller. For several variables, expanding the square gives the corresponding rule: the variance of a finite sum is the sum of the individual variances plus twice the sum of the covariances over all distinct pairs.

Moments from a Density

If \(X\) has density \(f_X\), the mean and second raw moment are integrals: \(\mathbb{E}[X]=\int_{-\infty}^{\infty}x f_X(x)\,dx\) and \(\mathbb{E}[X^2]=\int_{-\infty}^{\infty}x^2 f_X(x)\,dx\), when they are finite. The density must be nonnegative and integrate to \(1\). The variance can then be found either by integrating \((x-\mu_X)^2f_X(x)\) or by subtracting the square of the mean from the second raw moment.

Covariance and Dependent Sums

To calculate covariance from a joint discrete law, multiply each pair of values by its joint probability and sum to obtain \(\mathbb{E}[XY]\). Then subtract the product of the means. For a joint density, replace that sum by the corresponding double integral. In either case, the joint law matters because it determines which pairs of values occur together.

Common Pitfalls and a Practical Check

A common mistake is to call the second raw moment the variance. In the first example, \(\mathbb{E}[X^2]=\tfrac54\), but the variance is \(\tfrac54-(\tfrac14)^2=\tfrac{19}{16}\). The missing term is the square of the mean. This distinction matters whenever the mean is not zero.

Another mistake is to calculate the variance of a sum by adding only the individual variances. For the dependent pair above, that incorrect calculation gives \(\tfrac{15}{64}+\tfrac{15}{64}=\tfrac{30}{64}\), whereas the correct answer is \(\tfrac{44}{64}\). The omitted term is \(2\operatorname{Cov}(X,Y)=\tfrac{14}{64}\). The formula for the sum always includes covariance; it is not permissible to omit it merely because the variables are being added.

For any proposed calculation, useful checks include verifying that discrete probabilities sum to \(1\) or that a density integrates to \(1\), checking that the variance is nonnegative, and substituting the mean and second moment into the centering identity. For a sum of two variables, a direct calculation from the joint law can also check the covariance-based result, as in the example.