Tutorials › Statistical Theory I - Probability and Distribution Theory › Variance Addition for Independent Variables

Home  /  Tutorials  /  Variance and Covariance Structure

Variance and Covariance Structure · Tutorial 362 of 994

Variance Addition for Independent Variables

See why independence removes cross terms in the variance of a sum, then calculate variances for weighted sums of independent variables.

Intermediate 10 min read

What You'll Learn

  • State the finite-variance conditions for variance addition.
  • Prove that independent variables have zero covariance.
  • Derive the variance formula for a weighted sum.
  • Distinguish pairwise independence from mutual independence for this result.
  • Calculate weighted-sum variances for Bernoulli, Poisson, and discrete uniform variables.
  • Identify why independence is sufficient but not necessary.

Why Independence Simplifies Variance

When several sources of random variation are added together, their individual variances do not always add. The way the variables move together matters: one variable may tend to be high when another is low, partly cancelling the total variation, or they may tend to be high together. Independence rules out this kind of linear association and makes the variance of a sum especially simple.

The result is useful in many settings: combining independent measurement errors, adding independent counts, or studying a weighted total. The weights matter because multiplying a variable by \(a\) multiplies its variance by \(a^2\), not by \(a\). We will establish the addition rule, derive its weighted form, and check the calculations in several models.

Definition: A finite collection \(X_1,\ldots,X_n\) is mutually independent if their joint distribution is the product of their marginal distributions. The collection is pairwise independent if every pair \(X_i,X_j\), with \(i\ne j\), is independent. Mutual independence implies pairwise independence, but the reverse need not hold in general.

Throughout, each variable considered in a variance formula has a finite second moment: \(E[X_i^2]<\infty\). This guarantees that its mean and variance are finite. The number of variables \(n\) is finite, and all weights in a weighted sum are fixed real numbers.

Variance Addition for Independent Variables

The key step is that independent variables with finite second moments have zero covariance. To see why the moment condition is relevant, let \(U\) and \(V\) be independent and integrable. Their joint distribution is the product of their marginal distributions. For nonnegative measurable functions \(g\) and \(h\), Tonelli's theorem therefore gives

$$ E[g(U)h(V)] =\int g(u)h(v)\,dP_U(u)\,dP_V(v) =E[g(U)]E[h(V)]. $$

For signed integrable functions, apply this identity to positive and negative parts whenever the product is integrable. In particular, if \(U\) and \(V\) have finite second moments, then their centered product is integrable by the Cauchy–Schwarz inequality. Taking \(g(U)=U-E[U]\) and \(h(V)=V-E[V]\) gives the zero-covariance conclusion.

Theorem: If \(X_1,\ldots,X_n\) are pairwise independent and have finite variances, then $$ \operatorname{Var}\!\left(\sum_{i=1}^n X_i\right) =\sum_{i=1}^n \operatorname{Var}(X_i). $$ In particular, the result holds if the variables are mutually independent. Pairwise independence is enough for this variance identity; mutual independence is not required.

Proof. Write \(\mu_i=E[X_i]\), which is finite because \(E[X_i^2]<\infty\). Pairwise independence and the factorization above imply, for \(i\ne j\),

$$ E[(X_i-\mu_i)(X_j-\mu_j)] =E[X_i-\mu_i]E[X_j-\mu_j] =0. $$

Each centered variable is square-integrable. Since the sum is finite, its centered sum is also square-integrable, and we can expand its square:

$$ \begin{aligned} \operatorname{Var}\!\left(\sum_{i=1}^n X_i\right) &=E\!\left[\left(\sum_{i=1}^n (X_i-\mu_i)\right)^2\right]\\ &=\sum_{i=1}^n E[(X_i-\mu_i)^2] +2\sum_{1\le i<j\le n}E[(X_i-\mu_i)(X_j-\mu_j)]\\ &=\sum_{i=1}^n \operatorname{Var}(X_i). \end{aligned} $$

The cross terms vanish by pairwise independence, while each diagonal term is the variance of one variable. This proves the result. \(\square\)

This argument also shows exactly where independence enters: it removes each cross term in the centered-square expansion. The variance of a sum is not determined by the individual variances alone when those cross terms are nonzero.

Weighted Sums

A weighted sum has the form \(S=\sum_{i=1}^n a_iX_i\), where the \(a_i\) are fixed constants. Scaling a centered variable by \(a_i\) scales its square by \(a_i^2\). Applying the same argument as above yields the weighted addition rule.

Theorem: If \(X_1,\ldots,X_n\) are pairwise independent with finite variances and \(a_1,\ldots,a_n\) are real constants, then $$ \operatorname{Var}\!\left(\sum_{i=1}^n a_iX_i\right) =\sum_{i=1}^n a_i^2\operatorname{Var}(X_i). $$

Derivation. Let \(\mu_i=E[X_i]\). The centered weighted sum is \(\sum_i a_i(X_i-\mu_i)\). Expand its square, take expectations, and separate diagonal and cross terms:

$$ \begin{aligned} \operatorname{Var}\!\left(\sum_{i=1}^n a_iX_i\right) &=E\!\left[\left(\sum_{i=1}^n a_i(X_i-\mu_i)\right)^2\right]\\ &=\sum_{i=1}^n a_i^2 E[(X_i-\mu_i)^2] +2\sum_{1\le i<j\le n}a_i a_j E[(X_i-\mu_i)(X_j-\mu_j)]\\ &=\sum_{i=1}^n a_i^2\operatorname{Var}(X_i). \end{aligned} $$

The last equality follows because every cross expectation is zero for a pair of independent variables. A negative weight changes the direction in which a variable contributes to the sum, but its variance contribution remains nonnegative because the weight is squared. The mean, unlike the variance, is weighted without squaring: \(E[S]=\sum_i a_iE[X_i]\).

Worked Example: A Weighted Sum of Independent Bernoulli Variables

Let \(X_1,X_2,X_3\) be independent Bernoulli variables with success probabilities \(0.2\), \(0.5\), and \(0.8\), respectively. Find the mean and variance of \(S=X_1+2X_2-X_3\).

For a Bernoulli variable with success probability \(p\), the mean is \(p\) and the variance is \(p(1-p)\). Thus

$$ \begin{aligned} E[X_1]&=0.2, &\operatorname{Var}(X_1)&=0.2(0.8)=0.16,\\ E[X_2]&=0.5, &\operatorname{Var}(X_2)&=0.5(0.5)=0.25,\\ E[X_3]&=0.8, &\operatorname{Var}(X_3)&=0.8(0.2)=0.16. \end{aligned} $$

By linearity of expectation,

$$ E[S]=0.2+2(0.5)-0.8=0.4. $$

For the variance, the weights are \(1,2,-1\), so their squares are \(1,4,1\):

$$ \operatorname{Var}(S) =1^2(0.16)+2^2(0.25)+(-1)^2(0.16) =0.16+1+0.16 =1.32. $$

The negative sign on \(X_3\) affects the mean but not the size of its variance contribution. As a check, \(S\) has possible values from \(-1\) to \(3\), so a variance of \(1.32\) is consistent with its scale.

Worked Example: Unequal Weights for Independent Poisson Counts

Suppose \(X\sim\operatorname{Poisson}(2)\) and \(Y\sim\operatorname{Poisson}(5)\) are independent. Find the mean and variance of \(T=3X-2Y\).

A Poisson variable with parameter \(\lambda\) has mean and variance both equal to \(\lambda\). Therefore \(E[X]=2\), \(\operatorname{Var}(X)=2\), \(E[Y]=5\), and \(\operatorname{Var}(Y)=5\). The mean is

$$ E[T]=3E[X]-2E[Y]=3(2)-2(5)=-4. $$

The weighted variance rule gives

$$ \operatorname{Var}(T) =3^2\operatorname{Var}(X)+(-2)^2\operatorname{Var}(Y) =9(2)+4(5) =38. $$

The result is not \(3(2)+(-2)(5)\): variance scales with the square of each weight. The mean \(-4\) and variance \(38\) describe different features of \(T\), and a negative mean does not imply a negative variance.

Worked Example: Discrete Uniform Variables With Fractional Weights

Let \(X\) and \(Y\) be independent. Let \(X\) assign probability \(1/3\) to each of \(-1,0,1\), and let \(Y\) assign probability \(1/2\) to each of \(0,2\). Find the variance of \(R=2X+\tfrac12Y\).

For \(X\), symmetry gives \(E[X]=0\), and

$$ \operatorname{Var}(X)=E[X^2] =\frac{1+0+1}{3} =\frac{2}{3}. $$

For \(Y\), \(E[Y]=(0+2)/2=1\), and

$$ \operatorname{Var}(Y) =E[(Y-1)^2] =\frac{(0-1)^2+(2-1)^2}{2} =1. $$

Both variables have finite variances, and their independence allows us to add their weighted contributions:

$$ \operatorname{Var}(R) =2^2\left(\frac{2}{3}\right) +\left(\frac12\right)^2(1) =\frac{8}{3}+\frac14 =\frac{32}{12}+\frac{3}{12} =\frac{35}{12}. $$

This calculation works just as it did with integer weights: each variance is multiplied by the square of its coefficient.

What Independence Does—and Does Not—Guarantee

Independence is a sufficient condition for variance addition, not a necessary one. The expansion of the centered square shows that the same result holds whenever the cross expectations are zero—that is, whenever the variables are pairwise uncorrelated and have finite variances. Independence guarantees this zero-covariance condition, but zero covariance alone does not generally guarantee independence. In applications, do not infer independence merely because a variance-addition formula looks convenient.

The moment conditions also matter. The identity concerns variables with finite variances; if a variance is infinite or undefined, the finite-variance expansion above cannot be used as stated. Finally, the result concerns a finite sum. A series with infinitely many random terms requires additional convergence and integrability conditions, so the finite-sum proof does not by itself justify an infinite variance sum.

Common Mistakes

  • Using the weights instead of their squares. For \(aX\), the variance is \(a^2\operatorname{Var}(X)\). A weight of \(-2\) contributes four times the original variance, not negative twice the variance.
  • Adding variances just because the variables appear in a sum. The rule requires zero cross terms; independence is a standard sufficient condition. Dependent variables can have nonzero covariance, so their individual variances alone do not determine the total.
  • Assuming pairwise independence and mutual independence are interchangeable. Pairwise independence is enough for this variance identity, but it does not imply mutual independence in general.
  • Confusing the variance rule with the mean rule. Means use the original weights, \(E[\sum_i a_iX_i]=\sum_i a_iE[X_i]\); variances use squared weights when cross terms vanish.
Key takeaway: For a finite weighted sum of pairwise independent, finite-variance variables, each variable contributes \(a_i^2\operatorname{Var}(X_i)\). Independence eliminates cross terms; squaring the weights keeps every variance contribution nonnegative.

Check Your Understanding

Answer each question by identifying the relevant assumptions and showing the calculation where appropriate.

  1. Three pairwise independent variables have variances \(2\), \(5\), and \(1\). What is the variance of their sum?
  2. If \(X\) and \(Y\) are independent with variances \(4\) and \(9\), find \(\operatorname{Var}(2X-\tfrac13Y)\).
  3. Why does a negative coefficient not create a negative contribution to a weighted-sum variance?
  4. Is mutual independence necessary for the variance-addition identity for a finite collection? Explain which independence condition suffices.
  5. State one condition weaker than independence that still makes the variance of a finite sum equal the sum of its component variances.

Previous

Next

Back to Tutorials