Total Expectation and Total Variance Calculations

From Conditional Summaries to Overall Summaries

A model may describe a population as a collection of groups, or describe an outcome conditional on an unobserved class or random parameter. In either case, the overall mean depends on both the group-specific means and their probabilities. The overall variance has two sources: variation within groups, and variation because different groups have different means. Separating these contributions makes mixture and hierarchical calculations easier to organize and easier to check.

Work on a probability space $(\Omega,\mathcal{F},\mathbb{P})$. Let $X$ be a random variable with finite second moment, and let $C$ be a label taking values in a finite set $\{1,\ldots,r\}$. Write $A_g=\{C=g\}$ and $w_g=\mathbb{P}(A_g)$. When $w_g>0$, define the group-specific mean and variance by $\mu_g=\mathbb{E}[X\mid A_g]$ and $\sigma_g^2=\operatorname{Var}(X\mid A_g)$. Groups with $w_g=0$ contribute nothing to unconditional averages, and their conditional summaries need not be defined.

Definition: The within-group variance is the weighted average $\sum_g w_g\sigma_g^2$. The between-group variance is the variance of the group means under the group probabilities: $\sum_g w_g(\mu_g-\bar{\mu})^2$, where $\bar{\mu}=\sum_g w_g\mu_g$. The weights are probabilities, so $\sum_g w_g=1$.

The group label can be observed, as in a comparison of known classes, or latent, as in a mixture model where class membership is not observed. The calculation is the same: first find each group's probability, conditional mean, and conditional variance; then combine them. These formulas apply when the overall first and second moments exist. For the finite mixtures below, finite conditional second moments in each positive-probability group ensure that condition.

The Two Total-Law Calculations

Theorem (Law of total expectation for a finite partition): If $A_1,\ldots,A_r$ form a partition of $\Omega$ and $X$ is integrable, then $$ \mathbb{E}[X]=\sum_{g:w_g>0}w_g\mathbb{E}[X\mid A_g]. $$

Proof. Since the events partition the space, $X=\sum_{g=1}^r X\mathbf{1}_{A_g}$. Taking expectations and using linearity gives $\mathbb{E}[X]=\sum_g\mathbb{E}[X\mathbf{1}_{A_g}]$. For $w_g>0$, the definition of conditional expectation on an event gives $\mathbb{E}[X\mathbf{1}_{A_g}]=\mathbb{P}(A_g)\mathbb{E}[X\mid A_g]$. If $w_g=0$, then $\mathbf{1}_{A_g}=0$ almost surely, so that term is zero. Combining the terms proves the formula. $\square$

Theorem (Law of total variance for a finite partition): If $A_1,\ldots,A_r$ form a partition and $X$ has finite second moment, then $$ \operatorname{Var}(X)=\sum_{g:w_g>0}w_g\sigma_g^2 +\sum_{g:w_g>0}w_g(\mu_g-\bar{\mu})^2, \qquad \bar{\mu}=\sum_{g:w_g>0}w_g\mu_g. $$

Proof. The expectation theorem gives $\mathbb{E}[X]=\bar{\mu}$. On each positive-probability event $A_g$, write $X-\bar{\mu}=(X-\mu_g)+(\mu_g-\bar{\mu})$, square, and take the conditional expectation given $A_g$. The first squared term has conditional expectation $\sigma_g^2$. The cross term has conditional expectation zero because $\mathbb{E}[X-\mu_g\mid A_g]=0$. The last term is constant on $A_g$. Thus $\mathbb{E}[(X-\bar{\mu})^2\mid A_g]=\sigma_g^2+(\mu_g-\bar{\mu})^2$. Averaging over the partition, with zero-probability groups contributing zero, gives the stated identity. $\square$

The same decomposition can be written for a conditioning random variable $Z$, not just a finite label. Provided $X$ has finite second moment, the conditional expectation and variance formulas are $$ \mathbb{E}[X]=\mathbb{E}\big[\mathbb{E}[X\mid Z]\big], \qquad \operatorname{Var}(X)=\mathbb{E}\big[\operatorname{Var}(X\mid Z)\big] +\operatorname{Var}\big(\mathbb{E}[X\mid Z]\big). $$

Here conditioning on $Z$ means conditioning on the information $\sigma(Z)$ generated by $Z$. The finite-partition formulas are the special case in which $Z$ is a finite-valued group label. For a continuous or otherwise non-finite $Z$, conditional means and variances are random variables, and the outer expectation or variance averages over the distribution of $Z$. These equalities hold almost surely for conditional quantities; they do not require conditioning on a particular value $Z=z$ of probability zero.

One way to see the variance identity is to set $M=\mathbb{E}[X\mid Z]$ and expand $X-\mathbb{E}[X]=(X-M)+(M-\mathbb{E}[X])$. The cross term has expectation zero: conditioning on $Z$, the factor $M-\mathbb{E}[X]$ is known and $\mathbb{E}[X-M\mid Z]=0$. The squared terms average to the expected conditional variance and the variance of $M$, respectively. The finite-partition proof above gives the same argument with each conditional calculation performed group by group.

Worked Calculations

Worked Example: A Two-Class Mixture

A latent class $C$ equals $1$ with probability $\tfrac13$ and $2$ with probability $\tfrac23$. Given $C=1$, $X$ is equally likely to be $0$ or $2$. Given $C=2$, $X$ is equally likely to be $2$ or $6$. Find the mean and variance of $X$, and separate the two sources of variance.

Within class 1, the mean is $\mu_1=\tfrac12(0+2)=1$, and the variance is $\sigma_1^2=\tfrac12(0-1)^2+\tfrac12(2-1)^2=1$. Within class 2, the mean is $\mu_2=\tfrac12(2+6)=4$, and the variance is $\sigma_2^2=\tfrac12(2-4)^2+\tfrac12(6-4)^2=4$.

The overall mean is $$ \bar{\mu}=\tfrac13(1)+\tfrac23(4)=\tfrac13+\tfrac83=3. $$ The within-group contribution is $\tfrac13(1)+\tfrac23(4)=\tfrac13+\tfrac83=3$. The between-group contribution is $$ \tfrac13(1-3)^2+\tfrac23(4-3)^2 =\tfrac13(4)+\tfrac23(1)=2. $$ Therefore $\operatorname{Var}(X)=3+2=5$.

Check by direct moments. Unconditionally, $\mathbb{P}(X=0)=\tfrac13\cdot\tfrac12=\tfrac16$, $\mathbb{P}(X=2)=\tfrac13\cdot\tfrac12+\tfrac23\cdot\tfrac12=\tfrac12$, and $\mathbb{P}(X=6)=\tfrac23\cdot\tfrac12=\tfrac13$. These probabilities sum to $\tfrac16+\tfrac12+\tfrac13=1$. Hence $\mathbb{E}[X]=0\cdot\tfrac16+2\cdot\tfrac12+6\cdot\tfrac13=3$, and $\mathbb{E}[X^2]=0^2\cdot\tfrac16+2^2\cdot\tfrac12+6^2\cdot\tfrac13=2+12=14$. Thus $\operatorname{Var}(X)=14-3^2=5$, agreeing with the decomposition.

Worked Example: A Binomial Model with a Random Success Probability

Let a latent success probability $\Theta$ equal $\tfrac14$ or $\tfrac34$, each with probability $\tfrac12$. Given $\Theta=\theta$, let $X$ be the number of successes in four conditionally independent trials, each with success probability $\theta$. Find the unconditional mean and variance.

For fixed $\theta$, $X$ is binomial with mean $4\theta$ and variance $4\theta(1-\theta)$. At $\theta=\tfrac14$, these are $1$ and $4\cdot\tfrac14\cdot\tfrac34=\tfrac34$. At $\theta=\tfrac34$, they are $3$ and $4\cdot\tfrac34\cdot\tfrac14=\tfrac34$. Therefore $$ \mathbb{E}[X]=\tfrac12(1)+\tfrac12(3)=2. $$

The expected within-$\Theta$ variance is $\tfrac12(\tfrac34)+\tfrac12(\tfrac34)=\tfrac34$. The conditional mean $4\Theta$ takes values $1$ and $3$ equally often, so its mean is $2$ and its variance is $\tfrac12(1-2)^2+\tfrac12(3-2)^2=1$. Thus $$ \operatorname{Var}(X)=\tfrac34+1=\tfrac74. $$

Check by conditional second moments. When $\Theta=\tfrac14$, $\mathbb{E}[X^2\mid\Theta]=\tfrac34+1^2=\tfrac74$. When $\Theta=\tfrac34$, it is $\tfrac34+3^2=\tfrac{39}4$. Their average is $\tfrac12(\tfrac74)+\tfrac12(\tfrac{39}4)=\tfrac{23}4$. Subtracting $\mathbb{E}[X]^2=4$ gives $\tfrac{23}4-\tfrac{16}4=\tfrac74$. The extra variance compared with a fixed success probability reflects uncertainty about $\Theta$.

Worked Example: A Poisson Count with a Random Rate

Suppose a count $N$, conditional on a random rate $\Lambda$, has a Poisson distribution with parameter $\Lambda$. Let $\Lambda=1$ or $3$, each with probability $\tfrac12$. Find the mean and variance of $N$.

For a Poisson variable with parameter $\lambda$, the conditional mean and variance are both $\lambda$. Thus $\mathbb{E}[N\mid\Lambda]$ equals $1$ or $3$, and $\operatorname{Var}(N\mid\Lambda)$ also equals $1$ or $3$. The overall mean is $\tfrac12(1)+\tfrac12(3)=2$. The expected conditional variance is $\tfrac12(1)+\tfrac12(3)=2$. The variance of the conditional mean is $$ \tfrac12(1-2)^2+\tfrac12(3-2)^2=1. $$ Consequently, $\operatorname{Var}(N)=2+1=3$.

Check by second moments. For fixed $\lambda$, $\mathbb{E}[N^2\mid\Lambda=\lambda]=\lambda+\lambda^2$, since variance equals the second moment minus the squared mean. At $\lambda=1$, this is $1+1=2$; at $\lambda=3$, it is $3+9=12$. Hence $\mathbb{E}[N^2]=\tfrac12(2)+\tfrac12(12)=7$, and $\operatorname{Var}(N)=7-2^2=3$. The random rate creates variation between conditional means in addition to the Poisson variation within each rate.

Interpreting the Decomposition

The within-group term answers: how variable is $X$ on average after the group or latent parameter is known? The between-group term answers: how much do the conditional means vary across groups? Both are nonnegative. Therefore the overall variance is at least the expected conditional variance. This does not mean every group has variance below the overall variance; the comparison is between the overall variance and the weighted average of conditional variances.

Common mistake: omitting the between-group term. In the two-class example, reporting only $\tfrac13(1)+\tfrac23(4)=3$ as the overall variance drops the variation in the group means. The correction is to add $\tfrac13(1-3)^2+\tfrac23(4-3)^2=2$, giving $5$. Directly, the unconditional second moment is $14$ and the squared overall mean is $9$, so the variance must be $14-9=5$, not $3$. Another tempting but invalid shortcut is to average group standard deviations: standard deviations do not add in this decomposition. The quantities to combine are variances, with the between-group variance included.

For a model with several levels of conditioning, apply the same principle at the level relevant to the question. First compute conditional means and variances at the finest level specified by the model; then average the conditional variances and calculate how the conditional means vary. If the model has nested levels, the tower property lets you move between levels, but the information being conditioned on must be stated: variation between broad groups and variation between subgroups are distinct summaries.

Check Your Understanding

For each question, identify the conditional means and variances before combining them.

A label takes values $a$ and $b$ with probabilities $\tfrac14$ and $\tfrac34$. The conditional means are $2$ and $6$, and the conditional variances are $1$ and $3$. Find the overall mean, within-group variance, between-group variance, and total variance.
In a two-class model, both classes have the same conditional mean but different conditional variances. What is the between-group variance of the conditional means?
For the random-rate count example, identify which term in the variance decomposition would disappear if the rate were fixed at its overall mean.
Explain why averaging conditional standard deviations does not give the total-variance formula.
State the moment condition needed to use the total-variance calculation for $X$.

Total Expectation and Total Variance Calculations

What You'll Learn