Within-Group and Between-Group Variance

Why Separate Within-Group and Between-Group Variation?

A population can vary because observations differ inside the groups to which they belong, because the groups have different averages, or both. For example, the variation in a hypothetical portfolio's annual claim amounts might reflect differences among policyholders within a risk class and differences in average claim amounts between classes. A variance calculation that keeps these sources separate can explain where the overall variability comes from.

The same calculation applies when a group is selected randomly and a value is then generated according to that group's distribution. The group is a conditioning variable: once its value is known, the conditional mean and variance describe the selected group's behavior. The overall variance combines those conditional variances with the variation in the group means.

The Total-Variance Formula

Suppose the possible groups are \(g_1,\ldots,g_k\), with \(P(G=g_j)=p_j\). Write \(\mu_j=E[X\mid G=g_j]\) and \(\sigma_j^2=\operatorname{Var}(X\mid G=g_j)\). The group probabilities satisfy \(p_j\geq 0\) and \(\sum_{j=1}^k p_j=1\). The overall mean is the weighted mean of the group means, and total variance separates as follows.

The weights are essential. A group with probability \(0.8\) contributes four times as much as a group with probability \(0.2\), all else equal. Neither the conditional variances nor the squared deviations of group means are generally averaged with equal weights.

Why the Decomposition Holds

The key identity is the conditional version of the variance definition. For a group \(G=g_j\), the squared deviation from the overall mean can be split into a deviation from the group's mean and a deviation of the group's mean from the overall mean:

Square this identity and take expectations. The cross-product has expectation zero. To see why, condition on \(G\): the term \(E[X\mid G]-E[X]\) is fixed within a group, while

Consequently, the expected cross-product is zero. The expected square of the first term is \(E[\operatorname{Var}(X\mid G)]\); the expected square of the second is \(\operatorname{Var}(E[X\mid G])\). This proves the total-variance formula:

This justification requires finite second moments so the variances and expectations in the identity are defined. In a finite population with a uniformly selected member, these moments exist automatically.

Finite Populations: Group Sizes Become Weights

For a finite population of \(N\) values, imagine selecting one member uniformly at random and calling that member's value \(X\). If group \(j\) contains \(N_j\) members, then \(P(G=g_j)=N_j/N\). Thus group size, not the number of groups, determines the weight.

The variance here is the probability variance for a uniformly selected population member, with divisor \(N\). It is not the sample variance computed from observed data using divisor \(N-1\). The latter is a different statistic, often used to estimate an unknown population variance.

Mixtures: Group Means Need Not Be Observed Values

In a mixture model, the group label is selected with specified probabilities and then \(X\) follows the conditional distribution for that group. The same weighted formulas apply whether the groups represent categories in a finite population or components of a probability model. Conditional means and variances are enough to calculate the overall variance; the full conditional distributions are not needed for this calculation.

Interpreting the Two Contributions

The within-group term measures the average residual spread after the group is known. It is a probability-weighted average, not the variance of all observations after subtracting a single group-specific quantity. The between-group term measures the spread in the group means around the overall mean. It is zero exactly when all groups with positive probability have the same conditional mean.

A zero between-group term does not imply zero overall variance: groups can have identical means and still have positive conditional variances. Conversely, if each group has no variation internally, the entire overall variance comes from differences among the group means. Both contributions are nonnegative, so the total variance cannot be less than either contribution individually.

For a numerical check, the second-moment identity is also useful:

This follows from \(E[X^2\mid G=g_j]=\sigma_j^2+\mu_j^2\), followed by \(\operatorname{Var}(X)=E[X^2]-(E[X])^2\). It provides an independent way to check arithmetic, but it does not replace interpreting the within- and between-group terms.

Common Mistakes and Exam Traps

• Using the wrong center for the between-group term. The group means are compared with the overall mean. In the two-group example, using zero would produce \(0.4(10^2)+0.6(16^2)\), which is not the between-group variance. The correct center is \(13.6\), giving \(0.4(10-13.6)^2+0.6(16-13.6)^2=8.64\).
• Giving every group equal weight. In the finite-population example, the groups have sizes 3 and 2, so their weights are \(0.6\) and \(0.4\), not \(0.5\) and \(0.5\).
• Adding group variances without weighting them. The within-group contribution is \(\sum_j p_j\sigma_j^2\), not \(\sum_j\sigma_j^2\) and not an unweighted average unless all group probabilities are equal.
• Mixing probability variance with sample variance. A uniformly selected member of a population of size \(N\) has variance with divisor \(N\). Replacing it with an \(N-1\) sample-variance calculation changes the quantity being computed.

This within-versus-between interpretation is a useful way to organize conditional-variance calculations aligned with the SOA Exam P (Probability) syllabus: group proportions determine the weights, conditional variances describe internal spread, and group means determine the additional spread between groups.