Deriving the Within-Between Sum-of-Squares Split

Why Split Variation Across Risks?

In balanced Bühlmann data, each risk is observed in the same number of periods. The observed variation has two sources: outcomes for a given risk vary from period to period, and the risks have different hypothetical means. A sum-of-squares split separates these sources in the data. Its mean squares then connect the observed variation to the model’s expected process variance and variance of hypothetical means.

The split is both an algebraic identity and a model-based result. The identity holds for any balanced data set. The interpretation of the mean squares requires assumptions about how observations are generated. Keeping those claims separate prevents a common error: treating one observed mean square as if it were exactly equal to a model variance.

Definition: Suppose there are \(r\) risks, each with \(n\) observations. Write \(X_{ij}\) for observation \(j\) on risk \(i\), \(\bar X_i=\frac{1}{n}\sum_{j=1}^n X_{ij}\) for risk \(i\)'s average, and \(\bar X_{\cdot\cdot}=\frac{1}{r}\sum_{i=1}^r\bar X_i\) for the grand average. The within-risk sum of squares is \(SS_W=\sum_{i=1}^r\sum_{j=1}^n(X_{ij}-\bar X_i)^2\). The between-risk sum of squares is \(SS_B=n\sum_{i=1}^r(\bar X_i-\bar X_{\cdot\cdot})^2\). The total sum of squares is \(SS_T=\sum_{i=1}^r\sum_{j=1}^n(X_{ij}-\bar X_{\cdot\cdot})^2\).

The factor \(n\) in \(SS_B\) matters: each risk average represents \(n\) observations. The total number of observations is \(rn\). The degrees of freedom are \(r(n-1)\) for within-risk variation, \(r-1\) for between-risk variation, and \(rn-1\) for total variation.

Deriving the Exact Decomposition

For each observation, add and subtract its risk average: \(X_{ij}-\bar X_{\cdot\cdot}=(X_{ij}-\bar X_i)+(\bar X_i-\bar X_{\cdot\cdot})\). Squaring and summing gives a within-risk term, a between-risk term, and a cross-product. The cross-product is zero because the deviations within each risk sum to zero.

\begin{aligned} SS_T &=\sum_{i=1}^r\sum_{j=1}^n \left[(X_{ij}-\bar X_i)+(\bar X_i-\bar X_{\cdot\cdot})\right]^2\\ &=SS_W+n\sum_{i=1}^r(\bar X_i-\bar X_{\cdot\cdot})^2\\ &\quad+2\sum_{i=1}^r(\bar X_i-\bar X_{\cdot\cdot}) \sum_{j=1}^n(X_{ij}-\bar X_i)\\ &=SS_W+SS_B. \end{aligned}

The last equality follows from \(\sum_{j=1}^n(X_{ij}-\bar X_i)=0\) for every risk \(i\). Thus \(SS_T=SS_W+SS_B\) for every balanced data set, regardless of whether the Bühlmann model assumptions hold.

Key result: In a balanced one-way decomposition, \(SS_T=SS_W+SS_B\). The corresponding mean squares are \(MS_W=SS_W/[r(n-1)]\) and \(MS_B=SS_B/(r-1)\). Their expected values have Bühlmann interpretations only under an appropriate repeated-observation model.

Connecting Mean Squares to Bühlmann Components

Let \(\Theta_i\) denote the risk parameter for risk \(i\), and let \(\mu_i=E[X_{ij}\mid\Theta_i]\) be its hypothetical mean. Define the expected process variance and variance of hypothetical means by \(\nu=E[\operatorname{Var}(X_{ij}\mid\Theta_i)]\) and \(\tau^2=\operatorname{Var}(\mu_i)\), respectively. The notation \(\nu\) is used here to keep the process variance distinct from the discount factor \(v\).

For the mean-square results below, assume risks are independent draws from the same risk population. Conditional on a risk parameter, its period observations are independent and have a common conditional mean and variance. The expected conditional variance across risks is finite and equals \(\nu\), and the hypothetical means have finite variance \(\tau^2\). These assumptions allow conditional variances to differ by risk; the formulas use their population average. If observations are dependent across periods, or risks are not sampled from a common population, these expected mean-square relationships need not hold as stated.

Within-risk mean square.
Conditional on \(\Theta_i\), centering \(n\) independent observations at their own average gives expected within-risk sum of squares \((n-1)\operatorname{Var}(X_{ij}\mid\Theta_i)\). Averaging over risks and dividing by \(r(n-1)\) gives \(E[MS_W]=\nu\).

Variation of a risk average.
The law of total variance gives \(\operatorname{Var}(\bar X_i)=\tau^2+\nu/n\): the risk average varies because the hypothetical means differ and because each average contains process variation.

Between-risk mean square.
For \(r\) independent, identically distributed risk averages, the expected sum of squared deviations from their grand average is \((r-1)\operatorname{Var}(\bar X_i)\). Multiplication by \(n\) and division by \(r-1\) yields \(E[MS_B]=\nu+n\tau^2\).

The derivation in Step 3 uses the standard identity \(E[\sum_{i=1}^r(Y_i-\bar Y)^2]=(r-1)\operatorname{Var}(Y_i)\) for independent, identically distributed \(Y_i\) with finite variance. Applying it to \(Y_i=\bar X_i\) gives \(E[SS_B]=n(r-1)(\tau^2+\nu/n)=(r-1)(n\tau^2+\nu)\). This proves the stated expected between-risk mean square.

E[MS_W]=\nu,\qquad E[MS_B]=\nu+n\tau^2,\qquad E[MS_B]-E[MS_W]=n\tau^2.

The within-risk mean square reflects expected process variance. The between-risk mean square contains both that same process contribution and the additional variation among hypothetical means, scaled by \(n\). The difference in expected mean squares isolates \(n\tau^2\). These are expectations under the model, not claims that a particular data set’s mean squares equal the parameters exactly.

Worked Calculations

Worked Example: Decompose a Small Balanced Data Set

A hypothetical insurer groups three commercial risks and records three annual loss amounts for each, in thousands of dollars. The observations are: Risk A, 8, 10, 12; Risk B, 10, 12, 14; Risk C, 12, 14, 16. Calculate the three sums of squares and both mean squares. Use the balanced one-way decomposition; do not interpret the observed mean squares as exact model components.

Step 1: Find the risk averages and grand average. The risk averages are \(\bar X_A=10\), \(\bar X_B=12\), and \(\bar X_C=14\). Therefore \(\bar X_{\cdot\cdot}=(10+12+14)/3=12\).

Step 2: Calculate within-risk variation. For Risk A, the squared deviations from 10 are \((8-10)^2+(10-10)^2+(12-10)^2=4+0+4=8\). Risk B gives \((10-12)^2+(12-12)^2+(14-12)^2=4+0+4=8\). Risk C gives \((12-14)^2+(14-14)^2+(16-14)^2=4+0+4=8\). Thus \(SS_W=8+8+8=24\).

Step 3: Calculate between-risk variation. There are \(n=3\) observations per risk, so \(SS_B=3[(10-12)^2+(12-12)^2+(14-12)^2]=3(4+0+4)=24\). The exact decomposition gives \(SS_T=SS_W+SS_B=24+24=48\).

Step 4: Calculate degrees of freedom and mean squares. The within-risk degrees of freedom are \(3(3-1)=6\), and the between-risk degrees of freedom are \(3-1=2\). Hence \(MS_W=24/6=4\) and \(MS_B=24/2=12\), in squared thousands of dollars.

Check. Directly centering all nine observations at 12 gives squared deviations summing to \(20+8+20=48\), which agrees with \(SS_T=48\). The method avoids omitting the factor \(n\) from \(SS_B\). Under a Bühlmann model, the observed values 4 and 12 are not themselves guaranteed to equal \(\nu\) and \(\nu+3\tau^2\); those are their expected values.

Worked Example: Use the Correct Within-Risk Degrees of Freedom

A hypothetical portfolio has three risks with four annual observations per risk. Their risk averages are 10, 12, and 14, and the within-risk sum of squares is \(SS_W=36\). Find \(SS_B\), \(SS_T\), \(MS_W\), and \(MS_B\). Amounts are in thousands of dollars.

Step 1: Find the grand average. Because the design is balanced, \(\bar X_{\cdot\cdot}=(10+12+14)/3=12\).

Step 2: Find the between-risk and total sums of squares. With \(n=4\), \(SS_B=4[(10-12)^2+(12-12)^2+(14-12)^2]=4(4+0+4)=32\). Then \(SS_T=36+32=68\).

Step 3: Compute both mean squares using their own degrees of freedom. The within-risk degrees of freedom are \(r(n-1)=3(4-1)=9\), so \(MS_W=36/9=4\). The between-risk degrees of freedom are \(r-1=2\), so \(MS_B=32/2=16\).

Check. The degrees of freedom sum to \(9+2=11=rn-1\). Also, the sums of squares satisfy \(36+32=68\). The method avoids the trap of dividing \(SS_W\) by \(r\), by \(n\), or by the between-risk degrees of freedom; the correct denominator is \(r(n-1)\).

Worked Example: Calculate Expected Mean Squares Under a Model

Suppose a hypothetical balanced study has \(r=5\) risks and \(n=4\) independent annual observations per risk. Assume the expected process variance is \(\nu=25\) and the variance of hypothetical means is \(\tau^2=9\), in squared thousands of dollars. Calculate both expected mean squares and the expected sums of squares.

Step 1: Find the expected within-risk mean square. The model result is \(E[MS_W]=\nu=25\).

Step 2: Find the expected between-risk mean square. Here \(n\tau^2=4(9)=36\), so \(E[MS_B]=\nu+n\tau^2=25+36=61\).

Step 3: Convert to expected sums of squares. The within-risk degrees of freedom are \(5(4-1)=15\), giving \(E[SS_W]=15(25)=375\). The between-risk degrees of freedom are \(5-1=4\), giving \(E[SS_B]=4(61)=244\).

Check. The expected total sum of squares is \(E[SS_T]=375+244=619\), consistent with the exact identity \(SS_T=SS_W+SS_B\). The gap in expected mean squares is \(61-25=36=n\tau^2\), and \(36/4=9=\tau^2\). The method avoids treating \(E[MS_B]\) as just the variance of hypothetical means; it includes the process contribution too.

Interpretation and Exam Traps

This decomposition matters because it shows precisely how repeated observations distinguish within-risk process variation from variation across risks. More observations per risk do not change the expected within-risk mean square, but they multiply the hypothetical-mean contribution to the between-risk mean square by \(n\). That scaling is why the factor \(n\) must appear in the between-risk sum of squares and in the expected mean-square relationship.

Confusing an identity with a model result. \(SS_T=SS_W+SS_B\) is an algebraic identity for balanced data. The expected mean-square formulas require the stated model assumptions.
Forgetting the group-size factor. The between-risk sum of squares is \(n\sum_i(\bar X_i-\bar X_{\cdot\cdot})^2\), not just the sum of squared deviations of the risk averages.
Using the wrong degrees of freedom. \(MS_W\) divides by \(r(n-1)\); \(MS_B\) divides by \(r-1\). Check that the two counts sum to \(rn-1\).
Calling an observed mean square a component. The model states \(E[MS_W]=\nu\), not \(MS_W=\nu\) in every sample. Likewise \(E[MS_B]=\nu+n\tau^2\).
Ignoring balance. The formulas here use the same \(n\) for every risk. Unequal observation counts require a different treatment and are not covered by this balanced derivation.

Key takeaway: For balanced repeated-period data, \(SS_T=SS_W+SS_B\), with degrees of freedom \(r(n-1)\), \(r-1\), and \(rn-1\). Under the Bühlmann repeated-observation assumptions, \(E[MS_W]=\nu\) and \(E[MS_B]=\nu+n\tau^2\). Their expected difference is \(n\tau^2\).

Check Your Understanding

For each problem, use the balanced-design formulas and keep observed sums of squares distinct from model expectations.

Four risks each have three observations, and their risk averages are 5, 7, 9, and 11. Find the grand average and \(SS_B\). (Answer: \(\bar X_{\cdot\cdot}=8,\ SS_B=60\).)
Three risks each have five observations, with risk averages 20, 22, and 26, and \(SS_W=80\). Find \(SS_B\), \(SS_T\), \(MS_W\), and \(MS_B\). (Answer: \(SS_B=93.3333,\ SS_T=173.3333,\ MS_W=6.6667,\ MS_B=46.6667\).)
In a balanced model with \(n=6\), \(\nu=12\), and \(\tau^2=5\), what are the expected within-risk and between-risk mean squares? (Answer: \(E[MS_W]=12,\ E[MS_B]=42\).)
For a balanced design, the observed mean squares are \(MS_B=31\) and \(MS_W=11\), with \(n=4\). What value does their difference, divided by \(n\), produce? State what this quantity represents in the expected mean-square relationship. (Answer: \((31-11)/4=5\); it is the method-of-moments contrast for \(\tau^2\), corresponding to \([E(MS_B)-E(MS_W)]/n=\tau^2\), not a guarantee that the population component equals 5.)
Four risks each have two observations, with risk averages 3, 4, 7, and 10, and \(SS_W=24\). Calculate \(SS_B\), \(SS_T\), \(MS_W\), and \(MS_B\), and verify the degrees of freedom. (Answer: \(SS_B=60,\ SS_T=84,\ MS_W=6,\ MS_B=20\); degrees of freedom are 4 and 3, summing to \(7=rn-1\).)

What You'll Learn