Why Variance-Component Estimation Needs Care
Bühlmann-Straub credibility separates variation within a risk class from variation between risk classes. The within-class variation reflects random process noise; the between-class variation also reflects genuine differences in underlying class means. When exposures differ, each class mean has a different amount of process noise. A direct calculation of the spread of class means therefore overstates the structural variation unless that noise is removed.
The estimation procedure uses weighted observations to estimate the overall mean and process variance, then subtracts the process contribution from the observed between-class variation. The correction depends on the number of classes and the class exposure totals; it is not generally the same as subtracting one process variance from every class mean.
Model and Moment Estimates
Assume the Bühlmann-Straub model: conditional on the risk parameter \(\theta_i\), observations in class \(i\) have common mean \(\mu(\theta_i)\), variance \(\sigma^2/w_{ij}\), and are independent across periods. Across classes, the risk parameters are independent and identically distributed, with \(E[\mu(\theta_i)]=\mu\) and \(\operatorname{Var}(\mu(\theta_i))=\tau^2\). Thus, \(\sigma^2\) is the process variance parameter and \(\tau^2\) is the structural variance parameter. This moment-based method does not require normally distributed observations, but it does require the stated mean, variance, independence, and exposure-scaling assumptions.
Because the observations within a class share the same conditional mean, weighting by exposure gives the class mean \(\bar X_i\) process variance \(\sigma^2/W_i\). Unconditionally, its variance is \(\tau^2+\sigma^2/W_i\). This unequal variance across class means is the reason the between-class correction must use the actual \(W_i\).
The pooled process estimate uses \(N-I\) degrees of freedom: within each class, fitting its own weighted mean uses one degree of freedom. The numerator is the sum of squared deviations from those class means, with each squared deviation multiplied by its exposure weight.
Why the Between-Class Correction Works
The observed weighted spread \(B\) contains two sources of variation. To see their contributions, write \(m_i=\mu(\theta_i)\) and \(V_i=\tau^2+\sigma^2/W_i\), the unconditional variance of \(\bar X_i\). Independence across classes gives \(E[\hat\mu]=\mu\) and \(\operatorname{Var}(\hat\mu)=\sum_i W_i^2V_i/W^2\). Also, \(B=\sum_iW_i\bar X_i^2-W\hat\mu^2\). Taking expectations and substituting the class-mean variances yields:
Since \(\sum_iW_i=W\), the process contribution is exactly \((I-1)\sigma^2\). The remaining expected spread is \(\tau^2D\), so solving the moment equation for \(\tau^2\) gives the raw estimate above, replacing the unknown process variance with \(\hat\sigma^2\). This derivation also explains why the correction is based on \(I-1\), while the divisor for the structural estimate is \(D\).
For every class, calculate \(W_i\), \(\bar X_i\), and its weighted within-class sum of squares.
Use the exposure-weighted grand mean and divide the pooled within-class sum of squares by \(N-I\).
Find \(B\) and \(D\), subtract \((I-1)\hat\sigma^2\) from \(B\), then divide by \(D\).
A raw estimate can be negative. If the application requires a nonnegative variance estimate, use the stated truncation convention and report both values when informative.
In practice, a nonnegative method-of-moments convention is often to set \(\hat\tau^2=\max(0,\hat\tau^2_{\mathrm{raw}})\). This is a boundary adjustment, not a claim that the raw estimate is a valid negative variance. If \(D=0\), the structural component cannot be estimated by this formula. With positive class weights, \(D>0\) when there are at least two classes. If \(N=I\), there are no within-class degrees of freedom, so this pooled estimate of \(\sigma^2\) is unavailable.
Worked Calculations
Worked Example: Unequal Period Weights Within Three Classes
A hypothetical portfolio has three classes, each observed in two periods. The data are: class A has \((w,X)=(1,8),(3,10)\); class B has \((2,11),(2,13)\); and class C has \((1,14),(2,17)\). Estimate \(\mu\), \(\sigma^2\), and the raw \(\tau^2\).
Class summaries. For A, \(W_A=1+3=4\), \(\bar X_A=(1(8)+3(10))/4=38/4=9.5\), and its weighted within sum is \(1(8-9.5)^2+3(10-9.5)^2=2.25+0.75=3\). For B, \(W_B=4\), \(\bar X_B=(2(11)+2(13))/4=12\), and the within sum is \(2(11-12)^2+2(13-12)^2=4\). For C, \(W_C=3\), \(\bar X_C=(1(14)+2(17))/3=16\), and the within sum is \(1(14-16)^2+2(17-16)^2=6\).
Overall mean and process variance. There are \(N=6\) observations, \(I=3\) classes, and \(W=4+4+3=11\). Therefore \(\hat\mu=(4(9.5)+4(12)+3(16))/11=134/11\). The pooled within sum is \(3+4+6=13\), so \(\hat\sigma^2=13/(6-3)=13/3\).
Between-class quantities. The weighted sum of squared class means is \(4(9.5^2)+4(12^2)+3(16^2)=1705\). Thus \(B=1705-(134^2/11)=799/11\). Also, \(D=11-(4^2+4^2+3^2)/11=80/11\). The process correction is \((3-1)(13/3)=26/3\).
Structural estimate and check. The corrected numerator is \(799/11-26/3=2111/33\), giving \(\hat\tau^2_{\mathrm{raw}}=(2111/33)/(80/11)=2111/240\), or approximately \(8.7958\). The numerator is positive, so the nonnegative convention leaves this estimate unchanged. The computed \(D=80/11>0\) confirms that the structural estimate is defined.
Method and mistake avoided. This is unequal-weight method-of-moments estimation. Replacing the class weights with equal weights would change both the grand mean and \(D\), and would not estimate the stated model.
Worked Example: A Negative Raw Structural Estimate
Two hypothetical classes each have two unit-weight observations. Class A has values 4 and 8; class B has values 6 and 10. Calculate all three raw moment estimates, then apply the nonnegative convention to \(\tau^2\).
Overall mean. Both class weights are 2. The class means are \(\bar X_A=(4+8)/2=6\) and \(\bar X_B=(6+10)/2=8\). Hence \(\hat\mu=(2(6)+2(8))/4=7\).
Process variance. The within sum for A is \((4-6)^2+(8-6)^2=8\); for B it is \((6-8)^2+(10-8)^2=8\). With \(N-I=4-2=2\), \(\hat\sigma^2=(8+8)/2=8\).
Structural variance. The between sum is \(B=2(6-7)^2+2(8-7)^2=4\), and \(D=4-(2^2+2^2)/4=2\). The correction is \((2-1)8=8\). Thus \(\hat\tau^2_{\mathrm{raw}}=(4-8)/2=-2\), and the nonnegative estimate is \(\hat\tau^2=0\).
Method and mistake avoided. The raw method-of-moments estimate is \(-2\), not zero; zero is the result only after applying the nonnegative convention. A negative raw value signals that observed between-class variation is less than the estimated process contribution.
Worked Example: Use Every Class in the Grand Mean
A hypothetical data set has three classes with \((W_i,\bar X_i)\) equal to \((1,5)\), \((3,9)\), and \((2,8)\). Suppose \(\hat\sigma^2=2\). Calculate \(\hat\mu\), \(D\), and the raw structural estimate.
Grand mean. The total weight is \(W=1+3+2=6\). Therefore \(\hat\mu=(1(5)+3(9)+2(8))/6=48/6=8\).
Between sum and divisor. \(B=1(5-8)^2+3(9-8)^2+2(8-8)^2=9+3+0=12\). Also, \(D=6-(1^2+3^2+2^2)/6=6-14/6=11/3\).
Correction and estimate. There are \(I=3\) classes, so the process correction is \((3-1)2=4\). Consequently, \(\hat\tau^2_{\mathrm{raw}}=(12-4)/(11/3)=24/11\), approximately \(2.1818\). The estimate is positive and \(D>0\), so the nonnegative convention does not change it.
Method and mistake avoided. This is the unequal-total-weight between-class calculation. Omitting the third class would incorrectly produce a grand mean of \(8\) only by coincidence here, while still giving incorrect \(B\) and \(D\); all classes must be included in every total.
Worked Example: Estimation at the Nonnegative Boundary
Four hypothetical classes each have total weight 2. Their class means are 10, 10, 12, and 12, and the estimated process variance is 3. Calculate the raw structural estimate and its nonnegative version.
Grand mean and between sum. Since the weights are equal, \(\hat\mu=(10+10+12+12)/4=11\). The between sum is \(B=2(10-11)^2+2(10-11)^2+2(12-11)^2+2(12-11)^2=8\).
Divisor and correction. Total weight is \(W=8\), and \(D=8-(2^2+2^2+2^2+2^2)/8=8-2=6\). The process correction is \((4-1)3=9\).
Structural estimate and check. The raw estimate is \(\hat\tau^2_{\mathrm{raw}}=(8-9)/6=-1/6\). The nonnegative estimate is \(\hat\tau^2=0\). In particular, the between sum is 8, not 9; the correction exceeds it by 1, which produces the negative raw estimate.
Method and mistake avoided. This is a direct application of the corrected between-class moment. Rounding or setting a small-looking discrepancy to zero before calculating the raw value would obscure the reason the boundary convention is needed.
Common Exam Traps and Interpretation
The most important practical distinction is between the raw method-of-moments estimate and any constrained estimate used afterward. State the raw result first when asked for it. If a nonnegative estimate is requested, apply the boundary convention after computing the correction and division.
- Using an unweighted grand mean. The grand mean is weighted by \(W_i\), not by the number of periods in each class, unless all class weights happen to be equal.
- Forgetting the process correction. \(B\) includes process noise, so estimate \(\tau^2\) from \(B-(I-1)\hat\sigma^2\), not from \(B\) alone.
- Using the wrong divisor. The structural estimate divides by \(D=W-\sum_iW_i^2/W\), not by \(I-1\) or \(W\).
- Confusing within and between variation. The process estimate uses deviations of observations from their own class means and denominator \(N-I\). The structural estimate uses deviations of class means from the grand mean.
- Ignoring identifiability conditions. If \(N=I\), the within-class process estimate has no degrees of freedom. If \(D=0\), the weighted class means provide no between-class information for estimating \(\tau^2\).
These estimates matter because the credibility weight assigned to class experience depends on the relative size of process and structural variation. An overstated structural variance can make experience appear more informative than it is; omitting the correction can therefore affect the credibility calculation that uses the estimated components. The calculation is aligned with the CAS Exam 5 (Basic Ratemaking and Estimated Claim Liabilities) syllabus.
Check Your Understanding
For each problem, calculate the requested quantities before checking the final answer.
- Two classes have \((W_i,\bar X_i)=(2,4)\) and \((4,7)\). If \(\hat\sigma^2=3\), find \(\hat\mu\), \(D\), and \(\hat\tau^2_{\mathrm{raw}}\). (Answer: \(\hat\mu=6\), \(D=8/3\), \(\hat\tau^2_{\mathrm{raw}}=27/8=3.375\).)
- Three classes each have total weight 2 and means 5, 7, and 8. If \(\hat\sigma^2=2\), find the raw structural estimate. (Answer: \(\hat\mu=20/3\), \(D=4\), \(\hat\tau^2_{\mathrm{raw}}=4/3\).)
- Four classes each have total weight 1 and means 9, 9, 11, and 11. If \(\hat\sigma^2=2\), give the raw and nonnegative structural estimates. (Answer: \(B=4\), \(D=3\), \(\hat\tau^2_{\mathrm{raw}}=-2/3\), and \(\hat\tau^2=0\).)
- Class A has observations \((w,X)=(1,2),(3,4)\); class B has \((2,5),(2,7)\). Calculate \(\hat\sigma^2\), \(\hat\mu\), \(D\), and \(\hat\tau^2_{\mathrm{raw}}\). (Answer: \(\hat\sigma^2=7/2\), \(\hat\mu=19/4\), \(D=4\), and \(\hat\tau^2_{\mathrm{raw}}=9/4\).)