Why Unequal Class Sizes Change the Calculation
Bühlmann parameter estimation separates two sources of variation: variation among individual observations within a class, and variation among the underlying means of different classes. When classes have different numbers of observations, their sample means have different precision. A mean based on many observations fluctuates less from its underlying class mean than a mean based on only a few. Estimating variation among class means therefore requires an adjustment for this sampling variation.
The setting here is the classical Bühlmann model with repeated, equally weighted observations within each class. Class sizes may differ, but each observation has the same conditional variance. Unequal exposure weights are a different model and should not be substituted into these calculations.
Write \(N=\sum_{i=1}^m n_i\) for the total number of observations, \(\bar X_i\) for the mean in class \(i\), and \(\bar X_{\cdot\cdot}\) for the mean of all \(N\) observations. The class means are not equally precise when \(n_i\) differs: under the model, \(\operatorname{Var}(\bar X_i)=a+v/n_i\). The \(a\) term is genuine variation among classes; the \(v/n_i\) term is sampling noise in the observed class mean.
Method-of-Moments Estimates
The process variance is estimated from deviations around each class's own sample mean. These residuals remove the class-level mean, so their pooled variation estimates within-class process variation rather than variation between classes. The denominator is the total within-class degrees of freedom, not the total number of observations.
A natural estimate of the collective mean is the observation-weighted grand mean. Each observation has expected value \(\mu\), so this estimator is unbiased for fixed class sizes:
For the variance of hypothetical means, first consider the average squared difference between pairs of class means. The pairwise differences remove the unknown collective mean. They still include sampling noise, however, and that noise depends on the class sizes. The following derivation gives the required adjustment.
Conditional independence and common process variance imply \(\operatorname{Var}(\bar X_i)=a+v/n_i\). The first term is between-class variation; the second is the variance of the average of \(n_i\) process observations.
For distinct independent classes \(i\) and \(k\), both class means have expectation \(\mu\). Thus \(E[(\bar X_i-\bar X_k)^2]=\operatorname{Var}(\bar X_i)+\operatorname{Var}(\bar X_k)=2a+v(1/n_i+1/n_k)\).
There are \(m(m-1)/2\) pairs. Across this list, each reciprocal \(1/n_i\) occurs \(m-1\) times. Dividing the sum of expected squared differences by \(m(m-1)\) therefore gives \(a+(v/m)\sum_i1/n_i\).
Replace \(v\) by its pooled estimate and rearrange the pairwise moment equation. This yields the method-of-moments estimate of \(a\).
The first term is the sample variance of the \(m\) class means, expressed through pairwise differences. The second term estimates the average sampling variance of those means. If every class has the same size \(n\), the correction reduces to \(\widehat v/n\), giving the familiar balanced-class form. With unequal sizes, the reciprocals must be summed; substituting a single class size without justification changes the estimate.
Worked Calculations
Worked Example: Estimating Both Variance Components
A hypothetical portfolio is divided into four classes, with class sizes \(2,3,4,5\). The observed class means are \(10,12,11,15\), and the within-class sums of squared deviations are \(2,6,6,8\), respectively. Estimate \(\mu\), \(v\), and \(a\) by the unequal-class-size method of moments. Also calculate the resulting ratio \(a/v\). Avoid treating the four class means as equally precise observations of individual risks.
Method: calculate the pooled within-class variance first. The total within-class sum of squares is \(2+6+6+8=22\). The total number of observations is \(N=2+3+4+5=14\), and there are \(m=4\) classes, so the degrees of freedom are \(N-m=10\). Therefore \(\widehat v=22/10=2.2\).
Estimate the collective mean. Weight each class mean by its size: \(2(10)+3(12)+4(11)+5(15)=20+36+44+75=175\). Thus \(\widehat\mu=175/14=12.5\).
Calculate the pairwise term. The six squared differences are \((10-12)^2=4\), \((10-11)^2=1\), \((10-15)^2=25\), \((12-11)^2=1\), \((12-15)^2=9\), and \((11-15)^2=16\). Their sum is \(56\), so the pairwise term is \(56/[4(3)]=56/12=4.6667\).
Adjust for unequal sampling precision. The reciprocal-size sum is \(1/2+1/3+1/4+1/5=77/60=1.2833\). The adjustment is \((2.2/4)(77/60)=0.7058\). Hence \(\widehat a_{\mathrm{raw}}=4.6667-0.7058=3.9608\).
Calculate and check the ratio. The estimated ratio is \(\widehat a/\widehat v=3.9608/2.2=1.8004\). The result is positive, and the unadjusted pairwise term exceeds the estimated sampling-noise correction, as required for a positive heterogeneity estimate. Rounding is to four decimal places for the reported estimates and ratio.
Common mistake avoided: the unadjusted pairwise term, \(4.6667\), is not the estimate of \(a\). It includes sampling variation in the class means. Subtracting the correction is essential.
Worked Example: A Small Class Can Matter More to the Correction
Three hypothetical classes have sizes \(1,4,9\), means \(20,24,22\), and within-class sums of squares \(0,6,16\). Estimate \(v\), the raw method-of-moments estimate of \(a\), and the collective mean. Compare the raw estimate of \(a\) with the unadjusted variance of the three class means.
Estimate process variance. There are \(1+4+9=14\) observations and \(3\) classes, so the within-class degrees of freedom total \(14-3=11\). The pooled sum of squares is \(0+6+16=22\), giving \(\widehat v=22/11=2\).
Compute the unadjusted class-mean variance. The squared pairwise differences are \((20-24)^2=16\), \((20-22)^2=4\), and \((24-22)^2=4\). The pairwise term is \((16+4+4)/[3(2)]=24/6=4\). This is also the ordinary sample variance of the three class means.
Apply the unequal-size correction. The reciprocal-size sum is \(1+1/4+1/9=49/36\). The correction is \((2/3)(49/36)=49/54=0.9074\). Therefore \(\widehat a_{\mathrm{raw}}=4-49/54=167/54=3.0926\).
Estimate the collective mean. The weighted sum is \(1(20)+4(24)+9(22)=20+96+198=314\), so \(\widehat\mu=314/14=22.4286\).
Check. The correction is smaller than the pairwise term, leaving a positive estimate. The class with one observation contributes \(1\) to the reciprocal-size sum, much more than the class with nine observations, whose contribution is \(1/9\). This reflects the greater sampling noise in the small class mean. Values are rounded to four decimal places.
Common mistake avoided: using \(4\), the unadjusted variance of the means, as the estimate of \(a\) ignores the process variation in those means. It overstates the estimate here by \(0.9074\).
Worked Example: Recognizing a Negative Raw Estimate
Three hypothetical classes have sizes \(2,3,6\), sample means \(100,101,100\), and within-class sums of squares \(4,8,20\). Calculate the raw moment estimates of \(v\) and \(a\). Then state what a nonnegative constraint would do to the estimate of \(a\).
Estimate \(v\). The total number of observations is \(11\), so the within-class degrees of freedom are \(11-3=8\). The pooled sum of squares is \(4+8+20=32\), giving \(\widehat v=32/8=4\).
Calculate the pairwise term. The squared differences are \((100-101)^2=1\), \((100-100)^2=0\), and \((101-100)^2=1\). Their sum is \(2\), and the pairwise term is \(2/[3(2)]=1/3\).
Subtract sampling noise. The reciprocal-size sum is \(1/2+1/3+1/6=1\), so the correction is \((4/3)(1)=4/3\). Thus \(\widehat a_{\mathrm{raw}}=1/3-4/3=-1\).
Interpret the result. The unconstrained method-of-moments estimate is \(-1\), although a variance cannot be negative. A common practical response is to constrain the estimate to \(\widehat a=0\). That constrained value is not the same as the raw unbiased moment estimate; it is a boundary adjustment. The data provide little observed separation among class means relative to their estimated sampling noise.
Common mistake avoided: do not silently report a negative variance as a valid structural parameter. State whether the result is raw or constrained. Under a zero constraint, the estimated heterogeneity is zero. Values are exact in this example.
Interpretation and Practical Checks
Unequal class sizes affect the calculations in two related but distinct ways. The estimate of \(\mu\) is based on all observations, so larger classes contribute more observations to the grand mean. By contrast, the variance component \(a\) describes variation across classes. Its pairwise term treats each class as a class-level unit, then corrects for the fact that a class mean based on fewer observations contains more process noise.
The reciprocal-size correction is not an arbitrary penalty. It follows from the sampling variance \(v/n_i\) of each class mean. As a check, if all \(n_i=n\), then \(\sum_i1/n_i=m/n\), and the correction becomes \(\widehat v/n\). If the sizes are unequal, replacing them all by the largest size understates the noise correction; replacing them all by the smallest size overstates it. Use the actual sizes in the formula.
These estimates rely on the stated model: classes are independent, observations are conditionally independent within class, and the process variance is common across classes and observations. If conditional variances differ systematically by class, the pooled residual variance does not estimate one common \(v\) in the intended sense. If observations carry different exposure weights rather than being equal-weight repeated measurements, the classical calculation is not the appropriate weighted model.
Check Your Understanding
For each problem, use the stated classical model and show the pooled degrees of freedom and sampling-noise correction in your work.
- Three hypothetical classes have sizes \(2,4,6\), means \(8,10,9\), and within-class sums of squares \(2,4,8\). Find \(\widehat v\), \(\widehat a_{\mathrm{raw}}\), and \(\widehat\mu\). (Answer: \(\widehat v=14/9=1.5556\), \(\widehat a_{\mathrm{raw}}=0.5247\), \(\widehat\mu=9.1667\).)
- Four classes have sizes \(1,2,4,8\), means \(30,32,31,35\), and known process variance \(v=3\). Estimate \(a\) using the pairwise formula. (Answer: pairwise term \(=56/12=4.6667\), correction \(=1.4063\), and \(\widehat a_{\mathrm{raw}}=3.2604\).)
- Three classes have sizes \(1,3,10\), means \(40,42,41\), and known process variance \(v=2\). Compare the unadjusted class-mean variance with the corrected estimate of \(a\). (Answer: unadjusted term \(=1.0000\), correction \(=0.9556\), and \(\widehat a_{\mathrm{raw}}=0.0444\).)
- Three classes have sizes \(2,4,8\), means \(50,50.5,50\), and known process variance \(v=2\). Find the raw estimate and its value after imposing a nonnegative constraint. (Answer: pairwise term \(=0.0833\), correction \(=0.5833\), raw \(\widehat a=-0.5000\), constrained \(\widehat a=0\).)
- Three classes have sizes \(2,3,5\), means \(4,8,10\), and within-class sums of squares \(2,6,8\). Find \(\widehat v\), \(\widehat a_{\mathrm{raw}}\), and the observation-weighted \(\widehat\mu\). (Answer: \(\widehat v=16/7=2.2857\), pairwise term \(=56/6=9.3333\), correction \(=248/315=0.7873\), \(\widehat a_{\mathrm{raw}}=8.5460\), and \(\widehat\mu=8.2000\).)