Why Estimating K Is Difficult in a Small Portfolio
In the Bühlmann model, \(K\) is the ratio of the expected process variance to the variance of hypothetical means. It measures how much individual experience is needed for the credibility weight to rise: \(Z=n/(n+K)\). The model’s formula is simple, but estimating its two variance components from a small portfolio is not. With few risks, the estimate of between-risk variation has little information behind it; with few periods per risk, the estimate of within-risk variation is also imprecise.
This tutorial develops the balanced-portfolio method and focuses on what the sample size tells you about the reliability of the answer. A calculated \(\widehat K\) is a plug-in estimate, not a precise measurement of a fixed quantity. Small changes in observed risk means can materially change the estimated VHM, and therefore both \(\widehat K\) and the implied credibility weight.
The Balanced-Portfolio Estimator
Suppose there are \(m\) risks, each observed for the same \(n\) periods. Assume the risks are independent and drawn from the same population of risks. Conditional on a risk’s parameter, its period observations are independent with a common conditional mean and conditional variance; across risks, those means have variance \(a\), and the expected conditional variance is \(v\). These assumptions are enough to derive unbiased estimators of the two variance components. Normality is not required for unbiasedness, but it does give the exact chi-square sampling results described below.
Write \(\bar X_i\) for risk \(i\)’s average over its \(n\) observations, and \(\bar X\) for the average across all \(mn\) observations. The within-risk mean square estimates EPV. The sample variance of the risk means includes both VHM and the sampling variance of each risk mean, so that sampling contribution must be removed to estimate VHM.
There are \(m(n-1)\) within-risk degrees of freedom because each of the \(m\) risk groups has \(n\) observations and uses one estimated group mean. There are \(m-1\) degrees of freedom among the risk means because their deviations from the estimated overall mean sum to zero. Thus \(SS_W\) is divided by \(m(n-1)\), while the sum of squared deviations among risk means is divided by \(m-1\).
Why the VHM Estimate Subtracts Sampling Variation
A risk’s observed mean is not its hypothetical mean: it also contains period-to-period process noise. Under the model, the variance of a risk mean is \(a+v/n\). The sample variance of the observed risk means therefore estimates this sum, not VHM alone.
Subtracting \(\bar X_i\) removes the risk’s conditional mean. The expected within-risk sum of squares is \((n-1)v\) for each risk, so \(E[SS_W]=m(n-1)v\) and \(E[MS_W]=v\).
Each \(\bar X_i\) varies because the underlying risk mean varies, contributing \(a\), and because it averages \(n\) observations, contributing \(v/n\). Hence \(E[S_{\bar X}^2]=a+v/n\).
Since \(MS_W\) estimates \(v\), subtracting \(MS_W/n\) from \(S_{\bar X}^2\) estimates \(a\). Dividing the component estimates then gives the plug-in estimate \(\widehat K=\widehat v/\widehat a\), provided \(\widehat a>0\).
For a balanced analysis, the same calculation can be expressed using the usual between-risk mean square \(MS_B=nS_{\bar X}^2\). Then \(\widehat a=(MS_B-MS_W)/n\), and
This is a ratio of estimated components; it is not generally an unbiased estimator of \(K\), even when \(\widehat v\) and \(\widehat a\) are individually unbiased. A small or noisy denominator can have a large effect on the ratio.
Worked Estimates and Their Limitations
Worked Example: Three Risks Observed for Two Periods
A hypothetical portfolio has the following losses, in thousands of dollars. Assume a balanced Bühlmann model with independent risks, and suppose \(X_{ij}=\mu+\alpha_i+\varepsilon_{ij}\), where \(\alpha_i\sim N(0,a)\) and \(\varepsilon_{ij}\sim N(0,v)\) are mutually independent. The unknown parameters are \(a\) and \(v\); estimate them from the observations. No interest or mortality assumption applies.
| Risk | Period 1 | Period 2 | Risk mean |
|---|---|---|---|
| A | 8 | 12 | 10 |
| B | 12 | 16 | 14 |
| C | 16 | 20 | 18 |
Step 1: Calculate within-risk variation. The deviations from each risk mean are \((-2,2)\), so each risk contributes \((-2)^2+2^2=8\). Thus \(SS_W=8+8+8=24\). There are \(m(n-1)=3(2-1)=3\) within-risk degrees of freedom, giving \(MS_W=24/3=8\) thousand-squared dollars.
Step 2: Calculate the variation among risk means. The overall mean of the three risk means is \((10+14+18)/3=14\). Their squared deviations sum to \((10-14)^2+(14-14)^2+(18-14)^2=16+0+16=32\). Therefore \(S_{\bar X}^2=32/(3-1)=16\).
Step 3: Estimate the components and K. Here \(\widehat v=8\), and \(\widehat a=16-8/2=12\). Therefore \(\widehat K=8/12=0.6667\), rounded to four decimal places. Using \(n=2\), the implied credibility weight is \(\widehat Z=2/(2+0.6667)=0.7500\), rounded to four decimal places.
Check and common mistake avoided. The between-risk sample variance of 16 includes \(v/n=8/2=4\) of process noise, leaving an estimated VHM of \(16-4=12\). Subtracting the full EPV of 8 instead would incorrectly give 8 for VHM and \(K=1\).
Worked Example: Keeping the Degrees of Freedom Straight
A hypothetical portfolio contains four risks with three observations each. Their observed values, in thousands of dollars, are shown below. Assume independent risks and the normal random-effects model \(X_{ij}=\mu+\alpha_i+\varepsilon_{ij}\), with \(\alpha_i\sim N(0,a)\) and independent \(\varepsilon_{ij}\sim N(0,v)\). The parameters \(a\) and \(v\) are unknown and are to be estimated; no interest or mortality assumption applies.
| Risk | Period 1 | Period 2 | Period 3 | Risk mean |
|---|---|---|---|---|
| A | 8 | 10 | 12 | 10 |
| B | 10 | 12 | 14 | 12 |
| C | 12 | 14 | 16 | 14 |
| D | 14 | 16 | 18 | 16 |
Step 1: Estimate EPV. Each risk’s deviations from its mean are \((-2,0,2)\), with squared deviations \(4+0+4=8\). Across four risks, \(SS_W=4(8)=32\). The correct within-risk degrees of freedom are \(m(n-1)=4(3-1)=8\), so \(MS_W=32/8=4\), giving \(\widehat v=4\).
Step 2: Estimate VHM. The grand mean of the four risk means is \((10+12+14+16)/4=13\). The squared deviations are \(9,1,1,9\), which sum to 20. Thus \(S_{\bar X}^2=20/(4-1)=20/3\). The process contribution to this variance is \(\widehat v/n=4/3\), so \(\widehat a=20/3-4/3=16/3\).
Step 3: Calculate K and the implied weight. The component ratio is \(\widehat K=4/(16/3)=12/16=0.7500\), rounded to four decimal places. For \(n=3\), \(\widehat Z=3/(3+0.75)=3/3.75=0.8000\), rounded to four decimal places.
Check and common mistake avoided. The among-risk sample variance has 3 degrees of freedom, not 4; the within-risk sum of squares has 8, not 12, because each group mean is estimated. Using 12 as the within-risk denominator would understate EPV and distort both component estimates.
Worked Example: A Small Change in Risk Means, a Large Change in K
Consider three hypothetical risks, each observed for two periods. In both versions of the data, the within-risk mean square is \(MS_W=9\) thousand-squared dollars. Assume independent risks and the normal model \(X_{ij}=\mu+\alpha_i+\varepsilon_{ij}\), with \(\alpha_i\sim N(0,a)\) and independent \(\varepsilon_{ij}\sim N(0,v)\); \(a\) and \(v\) are unknown. No interest or mortality assumption applies. Compare risk means \((10,14,18)\) with revised means \((11,14,17)\).
Original means. Their grand mean is 14, and the squared deviations sum to \(16+0+16=32\). With \(m-1=2\) degrees of freedom, \(S_{\bar X}^2=32/2=16\). Therefore \(\widehat a=16-9/2=11.5\), while \(\widehat v=9\). This gives \(\widehat K=9/11.5=0.7826\), rounded to four decimal places, and \(\widehat Z=2/(2+0.7826)=0.7188\), rounded to four decimal places.
Revised means. Their grand mean remains 14, but the squared deviations now sum to \(9+0+9=18\). Thus \(S_{\bar X}^2=18/2=9\) and \(\widehat a=9-9/2=4.5\). With the same \(\widehat v=9\), the new estimate is \(\widehat K=9/4.5=2.0000\), and the implied weight is \(\widehat Z=2/(2+2)=0.5000\).
Check and common mistake avoided. The revised risk means move by only one unit for the outer risks, yet the estimated VHM falls from 11.5 to 4.5 and K rises from 0.7826 to 2.0000. The resulting credibility weight falls from 0.7188 to 0.5000. Do not interpret a precise-looking ratio as proof that a small portfolio has estimated the structural components precisely.
What Small Samples Tell You
The degrees of freedom quantify how limited the information is. With \(m\) risks, only \(m-1\) degrees of freedom inform the between-risk component. Adding periods per risk increases the within-risk degrees of freedom and reduces the process-noise contribution \(v/n\) to each risk mean, but it does not create additional risks. Adding risks supplies more information about the distribution of hypothetical means.
Under the normal random-effects model, \(MS_W/v\) has a chi-square distribution divided by its degrees of freedom, with \(m(n-1)\) degrees of freedom. Also, \(MS_B/(v+na)\) has a chi-square distribution divided by \(m-1\) degrees of freedom, and these two mean squares are independent. When either degree count is small, its estimate can vary substantially from sample to sample. The exact distributions require normality; the component expectation relationships do not.
The ratio \(\widehat K=\widehat v/\widehat a\) can amplify this variation. In particular, when the estimated VHM is small and positive, a modest change in risk means can greatly increase \(\widehat K\), reducing the implied weight for a fixed \(n\). If \(\widehat a\) is zero or negative, the plug-in ratio does not provide a usable positive K estimate. Do not silently replace such a result with a convenient positive number; the treatment of boundary estimates is a separate modeling decision.
Check Your Understanding
For each problem, use the balanced-portfolio estimators. Unless noted otherwise, assume the component estimates are positive. Round K and credibility weights to four decimal places.
- A portfolio has three risks observed for two periods. The within-risk mean square is 6, and the risk means are 8, 12, and 16. Calculate the estimated VHM, K, and the implied weight for \(n=2\). (Answer: \(\widehat a=13,\ \widehat K=0.4615,\ \widehat Z=0.8125\))
- Four risks are observed for two periods. The within-risk mean square is 5, and the risk means are 9, 11, 15, and 17. Calculate the estimated VHM, K, and the implied weight for \(n=2\). (Answer: \(\widehat a=65/6=10.8333,\ \widehat K=0.4615,\ \widehat Z=0.8125\))
- Five risks are observed for three periods. The within-risk mean square is 9, and the risk means are 10, 12, 14, 16, and 18. Calculate the estimated VHM, K, and the implied weight for \(n=3\). (Answer: \(\widehat a=7,\ \widehat K=1.2857,\ \widehat Z=0.7000\))
- A balanced portfolio has \(m=3\) risks and \(n=4\) periods per risk. How many within-risk and between-risk degrees of freedom are available, and how many observations are in the portfolio? (Answer: 9 within-risk degrees of freedom, 2 between-risk degrees of freedom, and 12 observations)
- For three risks observed over two periods, \(MS_W=4\). Compare risk means \((10,13,16)\) with revised means \((11,13,15)\). Calculate K for each set and the factor by which the second K exceeds the first. (Answer: first \(\widehat K=0.5714\); second \(\widehat K=2.0000\); the second is 3.5 times the first)