Why a VHM Estimate Can Be Negative
In the Bühlmann model, the variance of a risk’s hypothetical mean is the variance of the conditional means across risks. It is called the variance of hypothetical means, or VHM. In a finite portfolio, the observed variation among risk-level averages reflects both genuine variation in those conditional means and the sampling variation of the observations within each risk. Method-of-moments estimation separates those two contributions. Because it subtracts an estimated within-risk contribution from an estimated between-risk variation, the result can be negative when the data are limited or noisy.
A negative estimate is not a possible value of a variance. It is a warning about the estimation procedure and the information in the sample—not evidence that the underlying risk population has negative heterogeneity. This tutorial develops the estimator, identifies the boundary choice commonly made in practice, and explains why that choice must be reported with care.
Model and Balanced-Portfolio Estimator
Use a balanced portfolio with \(r\) risks and \(n\) observations for each risk. Assume risks are independently sampled from the same risk population. For each risk, assume its \(n\) observations are conditionally identically distributed given that risk’s \(\theta_i\): they have the same conditional mean \(m(\theta_i)\) and the same conditional variance. Conditional independence is also assumed. These conditions matter: independence alone would not ensure that observations within a risk have a common mean or variance.
Write \(\bar X_i=\frac{1}{n}\sum_{j=1}^n X_{ij}\) for the mean of risk \(i\)’s observations, and \(\bar X=\frac{1}{r}\sum_{i=1}^r\bar X_i\) for the portfolio mean. Define the within-risk pooled variance and the sample variance of risk-level means as follows:
Under the stated model, \(s_W^2\) estimates EPV. The sample variance \(s_B^2\) estimates the variance of observed risk means, which includes VHM and the conditional sampling variance of each risk mean. Since that latter variance is EPV divided by \(n\), the method-of-moments estimate is:
Why the Subtraction Appears
The adjustment \(s_W^2/n\) is not arbitrary. Conditional on \(\theta_i\), the \(n\) observations for risk \(i\) have the same mean and variance and are independent. Therefore, the conditional mean of \(\bar X_i\) is \(m(\theta_i)\), and its conditional variance is \(\operatorname{Var}(X_{ij}\mid\theta_i)/n\). Applying the law of total variance to the risk mean gives:
The between-risk sample variance \(s_B^2\) estimates \(\operatorname{Var}(\bar X_i)\), while \(s_W^2\) estimates EPV. Subtracting the estimated sampling component \(s_W^2/n\) from \(s_B^2\) therefore estimates VHM. This derivation depends on the common conditional mean and variance within each risk, as well as conditional independence. If observations within a risk have different conditional means, the conditional expectation of their average need not be the risk’s hypothetical mean; if their variances differ, the simple \(1/n\) variance adjustment need not apply.
Worked Examples
Worked Example: Diagnose a Negative Estimate From a Small Portfolio
A hypothetical insurer records two annual loss-cost observations, in thousands of dollars, for each of three risks. The observations are A: 8 and 12; B: 9 and 11; C: 10 and 14. Calculate the method-of-moments VHM estimate and explain its sign.
Method. There are \(r=3\) risks and \(n=2\) observations per risk. First calculate each risk mean: \(\bar X_A=10\), \(\bar X_B=10\), and \(\bar X_C=12\). The portfolio mean is \(\bar X=(10+10+12)/3=32/3\).
For A, the within-risk sum of squared deviations is \((8-10)^2+(12-10)^2=8\). For B, it is \((9-10)^2+(11-10)^2=2\). For C, it is \((10-12)^2+(14-12)^2=8\). Thus the total within-risk sum is \(18\), and \(s_W^2=18/[3(2-1)]=6\).
The squared deviations of the risk means from \(32/3\) are \(4/9\), \(4/9\), and \(16/9\). Their sum is \(24/9=8/3\), so \(s_B^2=(8/3)/(3-1)=4/3\). Therefore:
The estimate is \(-5/3\) in squared thousand-dollar units. The method subtracts an estimated within-risk contribution of \(3\) from a between-risk variance of only \(4/3\). With just three risks, sampling variation can make the observed between-risk spread too small to support a positive method-of-moments estimate. The negative result does not imply a negative true VHM; it says this sample does not yield a positive estimate by this method. The common mistake to avoid is reporting \(-5/3\) as a possible variance.
Worked Example: Apply a Zero Boundary and Interpret It
For a hypothetical balanced portfolio, suppose \(n=4\), \(s_W^2=16\), and \(s_B^2=3\), with variance quantities expressed in consistent squared-dollar units. Calculate the raw VHM estimate, then apply the convention of truncating a negative estimate at zero. State what happens to \(K\) and to the Bühlmann factor for a finite observation count \(n\).
Method. The raw estimate is \(s_B^2-s_W^2/n=3-16/4=3-4=-1\). Under the stated truncation convention, report \(\widehat{\text{VHM}}_{\text{trunc}}=\max(0,-1)=0\).
The untruncated ratio would be \(\hat K=16/(-1)=-16\). That is not a valid positive Bühlmann parameter. At the zero boundary, the ratio \(s_W^2/\widehat{\text{VHM}}_{\text{trunc}}\) is not finite: division by zero is undefined. In the limiting interpretation of the Bühlmann factor \(Z=n/(n+K)\), as \(K\) grows without bound, \(Z\) tends to zero for finite \(n\). Thus a zero-boundary VHM estimate corresponds to no credibility assigned to the risk’s own experience by this limiting factor—not to a finite estimated \(K\).
A responsible report preserves both pieces of information: the raw method-of-moments estimate is \(-1\), and the chosen nonnegative boundary estimate is \(0\). Truncation enforces the variance constraint, but it does not establish that the true VHM is exactly zero. The common mistake is to call \(K\) equal to zero; when VHM is zero and EPV is positive, the ratio defining \(K\) is not zero.
Worked Example: Compare a Positive Estimate With the Boundary Case
A hypothetical insurer observes three annual loss costs, in thousands of dollars, for each of four risks: A: 8, 10, 12; B: 10, 12, 14; C: 14, 16, 18; and D: 16, 18, 20. Calculate the variance-component estimates and \(K\), then calculate the Bühlmann factor for \(n=3\).
Method. The risk means are \(10,12,16,18\), and the portfolio mean is \((10+12+16+18)/4=14\). Each risk has within-risk squared deviations \(4+0+4=8\). The pooled within-risk estimate is \(s_W^2=(8+8+8+8)/[4(3-1)]=32/8=4\).
The squared deviations of the four risk means from 14 are \(16,4,4,16\), summing to 40. Hence \(s_B^2=40/(4-1)=40/3\). Adjust for the sampling variation of a three-observation mean:
Here the estimated VHM is positive, so the ordinary ratio is defined. With \(n=3\), the factor is \(Z=3/(3+1/3)=3/(10/3)=9/10=0.9\). This is a contrast with a boundary estimate: a positive VHM gives a finite \(K\), whereas a zero-truncated VHM does not. The common mistake to avoid is treating the observed between-risk variance \(40/3\) itself as VHM; doing so would omit the \(s_W^2/n=4/3\) sampling contribution.
Choosing and Reporting a Boundary
The nonnegative constraint on a variance makes truncating a negative method-of-moments estimate at zero an understandable practical convention. But this convention is a choice about how to handle an inadmissible estimate, not a correction that recovers the true VHM. It also is not automatically equivalent to a constrained estimate obtained by a different statistical method. A reader should be able to distinguish the raw estimate from the constrained value and understand the limitations of both.
Confirm equal observation counts, correct risk-level means, and the common conditional distribution assumptions required by the balanced-portfolio formulas.
Compute \(s_W^2\) and \(s_B^2\) from their respective sums of squares, then use \(s_B^2-s_W^2/n\). Do not confuse the between-risk variance with VHM.
If the result is negative, state that it is an inadmissible method-of-moments variance estimate arising from the observed sample; do not present it as a possible population variance.
If the estimate is truncated at zero, label it as a zero-boundary estimate. Do not report a finite \(K\) obtained by dividing by zero, and do not claim that the data prove the absence of heterogeneity.
A negative estimate can indicate that the portfolio has too few risks, that within-risk variation is large relative to the observed spread among risk means, or simply that sampling noise has produced an unfavorable balance. It is evidence that these data and this estimator do not support a positive VHM estimate; it is not, by itself, a test that the true VHM equals zero. In professional work, the limitation should be reported rather than hidden by an unexplained truncation.
Check Your Understanding
For each item, distinguish the raw method-of-moments result from any boundary interpretation.
- A balanced portfolio has \(n=2\), \(s_W^2=5\), and \(s_B^2=2\). Calculate the raw VHM estimate and its zero-truncated value. (Answer: raw VHM \(=2-5/2=-0.5\); zero-truncated VHM \(=0\).)
- A balanced portfolio has \(n=3\), \(s_W^2=9\), and \(s_B^2=8\). Calculate VHM and \(K\). (Answer: VHM \(=8-9/3=5\); \(K=9/5=1.8\).)
- A balanced portfolio has \(n=5\), \(s_W^2=12\), and \(s_B^2=2\). Does the raw estimate justify a finite positive \(K\)? State the raw and truncated VHM values. (Answer: raw VHM \(=2-12/5=-0.4\); truncated VHM \(=0\); no finite positive \(K\) follows from the zero boundary.)
- Why is conditional independence alone insufficient to justify \(\operatorname{Var}(\bar X_i\mid\theta_i)=\operatorname{Var}(X_{ij}\mid\theta_i)/n\) in this derivation? (Answer: the observations must also share a common conditional mean and variance; otherwise the average’s conditional expectation or variance need not have the stated form.)
- A report gives a raw VHM estimate of \(-2\), replaces it with zero, and says the estimated \(K\) is zero. Identify the error and give the correct interpretation. (Answer: \(K=\text{EPV}/\text{VHM}\) is undefined at VHM \(=0\) when EPV is positive, not zero; report the raw negative estimate, label zero as a boundary convention, and state that the data do not support a positive VHM estimate.)