Negative VHM Estimates and Practical Remedies

Why a Variance Estimate Can Be Negative

In Bühlmann credibility, the variance of hypothetical means describes how much the underlying expected outcomes vary across risks. It is a variance parameter, so its true value cannot be negative. Yet a method-of-moments estimate can be negative: the observed variation between risk averages may be smaller than the amount expected from ordinary within-risk sampling variation. This is a feature of subtracting two noisy estimates, not evidence that the true variance is below zero.

The practical question is what to do next. Setting a negative estimate to zero is a simple and sometimes appropriate remedy, but it is not a diagnosis. It is important to distinguish sampling fluctuation in a suitable data set from a mismatch in exposure, time period, measurement, or model assumptions.

Definition: Let \(\Theta\) denote a risk's hypothetical parameter and let \(X\) be an observation for that risk. The variance of hypothetical means (VHM) is \(a=\operatorname{Var}(E[X\mid\Theta])\). The expected value of the process variance (EPV) is \(v=E[\operatorname{Var}(X\mid\Theta)]\). Both are nonnegative population quantities. A method-of-moments estimate of \(a\), however, need not be nonnegative.

The Balanced-Portfolio Method-of-Moments Estimate

First consider a balanced portfolio with \(I\) risks and \(n\) observations for each risk. Write \(X_{ij}\) for observation \(j\) on risk \(i\), and \(\bar X_i\) for that risk's average. Use the following assumptions for the formula: risks are independent draws from the same population; observations are conditionally independent across periods given a risk's \(\Theta\); each risk has conditional mean \(E[X_{ij}\mid\Theta_i]=\mu(\Theta_i)\); and the conditional variance is a common constant \(v\). There are no changes in the scale or mix of observations that make one period incomparable with another.

The within-risk estimator measures variation around each risk's own average. The between-risk sample variance measures variation among the risk averages. The method-of-moments estimate of VHM is the latter less the estimated sampling contribution to it:

\widehat v = \frac{\displaystyle\sum_{i=1}^{I}\sum_{j=1}^{n}(X_{ij}-\bar X_i)^2} {I(n-1)}, \qquad S_{\bar X}^{\,2} = \frac{\displaystyle\sum_{i=1}^{I}(\bar X_i-\bar X_{\cdot})^2} {I-1}, \qquad \widehat a_{\mathrm{raw}} = S_{\bar X}^{\,2}-\frac{\widehat v}{n}.

Here \(\bar X_{\cdot}\) is the average of all \(I\) risk averages. The subscript “raw” distinguishes the direct method-of-moments estimate from a constrained practical estimate such as one truncated at zero.

Why the Subtraction Is Required

Even if all risks had exactly the same underlying expected outcome, their observed averages would differ because each average is based on a finite number of observations. The between-risk variation therefore contains both true risk-to-risk variation and sampling variation. The subtraction removes the latter on average.

\begin{aligned} \operatorname{Var}(\bar X_i) &=\operatorname{Var}\big(E[\bar X_i\mid\Theta_i]\big) +E\big[\operatorname{Var}(\bar X_i\mid\Theta_i)\big]\\ &=\operatorname{Var}\big(\mu(\Theta_i)\big)+\frac{v}{n}\\ &=a+\frac{v}{n}. \end{aligned}

The first equality is the variance decomposition into variation of conditional means and expected conditional variation. Conditional independence and the common conditional variance give \(\operatorname{Var}(\bar X_i\mid\Theta_i)=v/n\). Across independent, identically distributed risks, the expected sample variance of their averages is \(\operatorname{Var}(\bar X_i)\). Thus \(S_{\bar X}^{\,2}-\widehat v/n\) estimates \(a\). In any particular sample, however, the observed \(S_{\bar X}^{\,2}\) can be less than \(\widehat v/n\), producing a negative result.

This derivation relies on the balanced design and stated assumptions. With unequal exposures, a different, exposure-consistent Bühlmann-Straub estimation procedure is needed; applying the balanced formula to unadjusted observations does not fix the design. Likewise, changing conditional variances, dependence between periods, or changing underlying means can invalidate the simple variance decomposition used here.

Worked Example: A Negative Estimate in a Balanced Portfolio

A hypothetical portfolio has four risks, each observed twice. The recorded outcomes are \((9,11)\), \((10,12)\), \((11,13)\), and \((10,12)\). Estimate the EPV and raw VHM by the balanced method of moments, then interpret the sign.

Method. Calculate the four risk averages, the pooled within-risk variation, and the sample variance of the risk averages. Use dollars in the same units throughout.

Risk averages. They are \(10,11,12,11\), with grand average \(\bar X_{\cdot}=11\).

Within-risk sum of squares. Each pair differs from its own average by \(-1\) and \(1\), so each contributes \(1^2+1^2=2\). The total is \(4(2)=8\). Since \(I(n-1)=4(1)=4\),

\widehat v=\frac{8}{4}=2.

Between-risk sample variance. The squared deviations of the four averages from 11 are \(1,0,1,0\). Therefore,

S_{\bar X}^{\,2}=\frac{1+0+1+0}{4-1}=\frac{2}{3}.

There are \(n=2\) observations per risk, so the estimated sampling contribution is \(\widehat v/n=2/2=1\). The raw VHM estimate is

\widehat a_{\mathrm{raw}}=\frac{2}{3}-1=-\frac{1}{3}.

The negative value is a valid outcome of the unconstrained estimator, but it cannot be interpreted as a negative population variance or inserted as a VHM in a credibility calculation. The between-risk variation in this small sample did not exceed its estimated sampling contribution. This result alone does not establish that the true VHM is zero.

Truncating at Zero: What It Does and Does Not Do

A common practical remedy is to use the nonnegative estimate \(\widehat a=\max(0,\widehat a_{\mathrm{raw}})\). This respects the parameter space and allows a conventional credibility calculation to proceed. It is a boundary adjustment to an estimate, not a discovery that the population VHM equals zero. Repeated use of truncation can also introduce upward bias in estimates that would otherwise fluctuate around a small true value.

For the balanced Bühlmann credibility factor, when \(a>0\) and \(v>0\), the weight on a risk's average is \(Z=n/(n+v/a)\). With a truncated estimate of zero and positive \(\widehat v\), use the limiting interpretation \(Z=0\): the estimated premium is the collective mean, not the risk's experience average. Do not evaluate the formula by dividing by zero.

Worked Example: Truncation and Its Effect on a Premium

A hypothetical insurer observes four risks twice each. Their data are \((19,21)\), \((20,22)\), \((20,22)\), and \((21,23)\). Use the balanced method of moments, truncate a negative VHM estimate at zero, and find the credibility premium for a risk with average 22 when the collective mean is 21.

Within-risk estimate. Each pair has deviations \(-1,1\) from its average, giving within sum of squares 2 per risk and 8 in total. The denominator is \(4(2-1)=4\), so \(\widehat v=8/4=2\).

Between-risk estimate. The risk averages are \(20,21,21,22\), and their grand average is 21. The squared deviations sum to \(1+0+0+1=2\), so \(S_{\bar X}^{\,2}=2/3\). The sampling contribution is \(\widehat v/n=2/2=1\). Thus \(\widehat a_{\mathrm{raw}}=2/3-1=-1/3\), and the truncated estimate is \(\widehat a=0\).

Premium. Because \(\widehat a=0\) and \(\widehat v=2>0\), the limiting credibility weight is \(Z=0\). The premium is \(Z(22)+(1-Z)(21)=0(22)+1(21)=21\).

Check. The weights are 0 and 1, which sum to 1; with zero estimated VHM, the risk average has no weight under this fitted model. The answer is 21 in the same units as the observations.

When a Negative Result Calls for a Data Check

Truncation is most defensible when the observations fit the balanced model, the calculation has been checked, and a small negative estimate is plausibly due to sampling noise. Before using it, check that the data are comparable and that the estimation formula matches the design. A negative estimate can be a useful warning that something besides sampling variation is affecting the two variance estimates.

Check exposure and scale. Do not treat raw counts from unequal exposure periods as if they were comparable rates. Use an estimator suited to the exposure structure.
Check time effects. A shared trend, calendar-year shock, or change in operations can create dependence or make periods non-identically distributed.
Check measurement and data handling. Inconsistent units, coding errors, duplicated records, or omitted observations can distort within-risk or between-risk variation.
Check the risk definition. If the groups combine materially different types of risks, a single common population model may not describe their means or variances adequately.
Check the arithmetic. The within-risk denominator is \(I(n-1)\) in the balanced case, whereas the between-risk sample variance uses \(I-1\). Confusing these changes the estimate.

If a pattern suggests a shared time effect, dependence, or heterogeneous exposures, reconsider the data structure and assumptions and estimate using a model appropriate to that structure. Do not keep applying a balanced estimator to altered data and interpret a zero-truncated result as a cure. Conversely, with few risks, a negative value can occur even when the assumptions are reasonable; it is not, by itself, proof that the model is wrong.

Worked Example: A Common Calendar Shift Can Inflate Within-Risk Variation

A hypothetical balanced portfolio records two calendar years for each of four risks: \((8,12)\), \((9,13)\), \((10,14)\), and \((9,13)\). Calculate the raw balanced estimate, then identify a feature of the data that should be investigated before treating zero truncation as the full response.

Calculate the estimate. The risk averages are \(10,11,12,11\), with grand average 11. Their squared deviations sum to \(1+0+1+0=2\), so \(S_{\bar X}^{\,2}=2/3\).

Within each risk, the observations are 2 below and 2 above the risk average. Each risk contributes \(2^2+2^2=8\), and the total within sum of squares is \(4(8)=32\). The denominator is \(4(2-1)=4\), giving \(\widehat v=32/4=8\). Since \(n=2\), the estimated sampling contribution is \(8/2=4\).

\widehat a_{\mathrm{raw}}=\frac{2}{3}-4=-\frac{10}{3}.

Investigate the pattern. For every risk, the second-year outcome is exactly 4 greater than the first-year outcome. That common shift is a strong reason to check for a calendar-year effect. The simple balanced model treats the within-risk changes as process variation, but a shared period effect would make the observations dependent across risks and could inflate the pooled within-risk estimate.

Setting \(-10/3\) to zero makes the estimate nonnegative, but does not explain the repeated year-to-year shift. The appropriate next step is to check how calendar effects should be represented and whether a model accounting for them is needed. The computed negative value is correct under the stated arithmetic; its interpretation depends on whether the model assumptions are suitable.

Common Exam Trap

A negative raw estimate must not be used as though it were a valid variance parameter. For example, in the first worked example, substituting \(\widehat a_{\mathrm{raw}}=-1/3\), \(\widehat v=2\), and \(n=2\) into \(Z=n/(n+\widehat v/\widehat a)\) gives \(\widehat v/\widehat a=-6\), and then \(Z=2/(2-6)=-0.5\). This is not a meaningful credibility weight: it is negative and does not represent a weighted average of the collective mean and the risk average. The correction is to recognize that the raw estimate is outside the parameter space, investigate the data and assumptions, and, if proceeding with the simple model, apply an explicit nonnegative estimation rule such as truncation. Report which estimate was used.

Key takeaway: The balanced method-of-moments VHM estimate subtracts estimated within-risk sampling variance from observed between-risk variance. A negative result is possible because both components are estimated from finite data. Truncation at zero can make a credibility calculation feasible, but it cannot replace checking the data structure, assumptions, and arithmetic.

Check Your Understanding

For each problem, distinguish the raw method-of-moments estimate from any constrained estimate and state what the result implies.

A balanced portfolio has three risks observed twice: \((8,10)\), \((9,11)\), and \((10,12)\). Calculate \(\widehat v\), \(S_{\bar X}^{\,2}\), and \(\widehat a_{\mathrm{raw}}\). What is the estimate after truncation at zero? (Answer: \(\widehat v=2\), \(S_{\bar X}^{\,2}=1\), \(\widehat a_{\mathrm{raw}}=0\), and the truncated estimate is 0.)
Three risks each have two observations: \((3,5)\), \((4,6)\), and \((6,8)\). Calculate the balanced raw VHM estimate. (Answer: \(\widehat v=2\), \(S_{\bar X}^{\,2}=7/3\), and \(\widehat a_{\mathrm{raw}}=4/3\).)
A portfolio has one observation for each risk and no repeated periods. Can the balanced within-risk estimator \(\widehat v\) in this tutorial be calculated from those data? Explain the limitation. (Answer: No. With \(n=1\), there are no within-risk residual degrees of freedom, so this estimator cannot separate EPV from VHM.)
A negative raw VHM estimate is truncated to zero, while the estimated EPV is positive. Under the standard Bühlmann credibility formula, what is the limiting credibility weight on individual experience? (Answer: \(Z=0\); the collective mean receives all the weight under this fitted boundary estimate.)
For four risks observed twice, the second-period value is exactly 5 higher than the first-period value for every risk. Is truncation alone an adequate diagnosis of a negative VHM estimate? State the first feature to investigate. (Answer: No. Investigate a common calendar-period effect or other shared time change, since the repeated shift may violate the simple model's assumptions.)

What You'll Learn