Deriving the Optimal Linear Weight

Why Minimize the Error in the Risk Mean?

A credibility estimate combines an individual risk’s observed experience with a collective mean. The central question here is how much weight to give the individual experience. The answer depends on two kinds of variation: fluctuation in observations for a fixed risk, and genuine differences in expected outcomes from one risk to another. The optimal linear weight balances those sources of uncertainty.

The target is the risk’s hypothetical mean, not necessarily the next observed loss. The hypothetical mean is the expected outcome for that risk given its underlying risk characteristics. A sample mean can be close to that target, but it also contains random process fluctuation. The collective mean provides a stable reference point, but it may differ from the target for a particular risk.

Definition: Let $\Theta$ represent an individual risk’s underlying characteristics. For period observations $X_1,\ldots,X_n$, define the risk’s hypothetical mean as $\mu(\Theta)=E[X_j\mid\Theta]$, and the collective mean as $m=E[\mu(\Theta)]$. The process variance per observation is $\text{EPV}=E[\operatorname{Var}(X_j\mid\Theta)]$, and the variance of hypothetical means is $\text{VHM}=\operatorname{Var}(\mu(\Theta))$. Both are assumed finite.

For the derivation, assume the observations are conditionally independent and identically distributed given $\Theta$, with the same conditional mean and variance in each period. Thus, for a fixed risk, the observations share a hypothetical mean, and their process fluctuations are independent. Assume also that the collective mean is known. These are the Bühlmann model assumptions for the calculation below; when periods have different exposures or conditional variances, the sample mean’s variance needs a different treatment.

Linear credibility estimate: For a weight $\alpha$, the estimate of the risk’s hypothetical mean is $\widehat{\mu}_{\alpha}=\alpha\bar X+(1-\alpha)m$, where $\bar X=(X_1+\cdots+X_n)/n$. The optimal linear weight minimizes $E[(\widehat{\mu}_{\alpha}-\mu(\Theta))^2]$, the unconditional mean squared error for estimating the risk’s hypothetical mean.

Deriving the Optimal Weight

The sample mean varies around the risk’s hypothetical mean because of process fluctuation. Conditional independence gives its conditional variance as the one-observation process variance divided by $n$. Averaging over risks gives $\operatorname{Var}(\bar X-\mu(\Theta))=\text{EPV}/n$. By contrast, the collective mean’s difference from a particular risk’s hypothetical mean has mean square $\text{VHM}$.

To combine these facts correctly, write the estimation error as the sum of a process term and a hypothetical-mean term. The cross-product has expectation zero: conditional on $\Theta$, the sample mean’s deviation from $\mu(\Theta)$ has expectation zero, while $m-\mu(\Theta)$ is fixed.

Write the estimation error.
Subtract the target $\mu(\Theta)$ from the weighted estimate. The result separates into a sample fluctuation and a collective-mean deviation.

Use conditional independence.
The sample mean’s process variance is $\text{EPV}/n$. Its process deviation has zero conditional mean, so its cross-product with the hypothetical-mean deviation contributes zero in expectation.

Minimize the resulting quadratic.
Differentiate the mean squared error with respect to $\alpha$, set the derivative to zero, and solve for the weight.

\begin{aligned} \widehat{\mu}_{\alpha}-\mu(\Theta) &=\alpha\bigl(\bar X-\mu(\Theta)\bigr) +(1-\alpha)\bigl(m-\mu(\Theta)\bigr),\\ E\!\left[\bigl(\widehat{\mu}_{\alpha}-\mu(\Theta)\bigr)^2\right] &=\alpha^2\frac{\text{EPV}}{n}+(1-\alpha)^2\text{VHM},\\ \frac{d}{d\alpha}E\!\left[\bigl(\widehat{\mu}_{\alpha}-\mu(\Theta)\bigr)^2\right] &=2\alpha\frac{\text{EPV}}{n}-2(1-\alpha)\text{VHM}=0,\\ \alpha^* &=\frac{\text{VHM}}{\text{VHM}+\text{EPV}/n}. \end{aligned}

The second derivative of the mean squared error is $2(\text{EPV}/n+\text{VHM})$, which is positive when at least one component is positive. Therefore the stationary point is the minimum. When both components are positive, $\alpha^*$ lies strictly between zero and one. The result explains the direction of the trade-off: more process variance per sample mean reduces the weight on individual experience, while more variation in hypothetical means increases it.

This derivation concerns the best estimate within the stated linear class under squared-error loss. It does not, by itself, assert that the estimate is a full Bayesian posterior mean for every possible distribution of $\Theta$ and $X_j$. Nor does it make process fluctuation disappear: it accounts for its contribution to estimation error.

Worked Applications

Worked Example: Balancing Two Sources of Variance

A hypothetical portfolio has known $\text{EPV}=400$ and $\text{VHM}=100$, in squared dollars per observation and squared dollars, respectively. A risk has four conditionally independent annual observations with sample mean $\bar X=\$130$. The collective mean is $m=\$110$. Find the optimal weight and credibility estimate. No interest or mortality assumption is involved. The variance components and conditional independence are the model assumptions; no particular distributional form is needed.

Step 1: Calculate the process variance of the sample mean. There are four observations, so this variance is $400/4=100$.

Step 2: Calculate the weight. The VHM is 100 and the sample-mean process variance is also 100. Therefore $\alpha^*=100/(100+100)=0.5$.

Step 3: Apply the weight. The estimate is $0.5(\$130)+0.5(\$110)=\$120$. It gives equal weight to the individual sample mean and the collective mean.

Step 4: Check against the minimized error. At this weight, the process contribution is $0.5^2(100)=25$, and the hypothetical-mean contribution is $(1-0.5)^2(100)=25$. The total mean squared error is $25+25=50$ squared dollars. This also agrees with $100(100)/(100+100)=50$, the minimum of the quadratic.

Method and trap avoided. Divide EPV by the number of observations before comparing it with VHM. Using EPV $400$ directly would incorrectly give a weight of $100/(100+400)=0.2$, as if the estimate used only one observation.

Worked Example: Obtaining the Components From a Two-Type Model

Consider a hypothetical risk population with two equally likely types. Conditional on type A, each annual outcome has mean $\$80$ and variance $900$ squared dollars; conditional on type B, the mean is $\$120$ and the variance is $1{,}600$ squared dollars. Given type, annual observations are independent and identically distributed. One risk has $n=5$ years of data and a sample mean of $\$118$. Find the optimal weight, estimate, and minimum mean squared error. There is no interest or mortality assumption. The stated type probabilities and conditional first and second moments define the needed model; no further distributional assumption is required.

Step 1: Find the collective mean. The types have probability $1/2$ each, so $m=(80+120)/2=\$100$.

Step 2: Find the VHM. The hypothetical means are 20 dollars below and above the collective mean. Thus $\text{VHM}=[(80-100)^2+(120-100)^2]/2=(400+400)/2=400$ squared dollars.

Step 3: Find the EPV. Average the conditional variances using the type probabilities: $\text{EPV}=(900+1{,}600)/2=1{,}250$ squared dollars per observation. The process variance of the five-year sample mean is $1{,}250/5=250$.

Step 4: Find the weight and estimate. The optimal weight is $400/(400+250)=8/13$, approximately $0.6154$. The estimate is $(8/13)(\$118)+(5/13)(\$100)=\$1{,}444/13=\$111.0769$, or $\$111.08$ to the nearest cent.

Step 5: Check the error. The minimized mean squared error is $(8/13)^2(250)+(5/13)^2(400)=26{,}000/169$, approximately $153.8462$ squared dollars. Direct substitution into the minimum formula gives $400(250)/(400+250)=1{,}000/6.5=153.8462$, as required.

Method and trap avoided. Use the probability-weighted variance of the type means for VHM, and the probability-weighted conditional variance for EPV. Neither component is obtained by simply averaging the type means or variances without accounting for how the types differ.

Worked Example: Unequal Conditional Variances Across Risk Types

A hypothetical portfolio contains three equally likely risk types with annual conditional means $\$50$, $\$80$, and $\$110$, and conditional variances $400$, $900$, and $1{,}600$ squared dollars, respectively. Annual observations are conditionally independent and identically distributed within each type. A risk has five years of data and sample mean $\$92$. Find its optimal linear estimate. No interest or mortality assumption applies; the type probabilities and conditional moments are specified, and a full distributional form is unnecessary.

Step 1: Calculate the collective mean. The average of the three equally likely hypothetical means is $m=(50+80+110)/3=\$80$.

Step 2: Calculate the VHM. The deviations from $\$80$ are $-\$30$, $0$, and $\$30$. Thus $\text{VHM}=(900+0+900)/3=600$ squared dollars.

Step 3: Calculate the EPV and sample-mean process variance. The EPV is the equally weighted conditional variance: $(400+900+1{,}600)/3=2{,}900/3$, approximately $966.6667$. With five observations, the sample-mean process variance is $(2{,}900/3)/5=580/3$, approximately $193.3333$.

Step 4: Calculate the weight and estimate. The weight is $600/(600+580/3)=1{,}800/2{,}380=90/119$, approximately $0.7563$. The estimate is $(90/119)(\$92)+(29/119)(\$80)=\$10{,}600/119=\$89.0756$, or $\$89.08$ to the nearest cent.

Step 5: Check the direction. The estimate of about $\$89.08$ lies between the individual sample mean of $\$92$ and the collective mean of $\$80$, and is closer to the sample mean because its weight is greater than one-half.

Method and trap avoided. Average the conditional variances to obtain EPV, then divide by five. Do not treat the variance for the highest-mean type as the process variance for every risk; the portfolio’s types are equally likely.

Boundary Cases and Common Exam Traps

The formula also clarifies limiting cases. If VHM is zero and EPV is positive, all risks have the same hypothetical mean, so the collective mean is the optimal estimate and $\alpha^*=0$. If EPV is zero and VHM is positive, observations reveal each risk’s hypothetical mean without process noise, so $\alpha^*=1$. If both components are zero, every risk has the same known mean and the two inputs coincide; the mean squared error is zero for every weight, so the weight is not uniquely identified by minimization.

The most important practical check is to ask what varies and at what level. EPV describes variation among observations for the same risk; VHM describes variation in the risk-level means across risks. The sample mean reduces the first source by a factor of $n$, but it does not reduce the VHM. Confusing these levels leads to a wrong comparison and therefore a wrong weight.

Using EPV instead of EPV divided by $n$. The relevant process variance is that of the sample mean, not one observation.
Reversing the numerator. The weight on individual experience has VHM in its numerator. A larger VHM means risk-level means differ more, making the collective mean less representative of a particular risk.
Targeting the wrong quantity. The derivation minimizes error in estimating $\mu(\Theta)$, not the error in predicting a future observation considered by itself.
Assuming unequal periods still have variance EPV divided by $n$. That expression depends on equal conditional variances and conditional independence. Unequal exposures or dependence require an appropriate variance calculation before minimizing error.

Key takeaway: Under the stated Bühlmann assumptions, the mean squared error is $\alpha^2\text{EPV}/n+(1-\alpha)^2\text{VHM}$. Minimizing it gives $\alpha^*=\text{VHM}/(\text{VHM}+\text{EPV}/n)$. Compare VHM with the process variance of the sample mean, not with the process variance of a single observation.

The derivation is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus. When applying it, state the target, the model assumptions, and how the variance components and observation count enter the calculation.

Check Your Understanding

For each problem, use the optimal weight on the risk’s sample mean. Unless a problem specifies otherwise, observations are conditionally independent and identically distributed, the collective mean is known, and the target is the risk’s hypothetical mean.

A portfolio has $\text{EPV}=225$ and $\text{VHM}=75$. A risk has three observations. What is the optimal weight? (Answer: $0.5000$)
A risk’s sample mean is $\$140$, the collective mean is $\$100$, $\text{EPV}=600$, $\text{VHM}=150$, and $n=4$. Find the optimal weight and estimate. (Answer: weight $0.5000$; estimate $\$120.00$)
Two equally likely hypothetical types have conditional means $\$40$ and $\$100$, and conditional variances $400$ and $900$ squared dollars. A risk has five observations. Find EPV, VHM, and the optimal weight. (Answer: EPV $=650$, VHM $=900$, weight $=90/103\approx0.8738$)
A risk has sample mean $\$125$, while the collective mean is $\$95$. Given $\text{EPV}=360$, $\text{VHM}=90$, and $n=2$, calculate the credibility estimate. (Answer: weight $=1/3$; estimate $\$105.00$)
For $\text{EPV}=500$, $\text{VHM}=200$, and $n=5$, calculate both the optimal weight and the minimized mean squared error. (Answer: weight $=2/3$; minimum mean squared error $=200(100)/(200+100)=200/3\approx66.6667$)

What You'll Learn