Estimating Hypothetical-Mean Variance From Risk Averages

Why Risk Averages Have More Variation Than Hypothetical Means

A risk average is not the same thing as the risk’s hypothetical mean. It is an estimate of that mean, based on a finite number of observations. Consequently, averages from different risks vary for two reasons: the underlying risks may have different hypothetical means, and each observed average contains process variation from the finite sample used to calculate it. Estimating the variance of hypothetical means requires separating these sources of variation.

This distinction is central to the Bühlmann model components. In the model, the variance of hypothetical means describes how much expected outcomes differ across risks; the process variance describes variation around a risk’s own hypothetical mean. The goal here is to estimate the first component from observed risk averages while accounting for the second.

Definition: For risk $j$, let $\mu_j=E[X_{jt}\mid \text{risk }j]$ be its hypothetical mean, and let $\bar X_j$ be the average of its $n_j$ observed outcomes. The variance of hypothetical means is $\tau^2=\operatorname{Var}(\mu_j)$. The expected process variance is $\phi=E[\operatorname{Var}(X_{jt}\mid \text{risk }j)]$. Under a common-process-variance model, each risk has conditional variance $\phi$.

For the formulas below, assume risks are independent draws from the same population, so their hypothetical means have a common mean and variance $\tau^2$. Within each risk, observations are conditionally independent given that risk’s characteristics, and their conditional mean is $\mu_j$. Assume also that process variance is common across risks, or that the supplied value of $\phi$ represents the relevant expected process variance. These assumptions give each average a clear sampling-variance contribution. If observations are dependent within a risk, or process variances differ in a way not represented by $\phi$, the simple correction may not apply.

The Basic Correction for Balanced Risk Samples

Suppose every one of $r$ risks has the same number $n$ of observations. For a particular risk, conditional independence gives $\operatorname{Var}(\bar X_j\mid \text{risk }j)=\phi/n$. The total variance of $\bar X_j$ is therefore the variance among hypothetical means plus the sampling variance remaining in its average.

\operatorname{Var}(\bar X_j)=\tau^2+\frac{\phi}{n}

Let $S_{\bar X}^2$ be the ordinary sample variance of the $r$ risk averages, calculated with denominator $r-1$. Since the risk averages are independent and identically distributed under these assumptions, this sample variance estimates their variance. Thus, if $\phi$ is known, subtract the sampling variance $\phi/n$ from the observed variance of the averages.

Estimation rule: With balanced sample size $n$, the raw estimate of variance of hypothetical means is $$ \widehat{\tau^2}=S_{\bar X}^2-\frac{\phi}{n}. $$ If process variance is estimated by an unbiased estimate $\hat\phi$, the corresponding corrected estimate is $S_{\bar X}^2-\hat\phi/n$. The correction removes the estimated sampling variance in each risk average; it does not subtract $\hat\phi$ itself.

S_{\bar X}^2=\frac{1}{r-1}\sum_{j=1}^{r}(\bar X_j-\bar{\bar X})^2, \qquad \bar{\bar X}=\frac{1}{r}\sum_{j=1}^{r}\bar X_j

Worked Example: Correcting Four Risk Averages

A hypothetical portfolio has four independently sampled risks, each observed for two periods. Their period averages are $8, 10, 12,$ and $14$ thousand dollars. Suppose the process variance is known to be $4$ (thousand dollars)$^2$. Estimate the variance of hypothetical means.

Method. Calculate the sample variance of the risk averages, then subtract the process variance divided by the number of observations per risk.

Step 1: Average the risk averages. $$ \bar{\bar X}=\frac{8+10+12+14}{4}=11 $$

Step 2: Calculate their sample variance. The deviations from 11 are $-3,-1,1,3$, and their squared deviations sum to $9+1+1+9=20$. With $r-1=3$, $$ S_{\bar X}^2=\frac{20}{3}=6.6667 $$

Step 3: Calculate the sampling-variance correction. $$ \frac{\phi}{n}=\frac{4}{2}=2 $$

Step 4: Subtract the correction. $$ \widehat{\tau^2}=6.6667-2=4.6667 $$ The estimated variance of hypothetical means is $4.6667$ (thousand dollars)$^2$. As a check, adding back the estimated sampling variance gives $4.6667+2=6.6667$, the observed sample variance of the risk averages. The method avoids treating all variation among observed averages as true variation among risks.

Why the Correction Works

The formula follows directly from the variance of a risk average. Write each average as its risk’s hypothetical mean plus its within-risk sampling error: $$ \bar X_j=\mu_j+(\bar X_j-\mu_j). $$ The sampling error has conditional mean zero. Therefore its covariance with $\mu_j$ is zero: conditioning on the risk, the expected sampling error is zero, so $E[\mu_j(\bar X_j-\mu_j)]=0$. The variance of a sum with zero covariance is the sum of the variances.

Separate the two sources.
$\operatorname{Var}(\bar X_j)=\operatorname{Var}(\mu_j)+\operatorname{Var}(\bar X_j-\mu_j)$, because the covariance term is zero.

Evaluate each variance.
$\operatorname{Var}(\mu_j)=\tau^2$. Conditional independence gives $\operatorname{Var}(\bar X_j\mid \text{risk }j)=\phi/n$, so the unconditional average sampling variance is $\phi/n$.

Relate the result to the observed sample variance.
Thus $\operatorname{Var}(\bar X_j)=\tau^2+\phi/n$. The sample variance of independent risk averages estimates this total, so subtracting $\phi/n$ isolates $\tau^2$.

This is an unbiased correction when the assumptions hold and $\phi$ is known. If $\phi$ is replaced by an unbiased estimate, the difference also has the correct expectation, provided the estimate targets the relevant process variance. The quality of a particular estimate can still be affected by sampling error; a corrected estimate is not guaranteed to be positive in every finite data set.

Unequal Numbers of Observations per Risk

When risks have different sample sizes, their averages have different sampling variances. Keep the risk averages equally weighted in the sample variance $S_{\bar X}^2$. Under the stated common-variance assumptions, risk $j$’s average has variance $\tau^2+\phi/n_j$. The expected sample variance across the $r$ averages is then the between-risk variance plus the average sampling variance across risks.

E[S_{\bar X}^2]=\tau^2+\frac{\phi}{r}\sum_{j=1}^{r}\frac{1}{n_j}

The correction is therefore the mean of the $r$ risk-specific sampling variances, not the correction for a single typical risk unless that choice gives the same value. If $\phi$ is estimated, use $\hat\phi$ in place of $\phi$.

Unequal-sample-size rule: For equally weighted risk averages with sample sizes $n_1,\ldots,n_r$, the raw corrected estimate is $$ \widehat{\tau^2}=S_{\bar X}^2-\frac{\phi}{r}\sum_{j=1}^{r}\frac{1}{n_j}. $$ This rule does not apply unchanged to a weighted variance of the risk averages. Changing the weights changes the expected sampling contribution and requires a matching derivation.

To justify the unequal-size adjustment, let $Y_j=\bar X_j$, with common expectation $m=E[\mu_j]$ and variance $v_j=\tau^2+\phi/n_j$. The sample-variance numerator is $\sum_j(Y_j-\bar Y)^2=\sum_j(Y_j-m)^2-r(\bar Y-m)^2$. Taking expectations gives $\sum_jv_j-r(r^{-2}\sum_jv_j)=(r-1)\sum_jv_j/r$, since the $Y_j$ are independent. Dividing by $r-1$ yields $E[S_{\bar X}^2]=\sum_jv_j/r=\tau^2+\phi\sum_j(1/n_j)/r$, as claimed.

Worked Example: Unequal Exposure Counts

Three hypothetical risks have risk averages of $20, 23,$ and $29$ thousand dollars, based on $n_1=1,n_2=2,n_3=3$ observations, respectively. Assume a common process variance of $18$ (thousand dollars)$^2$. Estimate the variance of hypothetical means.

Method. Use the equally weighted sample variance of the three averages and subtract the average of their three sampling variances.

Step 1: Calculate the mean and observed sample variance. $$ \bar{\bar X}=\frac{20+23+29}{3}=24 $$ The deviations are $-4,-1,5$, with squared deviations $16,1,25$. Therefore, $$ S_{\bar X}^2=\frac{16+1+25}{3-1}=21 $$

Step 2: Calculate the unequal-size correction. $$ \frac{\phi}{r}\left(\frac{1}{n_1}+\frac{1}{n_2}+\frac{1}{n_3}\right) =\frac{18}{3}\left(1+\frac{1}{2}+\frac{1}{3}\right) =6\left(\frac{11}{6}\right)=11 $$

Step 3: Subtract. $$ \widehat{\tau^2}=21-11=10 $$ The estimate is $10$ (thousand dollars)$^2$. Adding the correction back gives $10+11=21$, matching the observed sample variance. The method avoids using one sample size for all three risks when their sampling variances differ.

Estimating Process Variance from Replicates

In applications, $\phi$ may not be supplied. If each risk has repeated observations, process variation can be estimated from variation within risks, where differences in hypothetical means do not contribute to the within-risk deviations. For balanced groups of size $n$, a pooled within-risk estimate is $$ \hat\phi=\frac{\sum_{j=1}^{r}\sum_{t=1}^{n}(X_{jt}-\bar X_j)^2}{r(n-1)}. $$ The denominator counts the within-risk degrees of freedom. This estimate can then be used in the balanced correction. With unequal group sizes, the pooled within-risk estimate uses the total within-risk sum of squares divided by $\sum_j(n_j-1)$, when the common-process-variance assumptions are appropriate.

Worked Example: Estimating Both the Correction and the Target

Four hypothetical risks each have three annual observations, measured in thousands of dollars. Their risk averages are $12,16,18,$ and $22$. The within-risk sums of squared deviations are $8,4,12,$ and $6$, respectively. Estimate the process variance from the within-risk data, then estimate the variance of hypothetical means.

Method. Pool within-risk squared deviations to estimate process variance, then subtract the resulting average-level sampling variance from the sample variance of risk averages.

Step 1: Estimate process variance. There are $4(3-1)=8$ within-risk degrees of freedom, and the total within-risk sum of squares is $8+4+12+6=30$. Hence, $$ \hat\phi=\frac{30}{8}=3.75 $$

Step 2: Calculate the variance of the risk averages. $$ \bar{\bar X}=\frac{12+16+18+22}{4}=17 $$ The deviations are $-5,-1,1,5$, whose squares sum to $25+1+1+25=52$. Thus, $$ S_{\bar X}^2=\frac{52}{4-1}=\frac{52}{3}=17.3333 $$

Step 3: Correct for the sampling variance in each average. $$ \frac{\hat\phi}{n}=\frac{3.75}{3}=1.25 $$

Step 4: Estimate the variance of hypothetical means. $$ \widehat{\tau^2}=17.3333-1.25=16.0833 $$ The estimate is $16.0833$ (thousand dollars)$^2$. As a check, $16.0833+1.25=17.3333$, which recovers the observed variance of the averages. The method avoids treating within-risk process fluctuation as between-risk variation.

Common Mistakes and Interpretation

A frequent error is to set the variance of hypothetical means equal to the sample variance of risk averages. That overstates the component whenever the risk averages retain positive sampling variance. The opposite error is subtracting the full process variance instead of the process variance of an average.

Common exam trap: In Worked Example 1, the incorrect calculation $6.6667-4=2.6667$ subtracts the process variance itself. The correct sampling contribution is $\phi/n=4/2=2$, so the corrected estimate is $6.6667-2=4.6667$. Averaging $n$ independent observations reduces process variance by a factor of $n$; it does not leave the full process variance in the average.

Use the sample-variance denominator. The ordinary sample variance across $r$ risk averages divides by $r-1$, not $r$.
Match units. If the observations are measured in thousands of dollars, both component variances are in squared thousands of dollars.
Respect unequal sample sizes. For equally weighted averages, use the average of $1/n_j$, not $1/\bar n$ without checking that it gives the same correction.
Distinguish raw estimates from constraints. The subtraction can produce a negative raw estimate in a small sample. Since a variance cannot be negative, that result signals that the estimated sampling contribution exceeds the observed spread; any constrained reporting rule should be stated separately rather than hidden in the arithmetic.

Estimating variance of hypothetical means matters because it quantifies genuine differences in expected outcomes among risks, after separating out ordinary process noise. The correction is simple, but its validity depends on the data structure: identify the risk averages, determine their sample sizes, estimate the relevant process variance, and use weights and assumptions that match the formula.

Check Your Understanding

For each problem, calculate the raw corrected estimate of variance of hypothetical means. Use the stated common-process-variance assumptions.

Five risks each have four observations. The sample variance of their risk averages is $9$, and the process variance is $12$. What is the corrected estimate? (Answer: $6$)
Three risks each have three observations, and their averages are $5,7,$ and $11$. The process variance is $6$. Calculate the sample variance of the averages and the corrected estimate. (Answer: $S_{\bar X}^2=28/3$, and $\widehat{\tau^2}=22/3$, approximately $7.3333$)
Three equally weighted risk averages are $4,8,$ and $10$, based on $1,2,$ and $4$ observations, respectively. The process variance is $9$. Calculate the unequal-size correction and the raw estimate. (Answer: correction $=21/4=5.25$; $\widehat{\tau^2}=49/12$, approximately $4.0833$)
Three risks each have two observations and averages $10,14,$ and $16$. Their within-risk sums of squares are $6,10,$ and $8$. Estimate process variance by pooling within-risk sums of squares, then find the corrected estimate of variance of hypothetical means. (Answer: $\hat\phi=8$, $S_{\bar X}^2=28/3$, and $\widehat{\tau^2}=16/3$, approximately $5.3333$)
Four risks each have two observations. Their risk-average sample variance is $3$, while process variance is $8$. Find the raw corrected estimate and interpret its sign. (Answer: $3-8/2=-1$; this is a negative raw estimate, not a possible negative population variance.)

What You'll Learn