Estimating EPV from Repeated Observations

Why Repeated Observations Estimate EPV

In the Bühlmann model, observations vary for two distinct reasons: risks may have different expected outcomes, and outcomes for the same risk may fluctuate from observation to observation. The expected process variance, or EPV, describes the second source of variation, averaged across the risk population. Repeated observations on each risk let us estimate that within-risk variation without treating differences in risk means as process noise.

For example, two hypothetical insured risks might have different expected annual claim costs. A difference between their observed averages is not, by itself, evidence about the process variance. EPV is estimated from how observations fluctuate within each risk, then summarized across risks. The result is used as a component of Bühlmann credibility, but this tutorial focuses on estimating EPV rather than estimating the other model component or constructing a credibility premium.

Definition: Let $\Theta$ denote a risk parameter drawn from the risk population. Given $\Theta$, let $X_1,\ldots,X_n$ be repeated observations with conditional mean $\mu(\Theta)=E[X_j\mid\Theta]$ and conditional variance $\sigma^2(\Theta)=\operatorname{Var}(X_j\mid\Theta)$. The expected process variance is $$ \mathrm{EPV}=E[\operatorname{Var}(X_j\mid\Theta)]=E[\sigma^2(\Theta)]. $$ It is the average conditional variance over the risk population; its units are the square of the units of $X_j$.

The usual estimation setup assumes that, conditional on a risk's parameter, its repeated observations are independent and identically distributed, with finite variance. Risks in the sample are taken to represent the target risk population. Each risk must have at least two observations to calculate a sample variance. For the simple equal-risk average, the number of observations can differ, provided the observation counts are fixed or otherwise not informative about risk characteristics and each sampled risk is intended to receive equal weight.

Risk-Specific Sample Variance

For risk $i$, write its observations as $X_{i1},\ldots,X_{in_i}$, and its sample mean as $\bar X_i$. The sample variance for that risk is $$ s_i^2=\frac{\sum_{j=1}^{n_i}(X_{ij}-\bar X_i)^2}{n_i-1}. $$ The denominator is $n_i-1$, not $n_i$. This is the usual unbiased sample variance: conditional on the risk's parameter, its expected value is the risk's conditional variance.

E[s_i^2\mid\Theta_i]=\sigma^2(\Theta_i) \qquad\Longrightarrow\qquad E[s_i^2]=E[\sigma^2(\Theta_i)]=\mathrm{EPV}.

This result explains the basic estimator. Calculate one sample variance for each risk, then average those risk-specific variances to estimate the population-average process variance. A sample variance is noisy: it does not reveal the exact conditional variance for its particular risk. Averaging across representative risks reduces the effect of that sampling noise.

Estimator: If $m$ sampled risks represent the target population and each risk is to receive equal weight, an unbiased estimator of EPV is $$ \widehat{\mathrm{EPV}}_{\mathrm{equal\ risk}}=\frac{1}{m}\sum_{i=1}^{m}s_i^2. $$ When every risk has the same number $n$ of observations, this is also the pooled within-risk sum of squares divided by the total within-risk degrees of freedom: $$ \widehat{\mathrm{EPV}}=\frac{\sum_{i=1}^{m}(n-1)s_i^2}{m(n-1)}. $$

Worked Example: Estimating EPV for Three Risks

A hypothetical insurer observes four annual loss amounts, in thousands of dollars, for each of three risks. Assume the annual observations are conditionally independent and identically distributed within each risk, all have the same exposure basis, and the three risks represent the target population. Estimate EPV by the equal-risk method.

Risk A: The observations are $4,6,5,5$. Their mean is $(4+6+5+5)/4=5$. The squared deviations sum to $ (4-5)^2+(6-5)^2+(5-5)^2+(5-5)^2=1+1+0+0=2$. Thus $s_A^2=2/(4-1)=2/3$.

Risk B: The observations are $6,10,8,8$. Their mean is $32/4=8$. The squared deviations sum to $4+4+0+0=8$, so $s_B^2=8/3$.

Risk C: The observations are $1,3,2,2$. Their mean is $8/4=2$. The squared deviations sum to $1+1+0+0=2$, so $s_C^2=2/3$.

The equal-risk estimate is $(2/3+8/3+2/3)/3=(12/3)/3=4/3$ thousand-dollars-squared. As a check, the within-risk sums of squares total $2+8+2=12$, and there are $3(4-1)=9$ within-risk degrees of freedom. Thus $12/9=4/3$, the same estimate. This method avoids using variation between the three risk means as though it were process variation.

Why the Sample Variance Works

The justification follows from the standard within-risk decomposition. Conditional on $\Theta_i$, suppose the observations for risk $i$ share mean $\mu_i$ and variance $\sigma_i^2$. For $n_i$ observations, the sum of squared deviations from their sample mean can be written as $$ \sum_{j=1}^{n_i}(X_{ij}-\bar X_i)^2 = \sum_{j=1}^{n_i}(X_{ij}-\mu_i)^2 - n_i(\bar X_i-\mu_i)^2. $$ Taking the conditional expectation, the first sum has expectation $n_i\sigma_i^2$. Since $\operatorname{Var}(\bar X_i\mid\Theta_i)=\sigma_i^2/n_i$, the second term has expectation $n_i(\sigma_i^2/n_i)=\sigma_i^2$. Therefore $$ E\left[\sum_{j=1}^{n_i}(X_{ij}-\bar X_i)^2\middle|\Theta_i\right] =(n_i-1)\sigma_i^2. $$ Dividing by $n_i-1$ gives $E[s_i^2\mid\Theta_i]=\sigma_i^2$. Averaging this conditional equality across the risk population gives $E[s_i^2]=\mathrm{EPV}$.

The subtraction of $n_i(\bar X_i-\mu_i)^2$ is important: estimating the mean from the same observations costs one degree of freedom. Dividing by $n_i$ instead would, on average, underestimate the within-risk variance by the factor $(n_i-1)/n_i$.

Unequal Observation Counts and Pooling

If the number of observations differs by risk, two calculations that coincide in a balanced sample need not agree. The equal-risk estimator averages the $s_i^2$ values, giving each sampled risk the same contribution. The familiar pooled within-risk estimator instead is $$ \widehat{\mathrm{EPV}}_{\mathrm{pooled}} = \frac{\sum_{i=1}^{m}(n_i-1)s_i^2}{\sum_{i=1}^{m}(n_i-1)}. $$ It gives more weight to a risk with more within-risk degrees of freedom. This is appropriate for a common conditional variance shared by all risks. In the Bühlmann setting, where conditional variances can vary with $\Theta_i$, it generally estimates a degrees-of-freedom-weighted average of the sampled risks' conditional variances, not necessarily the equal-risk EPV. If observation counts are associated with risk characteristics, that distinction can matter even more.

Accordingly, decide what population average the estimate is intended to represent and how risks were sampled. If the target is the average conditional variance for a randomly selected risk, and the sampled risks represent that population equally, average the risk-specific sample variances. Do not assume that pooling by degrees of freedom automatically estimates that same target when $n_i$ differs.

Worked Example: Unequal Counts Change the Pooled Estimate

A hypothetical study samples three risks equally from a target population but takes different numbers of observations from them. The observations are on the same scale, and within each risk they are conditionally independent and identically distributed. Risk A has two observations, $1,5$; Risk B has three, $2,4,6$; and Risk C has five, $1,3,5,7,9$. Estimate the equal-risk EPV and compare it with degrees-of-freedom pooling.

Risk A: Its mean is $3$, and its squared deviations sum to $4+4=8$. Thus $s_A^2=8/(2-1)=8$.

Risk B: Its mean is $4$, and its squared deviations sum to $4+0+4=8$. Thus $s_B^2=8/(3-1)=4$.

Risk C: Its mean is $5$, and its squared deviations sum to $16+4+0+4+16=40$. Thus $s_C^2=40/(5-1)=10$.

The target gives each risk equal weight, so the estimate is $(8+4+10)/3=22/3$, or approximately $7.333$ squared units. The degrees-of-freedom pooled calculation is $(1\cdot8+2\cdot4+4\cdot10)/(1+2+4)=56/7=8$ squared units. These are different because the pooled estimate weights the risks in proportion to $n_i-1$. The common mistake is to report 8 as the equal-risk estimate; for the stated target, the correction is the unweighted average, $22/3$.

Equal Observation Units and Exposure Scales

Repeated observations must measure the same kind of quantity on a comparable basis. If an observation represents one year of loss per policy, another observation that represents five years of aggregate loss is not automatically comparable. A change in observation duration or exposure can change the variance even when the underlying risk process is unchanged.

The equal-observation assumption means that each $X_{ij}$ is defined on the same measurement and exposure unit for the intended EPV. If observations are related by a known deterministic scale factor, first express them on a common basis, or explicitly account for how that scaling changes the variance. For a constant $c$, $\operatorname{Var}(cX\mid\Theta)=c^2\operatorname{Var}(X\mid\Theta)$; variance scales by the square of the factor. If exposure effects are more complex, simply dividing by exposure may not produce observations with a common conditional variance.

Worked Example: Standardizing a Deterministic Exposure Scale

A hypothetical portfolio study records observations as exposure-multiplied amounts $Y=eR$, where $R$ is the comparable per-unit loss measure and $e$ is a fixed multiplier within each risk. Three standardized observations for Risk A are $8,10,12$, with $e=2$; for Risk B they are $6,10,14$, with $e=3$. Assume the risks receive equal weight and the standardized observations satisfy the repeated-observation assumptions. Estimate EPV for $R$, not for $Y$.

For Risk A, the mean of $R$ is $10$, and its squared deviations sum to $4+0+4=8$. Thus $s_{R,A}^2=8/(3-1)=4$. Its recorded $Y$ observations are $16,20,24$; their sample variance is $16$. This confirms $s_{Y,A}^2=e^2s_{R,A}^2=2^2(4)=16$.

For Risk B, the mean of $R$ is $10$, and its squared deviations sum to $16+0+16=32$. Thus $s_{R,B}^2=32/2=16$. The recorded $Y$ observations are $18,30,42$; their squared deviations from $30$ sum to $144+0+144=288$, so $s_{Y,B}^2=288/2=144=3^2(16)$.

The equal-risk EPV estimate on the standardized $R$ scale is $(4+16)/2=10$ squared per-unit-loss units. Averaging the raw variances of $Y$ would give $(16+144)/2=80$, which estimates variation on the exposure-multiplied scale, not the requested scale. The correction is to standardize each variance by $e^2$ before averaging. This method applies because the problem specifies a deterministic multiplier; it should not be presumed for arbitrary exposure patterns or loss processes.

Common Errors and Conditions for Use

An EPV estimate is meaningful only relative to its data construction and target population. Before calculating it, check that observations are repeated measurements for the same risk, expressed in comparable units, and that the conditional independence and equal-distribution assumptions are reasonable. If the observations are serially dependent or have changing conditional variances over time, the ordinary sample variance need not be an unbiased estimate of the intended process variance.

Mixing within- and between-risk variation. Calculate deviations from each risk's own mean, not from the overall mean, when estimating process variance.
Using $n_i$ in the sample-variance denominator. The unbiased risk-specific sample variance uses $n_i-1$.
Pooling unequal groups without checking the target. Degrees-of-freedom pooling weights risks differently from an equal-risk average.
Comparing different exposure units directly. A deterministic multiplier changes variance by its square; raw variances may not be on the target scale.
Interpreting a risk-specific sample variance as exact. It is an estimate with sampling error, not the known conditional variance for that risk.

Key takeaway: Under conditional independence and identical distribution within risk, each risk-specific sample variance is unbiased for that risk's conditional variance. Average those estimates across representative risks to estimate EPV for the corresponding risk population and scale. Pool by degrees of freedom only when that weighting matches the variance target.

Check Your Understanding

For each problem, use the stated target and observation scale when selecting the estimator.

Two equally weighted risks each have three repeated observations. Their observations are $2,4,6$ and $1,3,5$. Calculate each sample variance and estimate the equal-risk EPV. (Answer: $4$ squared units.)
Three equally weighted risks have equal observation counts, and their risk-specific sample variances are $2,5,11$. Estimate EPV. (Answer: $6$ squared units.)
Three equally weighted risks have sample sizes $2,4,5$ and sample variances $6,3,9$, respectively. Calculate the equal-risk estimate and the degrees-of-freedom pooled estimate. (Answer: Equal-risk estimate $=6$; pooled estimate $=51/8=6.375$, both in squared units.)
Two equally weighted risks have standardized per-unit sample variances $3$ and $7$. Their recorded observations are deterministic multiples of those standardized observations, with multipliers $2$ and $5$, respectively. What is the equal-risk EPV estimate on the standardized scale, and what would averaging the raw recorded variances produce if both risks have the same number of observations? (Answer: Standardized estimate $=5$; raw-scale average $=(2^2\cdot3+5^2\cdot7)/2=93.5$ squared recorded units.)
Two equally weighted risks each have four observations. Risk A's data are $10,10,12,12$; Risk B's are $1,3,5,7$. Calculate the two sample variances and the equal-risk EPV estimate. (Answer: $s_A^2=4/3$, $s_B^2=20/3$, and the estimate is $4$ squared units.)

What You'll Learn