Decomposing Predictive Variance for Claim Counts

Why a Forecast Has More Than Process Variation

A Poisson model with a known claim rate describes the random fluctuation in future counts around that rate. In ratemaking, however, the rate is often not known: it is estimated from experience and remains uncertain after updating. A forecast must allow for both sources of variation. Ignoring uncertainty in the rate generally understates the variance of future claims.

This tutorial considers a risk with claim rate $\lambda$ per unit of exposure. Conditional on $\lambda$, claims follow a Poisson process. The rate itself has a posterior distribution given the observed data. Future exposure is treated as fixed and known; if exposure is uncertain, that adds another source of predictive variation and is not included in the formulas below.

Definition: Let $N$ be the future claim count over fixed exposure $e$, and let the posterior distribution of the risk’s rate $\lambda$, after observing the available data, have mean $m$ and variance $s^2$. The process variance is the expected conditional variance $E[\operatorname{Var}(N\mid\lambda)]$. The parameter-uncertainty variance is the variance of the conditional mean $\operatorname{Var}(E[N\mid\lambda])$. Their sum is the predictive variance of $N$.

The Variance Decomposition

Assume that, conditional on the rate, the future count is Poisson with mean proportional to exposure:

N\mid\lambda \sim \operatorname{Poisson}(e\lambda), \qquad e\geq 0.

For a Poisson random variable, the conditional mean and variance are both $e\lambda$. Applying the law of total variance gives the predictive variance:

\begin{aligned} \operatorname{Var}(N) &= E[\operatorname{Var}(N\mid\lambda)] + \operatorname{Var}(E[N\mid\lambda])\\ &= E[e\lambda]+\operatorname{Var}(e\lambda)\\ &= em+e^2s^2. \end{aligned}

The first term, $em$, is expected process variance: even if the rate were known, Poisson counts would fluctuate. The second term, $e^2s^2$, reflects uncertainty about the rate. The square on exposure in the second term follows because the conditional mean is $e\lambda$, and $\operatorname{Var}(e\lambda)=e^2\operatorname{Var}(\lambda)$.

Theorem: If $N\mid\lambda\sim\operatorname{Poisson}(e\lambda)$, and the posterior distribution of $\lambda$ has finite mean $m$ and variance $s^2$, then $$ E[N]=em,\qquad \operatorname{Var}(N)=em+e^2s^2. $$ The first identity follows from $E[N]=E[E(N\mid\lambda)]=E[e\lambda]$. The variance identity follows from the law of total variance. The formula does not require a Gamma posterior, but it does require the stated conditional Poisson model and fixed exposure.

Using a Gamma Posterior

For a common conjugate model, use the shape-rate convention: a Gamma distribution with shape $\alpha$ and rate $\beta$ has mean $\alpha/\beta$ and variance $\alpha/\beta^2$. If the prior is $\operatorname{Gamma}(\alpha_0,\beta_0)$, and the observed count is $n$ over exposure $w$, the Poisson likelihood updates the posterior to $\operatorname{Gamma}(\alpha_0+n,\beta_0+w)$. This update assumes a constant rate over the observed and forecast periods, conditional independence of counts given that rate, and exposure measured in the same units throughout.

Writing the posterior parameters as $\alpha$ and $\beta$, substitute $m=\alpha/\beta$ and $s^2=\alpha/\beta^2$ into the decomposition:

E[N]=e\frac{\alpha}{\beta},\qquad \operatorname{Var}(N)=e\frac{\alpha}{\beta} +e^2\frac{\alpha}{\beta^2}.

A plug-in forecast replaces $\lambda$ by its posterior mean $\alpha/\beta$ and treats that estimate as if it were known. Its Poisson variance is $e\alpha/\beta$, which includes process variation but omits $e^2\alpha/\beta^2$. The plug-in variance is appropriate only when the rate is being treated as known, or when posterior uncertainty is negligible for the question at hand.

Worked Calculations

Worked Example: Updating the Rate and Decomposing Forecast Variance

A hypothetical portfolio segment has a claim rate with prior distribution $\operatorname{Gamma}(2,1)$, using the shape-rate convention. It produces 4 claims over 2 units of exposure. Calculate the predictive mean and variance of the claim count over the next 3 units of exposure, and compare the variance with the plug-in result. This is a frequency-only model: interest and mortality assumptions do not apply. The observed count likelihood is Poisson conditional on $\lambda$, and the future count is conditionally Poisson with fixed exposure 3.

Step 1: Update the posterior. The posterior shape is $2+4=6$, and the posterior rate is $1+2=3$. Thus $\lambda\mid\text{data}\sim\operatorname{Gamma}(6,3)$.

Step 2: Find the posterior rate moments. The posterior mean rate is $m=6/3=2$, and its variance is $s^2=6/3^2=6/9=2/3$.

Step 3: Calculate the predictive mean. For future exposure $e=3$, $E[N]=3(2)=6$ claims.

Step 4: Separate the variance components. The expected process variance is $3(2)=6$. The parameter-uncertainty component is $3^2(2/3)=9(2/3)=6$. Therefore, $\operatorname{Var}(N)=6+6=12$ claims squared.

Check and method. A plug-in Poisson calculation at rate 2 gives variance $3(2)=6$, exactly the process component and only half the full predictive variance. The method is the law of total variance; the common mistake is to report the plug-in variance as the full predictive variance. The result is internally consistent: the predictive mean is 6 and the predictive variance, 12, exceeds it because the rate is uncertain.

Worked Example: The Exposure Unit Changes Both Components

For a hypothetical risk, the posterior claim rate is $\operatorname{Gamma}(4,2)$, with shape 4 and rate 2. Forecast claims over 1.5 units of exposure. Find the posterior rate moments, the predictive mean, both variance components, and the total variance. This is a frequency-only setting, so interest and mortality assumptions do not apply. Conditional on the rate, the future count is Poisson; exposure is fixed at 1.5 units.

The posterior mean rate is $m=4/2=2$. The posterior rate variance is $s^2=4/2^2=4/4=1$. The predictive mean is $1.5(2)=3$ claims.

The expected process variance is $1.5(2)=3$ claims squared. The parameter-uncertainty variance is $1.5^2(1)=2.25$ claims squared. Adding the components gives $\operatorname{Var}(N)=3+2.25=5.25$ claims squared. The plug-in Poisson variance would be 3.

Check and method. The decomposition gives a predictive variance of 5.25, greater than the plug-in value by 2.25, exactly the parameter component. The common mistake is to multiply the rate variance by exposure only; the conditional mean scales as $e\lambda$, so the variance of that mean scales as $e^2s^2$, not $es^2$.

Worked Example: Combining Forecast Periods for One Risk

After observing experience, a hypothetical risk has posterior rate $\lambda\sim\operatorname{Gamma}(3,2)$. Forecast counts $N_1$ and $N_2$ over two nonoverlapping future periods with exposures 1 and 2. Conditional on the same $\lambda$, the period counts are independent Poisson variables. Calculate the variance of the total count and show why adding the two marginal predictive variances alone is insufficient. Interest and mortality assumptions do not apply to this frequency-only model.

The posterior rate mean is $m=3/2=1.5$, and its variance is $s^2=3/2^2=3/4=0.75$. The combined exposure is $1+2=3$. Conditional on $\lambda$, the total count is Poisson with mean $3\lambda$, so its process variance is $3(1.5)=4.5$. The parameter component is $3^2(0.75)=9(0.75)=6.75$. The total predictive variance is therefore $4.5+6.75=11.25$ claims squared.

The separate predictive variances are $\operatorname{Var}(N_1)=1(1.5)+1^2(0.75)=2.25$ and $\operatorname{Var}(N_2)=2(1.5)+2^2(0.75)=3+3=6$. Their sum is 8.25, not 11.25. Although the counts are conditionally independent given $\lambda$, they are not independent after $\lambda$ is integrated out. Their covariance is $\operatorname{Cov}(N_1,N_2)=1(2)\operatorname{Var}(\lambda)=2(0.75)=1.5$. Hence

\operatorname{Var}(N_1+N_2) =2.25+6+2(1.5)=11.25.

Check and method. This agrees with the calculation using combined exposure 3. The common mistake is to add marginal variances while treating the forecasts as unconditionally independent. Sharing the same uncertain rate makes high-rate outcomes tend to raise both period counts, so the covariance must be included.

Derivation and Interpretation

The decomposition can be understood by separating deviations around the conditional mean from deviations of that mean around the overall predictive mean. Write

N-E[N]=\{N-E[N\mid\lambda]\}+\{E[N\mid\lambda]-E[N]\}.

The first bracket is process variation conditional on the rate; the second is variation in the conditional mean as the uncertain rate varies. When the squared expression is averaged, the cross-product has expectation zero. In fact,

E[(N-E[N\mid\lambda])(E[N\mid\lambda]-E[N])] =E[(E[N\mid\lambda]-E[N])E[N-E[N\mid\lambda]\mid\lambda]]=0,

because $E[N-E[N\mid\lambda]\mid\lambda]=0$. The remaining two squared terms are $E[\operatorname{Var}(N\mid\lambda)]$ and $\operatorname{Var}(E[N\mid\lambda])$, respectively. This establishes why the components add rather than multiply or replace one another.

For a Gamma posterior, the predictive variance can also be written as

\operatorname{Var}(N) =e\frac{\alpha}{\beta}\left(1+\frac{e}{\beta}\right).

This form makes the exposure effect visible, but the two-term expression is usually more useful for interpretation. Process variance grows linearly with exposure. Parameter uncertainty grows quadratically with exposure when the same uncertain rate applies to the entire forecast. Therefore, extrapolating far beyond the observed experience can make the omitted parameter component particularly material.

Common Exam Traps

Using only the plug-in Poisson variance. The quantity $em$ is the process component, not the full predictive variance. Add $e^2s^2$ when the rate remains uncertain.
Using the wrong exposure power. Conditional variance is $e\lambda$, so its posterior expectation uses $e$. The variance of the conditional mean $e\lambda$ uses $e^2$.
Mixing up the Gamma rate and scale. Under the shape-rate convention used here, $E[\lambda]=\alpha/\beta$ and $\operatorname{Var}(\lambda)=\alpha/\beta^2$. These are not the formulas for a shape-scale parameterization.
Adding forecast-period variances without covariance. Counts in separate periods may be conditionally independent but dependent predictively when they share an uncertain rate.
Changing exposure units partway through. Observed and future exposures must use consistent units. If one exposure is measured in policy-years and another in months, convert before updating or forecasting.

Key takeaway: A claim-count forecast with an uncertain Poisson rate has predictive variance equal to expected process variance plus variance in the conditional mean. For fixed exposure $e$ and posterior rate moments $m$ and $s^2$, that is $em+e^2s^2$. A plug-in estimate omits the second term.

Check Your Understanding

For each problem, use fixed exposure and the conditional Poisson model. Gamma distributions use the shape-rate convention.

A hypothetical risk has prior rate $\operatorname{Gamma}(2,1)$ and records 3 claims over 2 exposure units. Find the predictive variance over the next 2 exposure units, showing the posterior parameters and both variance components. (Answer: posterior $\operatorname{Gamma}(5,3)$; process $10/3$, parameter $20/9$, total $50/9\approx5.5556$ claims squared.)
A risk’s posterior rate is $\operatorname{Gamma}(4,2)$. Find the predictive mean and variance over 0.5 exposure units, and state the plug-in variance. (Answer: mean $1$ claim; process and plug-in variance $1$; parameter variance $0.25$; total variance $1.25$ claims squared.)
A posterior rate is $\operatorname{Gamma}(5,4)$. Two future counts have exposures 0.5 and 1.5 and are conditionally independent given the common rate. Find their predictive covariance. (Answer: $0.5(1.5)(5/16)=15/64=0.2344$ claims squared.)
A posterior rate is $\operatorname{Gamma}(6,3)$, and future exposure is 2. Calculate the process component, parameter component, total predictive variance, and the parameter component’s percentage of the total. (Answer: process $4$, parameter $8/3$, total $20/3\approx6.6667$ claims squared; parameter component $40\%$.)
For posterior rate $\operatorname{Gamma}(4,2)$, find the positive future exposure at which the process and parameter-uncertainty components are equal. Give each component and the total variance at that exposure. (Answer: exposure $e=2$; each component is $4$, and total variance is $8$ claims squared.)

What You'll Learn