Two Prediction Targets in a Hierarchical Model
A forecast for a known risk and a forecast for a new risk answer different questions. For an existing risk, the analyst has observations from that risk and uses them to update what is known about its latent claim tendency. For a new risk, there is no risk-specific history to update: its latent tendency is drawn from the population distribution. Confusing these targets can change both the forecasted mean and the forecasted distribution.
The distinction matters even when the risks belong to the same class and share the same population model. An existing risk may have unusually high or low observed experience. A new risk has not yet supplied evidence of its own, so its forecast averages over the population of possible risk parameters instead.
These formulas assume that, conditional on the latent risk parameter, the past observations and future observations follow the specified model, and that the risk’s parameter remains stable over the prediction period. For the new-risk forecast, the population distribution is treated as known. If population parameters are themselves uncertain, that additional uncertainty must also be modeled; it is not included in the examples below.
Existing-Risk and New-Risk Means
For either target, the predictive mean is an average of the conditional mean over the appropriate distribution of the latent parameter. Thus, for an existing risk, \(E[Y\mid D]=E[E(Y\mid\Theta,D)\mid D]\); for a new risk, \(E[Y_{\mathrm{new}}]=E[E(Y_{\mathrm{new}}\mid\Theta)]\). The conditional model may be the same in both cases, but the distribution used for the averaging is different.
A useful diagnostic is to state the target before doing any calculation: Is the future observation from the risk that generated the data, or from a newly sampled risk? If it is the existing risk, use its posterior distribution. If it is a new risk with no individual history, use the population distribution—not the posterior of a different risk.
Worked Example: Two Future Bernoulli Exposures
In a hypothetical coverage class, each risk has a fixed claim probability \(p\). For each exposure, a claim indicator is Bernoulli conditional on \(p\), independently across exposures. The population prior is \(p\sim\operatorname{Beta}(1,2)\). An existing risk has one claim and one claim-free exposure. Compare its forecast for two future exposures with the forecast for a new risk. There is no discounting or mortality; the distributions and parameters are as stated.
Existing risk. Beta-Bernoulli conjugacy gives posterior parameters equal to prior parameters plus the observed successes and failures: \(\ p\mid D\sim\operatorname{Beta}(1+1,2+1)=\operatorname{Beta}(2,3)\). Its posterior mean claim probability is \(2/(2+3)=0.4000\). The expected number of claims in two future exposures is \(2(0.4000)=0.8000\).
Let \(M\) be the number of claims in those two exposures. Given \(p\), \(P(M=0\mid p)=(1-p)^2\). Integrating over the existing risk’s posterior, \(P(M=0\mid D)=E[(1-p)^2\mid D]\). For \(p\sim\operatorname{Beta}(a,b)\), this expectation is \(b(b+1)/[(a+b)(a+b+1)]\). Substituting \(a=2,b=3\) gives \(3\cdot4/(5\cdot6)=0.4000\).
New risk. With no risk-specific observations, use \(p\sim\operatorname{Beta}(1,2)\). The mean number of claims over two exposures is \(2(1/3)=0.6667\). Its probability of no claims is \(2\cdot3/(3\cdot4)=0.5000\).
The existing-risk forecast is therefore \(0.8000\) expected claims and probability \(0.4000\) of no claims; the new-risk forecast is \(0.6667\) expected claims and probability \(0.5000\) of no claims. The method is posterior predictive averaging for the existing risk and prior predictive averaging for the new risk. The common mistake is to use the prior for the existing risk or the posterior for the new risk. Both calculations are checked against their respective Beta parameters and two future exposures.
Why Predictive Variance Includes Risk Uncertainty
The forecast is a mixture over possible latent parameters. Even if the future count is Poisson conditional on a fixed rate, uncertainty about that rate adds variation to the predictive count. The law of total variance gives the decomposition directly.
The first term is average process variance: variation in future outcomes even if the latent parameter were known. The second term is variation in the conditional mean across plausible latent parameters. For a new risk, the same decomposition applies with the population distribution in place of the posterior.
For example, suppose \(Y\mid\lambda\sim\operatorname{Poisson}(s\lambda)\), where \(s\) is future exposure. Conditional on \(\lambda\), both the mean and variance are \(s\lambda\). Therefore, for any distribution of \(\lambda\) with finite variance,
This result follows by substituting \(E[Y\mid\lambda]=s\lambda\) and \(\operatorname{Var}(Y\mid\lambda)=s\lambda\) into the total-variance formula. It applies whether the distribution for \(\lambda\) is a posterior for an existing risk or a population distribution for a new risk, provided the corresponding moments exist.
Worked Example: Poisson-Gamma Forecasts for an Existing and a New Risk
Assume a hypothetical risk’s annual claim count, conditional on its fixed rate \(\lambda\), is Poisson with mean equal to exposure times \(\lambda\). The population distribution is \(\lambda\sim\operatorname{Gamma}(2,1)\), using shape-rate notation, so its mean is \(2/1=2\). An existing risk has three claims over two years of exposure. Compare one future year for that risk with one year for a new risk. No interest or mortality assumptions enter this claim-count calculation.
Existing risk. With Poisson observations and a Gamma(shape \(\alpha\), rate \(\beta\)) prior, data consisting of \(n\) claims over exposure \(t\) produce a Gamma(shape \(\alpha+n\), rate \(\beta+t\)) posterior. Here the posterior is \(\operatorname{Gamma}(2+3,1+2)=\operatorname{Gamma}(5,3)\), with mean \(5/3=1.6667\). The expected one-year count is therefore \(1.6667\).
For a Gamma(shape \(a\), rate \(b\)) rate and Poisson future count over exposure \(s\), the predictive probability of zero claims is \(E[e^{-s\lambda}]=(b/(b+s))^a\). For the existing risk, this is \((3/(3+1))^5=(3/4)^5=243/1024=0.2373\).
New risk. Use the population distribution \(\operatorname{Gamma}(2,1)\), not the existing risk’s posterior. Its expected one-year count is \(2/1=2.0000\), and its probability of zero claims is \((1/(1+1))^2=1/4=0.2500\).
Thus, the existing risk has a lower expected count and a lower probability of zero claims than the new risk under these data. The calculation uses the posterior for the known risk and the population prior for the new risk. A mistake would be to apply the existing risk’s three claims to the new risk’s forecast; that would answer a different question. The means and zero-count probabilities have been checked using the respective Gamma parameters.
Predictive Distributions Are Not Plug-In Distributions
A plug-in forecast replaces the latent parameter by a single estimate, often its posterior mean. A predictive forecast instead averages the conditional distribution over all parameter values supported by the posterior or population distribution. These are not generally the same distribution. For instance, a Poisson count mixed over a Gamma-distributed rate has a negative-binomial predictive distribution, not a Poisson distribution at the Gamma mean.
Worked Example: A Tail Probability and the Plug-In Trap
Use the Poisson-Gamma assumptions and the existing risk’s posterior from the previous example: \(\lambda\mid D\sim\operatorname{Gamma}(5,3)\). For one future year, calculate the predictive probability of at least two claims, then compare it with a Poisson plug-in calculation at the posterior mean. There is no discounting or mortality; the rate distribution and one-year exposure are unchanged.
For a Poisson count mixed over a Gamma(shape \(a\), rate \(b\)) rate, the probabilities of zero and one claim over exposure \(s\) are \(P(Y=0)=(b/(b+s))^a\) and \(P(Y=1)=P(Y=0)\,as/(b+s)\). With \(a=5,b=3,s=1\), \(P(Y=0)=(3/4)^5=243/1024=972/4096\), and \(P(Y=1)=(243/1024)(5/4)=1215/4096\).
Therefore, \(P(Y\ge2\mid D)=1-972/4096-1215/4096=1909/4096=0.4661\). For the plug-in calculation, use a Poisson mean of \(5/3\). It gives \(1-e^{-5/3}(1+5/3)\approx0.4963\). The plug-in value is not the predictive probability: it omits variation in \(\lambda\). The correct method is to mix the conditional probabilities over the posterior. The two probabilities are checked by subtracting the explicitly calculated zero- and one-claim probabilities from one.
Practical Interpretation and Common Traps
The phrase “new risk” means a risk with no individual observations used in the forecast. It does not necessarily mean a risk with no information at all. If a new-to-the-book risk arrives with credible external or prior-period experience that is modeled as its own data, those observations can update its parameter distribution. In that case, the forecast uses the resulting risk-specific posterior. Conversely, if no such individual information is used, the appropriate distribution is the population distribution.
- Do not carry one risk’s posterior to another risk. The observed risk’s posterior describes uncertainty about that risk’s latent parameter, not a generic member of the population.
- Do not mistake a predictive mean for a predictive distribution. A mean alone does not give a tail probability; integrating over parameter uncertainty can change the distribution’s spread and shape.
- Keep exposure units consistent. In the Poisson model, both the posterior rate update and future mean use exposure in the same units.
- State what is treated as known. The examples treat population distribution parameters as fixed. If they are estimated with uncertainty, the forecast must account for that additional layer.
Check Your Understanding
For each problem, identify whether the forecast uses a risk-specific posterior or the population distribution before calculating.
- A population claim probability has prior distribution \(\operatorname{Beta}(2,3)\). An existing risk has two claims in four Bernoulli exposures. Find its posterior mean claim probability and the mean for a new risk with no individual history. (Answer: existing \(4/9=0.4444\); new \(2/5=0.4000\).)
- For a hypothetical Poisson-Gamma model, \(\lambda\sim\operatorname{Gamma}(3,2)\) in shape-rate form. An existing risk has two claims over one year. Find its expected count and zero-claim probability over the next two years; then find the same quantities for a new risk. (Answer: existing posterior \(\operatorname{Gamma}(5,3)\), mean \(10/3=3.3333\), zero probability \((3/5)^5=0.0778\); new mean \(3.0000\), zero probability \(1/8=0.1250\).)
- A risk has \(p\sim\operatorname{Beta}(1,3)\) and one claim in three Bernoulli exposures. Find the probability of no claims in its next three exposures, both for this existing risk and for a new risk. (Answer: existing posterior \(\operatorname{Beta}(2,5)\), probability \(5/12=0.4167\); new risk probability \(1/2=0.5000\).)
- A hypothetical Poisson-Gamma population has \(\lambda\sim\operatorname{Gamma}(2,1)\). An existing risk has five claims over four years. Find its one-year expected count and zero-claim probability, and the corresponding values for a new risk. (Answer: existing posterior \(\operatorname{Gamma}(7,5)\), mean \(1.4000\), zero probability \((5/6)^7=0.2791\); new mean \(2.0000\), zero probability \(0.2500\).)
- Let \(\lambda\sim\operatorname{Gamma}(2,1)\), with one claim observed over one year for an existing risk. For two future years, calculate predictive variance for that risk and for a new risk. Use \(\operatorname{Var}(Y)=sE[\lambda]+s^2\operatorname{Var}(\lambda)\). (Answer: existing posterior \(\operatorname{Gamma}(3,2)\), variance \(2(3/2)+4(3/4)=6\); new risk variance \(2(2)+4(2)=12\).)