Diagnosing Prior-Data Conflict in Credibility Models

Why Diagnose Prior-Data Conflict?

A credibility model combines prior information about a risk parameter with observed experience. Before updating, it is useful to ask a different question: if the prior and sampling model were appropriate, how unusual would experience like this be? This is a prior-data conflict diagnostic. It checks whether the observations look plausible under the model as specified, rather than merely calculating a posterior premium.

A small prior predictive tail probability can flag a mismatch. Possible explanations include an unrepresentative prior, an unsuitable likelihood, an incorrect exposure measure, a data-quality problem, or chance variation. The diagnostic does not identify which explanation is responsible, and it does not by itself prove that the model is wrong.

Definition: Let $\theta$ have prior distribution $\pi(\theta)$, and let $D$ denote data with sampling distribution $p(D\mid\theta)$. The prior predictive distribution of $D$ is $$ p(D)=\int p(D\mid\theta)\pi(\theta)\,d\theta. $$ A prior-data conflict is indicated when an observed feature of $D$ falls in a region that has low probability under this distribution. The feature or discrepancy being assessed should be stated, such as a high count, a large sample mean, or an unusually large loss.

This is a check of the joint assumptions: both the prior and the sampling model contribute to the predictive distribution. The same observation can be surprising under one stated model and ordinary under another. A tail probability is therefore meaningful only after the prior, likelihood, exposure, and chosen discrepancy have been specified.

Normal-Normal Model: Predicting a Sample Mean

Suppose $X_1,\ldots,X_n$ are conditionally independent observations with $X_i\mid\theta\sim N(\theta,\sigma^2)$, where $\sigma^2$ is known, and suppose $\theta\sim N(\mu_0,\tau^2)$. These assumptions imply that the sample mean has the prior predictive distribution

\bar X\sim N\left(\mu_0,\tau^2+\frac{\sigma^2}{n}\right).

The variance has two parts. Variation in $\theta$ across the prior contributes $\tau^2$; sampling variation in $\bar X$, conditional on $\theta$, contributes $\sigma^2/n$. Adding only the conditional sampling variance would ignore uncertainty about the parameter and make the predictive distribution too narrow.

Derivation: Write $\bar X=\theta+(\bar X-\theta)$. Under the model, $E[\bar X\mid\theta]=\theta$ and $\operatorname{Var}(\bar X\mid\theta)=\sigma^2/n$. The law of total expectation gives $E[\bar X]=E[\theta]=\mu_0$. The law of total variance gives $$ \operatorname{Var}(\bar X) =E[\operatorname{Var}(\bar X\mid\theta)] +\operatorname{Var}(E[\bar X\mid\theta]) =\frac{\sigma^2}{n}+\tau^2. $$ Because $\theta$ and the conditional sampling error are normal, their sum is normal, establishing the stated predictive distribution.

Worked Example: A Sample Mean Far Above Its Prior Center

A hypothetical portfolio has five independent claim costs $X_i\mid\theta\sim N(\theta,20)$, with known conditional variance 20 dollars squared. The prior is $\theta\sim N(10,1)$, where the second parameter is a variance. The observed costs are 11, 13, 15, 17, and 19 dollars. Assess the two-sided prior predictive tail probability for a sample mean at least as far from the prior mean as the observed mean.

Method. Use the normal prior predictive distribution of $\bar X$, then calculate the probability of a deviation at least as large as the observed absolute deviation.

Step 1: Calculate the observed mean and deviation. $$ \bar x=\frac{11+13+15+17+19}{5}=\frac{75}{5}=15, \qquad |\bar x-\mu_0|=|15-10|=5. $$

Step 2: Calculate the predictive variance and standard deviation. $$ \operatorname{Var}(\bar X)=1+\frac{20}{5}=5, \qquad \operatorname{SD}(\bar X)=\sqrt{5}. $$ Thus the predictive distribution is $N(10,5)$, with variance 5.

Step 3: Standardize and calculate both tails. The observed standardized deviation is $5/\sqrt{5}=\sqrt{5}$. By symmetry, $$ P\left(|\bar X-10|\geq 5\right) =2\left[1-\Phi(\sqrt{5})\right] \approx 0.0253473. $$ To four decimal places, the tail probability is 0.0253.

Check and interpretation. The predictive mean is 10, and the observed mean is five dollars above it, so this is an upper-and-lower-tail calculation, not a one-sided upper-tail probability. Under the stated model, deviations this large or larger are uncommon. That flags potential conflict but does not establish whether the prior, normal likelihood, or observations need review. The calculation avoids rounding $\Phi(\sqrt{5})$ before subtracting it from 1; premature rounding can change the final fourth decimal place.

Gamma-Poisson Model: Predicting Claim Counts

For a frequency diagnostic, let the claim count $N$, conditional on a risk’s rate $\lambda$, be Poisson with mean $e\lambda$, where $e$ is exposure measured in the rate’s time units. Use a gamma prior with shape $\alpha$ and rate $\beta$, so its density is $\beta^\alpha\lambda^{\alpha-1}e^{-\beta\lambda}/\Gamma(\alpha)$. Integrating out $\lambda$ gives a negative binomial prior predictive count distribution:

P(N=k)= \frac{\Gamma(k+\alpha)}{\Gamma(\alpha)k!} \left(\frac{\beta}{\beta+e}\right)^\alpha \left(\frac{e}{\beta+e}\right)^k, \qquad k=0,1,2,\ldots

Its mean and variance are

E[N]=\frac{e\alpha}{\beta}, \qquad \operatorname{Var}(N)=\frac{e\alpha}{\beta} +\frac{e^2\alpha}{\beta^2}.

The variance exceeds the mean when the rate itself varies across the prior: this extra variation is part of the predictive distribution and should not be discarded by treating the prior mean rate as known. Exposure must also be in the same units used for $\lambda$; using a count from a longer period without adjusting $e$ can create a false conflict.

Worked Example: An Unusually High Annual Claim Count

A hypothetical risk has annual claim rate $\lambda$, with $N\mid\lambda\sim\operatorname{Poisson}(\lambda)$ for one year and $\lambda\sim\operatorname{Gamma}(2,1)$ in shape-rate form. The observed one-year count is 8. Calculate the prior predictive probability of observing 8 or more claims.

Method. Substitute $\alpha=2$, $\beta=1$, and $e=1$ into the negative binomial predictive probability, then sum its upper tail.

Step 1: Write the predictive probability for an individual count. $$ P(N=k)= \frac{\Gamma(k+2)}{\Gamma(2)k!} \left(\frac12\right)^2\left(\frac12\right)^k =\frac{k+1}{2^{k+2}}. $$ Here $\Gamma(k+2)/(\Gamma(2)k!)=k+1$.

Step 2: Sum from 8 upward. For this distribution, the tail beginning at integer $m$ is $$ P(N\geq m)=\sum_{k=m}^{\infty}\frac{k+1}{2^{k+2}} =\frac{m+2}{2^{m+1}}. $$ For example, the identity follows by summing the geometric series and its weighted series, or by subtracting the cumulative probabilities for $k=0,\ldots,m-1$ from 1. With $m=8$, $$ P(N\geq 8)=\frac{8+2}{2^9}=\frac{10}{512}=0.01953125. $$ To four decimal places, this is 0.0195.

Step 3: Check the scale. The prior predictive mean is $e\alpha/\beta=2$, and its variance is $2+2=4$. A count of 8 is well above the predictive mean, and the calculated upper-tail probability is small. The result is a reason to examine the risk assumptions and count data, not a conclusion that the risk must be misclassified. This calculation avoids substituting a Poisson distribution with fixed rate 2, which would omit uncertainty in $\lambda$.

Gamma-Exponential Model: Predicting a Severity

For a severity diagnostic, suppose a positive loss $X$, conditional on rate $\lambda$, has an exponential distribution with density $f_X(x\mid\lambda)=\lambda e^{-\lambda x}$ for $x\geq0$. Let $\lambda\sim\operatorname{Gamma}(\alpha,\beta)$ in shape-rate form. Integrating the likelihood over the prior gives the prior predictive density and survival function:

f_X(x)=\frac{\alpha\beta^\alpha}{(\beta+x)^{\alpha+1}}, \qquad P(X\geq x)=\left(\frac{\beta}{\beta+x}\right)^\alpha, \qquad x\geq 0.

The survival formula follows by integrating the density from $x$ to infinity. This is a marginal distribution: it incorporates uncertainty in the rate rather than evaluating an exponential tail at a single fixed rate. The units of $\beta$ and $x$ must match; $\lambda$ has units inverse to the loss amount.

Worked Example: Checking a Large Individual Loss

A hypothetical severity model assumes $X\mid\lambda\sim\operatorname{Exponential}(\lambda)$ and $\lambda\sim\operatorname{Gamma}(2,100)$ in shape-rate form, with losses and $\beta=100$ measured in dollars. One observed loss is 400 dollars. Find its prior predictive upper-tail probability.

Method. Use the gamma-exponential prior predictive survival function. Since the diagnostic concerns whether a loss is at least as large as the observed amount, use the upper tail.

Step 1: Substitute the shape, rate, and observed loss. $$ P(X\geq 400) =\left(\frac{100}{100+400}\right)^2 =\left(\frac{100}{500}\right)^2. $$

Step 2: Evaluate and check. $$ P(X\geq 400)=(0.2)^2=0.04. $$ The upper-tail probability is 0.0400 to four decimal places. The loss is four times $\beta$, and the calculation gives a 4% tail probability under the mixture model. A single such observation is not conclusive evidence of a poor model; the severity distribution, data handling, and reason for examining this loss all matter. The calculation avoids using an exponential survival probability at the prior mean rate $E[\lambda]=2/100$, which would fail to account for rate uncertainty.

Interpreting a Diagnostic Carefully

A tail probability summarizes the extremeness of a chosen feature under a specified prior predictive model. It is not the probability that the prior is false, nor is it automatically a formal rejection threshold. A small value can arise by chance, particularly when many risks or many features are inspected and only the most unusual result is reported.

The direction and construction of the diagnostic matter. If the concern is specifically an unusually high count and that direction was chosen in advance, an upper-tail probability is appropriate. If departures in either direction matter, use a two-sided or otherwise explicitly defined discrepancy. For discrete counts, tail probabilities are not generally doubled from one side as if the distribution were continuous and symmetric; sum the relevant probabilities under the actual predictive distribution.

Before interpreting an unusual result, check that the data correspond to the assumed observation period, exposure, coverage, and loss definition. For example, a claim count covering two years cannot be compared with a one-year predictive count without changing the exposure. For severity, confirm that the observed amount is on the same scale and has been treated consistently with the model. A discrepancy may reflect a data or model specification issue rather than a surprising risk parameter alone.

State the model and the feature being checked.
Specify the prior, conditional sampling distribution, exposure, and whether the concern is an upper tail or a departure in either direction.

Form the prior predictive distribution.
Integrate out the unknown parameter; retain the variation contributed by both the prior and the sampling process.

Calculate the appropriate tail probability.
Use the observed statistic and the predictive distribution, with sufficient numerical precision before final rounding.

Investigate rather than overclaim.
Review data quality, exposure and model assumptions, and consider whether multiple diagnostics were examined before treating a small tail as evidence of conflict.

Common Exam Traps

Leaving out prior uncertainty. In the normal-normal example, the variance of $\bar X$ is $\tau^2+\sigma^2/n$, not merely $\sigma^2/n$.
Using the wrong time unit. In a Poisson frequency model the mean is $e\lambda$; exposure $e$ must match the rate’s time unit.
Ignoring rate uncertainty. A gamma-Poisson predictive count is overdispersed relative to a Poisson count at the prior mean rate. A gamma-exponential predictive severity is also a mixture, not an exponential distribution at the prior mean rate.
Rounding a tail calculation too early. For the normal example, $2[1-\Phi(\sqrt{5})]\approx0.0253473$, which rounds to 0.0253. Rounding the CDF first to 0.9873 and then subtracting produces 0.0254, an avoidable error.
Calling a small tail probability proof. It signals that the observed feature is unusual under the stated predictive model. It does not by itself identify the cause or assign a probability to the model being wrong.

Key takeaway: Prior-data conflict is assessed by comparing observed experience with the prior predictive distribution, which includes both parameter uncertainty and sampling variation. A small, correctly calculated tail probability is a diagnostic prompt to investigate assumptions, not a verdict on its own.

Check Your Understanding

For each problem, use the stated prior predictive model and keep exposure, variance, and tail direction explicit.

Suppose $X_i\mid\theta\sim N(\theta,9)$, $\theta\sim N(20,4)$, and $n=4$. If $\bar x=25$, calculate the one-sided prior predictive probability $P(\bar X\geq25)$. (Answer: Predictive distribution $N(20,6.25)$; $z=2$; probability $0.0228$.)
Let $\lambda\sim\operatorname{Gamma}(1,2)$ in shape-rate form and $N\mid\lambda\sim\operatorname{Poisson}(2\lambda)$ for two years of exposure. Find $E[N]$ and $P(N\geq6)$. (Answer: $E[N]=1$; $P(N\geq6)=(2/4)^6=0.015625$, or $0.0156$ to four decimals.)
Let $X\mid\lambda$ be exponential with rate $\lambda$, and let $\lambda\sim\operatorname{Gamma}(3,150)$ in shape-rate form. Find the prior predictive probability that one loss is at least 150 dollars. (Answer: $(150/(150+150))^3=0.1250$.)
In a normal-normal model, $\theta\sim N(50,9)$, $X_i\mid\theta\sim N(\theta,64)$, and $n=4$. An observed sample mean is 60. Find the two-sided prior predictive probability of a deviation at least this large from 50. (Answer: Predictive variance $9+64/4=25$, $z=2$, and two-sided probability $2[1-\Phi(2)]=0.0455$.)
Suppose annual counts have $N\mid\lambda\sim\operatorname{Poisson}(\lambda)$, with $\lambda\sim\operatorname{Gamma}(3,1.5)$ in shape-rate form. Find the prior predictive mean and the probability of zero claims in one year. (Answer: Mean $3/1.5=2$; $P(N=0)=(1.5/(1.5+1))^3=0.2160$.)

What You'll Learn