Predicting Zero Probabilities and Positive Amounts

From a Fitted Tweedie Model to Predictions

A Tweedie response with power $1<p<2$ can be exactly zero or positive. A fitted mean alone does not describe how often zero occurs: the probability of zero depends on the mean, dispersion, and power together. This distinction matters in prediction. For example, predicting an average cost across all policies is not the same as predicting the average cost among policies with a positive claim.

The compound Poisson-Gamma representation gives the needed probabilities and positive-amount distribution directly. The calculations below use fitted parameter values as plug-in values. They describe the model’s predictive distribution at a specified covariate pattern, without incorporating uncertainty from estimating the regression coefficients, dispersion, or power.

Definition: For a Tweedie response with $1<p<2$, mean $\mu>0$, and dispersion $\phi>0$, the compound Poisson-Gamma representation is $Y=\sum_{j=1}^{N}J_j$, where $N\sim\operatorname{Poisson}(\lambda)$, the $J_j$ are independent Gamma variables independent of $N$, and $$ \lambda=\frac{\mu^{2-p}}{\phi(2-p)},\qquad J_j\sim\operatorname{Gamma}(\alpha,\text{scale }\beta),\qquad \alpha=\frac{2-p}{p-1},\quad \beta=\phi(p-1)\mu^{p-1}. $$ The sum is defined to be zero when $N=0$. In this parameterization a Gamma variable with shape $\alpha$ and scale $\beta$ has mean $\alpha\beta$.

For observation $i$, replace the unknown mean by the fitted mean $\hat\mu_i$, and use the fitted dispersion $\hat\phi$ and power $\hat p$. The fitted Poisson rate is $$ \hat\lambda_i=\frac{\hat\mu_i^{\,2-\hat p}}{\hat\phi(2-\hat p)}. $$ A zero response occurs exactly when the Poisson count is zero; every Gamma jump is strictly positive. Thus the fitted probability of zero is $e^{-\hat\lambda_i}$, and the fitted probability of a positive response is $1-e^{-\hat\lambda_i}$. The latter probability is often denoted $\hat q_i$.

Zero and Positive Probabilities

The link function is used to obtain the fitted mean at the covariate pattern; once $\hat\mu_i$ is available, the zero probability follows from the compound Poisson rate. In particular, a larger fitted mean does not automatically mean a smaller zero probability in every comparison: the rate also depends on $\hat\phi$ and $\hat p$. Comparisons should be made using the fitted $\hat\lambda_i$, not the mean alone.

Result: Under the Tweedie compound Poisson-Gamma model with $1<p<2$, $$ P(Y=0)=e^{-\lambda},\qquad P(Y>0)=q=1-e^{-\lambda}. $$ At fitted values, $\hat P(Y_i=0)=e^{-\hat\lambda_i}$ and $\hat P(Y_i>0)=\hat q_i=1-e^{-\hat\lambda_i}$.

Proof. Each jump has a Gamma distribution supported on strictly positive amounts, so $Y=0$ if $N=0$. If $N\geq1$, the sum of one or more positive jumps is positive. Therefore the events $\{Y=0\}$ and $\{N=0\}$ are identical. Since $N$ is Poisson with rate $\lambda$, $P(N=0)=e^{-\lambda}$. Taking the complement gives $P(Y>0)=1-e^{-\lambda}$. $\square$

Worked Example: Zero Probability and Positive Amounts

Suppose a fitted Tweedie model at one covariate pattern has $\hat p=1.5$, $\hat\mu=4$, and $\hat\phi=0.8$. Calculate the fitted zero and positive probabilities.

The fitted Poisson rate is $$ \hat\lambda=\frac{4^{2-1.5}}{0.8(2-1.5)} =\frac{2}{0.4}=5. $$ Thus $$ \hat P(Y=0)=e^{-5}=0.006737947,\qquad \hat P(Y>0)=1-e^{-5}=0.993262053. $$ So the model assigns about $0.674\%$ probability to zero and about $99.326\%$ probability to a positive amount.

Summaries Conditional on a Positive Outcome

The Tweedie mean and variance are $E[Y]=\mu$ and $\operatorname{Var}(Y)=\phi\mu^p$. Since $Y=0$ contributes nothing to its first and second raw moments, conditioning on positivity divides those moments by $q=P(Y>0)$. This gives useful summaries without first evaluating the positive density.

Result: For $q=1-e^{-\lambda}>0$, $$ E[Y\mid Y>0]=\frac{\mu}{q}, \qquad \operatorname{Var}(Y\mid Y>0) =\frac{\phi\mu^p+\mu^2}{q}-\left(\frac{\mu}{q}\right)^2. $$ Equivalently, $\operatorname{Var}(Y\mid Y>0)=\frac{\phi\mu^p}{q}-\frac{\mu^2e^{-\lambda}}{q^2}$.

Derivation. Because $Y=0$ on the event $\{Y=0\}$, $E[Y]=E[Y\,\mathbf{1}_{\{Y>0\}}]$. By the definition of conditional expectation for an event of positive probability, $$ E[Y\mid Y>0]=\frac{E[Y\,\mathbf{1}_{\{Y>0\}}]}{P(Y>0)} =\frac{\mu}{q}. $$ Likewise $E[Y^2\mid Y>0]=E[Y^2]/q$. The variance identity gives $E[Y^2]=\operatorname{Var}(Y)+(E[Y])^2=\phi\mu^p+\mu^2$, so subtracting the square of the conditional mean yields the stated variance. For the equivalent form, use $1-q=e^{-\lambda}$ and simplify: $$ \frac{\phi\mu^p+\mu^2}{q}-\frac{\mu^2}{q^2} =\frac{\phi\mu^p}{q}-\frac{\mu^2(1-q)}{q^2} =\frac{\phi\mu^p}{q}-\frac{\mu^2e^{-\lambda}}{q^2}. $$

Worked Example: Conditional Positive-Amount Moments

Continue with $\hat p=1.5$, $\hat\mu=4$, and $\hat\phi=0.8$, for which $\hat q=0.993262053001$. The fitted unconditional variance is $$ \widehat{\operatorname{Var}}(Y)=0.8(4^{1.5})=0.8(8)=6.4, $$ so the fitted second raw moment is $6.4+4^2=22.4$.

The predicted mean among positive outcomes is $$ \widehat{E[Y\mid Y>0]}=\frac{4}{0.993262053001} \approx 4.0271346\approx 4.0271. $$ The conditional variance is $$ \widehat{\operatorname{Var}(Y\mid Y>0)} =\frac{22.4}{0.993262053001} -\left(\frac{4}{0.993262053001}\right)^2. $$ The two terms are approximately $22.5519539$ and $16.2178130$, respectively. Therefore the conditional variance is $6.3341409$, or approximately $6.3341$. These are positive-outcome summaries; they are not the unconditional mean $4$ and variance $6.4$.

The Predictive Distribution of a Positive Amount

Conditional moments summarize a positive amount, but they do not specify its full predictive distribution. The compound representation gives that distribution as a mixture. Given $N=n$, the sum of $n$ independent Gamma jumps with common shape $\alpha$ and scale $\beta$ is Gamma with shape $n\alpha$ and scale $\beta$. Conditional on a positive response, the count $N$ has a zero-truncated Poisson distribution.

Result: For $y>0$, the distribution of $Y\mid Y>0$ is a mixture over $n=1,2,\ldots$, with weights $$ P(N=n\mid N\geq1)=\frac{e^{-\lambda}\lambda^n}{n!(1-e^{-\lambda})}, $$ and component distributions $Y\mid N=n\sim\operatorname{Gamma}(n\alpha,\text{scale }\beta)$.

Derivation. Since $Y>0$ is equivalent to $N\geq1$, the conditional probability of count $n\geq1$ is $P(N=n)/P(N\geq1)$. Substituting the Poisson probability gives the displayed weight. Given $N=n$, $Y$ is a sum of $n$ independent Gamma jumps with common scale, so Gamma shapes add to $n\alpha$. The weights sum to one because $$ \sum_{n=1}^{\infty}\frac{e^{-\lambda}\lambda^n}{n!(1-e^{-\lambda})} =\frac{1-e^{-\lambda}}{1-e^{-\lambda}}=1. $$ Consequently the conditional cumulative distribution function at $y>0$ is the weighted sum of the Gamma cumulative distribution functions. A positive-outcome quantile can be obtained by solving this mixture CDF equation numerically. $\square$

Worked Example: Interpreting the Positive-Amount Mixture

Take $p=1.5$, $\mu=1$, and $\phi=1$. Then $$ \lambda=\frac{1^{0.5}}{1(0.5)}=2,\qquad \alpha=\frac{0.5}{0.5}=1,\qquad \beta=1(0.5)1^{0.5}=0.5. $$ Each jump is exponential with mean $0.5$, and $q=1-e^{-2}=0.8646647$.

The conditional weights for one, two, and three jumps are $$ P(N=1\mid N\geq1)=\frac{2e^{-2}}{q}\approx0.3130, $$ $$ P(N=2\mid N\geq1)=\frac{2e^{-2}}{q}\approx0.3130, \qquad P(N=3\mid N\geq1)=\frac{(8/6)e^{-2}}{q}\approx0.2087. $$ For these components, the positive amount has Gamma shapes $1,2,3$, respectively, all with scale $0.5$, and therefore component means $0.5,1.0,1.5$. The remaining conditional weight for four or more jumps is approximately $1-0.3130-0.3130-0.2087=0.1653$. This is why the positive distribution is not just a single Gamma distribution: it averages over a random positive number of jumps.

Unconditional Prediction and Interpretation

For an unconditional prediction, retain both the probability of zero and the amount distribution when positive. The fitted mean $\hat\mu_i$ is the expected response across all cases, including zeros. The fitted conditional positive mean is $\hat\mu_i/\hat q_i$, which is necessarily larger than $\hat\mu_i$ when $0<\hat q_i<1$. These answer different questions and should not be substituted for one another.

The fitted unconditional variance is $\hat\phi\hat\mu_i^{\,\hat p}$, and the fitted second raw moment is $\hat\phi\hat\mu_i^{\,\hat p}+\hat\mu_i^2$. For a prediction interval or a probability such as $P(Y_i>c)$, the full Tweedie distribution is needed; the mean and variance alone do not determine such probabilities. The zero mass and positive mixture together define the predictive distribution.

Worked Example: A Nearly Certain Positive Outcome

Suppose a fitted model gives $\hat p=1.5$, $\hat\mu=9$, and $\hat\phi=0.5$. Then $$ \hat\lambda=\frac{9^{0.5}}{0.5(0.5)}=\frac{3}{0.25}=12, \qquad \hat P(Y=0)=e^{-12}\approx0.0000061442. $$ Hence $\hat q=1-e^{-12}\approx0.9999938558$. The positive-outcome mean is $$ \widehat{E[Y\mid Y>0]}=\frac{9}{0.9999938558}\approx9.0000553. $$ The unconditional variance is $0.5(9^{1.5})=13.5$, and the second raw moment is $13.5+9^2=94.5$. Thus $$ \widehat{\operatorname{Var}(Y\mid Y>0)} =\frac{94.5}{0.9999938558} -\left(\frac{9}{0.9999938558}\right)^2 \approx13.4996. $$ The positive and unconditional summaries are close here because the fitted zero probability is very small, but they are not identical.

Common Pitfalls and Numerical Practice

Using the mean as the positive-outcome mean. The unconditional mean includes zeros. Divide by $q$ to obtain $E[Y\mid Y>0]$.
Confusing the zero probability with the positive probability. The zero probability is $e^{-\lambda}$; its complement $1-e^{-\lambda}$ is the positive probability.
Conditioning the variance incorrectly. Divide the second raw moment by $q$, then subtract the square of the conditional mean. Simply dividing the unconditional variance by $q$ omits the contribution of the mean and the conditional centering.
Treating positive amounts as a single Gamma distribution. Conditional on a positive outcome, the number of jumps is random, so the positive distribution is a mixture over jump counts.
Ignoring numerical cancellation. When $\lambda$ is small, computing $1-e^{-\lambda}$ by direct subtraction can lose precision. Numerically, use a stable evaluation of $-\operatorname{expm1}(-\lambda)$ when available.

Key takeaway: At each covariate pattern, first compute $\hat\lambda=\hat\mu^{\,2-\hat p}/\{\hat\phi(2-\hat p)\}$. Then the zero probability is $e^{-\hat\lambda}$, the positive probability is its complement, and conditional positive moments follow by dividing the corresponding raw moments by that positive probability.

Check Your Understanding

Answer each question using the compound Poisson-Gamma representation and distinguish conditional from unconditional summaries.

For $1<p<2$, why is the event $Y=0$ exactly the event $N=0$?
A fitted Tweedie model has $\hat\lambda=0.7$. Write the fitted probabilities of a zero response and a positive response.
Explain why $E[Y\mid Y>0]$ is greater than $E[Y]$ whenever $0<P(Y>0)<1$ and $E[Y]>0$.
Given $E[Y]=\mu$, $\operatorname{Var}(Y)=\phi\mu^p$, and $q=P(Y>0)$, derive $\operatorname{Var}(Y\mid Y>0)$ from the first two raw moments.
Why does the positive conditional distribution involve a mixture over Gamma distributions rather than just the jump-size Gamma distribution?