Family Choice for Data With Many Exact Zeros

Why Exact Zeros Change the Family-Choice Question

A continuous positive-response family, such as the Gamma family, assigns probability zero to any one exact value, including zero. That makes it unsuitable as a direct model for a response that has both a genuine point mass at zero and a continuous range of positive values. This pattern is called semicontinuous: the response can be exactly zero, or it can be positive and vary continuously.

Rainfall on a given day may be exactly zero when no rain falls, while positive rainfall amounts vary continuously. A patient’s annual out-of-pocket spending may be zero if no covered service is used, and positive otherwise. In these settings, a useful family must represent both parts of the observed support. For a Tweedie model with power between one and two, the zero probability is linked to the mean and dispersion; it is not a separate free parameter.

Definition: A response is semicontinuous when its distribution has a point mass at zero and a continuous distribution over positive values. A Tweedie distribution with mean $\mu>0$, dispersion $\phi>0$, and power $1<p<2$ has this support: it can equal zero or take a positive value, but it does not take negative values.

The Zero-Atom Tweedie Model

Write $Y$ for a response conditional on its covariates, and let $\mu=E[Y]$. The Tweedie family in this range of powers has variance $\operatorname{Var}(Y)=\phi\mu^p$. Its compound Poisson–Gamma representation makes the exact-zero mechanism explicit: a Poisson number of positive Gamma amounts are summed, and the sum is zero when that number is zero.

Definition: For $1<p<2$, a compound Poisson–Gamma Tweedie response can be written $Y=\sum_{j=1}^{N} Z_j$, where $N\sim\operatorname{Poisson}(\lambda)$, the $Z_j$ are independent Gamma random variables with shape $a=(2-p)/(p-1)$ and scale $b=\phi(p-1)\mu^{p-1}$, and $N$ is independent of the $Z_j$. The empty sum is defined to be zero. The Poisson mean is $\lambda=\mu^{2-p}/\{\phi(2-p)\}$.

The Gamma convention here is shape and scale, so $E[Z_j]=ab$. The zero probability follows directly from the event that there are no positive contributions:

P(Y=0)=P(N=0)=e^{-\lambda} =\exp\!\left\{-\frac{\mu^{2-p}}{\phi(2-p)}\right\}.

When $N\geq 1$, each Gamma contribution is strictly positive, so the sum is positive. Conditional on $N\geq 1$, the sum has a continuous distribution. Thus the model has an exact atom at zero and a continuous positive component, rather than a continuous density across zero.

Why the Representation Has the Tweedie Moments

The parameterization above is not an arbitrary construction: it gives the specified mean and variance. For a compound Poisson sum with rate $\lambda$ and independent jumps $Z_j$, conditioning on $N$ gives $E[Y]=\lambda E[Z_j]$ and $\operatorname{Var}(Y)=\lambda E[Z_j^2]$. These identities follow by applying the laws of total expectation and total variance to the random sum.

Result: With the compound Poisson–Gamma parameters above, $E[Y]=\mu$, $\operatorname{Var}(Y)=\phi\mu^p$, and $P(Y=0)=\exp\{-\mu^{2-p}/[\phi(2-p)]\}$.

Derivation. A Gamma variable with shape $a$ and scale $b$ has $E[Z_j]=ab$ and $E[Z_j^2]=a(a+1)b^2$. Substituting the specified parameters,

\lambda ab =\frac{\mu^{2-p}}{\phi(2-p)} \frac{2-p}{p-1}\phi(p-1)\mu^{p-1} =\mu.

For the variance, use $a+1=1/(p-1)$:

\begin{aligned} \lambda a(a+1)b^2 &=\frac{\mu^{2-p}}{\phi(2-p)} \frac{2-p}{(p-1)^2} \phi^2(p-1)^2\mu^{2p-2}\\ &=\phi\mu^p. \end{aligned}

Finally, because the jumps are positive, $Y=0$ exactly when $N=0$, which has probability $e^{-\lambda}$. This proves all three claims. $\square$

The formula also shows the family’s constraint: at fixed $\mu$, $\phi$, and $p$, the model determines the zero probability. One cannot tune the zero mass independently while keeping those parameters fixed. Conversely, an observed zero fraction can help assess whether the implied Poisson rate is plausible, but matching that fraction alone is not enough to establish a good family choice.

Positive Values and Their Implied Variability

The model also connects the positive observations to the overall mean and variance. Let $q=P(Y=0)$, so $1-q=P(Y>0)$. Since a zero contributes nothing to the first and second raw moments, conditioning on $Y>0$ gives $E[Y\mid Y>0]=\mu/(1-q)$ and $E[Y^2\mid Y>0]=(\phi\mu^p+\mu^2)/(1-q)$. Subtracting the squared conditional mean yields:

Result: For a zero-atom Tweedie response, $$ E[Y\mid Y>0]=\frac{\mu}{1-q}, \qquad \operatorname{Var}(Y\mid Y>0) =\frac{\phi\mu^p}{1-q}-\frac{\mu^2q}{(1-q)^2}. $$ Here $q=\exp\{-\mu^{2-p}/[\phi(2-p)]\}$.

This is a practical family check: compare the model-implied zero rate and the implied distribution of positive values with the data. A good match to the zero proportion can still come with positive values that are much too variable, or too little variable. The calculations concern what the family entails; they do not replace checking whether the mean structure is appropriate for the scientific question.

Worked Example: Rainfall With Many Dry Days

Suppose daily rainfall has an observed mean of $2$ mm, and $60\%$ of days have exactly zero rainfall. Consider a Tweedie candidate with $p=1.5$. What dispersion would make its zero probability match the observed fraction, and what positive-response moments would it then imply?

Here $q=0.60$, so the required Poisson rate is $\lambda=-\log(q)=-\log(0.60)=0.5108$. The rate formula, with $2-p=0.5$, gives $\lambda=\sqrt{2}/(0.5\phi)$. Solving, $\phi=\sqrt{2}/(0.5\times0.5108)=5.536$.

The model’s positive-response mean is $2/(1-0.60)=5$ mm. Its unconditional variance is $\phi\mu^p=5.536\times2^{1.5}=5.536\times2.828=15.66$ mm$^2$. Therefore the positive-response variance is $15.66/0.40-(2^2\times0.60)/(0.40^2)=39.15-15=24.15$ mm$^2$, giving a positive-response standard deviation of about $4.91$ mm.

Matching the dry-day fraction therefore commits the model to a positive mean of $5$ mm and a positive standard deviation of about $4.91$ mm. If the observed wet-day amounts are much less variable, this candidate is not a satisfactory joint description, even though its zero probability matches.

Worked Example: Many Zero Expenditures

Suppose annual spending has mean $\$120$, $40\%$ of people spend exactly zero, and the positive amounts have mean $\$200$ and variance $\$36{,}400$. Assess a Tweedie candidate with $p=1.5$.

The candidate’s zero fraction fixes $\lambda=-\log(0.40)=0.9163$. With $\mu=120$ and $2-p=0.5$, the rate equation gives $\phi=\sqrt{120}/(0.5\times0.9163)=10.954/0.4582=23.91$.

The implied positive mean is $120/(1-0.40)=\$200$, which matches the stated value. The unconditional variance is $23.91\times120^{1.5}=23.91\times1{,}314.53$, approximately $\$31{,}430$. The implied positive variance is $31{,}430/0.60-(120^2\times0.40)/(0.60^2)$, approximately $52{,}383-16{,}000=\$36{,}383$. This is close to the stated $\$36{,}400$, with the small difference due to rounding.

The zero frequency, positive mean, and positive variance are therefore mutually compatible with this candidate, to the precision of the rounded calculations. This makes the family worth considering; it does not by itself show that its full positive tail or its conditional mean pattern across covariates is adequate.

Worked Example: When the Zero Rate Is the Warning

A rainfall model at one covariate profile has fitted mean $\mu=10$ mm, dispersion $\phi=0.8$, and power $p=1.5$. The observed fraction of dry days at comparable profiles is about $0.20$. Calculate the model-implied zero probability.

The implied Poisson rate is $\lambda=10^{0.5}/\{0.8(0.5)\}=\sqrt{10}/0.4=7.906$. Thus $q=e^{-7.906}$, approximately $0.00037$. The candidate predicts about $0.037\%$ exact zeros, far below the observed $20\%$.

This discrepancy is not repaired merely by noting that the positive amounts are right-skewed: the zero frequency itself is constrained by the candidate’s mean, dispersion, and power. The fitted family, parameters, or mean specification should be reconsidered before treating its predictions as a useful account of these data.

Choosing Between a Tweedie and Other Zero-Response Models

The Tweedie family is especially natural when the response can be understood as an aggregate of a random number of positive contributions, with zero arising when there are none. It can be a useful single-distribution model when the same mechanism and covariate structure plausibly govern both whether a positive amount occurs and how large that amount is.

A two-part model is a different choice: it models $P(Y>0)$ and the distribution of $Y\mid Y>0$ separately. This can be more appropriate when the occurrence of any positive response has a distinct process, or when observed zero frequencies and positive amounts cannot be reconciled by the Tweedie constraint. The choice is substantive as well as statistical: ask whether a single compound process is credible, rather than selecting a family only because it accommodates zeros.

Check the recorded support carefully. A reported zero may be a genuine absence, a rounded small positive amount, or a value below a detection limit. These are not equivalent data-generating situations. If zeros are created by rounding or censoring, a model for the recording process may be needed; treating every recorded zero as a true compound-process zero can distort the family choice.

Common Mistake: Matching the Zero Fraction Alone

A tempting argument is: “The Tweedie model has an atom at zero, and its fitted zero probability is near the observed zero fraction, so the response family fits.” The correction is that the atom is only one implication of the joint distribution. Once $\mu$, $\phi$, and $p$ are set, the family also entails positive-response moments and a positive continuous distribution. Assess those implications against observed positive amounts and relevant predictive checks.

Key takeaway: For $1<p<2$, a Tweedie model can represent exact zeros and positive continuous responses through a compound Poisson–Gamma mechanism. Its zero probability and positive-response variability are linked to the same parameters, so family choice must assess both parts of the response.

Check Your Understanding

Use the support and parameter relationships to answer these questions.

Why does a Tweedie model with $1<p<2$ assign positive probability to an exact zero?
For fixed $\mu$ and $p$, how does increasing $\phi$ affect $\lambda$ and the model’s zero probability?
A candidate matches the observed zero fraction but implies far more variability among positive values than is observed. What does that suggest about family choice?
Why might a two-part model be preferable when occurrence and positive amount appear to arise from distinct processes?
How would rounding or a detection limit complicate the interpretation of recorded zeros?