Selecting a Tweedie Power for Semicontinuous Data

Why the Tweedie Power Matters for Semicontinuous Data

Some responses have a distinctive shape: many observations are exactly zero, while the nonzero observations are positive and continuous. Examples include costs that are zero when no event occurs and positive when one or more events occur. A model that assigns probability only to positive values cannot represent the exact zeros as part of its response distribution.

The Tweedie family offers a particular way to represent this combination. Its variance function is indexed by a power $p$; for $1<p<2$, the corresponding distribution has a compound Poisson-gamma representation. This range of powers is therefore relevant when exact zeros and positive continuous values are both plausible features of the response. The power does not simply describe how variable the observations are: it also determines the distributional form and support.

Definition: A Tweedie exponential-dispersion model with mean $\mu>0$, dispersion $\phi>0$, and power $p$ has variance $$ \operatorname{Var}(Y)=\phi\mu^p. $$ Here $Y$ is the response. In a regression model, this variance statement applies conditionally on the covariates, with $\mu$ denoting the conditional mean.

The variance power is one clue to family choice, not a guarantee that the model describes every feature of the data. In particular, the mean-variance relationship alone does not establish that the observed zeros, tail behavior, or covariate structure are adequately represented. The interpretation of $p$ is most useful when considered together with the response’s support.

How the Power Relates to Familiar Families

Several familiar distributions correspond to special Tweedie variance powers. These examples help place the semicontinuous case in context. The parameterization of each distribution matters: for instance, the ordinary Poisson distribution has dispersion fixed at one, whereas the Tweedie variance expression includes $\phi$.

Power	Variance form	Distributional connection
$p=0$	$\phi$	Gaussian variance is constant in the mean
$p=1$	$\phi\mu$	Poisson-type variance; the ordinary Poisson has $\phi=1$
$1<p<2$	$\phi\mu^p$	Compound Poisson-gamma, with an atom at zero and a continuous positive component
$p=2$	$\phi\mu^2$	Gamma-type variance
$p=3$	$\phi\mu^3$	Inverse-Gaussian-type variance

The useful distinction here is support. Gamma and inverse-Gaussian responses are strictly positive under their usual distributions. By contrast, the compound Poisson-gamma response can equal zero when no event occurs, and is positive when one or more events occur. Thus $1<p<2$ gives a distributional mechanism for exact zeros, rather than treating zero as an inconvenient value outside the positive-response family.

Worked Example: Using Support to Screen a Candidate Power

Suppose a response records the total cost of claims for each policyholder over one year. Some policyholders have no claims and hence cost exactly zero; others have one or more claims, producing a positive continuous total. Which of the candidate powers $p=2$ and $p=1.6$ is compatible with both parts of this response support?

At $p=2$, the Tweedie variance is $\phi\mu^2$, the Gamma-type case. A Gamma response takes positive values, so this family does not assign probability to an exact zero under its usual continuous distribution.

At $p=1.6$, the power lies between one and two. The associated compound Poisson-gamma distribution has positive probability of a zero total and continuous positive values when at least one event occurs. It is therefore support-compatible with the described response.

This comparison screens the candidate families; it does not prove that the $p=1.6$ model fits well. The frequency and size of claims, the mean-variance pattern, and the relationship with covariates still need to be considered.

The Compound Poisson-Gamma Representation

For $1<p<2$, the response can be represented as a random sum. Let $N$ be a Poisson random variable with mean $\lambda$, and let $X_1,X_2,\ldots$ be independent Gamma random variables, independent of $N$, with shape $a$ and scale $s$. The scale convention means $E[X_i]=as$. Define

Y=\sum_{i=1}^{N}X_i,

where the sum is zero when $N=0$. The Tweedie parameterization is

\lambda=\frac{\mu^{2-p}}{\phi(2-p)},\qquad a=\frac{2-p}{p-1},\qquad s=\phi(p-1)\mu^{p-1}.

The Poisson count represents the number of positive contributions; the Gamma variables represent their sizes. This construction explains the support directly. If $N=0$, then $Y=0$. If $N\geq1$, each $X_i$ is positive with probability one, so their sum is positive and continuous.

Result: For $1<p<2$, the compound Poisson-gamma representation has mean $\mu$, variance $\phi\mu^p$, and exact-zero probability $$ P(Y=0)=\exp\left\{-\frac{\mu^{2-p}}{\phi(2-p)}\right\}. $$

Deriving the Representation and Zero Probability

The following derivation verifies both the Tweedie distributional form and the key consequences of the compound representation. It also makes clear which parameter controls each aspect of the model.

Obtain the Tweedie cumulant generating function.
For an exponential-dispersion model with variance function $\mu^p$, the unit cumulant function satisfies $b''(\theta)=\mu^p$, with $b'(\theta)=\mu$. Since $d\theta/d\mu=\mu^{-p}$, integration gives $\theta=-\mu^{1-p}/(p-1)$ and $b(\theta)=\mu^{2-p}/(2-p)$, up to an irrelevant additive constant.

Write the cumulant generating function.
For values of $t$ in the moment-generating-function domain, the exponential-dispersion form gives $K_Y(t)=\{b(\theta+\phi t)-b(\theta)\}/\phi$. With the expressions in Step 1, this becomes $\lambda\{(1-st)^{-a}-1\}$, using the stated $\lambda,a,s$.

Recognize the compound Poisson-gamma law.
Conditional on $N$, the sum of $N$ independent Gamma(shape $a$, scale $s$) variables has moment generating function $(1-st)^{-aN}$. Averaging over $N\sim\operatorname{Poisson}(\lambda)$ gives $E[e^{tY}]=\exp\{\lambda[(1-st)^{-a}-1]\}$, the same cumulant generating function as in Step 2.

Calculate the mass at zero.
Each Gamma jump is strictly positive. Thus $Y=0$ exactly when $N=0$, whose Poisson probability is $e^{-\lambda}$. Substituting the expression for $\lambda$ gives the stated formula.

For completeness, the mean and variance can also be checked directly from the random-sum representation. Conditional on $N$, the expected total is $Nas$, and the conditional variance is $Nas^2$. The law of total expectation and the law of total variance therefore give

E[Y]=\lambda as,\qquad \operatorname{Var}(Y)=\lambda as^2+\lambda(as)^2 =\lambda a(1+a)s^2.

Substitute the parameter values. Since $a=(2-p)/(p-1)$, we have $as=\phi(2-p)\mu^{p-1}$, so $\lambda as=\mu$. Also $1+a=1/(p-1)$, and hence

\lambda a(1+a)s^2 = \frac{\mu^{2-p}}{\phi(2-p)} \frac{2-p}{(p-1)^2} \phi^2(p-1)^2\mu^{2p-2} =\phi\mu^p.

This verifies the mean and variance, while the matching cumulant generating functions establish the full distributional representation. The calculations require $1<p<2$, $\mu>0$, and $\phi>0$, so all three representation parameters are positive. $\square$

Worked Example: Calculating the Compound Poisson-Gamma Parameters

Take $p=1.75$, $\mu=2$, and $\phi=0.5$. Calculate the Poisson rate, Gamma shape and scale, mean, variance, and probability of an exact zero.

The Poisson rate is

\lambda=\frac{2^{2-1.75}}{0.5(2-1.75)} =\frac{2^{0.25}}{0.125} =\frac{1.1892071\ldots}{0.125} \approx 9.5137.

The Gamma shape and scale are

a=\frac{2-1.75}{1.75-1} =\frac{0.25}{0.75} =\frac{1}{3}, $$ $$ s=0.5(1.75-1)2^{1.75-1} =0.375(2^{0.75}) \approx 0.6307.

The expected number of jumps is about $9.5137$, while the mean size of one jump is $as\approx(1/3)(0.6307)=0.2102$. Their product is $E[Y]=\lambda as\approx9.5137(0.2102)\approx2.000$, as required by the specified mean.

The variance is

\operatorname{Var}(Y)=\phi\mu^p =0.5(2^{1.75}) \approx0.5(3.3636) \approx1.6818.

Finally, the zero probability is

P(Y=0)=e^{-\lambda} =e^{-9.5136569\ldots} \approx 0.0000738.

Thus this particular mean and dispersion imply a very small chance of an exact zero, even though the model allows zeros. A model’s support being compatible with observed zeros does not mean it will predict a large zero frequency.

Using the Implied Zero Probability

The expression $P(Y=0)=e^{-\lambda}$ shows that the zero mass depends on the mean, dispersion, and power jointly. At fixed $p$ and $\phi$, a larger $\mu$ gives a larger $\lambda$, because $2-p>0$. It therefore gives a smaller probability of no jumps. This direction is intuitive: a greater expected total is associated with more expected contributions.

Worked Example: Comparing Zero Probabilities at Two Means

Suppose two covariate profiles have the same $p=1.5$ and $\phi=1$, but conditional means $\mu_1=0.25$ and $\mu_2=1$. Calculate the zero probabilities implied by the model.

For $p=1.5$, $2-p=0.5$, so $\lambda=\mu^{0.5}/0.5=2\sqrt{\mu}$. At $\mu_1=0.25$, $\lambda_1=2\sqrt{0.25}=1$, giving

P(Y=0\mid \mu_1)=e^{-1}\approx0.3679.

At $\mu_2=1$, $\lambda_2=2\sqrt{1}=2$, giving

P(Y=0\mid \mu_2)=e^{-2}\approx0.1353.

The higher-mean profile has a lower model-implied probability of no contribution. These are probabilities under the specified Tweedie model, not empirical proportions or guaranteed predictions for every sample.

What the Power Can and Cannot Tell You

For semicontinuous data, a value of $p$ strictly between one and two offers a coherent single-response distribution with an exact-zero mass and a continuous positive part. The model connects the two parts through one compound process: zeros occur when the Poisson count is zero, and positive totals arise from one or more Gamma jumps. Consequently, the zero probability is not an independent parameter in this representation; it is tied to $\mu$, $\phi$, and $p$.

The power also determines how variance grows with the mean: $\operatorname{Var}(Y)=\phi\mu^p$. This is useful when comparing a semicontinuous candidate with positive-only families. But do not infer the correct family from a variance pattern alone. A model may have a suitable variance function yet fail to describe the observed zero frequency, the positive tail, or systematic differences across covariate groups.

Common mistake: “There are many zeros, so any model with $1<p<2$ must fit the zero pattern.” The correction is that this power range permits exact zeros, but it does not freely set their frequency. For example, the parameters in the preceding numerical calculation imply $P(Y=0)\approx0.0000738$, a very small zero probability. The predicted mass must be compared with the data at the relevant covariate values; support compatibility alone is not enough.

Key takeaway: The Tweedie power $1<p<2$ is distinctive because it yields a compound Poisson-gamma response: no Poisson events produce an exact zero, and one or more Gamma contributions produce a positive continuous total. The same parameters determine the mean, variance, and zero probability.

Check Your Understanding

Use the definitions and results above to answer the following questions.

Why can a Gamma response with its usual continuous distribution not represent an exact zero as part of the response distribution?
For $1<p<2$, what event in the compound representation is equivalent to $Y=0$?
With $p=1.5$, $\phi=1$, and $\mu=4$, calculate $\lambda$ and $P(Y=0)$.
State the Gamma shape and scale in the compound Poisson-gamma representation, and explain which parameter is the scale.
Why does having a power between one and two establish support compatibility but not, by itself, good model fit?

Power	Variance form	Distributional connection
\(p=0\)	\(\phi\)	Gaussian variance is constant in the mean
\(p=1\)	\(\phi\mu\)	Poisson-type variance; the ordinary Poisson has \(\phi=1\)
\(1<p<2\)	\(\phi\mu^p\)	Compound Poisson-gamma, with an atom at zero and a continuous positive component
\(p=2\)	\(\phi\mu^2\)	Gamma-type variance
\(p=3\)	\(\phi\mu^3\)	Inverse-Gaussian-type variance