Hurdle Models for Excess-Zero Counts

Two Processes Behind a Hurdle Count

In pharmacometric data, a count response may have many zero observations and a distribution among positive observations that a single count model does not describe well. For example, patients may have zero rescue-medication doses during an interval, while patients who take at least one dose have a separate pattern of dose counts. A hurdle model represents these features with two components: one determines whether the count is zero or positive, and the other describes the count conditional on being positive.

The term hurdle emphasizes that every positive count must first cross the zero-versus-positive boundary. Unlike a zero-inflated model, the hurdle formulation assigns all zero counts to the binary component and uses a count distribution explicitly conditioned to be positive for the second component.

Definition: Let $Y_{ij}$ be a nonnegative count for subject $i$ at occasion $j$, and let $b_i$ denote that subject's random effects. A hurdle model specifies $q_{ij}=P(Y_{ij}>0\mid b_i)$ and a count probability mass function $f_{ij}(y\mid b_i)$ on $y=0,1,2,\ldots$. Its response probabilities are $$ P(Y_{ij}=0\mid b_i)=1-q_{ij},\qquad P(Y_{ij}=y\mid b_i)=q_{ij}\frac{f_{ij}(y\mid b_i)}{1-f_{ij}(0\mid b_i)},\quad y=1,2,\ldots. $$ The positive-count distribution is the original count distribution conditional on being greater than zero.

The probabilities sum to one: the zero probability is $1-q_{ij}$, and the positive probabilities sum to $q_{ij}$, because $\sum_{y=1}^{\infty} f_{ij}(y\mid b_i)=1-f_{ij}(0\mid b_i)$. Thus $q_{ij}$ is exactly the probability of observing a positive count, not the probability of a latent count being positive before truncation.

Regression Components and Assumptions

A common specification uses logistic regression for the hurdle and a Poisson or negative-binomial model for the positive counts. For example, with subject random effects $u_i$ and $v_i$,

\operatorname{logit}(q_{ij})=\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+u_i,\qquad \log(\lambda_{ij})=\log(E_{ij})+\mathbf{z}_{ij}^{\mathsf T}\boldsymbol{\gamma}+v_i.

Here $\mathbf{x}_{ij}$ and $\mathbf{z}_{ij}$ are covariate vectors, $\boldsymbol{\beta}$ and $\boldsymbol{\gamma}$ are fixed-effect coefficients, and $E_{ij}>0$ is exposure when the count is a rate over differing observation times or opportunities. The term $\log(E_{ij})$ is an offset with coefficient fixed at one. The parameter $\lambda_{ij}$ is the mean of the untruncated Poisson distribution, not the mean among positive counts. The random effects can be jointly distributed, often as a mean-zero multivariate normal vector; allowing correlation can represent a relationship between a subject's propensity to have any events and their positive event burden.

The model assumes the count is measured on a nonnegative integer scale and the positive component assigns nonzero probability to positive counts. Conditional on the random effects and covariates, observations are usually taken to be independent across occasions. For a given occasion, the two-stage probability specification must be appropriate: the binary component models positivity, and the chosen truncated distribution must reasonably describe the positive values. Random effects are integrated over their distribution for population-level likelihoods and predictions; setting them to zero describes a conditional prediction for a subject at the reference random-effect value.

A negative-binomial baseline can replace the Poisson baseline when its variance structure is more suitable. In either case, truncation uses that distribution's own zero probability: if the baseline pmf is $f(y)$, the positive probability is $f(y)/(1-f(0))$. The hurdle construction does not require the Poisson distribution.

Worked Example: Separating Zero and Positive Probabilities

Suppose, hypothetically, that the probability of at least one rescue dose in an interval is $q=0.40$, and the positive-count component is a Poisson distribution with $\lambda=2$, truncated at zero. First, $P(Y=0)=1-q=0.60$. The baseline Poisson probability of zero is $e^{-2}=0.1353$, so the probability of a positive baseline count is $1-e^{-2}=0.8647$.

For exactly one dose, the baseline probability is $f(1)=e^{-2}2^1/1!=0.2707$. The hurdle-model probability is therefore $$ P(Y=1)=0.40\frac{0.2707}{0.8647}=0.1252. $$ Similarly, $f(2)=e^{-2}2^2/2!=0.2707$, so $P(Y=2)=0.40(0.2707/0.8647)=0.1252$. The zero probability remains 0.60; the Poisson zero probability is used only to normalize the distribution of positive counts.

Among observations that are positive, the probability of exactly one dose is $0.2707/0.8647=0.3130$. Among all intervals, its probability is 0.1252. Keeping these conditional and overall probabilities distinct is essential when interpreting the two model components.

Means and a Short Derivation

The mean of the complete hurdle response combines the probability of crossing the hurdle with the mean count after crossing it. In general,

E(Y_{ij}\mid b_i)=q_{ij}E(Y_{ij}\mid Y_{ij}>0,b_i).

For a Poisson baseline with parameter $\lambda_{ij}$, the conditional positive mean is $\lambda_{ij}/(1-e^{-\lambda_{ij}})$. To see this, condition the baseline Poisson distribution on being positive. The numerator in its conditional expectation is the sum of $y f(y)$ for $y\geq1$; the $y=0$ term in the full Poisson expectation is zero, so this sum equals the full Poisson mean $\lambda_{ij}$. The conditioning event has probability $1-e^{-\lambda_{ij}}$. Hence

E(Y_{ij}\mid Y_{ij}>0,b_i)= \frac{\sum_{y=1}^{\infty}y f_{ij}(y\mid b_i)} {1-f_{ij}(0\mid b_i)} =\frac{\lambda_{ij}}{1-e^{-\lambda_{ij}}}, \qquad E(Y_{ij}\mid b_i)=q_{ij}\frac{\lambda_{ij}}{1-e^{-\lambda_{ij}}}.

This formula shows why $\lambda_{ij}$ should not be reported as the expected count among positive observations. Truncation raises that conditional mean above $\lambda_{ij}$; the overall mean is then reduced by the probability $q_{ij}$ of a positive count.

Worked Example: Estimating the Components From Hypothetical Counts

Consider 20 hypothetical independent intervals with no covariates or random effects. Twelve counts are zero, and the eight positive counts are $1,1,2,1,3,1,2,1$. The estimated hurdle probability is the observed positive fraction: $$ \widehat q=\frac{8}{20}=0.400. $$

The positive counts sum to 12, so their sample mean is $12/8=1.50$. Under a zero-truncated Poisson component, the log likelihood for these positive observations, as a function of $\lambda$, is $$ \ell_+(\lambda)=12\log(\lambda)-8\lambda-\sum_{k=1}^{8}\log(y_k!)-8\log(1-e^{-\lambda}). $$

Differentiating with respect to $\lambda$ and setting the derivative to zero gives $$ \frac{12}{\lambda}-8-\frac{8}{e^\lambda-1}=0 \quad\Longleftrightarrow\quad \frac{12}{8}=\frac{\lambda}{1-e^{-\lambda}}. $$ Solving this one-dimensional equation gives $\widehat{\lambda}\approx0.874$. The fitted mean among positive intervals is $\widehat{\lambda}/(1-e^{-\widehat{\lambda}})\approx1.50$ counts, matching the positive sample mean by the likelihood equation. The estimated overall mean is $0.400(1.50)=0.600$ counts per interval. The estimate $\widehat{\lambda}$ is not itself the positive-count mean.

Interpreting Effects and Random Effects

A coefficient in the hurdle model has a component-specific interpretation. In the logistic component, a one-unit increase in a covariate changes the log odds of a positive count by its coefficient, conditional on the other covariates and random effects. In a log-linked count component, a one-unit increase multiplies the baseline Poisson rate by the exponential of its coefficient, conditional on the random effects. Because truncation changes the mean of positive counts, this rate ratio is not generally a rate ratio for the observed positive-count mean.

Worked Example: Exposure and Subject-Specific Propensity

Suppose a hypothetical model for a dose indicator $D$ has $\operatorname{logit}(q_{ij})=-0.4+0.6D+u_i$. For a subject with $u_i=-0.5$ and $D=0$, the linear predictor is $-0.4-0.5=-0.9$, giving $q_{ij}=1/(1+e^{0.9})=0.289$. For a subject with $u_i=0.5$ at the same dose, it is $0.1$, giving $q_{ij}=1/(1+e^{-0.1})=0.525$. These are conditional positive-count probabilities for the two specified subjects, not population-average probabilities.

For the positive component, suppose the rate is $0.60$ counts per hour at $D=0$, the dose coefficient is $0.30$, and exposure is $E_{ij}=2$ hours. At $D=1$, the rate is $0.60e^{0.30}=0.810$ counts per hour, and the baseline Poisson mean over the interval is $\lambda_{ij}=2(0.810)=1.62$ counts. Its positive-count mean is $1.62/(1-e^{-1.62})\approx2.02$ counts. The dose effect on positivity is governed by the logistic component, while the dose effect on the positive count distribution is governed by the count component.

Relationship to Alternatives and Model Limitations

A standard Poisson or negative-binomial model uses one count distribution for zeros and positive counts. It may be inadequate when the observed positivity probability and the positive-count distribution need separate covariate effects or separate random effects. A hurdle model allows this separation directly. Its essential restriction is that every zero comes from the binary component, while positive counts follow the selected zero-truncated distribution.

In a zero-inflated model, by contrast, zeros can arise both from a separate zero-generating state and from the zero outcome of the baseline count distribution. Positive outcomes also retain the baseline distribution's mixture weighting. Thus, the models assign different probabilities even when both can accommodate more zeros than a simple Poisson model. A hurdle model is a useful choice when the scientific or operational question naturally separates occurrence from positive burden; the data alone do not establish that these are distinct biological mechanisms.

The model can be limited by sparse positive counts, a poorly chosen positive-count distribution, or an overly elaborate random-effects structure. In mixed-effects fits, the two components may be weakly informed when few subjects have positive observations or when there are few repeated measures per subject. Estimates that are unstable, very imprecise, or sensitive to reasonable changes in the positive-count distribution warrant caution. Model checks should examine both the frequency of zeros and the distribution of positive counts; a good fit to one does not guarantee a good fit to the other.

Common Mistake

Wrong reasoning: “The positive-count component has Poisson mean $\lambda$, so the expected count among subjects with a positive count is $\lambda$.”
Correction: $\lambda$ is the mean of the untruncated baseline Poisson distribution. Conditioning on positivity changes the mean to $\lambda/(1-e^{-\lambda})$; the expected count over all observations is $q\lambda/(1-e^{-\lambda})$.
Wrong reasoning: “A hurdle model says that every observed zero is a structural zero.”
Correction: It assigns every zero to its binary component by model construction. That is a statistical representation, not by itself evidence for a particular biological cause of zero.

Key takeaway: A hurdle model combines a binary probability of positivity with a count distribution conditioned on being positive. Interpret the binary probability, the truncated positive-count distribution, and their combined mean as distinct quantities.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If $q=0.25$, what is the model probability of zero?
For a zero-truncated Poisson component with $\lambda=1$, write the conditional mean among positive counts and explain why it exceeds 1.
A covariate has coefficient $\log(1.5)$ in a log-linked positive-count component. What does this coefficient imply about the baseline rate, holding other terms fixed?
How does the source of zeros in a hurdle model differ from the source of zeros in a zero-inflated model?

Solutions. (1) $P(Y=0)=1-q=0.75$. (2) The conditional mean is $1/(1-e^{-1})\approx1.582$ counts. It exceeds 1 because conditioning a Poisson distribution on a positive value removes its zero outcomes. (3) The baseline rate is multiplied by $e^{\log(1.5)}=1.5$, or increased by 50%, conditional on the other model terms; this is not automatically a 50% increase in the mean among positive counts. (4) A hurdle model assigns all zeros to its binary component. A zero-inflated model allows zeros from both its extra-zero state and the baseline count distribution.