Mean and Variance of a Hurdle Count

Why the Hurdle Mean and Variance Need Two Pieces

A hurdle count is zero when an observation does not cross a hurdle and is positive when it does. Among positive observations, counts follow a positive-valued distribution. These two parts determine the overall mean and variance: the chance of a positive count controls how often counts contribute, while the conditional distribution describes their sizes once positive.

This separation matters when interpreting model predictions. A low overall mean could reflect rare positive outcomes, small counts among positive outcomes, or both. Likewise, overall variability includes variation among positive counts and variation between zero and positive outcomes. The formulas below make those contributions explicit.

The positive-count distribution can be specified directly on \(1,2,\ldots\), or obtained by taking a baseline count distribution and conditioning it to be positive. In either case, \(q\) is the hurdle-crossing probability; it need not equal the baseline distribution’s probability of being positive.

Deriving the Overall Moments

Because \(Y=0\) when \(A=0\), and \(Y\) has conditional mean \(m_+\) when \(A=1\), its conditional expectation given \(A\) is \(E(Y\mid A)=A m_+\). Taking expectations gives the overall mean:

For the variance, use the law of total variance: \(\operatorname{Var}(Y)=E\{\operatorname{Var}(Y\mid A)\}+\operatorname{Var}\{E(Y\mid A)\}\). When \(A=0\), the conditional variance is zero; when \(A=1\), it is \(v_+\). Thus the first term is \(qv_+\). The conditional mean is \(A m_+\), and \(A\) is Bernoulli with success probability \(q\), so the second term is \(m_+^2q(1-q)\). Therefore:

An equivalent check starts with the second moment. Since \(Y^2=0\) at zero and \(E(Y^2\mid Y>0)=v_++m_+^2\),

The two derivations agree. The conditional-variance derivation highlights the components; the second-moment derivation is a useful arithmetic check.

Calculating Positive Moments From a Baseline Distribution

For a baseline count variable \(X\) with mean \(\mu_0\), variance \(\sigma_0^2\), and zero probability \(\pi_0=P(X=0)<1\), the distribution conditioned on \(X>0\) has:

The denominators are the probability of retaining a baseline draw after restricting to positive values. The numerators are unchanged by removing zeros because a zero contributes zero to both \(X\) and \(X^2\). Subtracting the squared conditional mean from the conditional second raw moment yields the conditional variance. This is the calculation used in the Poisson example; it also applies whenever the baseline distribution has the stated moments.

Conditional and Marginal Moments With Random Effects

In a mixed model, hurdle parameters may depend on covariates and a random effect. The formulas first give moments conditional on those quantities. If \(B\) denotes a random effect, write \(q(B)\), \(m_+(B)\), and \(v_+(B)\) for the corresponding conditional parameters. Then:

To obtain marginal moments, average the conditional variance and add the variance of the conditional mean. This is the law of total variance, not a license to substitute average parameters into the conditional formula:

Assumptions, Diagnostics, and Common Errors

The moment identities require a well-defined hurdle probability and finite conditional moments. They do not require a Poisson positive-count distribution. When parameters vary with covariates, calculate moments at the same covariate values; for a marginal prediction, also average over the relevant random-effect distribution. These are algebraic identities, but using them for a fitted model’s interpretation depends on the model adequately describing the zero probability and positive counts.

Useful checks examine each part rather than only the overall mean. Compare the observed proportion of positive counts with the model-implied \(q\), and compare the distribution of positive counts with the model-implied conditional mean and variance. A discrepancy confined to positive counts points toward the positive-count component; a discrepancy in the frequency of zeros points toward the hurdle probability. In grouped or mixed data, distinguish conditional predictions from marginal predictions, since averaging over heterogeneity can change both moments.

A common mistake is to treat \(m_+\) as the overall mean, or to set the hurdle probability equal to the baseline distribution’s nonzero probability when the positive component is zero-truncated. The correction is to identify the conditioning explicitly: \(m_+=E(Y\mid Y>0)\), whereas \(E(Y)=q m_+\); and the hurdle probability \(q\) is specified separately from the truncation probability used to normalize the positive distribution.

Another error is to use \(qv_+\) as the entire variance. That omits \(q(1-q)m_+^2\), which is positive whenever \(0<q<1\) and \(m_+>0\). Conversely, with random effects, applying the conditional variance formula at averaged parameter values generally does not give the marginal variance. Average the conditional variance and add the variance of the conditional mean.

The moment formulas summarize a distribution but do not establish that its hurdle mechanism is appropriate, nor do they guarantee a good fit to the entire positive-count shape. A single mean and variance can conceal tail behavior or subgroups. Their role is to interpret the implied moments and to check calculations, alongside assessment of the model’s component distributions.