Random Effects and Marginal Zero Probabilities

Why Random Effects Change the Zero Probability

In a zero-inflated count model, an observed zero can arise from either the structural-zero process or the count process. In a GLMM, the count rate also varies with latent random effects. Consequently, a population-level zero probability must average over both sources of variation: first account for the mixture at each random-effect value, then average that conditional probability over the random-effects distribution.

This distinction matters when the prediction concerns a new member of the modeled population whose random effect is unknown. A prediction conditional on a particular known or estimated random effect answers a different question. Also, substituting an average count rate into a Poisson zero-probability formula generally does not reproduce the average zero probability.

The definition holds under a specified conditional model: the non-structural count is Poisson given the random effects, the structural-zero probability is correctly specified, and the random-effects distribution \(G\) is the one relevant to the target population. For example, a common log-rate specification is \(\lambda(x,b)=\exp(x^\mathsf{T}\beta+z^\mathsf{T}b)\), where \(x\) and \(z\) are covariate vectors. The derivation below only requires a positive rate function; it does not require this particular link.

Deriving the Population-Level Probability

At a fixed value \(b\), a structural zero occurs with probability \(\pi(x)\). If the structural-zero state does not apply, the Poisson count is zero with probability \(\exp\{-\lambda(x,b)\}\). The two ways of obtaining zero are disjoint, so the conditional probability is their sum, with the count-process probability weighted by the chance that the structural-zero state does not apply.

To obtain the probability for a randomly selected member of the population at covariate value \(x\), apply the law of total probability and integrate the conditional probability over \(G\). Since \(\pi(x)\) is constant with respect to \(b\), it can be taken outside the integral.

The integral is the population-average probability that the Poisson component produces zero. It is not generally the Poisson zero probability evaluated at an average rate. If the structural-zero probability itself varies with the random effect, say it is \(\pi(x,b)\), the general expression is instead \(\int[\pi(x,b)+\{1-\pi(x,b)\}\exp\{-\lambda(x,b)\}]\,dG(b)\). Both components must then be integrated together.

Worked Calculations

Why Plugging In an Average Rate Fails

A tempting shortcut is to find the mean rate \(E(\Lambda)\) and calculate \(\exp\{-E(\Lambda)\}\). That gives the zero probability for a Poisson variable with the mean rate, but it does not generally give the average zero probability across random effects. The function \(f(\lambda)=\exp(-\lambda)\) is strictly convex for \(\lambda\geq0\), because \(f''(\lambda)=\exp(-\lambda)>0\). Jensen's inequality therefore gives

When the rate has finite mean and \(\pi\) is fixed, this implies

The inequality is strict when \(\Lambda\) is non-degenerate. Thus, at the same mean rate, rate heterogeneity increases the count-process probability of zero relative to a single Poisson rate. This result does not say that every random-effects model predicts more zeros than every model without random effects: the comparison requires the same structural-zero probability and the same mean count rate.

Assumptions, Diagnostics, and Alternatives

The integration formula is only as credible as the conditional model and the distribution being integrated. In particular, it assumes the stated Poisson count distribution conditional on the random effects, a correctly specified structural-zero probability, and a suitable random-effects distribution. A misspecified rate link or random-effects tail can materially change the average of \(\exp\{-\lambda(x,B)\}\), even when predictions at typical random-effect values look reasonable.

For a fitted model, compare observed zero frequencies with predicted population-level zero probabilities for relevant covariate patterns or groups. Predictions should integrate over the random-effects distribution for the target population. A prediction conditional on an estimated group effect is useful for that group, but it does not replace the population-level calculation for a new group. Large systematic discrepancies between observed and predicted zero frequencies are a warning to examine the conditional count distribution, structural-zero component, covariates, and random-effects assumptions; they do not by themselves identify which assumption failed.

Numerical integration is also part of the calculation, not a cosmetic detail. Check quadrature convergence or simulation precision, particularly when the rate changes sharply over the random-effects distribution. Extremely large rates contribute values near zero to the integral, whereas very small rates contribute values near one; both regions can matter to the average. Parameter uncertainty should also be considered when reporting fitted population probabilities.

The main alternatives answer different questions. A conditional prediction fixes the random effect rather than averaging it. A model without random effects uses a single rate for each covariate pattern and omits latent heterogeneity. Either may be appropriate for a particular goal, but neither is interchangeable with a population-level GLMM prediction. The integration also does not establish that structural zeros are scientifically distinct from zeros generated by the count process; that interpretation depends on the model and study context.

Common Mistake