Latent Heterogeneity in Binomial and Poisson Responses

Why Unobserved Variation Changes the Variance

A binomial or Poisson GLM specifies a variance benchmark once its mean is given. That benchmark is conditional on the model’s predictors and assumptions. But observations that look alike on recorded predictors may still differ in unmeasured ways: patients may have different underlying risks, or sites may have different event rates. If a success probability or event rate varies across observational units, the variance after averaging over that unobserved variation can exceed the binomial or Poisson benchmark.

The key distinction is between a conditional model, given the latent probability or rate, and the marginal distribution after that latent quantity is averaged out. A conditional distribution can be exactly binomial or Poisson while its marginal mixture is more variable. The derivations below use the law of total expectation and the law of total variance.

Definition: In a binomial mixture, a random success probability \(P\) varies across units, and \(Y\mid P\sim\operatorname{Binomial}(m,P)\). In a Poisson mixture, a random event rate \(\Lambda\) varies across units, and \(Y\mid\Lambda\sim\operatorname{Poisson}(\Lambda)\). A marginal distribution is the distribution of \(Y\) after averaging over the latent \(P\) or \(\Lambda\).

For these variance calculations, assume the latent quantities have finite second moments. The binomial probability is automatically bounded, and the Poisson rate must have a finite second moment for the marginal count to have finite variance. The conditional models describe variation given the latent quantity; the marginal variance includes both that conditional variation and variation between latent quantities.

Random Success Probabilities and Binomial Counts

Suppose each unit contributes \(m\) trials, and its probability of success \(P\) is drawn from a population distribution with mean \(\bar p=E[P]\) and variance \(\tau^2=\operatorname{Var}(P)\). Given \(P\), the \(m\) trials are independent Bernoulli observations with success probability \(P\), so \(Y\mid P\sim\operatorname{Binomial}(m,P)\).

Result: Under this binomial mixture, \(E[Y]=m\bar p\), and \(\operatorname{Var}(Y)=m\bar p(1-\bar p)+m(m-1)\tau^2\). Thus the marginal variance exceeds the binomial variance at the same marginal mean whenever \(m>1\) and \(\tau^2>0\).

Derivation. The conditional mean and variance are \(E[Y\mid P]=mP\) and \(\operatorname{Var}(Y\mid P)=mP(1-P)\). Therefore,

E[Y]=E\{E[Y\mid P]\}=mE[P]=m\bar p.

By the law of total variance,

\begin{aligned} \operatorname{Var}(Y) &=E\{\operatorname{Var}(Y\mid P)\} +\operatorname{Var}\{E[Y\mid P]\}\\ &=mE[P(1-P)]+m^2\operatorname{Var}(P)\\ &=m\{E[P]-E[P^2]\}+m^2\tau^2\\ &=m\{\bar p-(\tau^2+\bar p^2)\}+m^2\tau^2\\ &=m\bar p(1-\bar p)+m(m-1)\tau^2. \end{aligned}

The first term is the binomial variance evaluated at the marginal success probability \(\bar p\). The second is the extra variance from differences in success probabilities across units. It is nonnegative, and is strictly positive when \(m>1\) and the latent probability is not constant. If \(m=1\), the second term is zero: mixing Bernoulli probabilities changes no marginal variance beyond the Bernoulli variance at the marginal mean.

Worked Example: A Beta-Mixed Binomial Count

Suppose each patient has 10 trials, such as 10 opportunities for a response, and the patient-specific success probability follows a \(\operatorname{Beta}(2,8)\) distribution. The beta distribution has mean \(a/(a+b)\) and variance \(ab/[(a+b)^2(a+b+1)]\). Here,

\bar p=\frac{2}{10}=0.2, \qquad \tau^2=\frac{2\cdot8}{10^2\cdot11} =\frac{16}{1100}=\frac{4}{275}.

A binomial model using the marginal probability \(0.2\) would have mean \(10(0.2)=2\) and variance \(10(0.2)(0.8)=1.6\). The mixture has the same mean, but its variance is

\operatorname{Var}(Y) =1.6+10(9)\frac{4}{275} =\frac{88}{55}+\frac{72}{55} =\frac{32}{11}\approx2.909.

Thus the marginal variance is about 1.82 times the binomial benchmark. The reason is not that trials are dependent after conditioning on the patient’s probability: they are conditionally independent. Rather, patients with higher probabilities tend to have more successes across all 10 trials, while patients with lower probabilities tend to have fewer.

The same mechanism can be seen at the trial level. Let \(X_j\) and \(X_k\) be two distinct Bernoulli outcomes from the same unit. Conditional independence gives \(\operatorname{Cov}(X_j,X_k\mid P)=0\), but marginally,

\operatorname{Cov}(X_j,X_k) =E\{\operatorname{Cov}(X_j,X_k\mid P)\} +\operatorname{Cov}\{E[X_j\mid P],E[X_k\mid P]\} =\operatorname{Var}(P)=\tau^2.

A shared latent probability therefore makes outcomes from the same unit positively associated marginally. This is another way to understand why the count becomes more variable.

Random Rates and Poisson Counts

For Poisson counts, let \(\Lambda\) be a unit-specific event rate over a fixed observation period, with \(E[\Lambda]=\mu\) and \(\operatorname{Var}(\Lambda)=\omega^2\). Conditional on that rate, suppose \(Y\mid\Lambda\sim\operatorname{Poisson}(\Lambda)\).

Result: Under this Poisson mixture, \(E[Y]=\mu\) and \(\operatorname{Var}(Y)=\mu+\omega^2\). If \(\omega^2>0\), the marginal variance exceeds the Poisson variance at the same mean.

Derivation. A Poisson variable with mean \(\Lambda\) has conditional mean and variance both equal to \(\Lambda\). Hence

E[Y]=E\{E[Y\mid\Lambda]\}=E[\Lambda]=\mu,

and, using the law of total variance,

\begin{aligned} \operatorname{Var}(Y) &=E\{\operatorname{Var}(Y\mid\Lambda)\} +\operatorname{Var}\{E[Y\mid\Lambda]\}\\ &=E[\Lambda]+\operatorname{Var}(\Lambda) =\mu+\omega^2. \end{aligned}

If units have different known exposures \(e_i\) and a latent rate \(R_i\) per exposure unit, then \(Y_i\mid R_i\sim\operatorname{Poisson}(e_iR_i)\). For fixed \(e_i\), the same calculation gives \(E[Y_i]=e_iE[R_i]\) and \(\operatorname{Var}(Y_i)=e_iE[R_i]+e_i^2\operatorname{Var}(R_i)\). The added term is scaled by the square of exposure.

Worked Example: A Gamma-Mixed Poisson Count

Suppose the latent rate \(\Lambda\) follows a gamma distribution with shape 4 and rate \(4/3\). For a gamma distribution in shape-rate parameterization, the mean is shape divided by rate and the variance is shape divided by rate squared. Thus,

E[\Lambda]=\frac{4}{4/3}=3, \qquad \operatorname{Var}(\Lambda)=\frac{4}{(4/3)^2}=\frac{9}{4}=2.25.

The marginal count has mean 3 and variance \(3+2.25=5.25\), whereas a Poisson model with mean 3 has variance 3. The gamma mixture also gives a concrete check on the distribution of small counts. Its probability of zero is

P(Y=0)=E[e^{-\Lambda}] =\left(\frac{4/3}{4/3+1}\right)^4 =\left(\frac{4}{7}\right)^4 =\frac{256}{2401}\approx0.107.

A Poisson variable with mean 3 has \(P(Y=0)=e^{-3}\approx0.050\). The larger zero probability is consistent with mixing: some units have low rates and contribute many zeros, while units with high rates contribute relatively large counts. The mixture also has a longer upper tail than the corresponding Poisson distribution.

What Residual Patterns Can Suggest

Latent heterogeneity is not directly visible in an ordinary GLM data set. It is inferred, if at all, from the distribution of observed responses relative to the fitted mean and variance. A recurring pattern is more squared residual variation than expected from the binomial or Poisson benchmark. In grouped binomial data, that can mean more groups near the extremes than the fitted binomial model predicts. In counts, it can mean both more zeros and a heavier upper tail than the fitted Poisson model predicts.

For example, consider eight groups with 10 trials each and a fitted success probability of \(0.2\). The fitted count mean is 2, and the binomial variance is 1.6. Suppose the observed counts are \(0,0,1,1,3,3,4,4\). Their sum is 16, so their average is exactly 2. The Pearson residual for group \(i\) is the observed-minus-fitted count divided by the square root of the fitted variance. The squared residuals are \(2.5\) for each count of 0 or 4 and \(0.625\) for each count of 1 or 3. Thus the Pearson sum of squares is

$$ 2(2.5)+2(0.625)+2(0.625)+2(2.5)=12.5. $$

The mean is correctly matched, but this small illustration has many counts farther from 2 than the fitted binomial variance would lead one to expect. Such a pattern can motivate investigation of latent heterogeneity. With only eight groups, however, it is not decisive evidence: sampling variation alone can produce an unusually large residual sum of squares.

Residual plots can also be suggestive. If observations sharing a site, patient, household, or other grouping tend to have residuals of similar sign, a common unmeasured feature within those units may be relevant. If residual spread grows with fitted means beyond the model’s benchmark, that may also suggest extra variation. Neither pattern identifies its cause. A misspecified mean, dependence, influential observations, or other departures can produce similar signals. Residuals should therefore be read as evidence about model adequacy, not as a direct measurement of a latent variable.

Worked Example: Why One Bernoulli Observation Cannot Show Extra Variance

Suppose each person has a latent response probability \(P\), with \(E[P]=0.2\), but only one binary outcome \(Y\) is observed per person. Conditional on \(P\), \(Y\sim\operatorname{Bernoulli}(P)\). The marginal mean is \(E[Y]=E[P]=0.2\). Since \(Y\) is still binary, its marginal variance is

\operatorname{Var}(Y)=0.2(1-0.2)=0.16.

That is exactly the Bernoulli variance at the marginal mean, even if \(P\) varies substantially across people. Equivalently, the binomial-mixture formula has \(m=1\), so its extra term \(m(m-1)\tau^2\) is zero. With only one binary response per person, marginal variance alone cannot distinguish heterogeneous probabilities from a common probability of 0.2. Repeated trials, grouping information, predictors, or additional assumptions may provide information, but the single-response variance does not.

Common Mistake

Wrong reasoning: “The conditional response distribution is binomial or Poisson, so the marginal variance must equal the fitted binomial or Poisson variance.”

Correction: Those variance benchmarks apply conditional on the probability or rate. When that quantity varies across units, the law of total variance adds the variance of the conditional mean. For a binomial mixture the added term is \(m(m-1)\operatorname{Var}(P)\); for a Poisson mixture it is \(\operatorname{Var}(\Lambda)\). Conversely, an elevated residual measure does not by itself prove latent heterogeneity, because other model failures can also increase residual variation.

Key takeaway: A random probability or rate preserves the conditional binomial or Poisson model but changes its marginal variance. Look for excess spread relative to the fitted benchmark, while treating residual patterns as clues that require interpretation rather than diagnoses of a unique cause.

Check Your Understanding

Use the conditional-versus-marginal distinction to answer these questions.

For \(Y\mid P\sim\operatorname{Binomial}(m,P)\), which part of the marginal variance formula represents variation between units?
If \(m=1\), why does random variation in \(P\) not increase the marginal Bernoulli variance beyond the variance at the marginal mean?
A Poisson mixture has mean 5 and latent-rate variance 3. What is its marginal variance, and what would a Poisson model with mean 5 use as its variance?
In grouped binomial data, what kind of observed count pattern might suggest probability heterogeneity?
Why does a large Pearson sum of squares not establish that latent heterogeneity is the cause of extra variation?