A Support-And-Variance Checklist For Distribution Choice

Why Use a Two-Stage Checklist?

A response distribution in a generalized linear mixed model (GLMM) must describe the values the outcome can take and the variation it can plausibly exhibit. Those are related but distinct questions. A distribution with a convenient variance formula is not appropriate if its support excludes valid outcomes; a distribution with the right support may still impose a mean-variance relationship that is implausible for the data.

A disciplined choice therefore proceeds in two stages: match the response support, then compare plausible conditional mean-variance behavior. The result is a shortlist of candidates, not an automatic declaration that one model is correct. Response-generation details, cluster structure, and model diagnostics remain important.

Definition: In a GLMM, a response distribution specifies the probability law of an outcome conditional on its covariates and random effects. Write \(Y_{ij}\) for observation \(i\) in cluster \(j\), and \(b_j\) for that cluster's random effect. The conditional mean and variance are \(E(Y_{ij}\mid b_j)\) and \(\operatorname{Var}(Y_{ij}\mid b_j)\). Distribution choice concerns the conditional response law, not merely the distribution of the observed values pooled across clusters.

Stage 1: Match the Support and Response Process

The support is the set of values a random variable can take with positive probability. Start by stating what one recorded response represents and its possible values. This prevents common mismatches: a count model for an ordinal rating, for example, treats the gaps between successive category codes as meaningful count increments when they may only represent order.

Binary response: a single outcome coded 0 or 1 is supported by a Bernoulli distribution. A binomial distribution is appropriate when the response is a number of successes among a known number \(m\) of trials under the relevant conditional trial assumptions.
Count response: a nonnegative integer count, potentially with no fixed upper bound, can be represented by a count distribution such as Poisson or a more flexible count family. A count that is capped by a known number of opportunities may instead have binomial support.
Ordinal response: a response in a finite set of ordered categories has categorical support with an order. The labels need not have equal spacing. Its distribution is described through category probabilities or an ordinal model, rather than by treating the labels automatically as counts.

Support alone does not settle every case. A recorded zero may mean a genuine absence, a structural impossibility, or no opportunity for the event; those are different response processes. Likewise, a total number of events is not necessarily an unrestricted count if each subject had a fixed number of trials. Clarify exposure, denominator, and category definitions before moving to variance.

Stage 2: Check Plausible Conditional Mean-Variance Behavior

For a proposed response family, write down its conditional mean \(\mu_{ij}=E(Y_{ij}\mid b_j)\) and conditional variance. Ask whether the variance implied by the family is plausible over the range of fitted means. The comparison is a screening device: raw sample variance at one mean, especially in a small sample, is not a conclusive test of a distribution.

Key result: For a Bernoulli response with conditional success probability \(p_{ij}\), the conditional mean is \(p_{ij}\) and the conditional variance is \(p_{ij}(1-p_{ij})\). For a binomial response from \(m_{ij}\) conditionally independent trials with common success probability \(p_{ij}\), the mean is \(m_{ij}p_{ij}\) and the variance is \(m_{ij}p_{ij}(1-p_{ij})\). A Poisson response has conditional mean and variance both equal to \(\mu_{ij}\). More flexible count distributions allow other variance patterns; for example, the NB2 variance is \(\mu_{ij}+\mu_{ij}^2/k\), for \(k>0\).

For ordinal responses, numerical variance is not generally a family-defining target. If categories are coded 1, 2, and 3, the variance of those codes depends on the assigned scores. The category probabilities are intrinsic to the response; the numerical variance is not, unless a scientifically justified score scale is part of the measurement.

In clustered data, observed variation pooled across clusters can exceed the conditional variance even when the conditional distribution is adequate. The law of total variance gives \( \operatorname{Var}(Y_{ij})=E\{\operatorname{Var}(Y_{ij}\mid b_j)\}+\operatorname{Var}\{E(Y_{ij}\mid b_j)\} \). The second term is variation in conditional means across random effects. Thus, a pooled variance-versus-mean comparison should not be read as a direct test of the conditional response family.

A Short Derivation: Why the Poisson Variance Is a Useful Screen

If \(Y\mid b\) is Poisson with conditional mean \(\mu\), its probability mass function is \(P(Y=y\mid b)=e^{-\mu}\mu^y/y!\), for nonnegative integers \(y\). The Poisson mean and variance are both \(\mu\), so a proposed Poisson GLMM commits to this equality conditional on the random effects. A systematic conditional variance substantially larger or smaller than that equality can signal that the Poisson response law is too restrictive.

For instance, under the NB2 variance function, \(\operatorname{Var}(Y\mid b)=\mu+\mu^2/k\), which exceeds \(\mu\) for finite positive \(k\). As \(k\) grows, the extra term approaches zero and the variance approaches the Poisson value. This algebra explains why NB2 is a candidate when extra-Poisson variation is plausible; it does not establish that NB2 is the generating distribution. Other sources of variation, including omitted predictors or an inadequate random-effects structure, can also produce an apparent mismatch.

Worked Example: Screening a Binary Response

Hypothetical data. Suppose 20 patients each have one binary response indicating whether a specified event occurred during follow-up. Four events are observed. The outcome support is exactly \(\{0,1\}\), so a Bernoulli response is compatible with the individual-level support. If the patients instead contributed counts of events over follow-up, this support assessment would be different.

The observed event proportion is \(\hat p=4/20=0.20\). Under a Bernoulli model with success probability \(p=0.20\), the conditional model variance for one response is

\operatorname{Var}(Y\mid b)=p(1-p)=0.20(0.80)=0.16.

The sample variance using the usual denominator \(n-1\) is

s^2=\frac{\sum_{i=1}^{20}(Y_i-\bar{x})^2}{19} =\frac{4(1-0.20)^2+16(0-0.20)^2}{19} =\frac{3.20}{19}=0.1684.

The sample variance is close to the Bernoulli variance, but this agreement is not a goodness-of-fit proof: the estimate is based on only 20 observations, and the patients may also be clustered. Before using a Bernoulli GLMM, check that each response is one event/no-event outcome and assess whether the conditional independence structure is credible after accounting for modeled covariates and random effects. The arithmetic screens the variance condition; it cannot verify those assumptions.

Worked Example: Comparing a Count Mean and Variance

Hypothetical data. Eight independent cluster-level summaries of a nonnegative event count are \(0,0,1,1,2,2,4,6\). Suppose each summary has the same exposure for this simple screening calculation. The support is nonnegative integers with no fixed upper bound, so a count family is plausible; an ordinal model would not preserve the meaning of count increments.

The sample mean is \(\bar{x}=(0+0+1+1+2+2+4+6)/8=16/8=2\). The empirical variance using denominator \(n\), for comparison with a population variance function, is

\frac{0^2+0^2+1^2+1^2+2^2+2^2+4^2+6^2}{8}-\bar{x}^{\,2} =\frac{62}{8}-2^2=7.75-4=3.75.

A Poisson model at mean 2 implies variance 2, whereas the NB2 variance at that mean is \(2+2^2/k\). Equating that expression to the empirical variance gives \(3.75=2+4/k\), so \(k=4/1.75=2.286\). This is only a moment-based illustration of how NB2 can accommodate a larger variance; eight observations do not establish an NB2 model or yield a reliable fitted estimate of \(k\).

For an actual GLMM, compare variation conditional on the modeled predictors and random effects, not just the pooled summaries. Verify exposure comparability, investigate influential large counts and zeros, and assess whether missing predictors or cluster heterogeneity could explain the apparent excess. If fitted conditional variation is below the Poisson pattern, the Poisson and NB2 comparison is not enough; a family allowing underdispersion may merit consideration.

Worked Example: Why Ordinal Scores Do Not Decide the Family

Hypothetical data. Forty respondents select one of three ordered categories: low, medium, or high. The counts are 12, 20, and 8. The support is these three categories, in that order. Their spacing is not specified, so choosing an ordinal response formulation is more defensible than declaring that the responses are counts from 1 to 3.

The observed category proportions are \(12/40=0.30\), \(20/40=0.50\), and \(8/40=0.20\), which sum to 1. If the categories are temporarily coded 1, 2, and 3, the resulting mean is \((12(1)+20(2)+8(3))/40=76/40=1.90\). The coded second moment is \((12(1^2)+20(2^2)+8(3^2))/40=164/40=4.10\), giving coded variance \(4.10-(1.90)^2=0.49\).

That variance is an arithmetic property of the assigned scores, not evidence that a count distribution is appropriate. If the same ordered categories were coded 1, 2, and 10, the numeric variance would change even though no respondent changed category. The support and response process therefore favor an ordinal categorical formulation; category probabilities and the intended ordinal structure should guide the model choice. Check that category labels really are ordered, that coding has not reversed the order, and that sparse categories do not make the proposed model unstable.

Apply the Checklist in Order

Define one response.
State the observational unit, what was recorded, the possible values, and any denominator or exposure. Determine whether values are binary, counts, or ordered categories.

Remove support mismatches.
Reject candidates that assign positive probability to impossible values or exclude valid ones. Consider whether a fixed number of trials, a cap, or ordered labels changes the support interpretation.

Write the conditional variance commitment.
For each remaining candidate, state how \(\operatorname{Var}(Y_{ij}\mid b_j)\) relates to \(E(Y_{ij}\mid b_j)\). For ordinal data, focus on category probabilities and ordering rather than arbitrary numeric variance.

Check the mismatch in context.
Compare observed and fitted behavior while accounting for predictors, exposure, and random effects. Look for systematic residual patterns, not just one large sample variance.

Keep more than one candidate when warranted.
Use the response process and diagnostics to narrow choices, then compare plausible models using appropriate fit checks and the scientific purpose of the analysis.

Diagnostics, Alternatives, and Limitations

The checklist is most useful when it directs attention to concrete ways a proposed family can fail. For binary outcomes, check whether each record really represents one trial and whether the conditional response pattern is compatible with the modeled predictors and random effects. For counts, inspect residual patterns across fitted means, the frequency of zeros and large values, and whether apparent dispersion changes with exposure or covariates. For ordinal outcomes, verify the category ordering and inspect category frequencies; very sparse or empty categories can undermine estimation even when the support is conceptually correct.

A variance mismatch is not uniquely diagnostic. Unmodeled covariates, unequal exposure, dependence not represented by the random effects, coding errors, and influential observations may all affect observed variation. In a GLMM, comparing a pooled sample variance directly to a conditional variance formula ignores the between-cluster contribution in the law of total variance. Use residual diagnostics and model-based checks as evidence about fit, not as mechanical tests that identify a single correct distribution.

The main alternative to this checklist is to choose a response family solely from the measurement process and then evaluate fit. That can be sensible when the response mechanism is well understood, but it can miss an important variance mismatch. Conversely, selecting a family only because its variance matches a sample estimate can overfit noise and ignore response meaning. Quasi-likelihood approaches can express a mean-variance relationship without specifying a full probability distribution, but a quasi-family is not itself a full response likelihood and cannot be substituted when likelihood-based inference or probability predictions are needed.

No finite checklist proves that a candidate family is true. The variance relationship may be imperfect, the random-effects distribution may be misspecified, and sparse data can provide little information to distinguish candidates. The appropriate conclusion is usually conditional: a candidate has compatible support and a plausible variance pattern under stated assumptions, and it should be checked against model diagnostics and the scientific response process.

Common Mistakes

Starting with variance before support. A convenient variance function cannot rescue a model that assigns the wrong possible values. First decide whether the response is a single binary outcome, a bounded trial count, an unrestricted count, or an ordered category.
Treating observed overdispersion as proof of a particular family. Larger-than-expected variation can arise from several causes. It supports investigation of alternatives; it does not uniquely identify one distribution.
Comparing pooled variance to a conditional formula. Cluster-to-cluster variation in fitted means contributes to marginal variation. Make the comparison at the conditional level or account for the clustering in the diagnostic.
Using category codes as if they were equally spaced measurements. Numeric labels do not turn an ordinal outcome into a count. Use numerical scores only when their spacing has substantive justification.
Treating a checklist as a final model-selection rule. It narrows candidates. Response-process knowledge, assumptions, fit diagnostics, and the analysis goal still matter.

Key takeaway: Choose in two stages. First, match the response support and how the outcome was generated. Then assess whether a candidate's conditional mean-variance behavior is plausible. The checklist narrows candidates; it does not replace model checking.

Check Your Understanding

For each item, decide what the checklist implies. Brief solutions follow the questions.

A subject has one yes/no outcome. Which support is appropriate, and what is its variance in terms of the conditional success probability \(p\)?
A count has conditional mean \(\mu=3\). What variance does a Poisson model imply? What does an NB2 model with \(k=3\) imply?
Why is a three-category ordered rating not automatically a count just because it can be coded 1, 2, 3?
A pooled count variance exceeds its pooled mean in clustered data. Does that alone establish conditional overdispersion? Explain.

Solutions. (1) A single yes/no response has Bernoulli support \(\{0,1\}\), with \(\operatorname{Var}(Y\mid b)=p(1-p)\). (2) Poisson implies variance \(3\). NB2 implies \(3+3^2/3=3+3=6\). (3) The labels specify order, not necessarily equal numerical spacing or count increments; use a formulation for ordered categories unless the score scale has meaningful spacing and count interpretation. (4) No. Between-cluster differences in conditional means can increase pooled variance, and other misspecification can also do so. Assess conditional fit while accounting for random effects and covariates.