Choose the Response Distribution From the Data-Generating Structure
A response family is not selected just because an outcome is recorded as a number. A value of 0 or 1, a count of successes out of a known total, an event count accumulated over exposure, and a ranked rating have different supports and sampling structures. Those distinctions determine which probability model makes sense before predictors or random effects are added.
The support-and-variance checklist is useful for organizing this decision: first ask what values the response can take, then how it was generated, and then whether its conditional variability is plausible under the candidate family. In a GLMM, these questions concern the response distribution conditional on the random effects. A common general setup is that observations are conditionally independent given the covariates and cluster-specific random effects \(b_j\), with \(b_j\) commonly modeled as multivariate normal with mean zero and covariance matrix \(D\). The selected response family describes the conditional distribution; it does not, by itself, determine the random-effects structure.
The four response types below are not interchangeable. In particular, a fixed-total success count is bounded by its known number of trials, whereas an event count is not. A ranked rating has ordered categories, but the distances between category labels need not be meaningful.
One Binary Outcome and a Fixed-Total Success Count
If each observational unit contributes one outcome in two exhaustive categories, code it as \(Y_{ij}\in\{0,1\}\). Conditional on predictors and random effects, the Bernoulli model is \(Y_{ij}\mid b_j\sim\operatorname{Bernoulli}(p_{ij})\), with \(\operatorname{logit}(p_{ij})=\eta_{ij}\). Its conditional variance is \(p_{ij}(1-p_{ij})\). The outcome should genuinely be binary; coding a multi-category response as 0 and 1 requires a defensible definition of the event and its complement.
When the recorded response is the number of successes \(Y_j\) among a known, fixed \(m_j\) trials, the natural candidate is \(Y_j\mid b_j\sim\operatorname{Binomial}(m_j,p_j)\), often with \(\operatorname{logit}(p_j)=\eta_j\). Its support is \(0,1,\ldots,m_j\), its conditional mean is \(m_jp_j\), and its conditional variance is \(m_jp_j(1-p_j)\). The binomial model assumes the number of trials is known and that, conditional on the model variables, trials have a common success probability and are independent. A shared random effect can induce dependence marginally across trials in a cluster while retaining conditional independence given that effect.
Worked Example: An Individual Binary Outcome
Suppose a hypothetical clinic records whether each patient has a positive test result, coded \(Y_{ij}=1\) for positive and \(0\) for negative. Consider one clinic with random intercept \(b_j=0.2\), and an illustrative fixed linear-predictor contribution \(x_{ij}^{\mathsf T}\beta=0.3\). The conditional logit is \(\eta_{ij}=0.3+0.2=0.5\), so \(\displaystyle p_{ij}=\frac{\exp(0.5)}{1+\exp(0.5)}=0.6225\).
For three patients in that clinic, suppose the observed outcomes are \(1,0,1\). Given the specified \(b_j\) and model contributions, the conditional likelihood contribution is \(\displaystyle (0.6225)^1(1-0.6225)^1(0.6225)^1 =(0.6225)^2(0.3775)\approx0.1463\). This calculation illustrates the Bernoulli likelihood; it is not an estimate of the regression coefficients. The binary coding is appropriate because each patient contributes one result, not a count out of a varying or fixed number of trials.
Worked Example: A Success Count With a Known Denominator
In a hypothetical batch, 9 of 12 inspected components pass a specified test. If all 12 components were inspected and each has the same conditional pass probability \(p=0.70\), the response is a binomial count with \(m=12\), not an unbounded event count. Its probability of exactly 9 passes is \(\displaystyle P(Y=9)=\binom{12}{9}(0.70)^9(0.30)^3\).
Substituting the values gives \(\displaystyle P(Y=9)=220(0.0403536)(0.027)=0.2397\). Under this model, the conditional mean is \(12(0.70)=8.4\) passes and the conditional variance is \(12(0.70)(0.30)=2.52\) passes squared. These calculations require a fixed denominator of 12 and the stated common-probability, conditional-independence assumptions. If the denominator were not known, or if the response instead counted events over time, the binomial support would not describe the data.
Unbounded Event Counts: Poisson and Negative Binomial
For a nonnegative integer count with no fixed upper bound, a Poisson distribution is a starting point: \(Y_{ij}\mid b_j\sim\operatorname{Poisson}(\mu_{ij})\), with \(\log(\mu_{ij})=\eta_{ij}\). It implies \(E(Y_{ij}\mid b_j)=\mu_{ij}\) and \(\operatorname{Var}(Y_{ij}\mid b_j)=\mu_{ij}\). If observation periods or population-at-risk sizes differ, the mean should reflect exposure. With exposure \(t_{ij}>0\) and a log rate predictor \(\eta_{ij}\), this is commonly written \(\log(\mu_{ij})=\log(t_{ij})+\eta_{ij}\); \(\log(t_{ij})\) is an offset with coefficient fixed at one.
The Poisson model assumes that the conditional mean-variance relationship and event-generating process are adequate after the predictors, exposure and random effects are accounted for. The negative-binomial family is a common alternative when conditional count variation exceeds the Poisson variance. Under the NB2 parameterization, \(\operatorname{Var}(Y_{ij}\mid b_j)=\mu_{ij}+\mu_{ij}^2/k\), where \(k>0\) is a dispersion parameter; as \(k\) grows, this variance approaches the Poisson variance. The earlier material on negative-binomial models covers this family in more depth. Choosing it is not a cure for omitted predictors, incorrect exposure, or dependence that remains unexplained.
Worked Example: Events Accumulated Over Exposure
A hypothetical unit is observed for 2 person-years and has 4 recorded events. Suppose the model gives a conditional event rate of \(0.80\) events per person-year. The expected count is \(\mu=t\lambda=2(0.80)=1.60\) events, equivalently \(\log(\mu)=\log(2)+\log(0.80)\).
Under the Poisson candidate, the probability of observing exactly 4 events is \(\displaystyle P(Y=4)=\frac{\exp(-1.60)(1.60)^4}{4!} =\frac{(0.2019)(6.5536)}{24}\approx0.0551\). The model permits counts greater than 2 because 2 is exposure, not a trial denominator. Its conditional variance is 1.60, equal to the conditional mean. If comparable observations show substantially greater conditional variability, or residual patterns indicate the Poisson model is inadequate, a negative-binomial family may be considered and the source of the mismatch investigated.
Ranked Ratings: Model the Order, Not Arbitrary Distances
An ordinal response has a finite set of categories with a meaningful order, such as ratings from 1 (lowest) to 5 (highest). A cumulative-logit model represents the probability of being at or below each cutoff. For \(K\) ordered categories, one common model is \(\operatorname{logit}\{P(Y_{ij}\leq k\mid b_j)\}=\theta_k-\eta_{ij}\), for \(k=1,\ldots,K-1\), with ordered thresholds \(\theta_1<\theta_2<\cdots<\theta_{K-1}\). Here \(\eta_{ij}\) can contain fixed and random effects. The minus sign means that increasing \(\eta_{ij}\) shifts probability toward higher categories.
This formulation uses ordering without assuming equal spacing between adjacent labels. Its proportional-odds assumption is that the predictor effects are the same at every cumulative cutoff; the thresholds differ, but the slope contribution \(\eta_{ij}\) does not. Categories must be exhaustive and correctly ordered, and observations are assumed conditionally independent given the modeled effects. If the common-slope assumption is not credible, a proportional-odds model may be too restrictive. Treating the labels as a continuous score instead makes a different assumption: that numerical distances between category codes are meaningful.
Worked Example: A Five-Level Ordered Rating
Suppose a hypothetical service evaluation uses ratings from 1 (very poor) to 5 (excellent). For one evaluation, let the linear-predictor value be \(\eta_{ij}=0.4\), and let the fitted thresholds be \(-1.2,-0.3,0.7,1.6\), in increasing order. The cumulative probability for rating at most 3 is \(\displaystyle P(Y_{ij}\leq3\mid b_j) =\operatorname{logit}^{-1}(0.7-0.4) =\frac{\exp(0.3)}{1+\exp(0.3)} =0.5744\).
Similarly, \(P(Y_{ij}\leq4\mid b_j)=\operatorname{logit}^{-1}(1.6-0.4)=0.7685\). Therefore the probability of rating exactly 4 is the difference between these cumulative probabilities: \(\displaystyle P(Y_{ij}=4\mid b_j)=0.7685-0.5744=0.1941\). The calculation depends on the categories being ordered and on the cumulative-logit model. It does not require ratings 1 through 5 to be equally spaced.
Justification, Diagnostics, and Alternatives
The probability calculations for an ordinal model follow from the fact that category probabilities are differences of cumulative probabilities. Write \(F_k=P(Y\leq k\mid b)\), with \(F_0=0\) and \(F_K=1\). Then \(P(Y=k\mid b)=F_k-F_{k-1}\). Because the thresholds increase and the logistic function is increasing, \(F_k\) increases with \(k\), so each difference is nonnegative. The probabilities sum to \(\sum_{k=1}^K(F_k-F_{k-1})=F_K-F_0=1\). This both justifies the category-probability calculation and shows why a valid ordering of thresholds matters.
The same principle—check what values are possible and how variation is generated—helps distinguish the main options:
| Observed response | Candidate family | Key structure to verify | Common concern |
|---|---|---|---|
| One outcome in two categories | Bernoulli | One binary observation per unit | Incorrect event coding or residual dependence |
| Successes out of a known total | Binomial | Fixed, known trials; common probability and conditional independence | Extra-binomial variation or a wrong denominator |
| Unbounded event count | Poisson; possibly negative binomial | Count support, suitable exposure, plausible conditional variance | Overdispersion, excess zeros or omitted structure |
| Ordered categorical rating | Cumulative ordinal model | Meaningful order and, for proportional odds, common predictor effects across cutoffs | Non-proportional effects or treating labels as equally spaced scores |
Diagnostics should be tied to the candidate model’s assumptions. For binary outcomes, inspect fitted probabilities and residual patterns for systematic lack of fit, and check whether dependence remains after the random-effects structure. For binomial counts, compare observed and model-implied variability while confirming that every denominator represents the actual number of trials. For event counts, examine residual patterns against fitted means and exposure, and assess whether conditional variation is compatible with the Poisson assumption. For ordinal outcomes, inspect category frequencies and compare observed cumulative patterns with fitted cumulative probabilities; evidence that predictor effects differ by cutoff challenges proportional odds.
A competing family should address a specific mismatch, not merely produce a preferred result. A binomial model is not interchangeable with a Bernoulli model when the denominator contains information about the number at risk. A negative-binomial model may accommodate overdispersed counts, but does not automatically resolve zero inflation or dependence. A nominal multinomial model does not use the order of a rating; it can be appropriate if categories have no defensible order, but it discards ordinal structure. Conversely, an ordinal model is unsuitable if the categories are not meaningfully ordered.
Is each response binary, a count bounded by known trials, an unbounded event count, or an ordered category?
Record trial denominators or exposure where relevant, and identify whether the response is a single outcome or an aggregated count.
Specify the response family and link, then state the assumptions that make its likelihood appropriate given the modeled effects.
Look for patterns inconsistent with the family, investigate their source, and compare alternatives that address that particular issue.
A frequent mistake is to select a family from the response’s storage format alone: for example, to use a Poisson model for every variable recorded as a count, or a linear model for ratings coded 1 through 5. The correction is to identify the outcome’s support and generating structure first. A count of successes has an upper bound determined by trials; an event count generally does not. A rating’s integers label ordered categories; they do not establish equal distances. No distribution choice repairs a poorly defined outcome, unmodeled exposure, or an implausible random-effects structure.
Check Your Understanding
For each question, decide on a response family and compare your reasoning with the solutions.
- A device either fails or does not fail during a specified inspection. Each device contributes one result. What family is a natural starting point?
- Out of 15 seeds planted in each tray, a researcher records the number that germinate. What family matches this response, and what is its support?
- A hypothetical site records 3 events over 1.5 years of observation. If the conditional rate is 2 events per year, calculate the Poisson mean and the probability of exactly 3 events.
- A survey uses ordered categories from “never” to “often.” Why is a cumulative ordinal model a more direct starting point than treating category codes as equally spaced measurements?
- What assumption of a proportional-odds model is challenged if a predictor has substantially different effects at different rating cutoffs?
Solutions. (1) Bernoulli, since each device contributes one binary result. (2) Binomial with \(m=15\), supported on \(0,1,\ldots,15\), provided the denominator is fixed and the conditional trial assumptions are plausible. (3) \(\mu=t\lambda=1.5(2)=3\); thus \(P(Y=3)=\exp(-3)3^3/3!=0.2240\). (4) The categories have a meaningful order but equal distances between labels are not established; a cumulative model uses their ordering without imposing such distances. (5) The common predictor-effect, or proportional-odds, assumption across cumulative cutoffs is questionable.