Start With What Was Recorded
A response model should represent the data actually recorded, not a convenient transformation of them. A toxicity indicator, an ordered severity grade, a choice among several unrelated outcomes, an event count, and a time to event do not have the same sampling structure. Treating them all as normally distributed measurements can distort both uncertainty and interpretation. In a nonlinear mixed-effects model, the response likelihood must also be combined with the model for subject-level random effects.
The key selection questions are: What values can the response take? Is there a meaningful ordering? Is the number of opportunities or the time at risk known? Are observations repeated within subjects? Is an event time observed exactly, or only known to exceed a censoring time? These questions determine the likelihood before any estimation method is chosen.
The conditional-independence statement is an assumption, not a consequence of using random effects. It says that, after conditioning on the modeled random effects and predictors, the remaining responses at different occasions have the specified product distribution. If residual dependence remains, the likelihood may need additional structure. A subject’s random effect is shared across that subject’s records; it does not make those records independent after the random effect is integrated out.
Match the Response Support to a Likelihood
A single binary outcome is commonly modeled as Bernoulli. If \(Y_{ij}\) is zero or one and \(p_{ij}\) is its conditional probability of one, then \(P(Y_{ij}=y\mid b_i)=p_{ij}^{y}(1-p_{ij})^{1-y}\). A logistic model sets \(\operatorname{logit}(p_{ij})=\eta_{ij}\), where \(\eta_{ij}\) includes the modeled predictors and may include a subject-specific random effect.
A binomial response is appropriate when the recorded outcome is a number of successes out of a known number of trials, and the trials have the same conditional success probability and the required conditional independence. Its likelihood includes the trial count \(m\): \(P(Y=y)=\binom{m}{y}p^y(1-p)^{m-y}\). A binomial denominator is not interchangeable with a count of subjects if each subject contributes multiple trials or if the trial structure differs.
If categories have a meaningful order, such as mild, moderate, and severe, an ordinal likelihood uses that order without assuming equal numerical spacing between categories. One common choice is the cumulative logit model. For categories \(1,\ldots,K\), it specifies \(\operatorname{logit}P(Y_{ij}\leq k\mid b_i)=c_k-\eta_{ij}\), for \(k=1,\ldots,K-1\), with ordered cutpoints \(c_1<\cdots<c_{K-1}\). This form assumes a common slope \(\eta_{ij}\) across cumulative splits, often called the proportional-odds assumption. It should not be used automatically when that assumption is implausible.
For nominal categories—distinct outcomes without a defensible order—a multinomial model is more appropriate. With category \(K\) as the reference, a baseline-category multinomial logit specifies \(\log\{P(Y_{ij}=k\mid b_i)/P(Y_{ij}=K\mid b_i)\}=\eta_{ijk}\) for \(k=1,\ldots,K-1\). The category probabilities sum to one. A numerical code for a category does not make the outcome quantitative or ordinal.
Worked Example: Turning Ordinal Cumulative Probabilities Into Category Probabilities
Suppose, hypothetically, that a three-level tolerability grade is modeled with a cumulative logit, cutpoints \(c_1=-1\) and \(c_2=1\), and \(\eta=0\) for a subject at the stated predictor values and random effect. The model gives \(P(Y\leq k)=\operatorname{logistic}(c_k-\eta)\), where \(\operatorname{logistic}(x)=1/(1+e^{-x})\).
The probability of grade 1 is \(P(Y=1)=P(Y\leq1)=\operatorname{logistic}(-1)=0.268941\). The cumulative probability through grade 2 is \(P(Y\leq2)=\operatorname{logistic}(1)=0.731059\). Therefore, the middle-grade probability is the difference of the unrounded cumulative probabilities:
The remaining probability is \(P(Y=3)=1-P(Y\leq2)=0.268941\). The three category probabilities sum to one, up to rounding. Rounding the cumulative probabilities before subtracting can change the reported middle-category probability.
Worked Example: A Nominal Multinomial Outcome
In a hypothetical pharmacometric analysis, a categorical response has three nominal outcomes: no change, improvement, and worsening. At one specified covariate and random-effect value, suppose the fitted probabilities are 0.20, 0.50, and 0.30, respectively. Taking worsening as the reference, the two log odds are:
These are category-versus-reference contrasts, not changes in a continuous response. Conversely, if the fitted logits are these values, the unnormalized category weights relative to worsening are \(e^{-0.4055}\approx0.6667\), \(e^{0.5108}\approx1.6667\), and \(1\). Their sum is \(3.3334\), so the normalized probabilities are approximately \(0.20\), \(0.50\), and \(0.30\). The multinomial likelihood uses the observed category for each response; it does not treat the category labels as equally spaced scores.
Counts Need Their Opportunity and Exposure Structure
A Poisson model is a basic likelihood for a nonnegative integer count \(Y_{ij}\), with conditional mean \(\mu_{ij}\) and variance \(\mu_{ij}\). A common log-mean specification for a count accumulated over exposure \(E_{ij}>0\) is \(\log(\mu_{ij})=\log(E_{ij})+\eta_{ij}\). The term \(\log(E_{ij})\) is an offset: its coefficient is fixed at one. Thus, \(\exp(\eta_{ij})\) represents the conditional expected rate per unit of exposure. The offset is appropriate only when the mean is proportional to exposure under the model.
This structure differs from a binomial denominator. Exposure such as patient-time scales an expected event count; a binomial denominator is a known number of trials, each with a success or failure. If counts are more variable than the Poisson model permits, or if the data show excess zeros, a different count likelihood may be needed. The earlier material on count dispersion and hurdle models covers those specific alternatives; the general requirement here is to check whether the selected count distribution is compatible with the observed data.
Worked Example: Applying a Count Offset
Suppose, hypothetically, that one subject has 3 patient-years of observation and the fitted conditional event rate is \(0.8\) events per patient-year. Under a Poisson model with the exposure offset, the conditional mean count is:
If the observed count is 6, its conditional probability under this fitted model is:
This calculation does not imply that the observed rate of \(6/3=2\) events per patient-year is the model’s fitted rate. The fitted rate is \(0.8\); the observed count is one realization around its model-based mean. If exposure were entered as an ordinary estimated covariate rather than an offset, the model would estimate its coefficient instead of fixing proportionality to exposure.
Event Times Require Event and Censoring Contributions
For an event time \(T\) with density \(f(t)\), survival function \(S(t)=P(T>t)\), and hazard \(h(t)=f(t)/S(t)\), an exact event at time \(t\) contributes \(f(t)=h(t)S(t)\). A right-censored observation at time \(c\) contributes \(S(c)\), because it establishes that the event time exceeds \(c\), not that an event occurred at \(c\). With event indicator \(\delta\), the contribution can be written \(f(t)^\delta S(t)^{1-\delta}\), where \(\delta=1\) denotes an observed event and \(\delta=0\) denotes right censoring.
In a mixed-effects event-time model, the hazard or survival function may depend on \(b_i\). Subject-level likelihood contributions must then be integrated over the random-effects distribution. The usual censoring contribution also relies on censoring being noninformative given the modeled information: roughly, the censoring process must not provide additional information about the event time after conditioning on that information. The likelihood must reflect the actual censoring and observation scheme; truncation is not the same as censoring.
Worked Example: One Event and One Right-Censored Time
Consider a hypothetical exponential event-time model with constant hazard \(0.2\) per year for one subject who has an event at 2 years, and constant hazard \(0.1\) per year for another subject who is right-censored at 3 years. Assume the stated hazards are the applicable conditional hazards and the censoring is noninformative under the model. For a constant hazard \(\lambda\), \(S(t)=e^{-\lambda t}\) and \(f(t)=\lambda e^{-\lambda t}\).
The event contribution is \(f(2)=0.2e^{-0.2(2)}=0.2e^{-0.4}\). The censored subject contributes \(S(3)=e^{-0.1(3)}=e^{-0.3}\). Assuming independent subjects, the joint likelihood contribution is:
The second subject contributes a survival probability, not the density of an event at year 3. Using an event density for that subject would assert that the event was observed at the censoring time and would encode the wrong data.
A Short Justification for the Mixed-Effects Likelihood
The conditional response model and the population likelihood answer different probability questions. Conditional on \(b_i\), the model gives probabilities or densities for subject \(i\)’s observed responses. Since \(b_i\) is unobserved, the likelihood for the observed data averages that conditional likelihood over the assumed distribution \(p(b_i)\). Under the conditional-independence assumption, this gives:
The product is inside the integral because the same \(b_i\) applies to all of subject \(i\)’s occasions. Multiplying separately integrated occasion probabilities would instead allow each occasion to have a newly drawn random effect, which is a different model. This distinction also explains why a coefficient’s subject-conditional interpretation generally cannot be obtained by simply reading it as a population-average effect.
Checking the Choice and Recognizing Failure
A model can converge and still use the wrong likelihood. Check the response coding and data collection first, then assess whether the fitted distribution reproduces important features of the observations. Diagnostics should respect the response type and repeated-subject structure; residuals should not automatically be treated as normally distributed measurements.
- Binary or binomial: verify which unit is a trial and that the denominator is the number of eligible trials. Check predicted event frequencies across relevant predictor ranges and subjects. Extra within-subject association or varying probabilities among nominally identical trials can invalidate a simple binomial specification.
- Ordinal: verify that category order is meaningful and that all modeled categories have adequate data. Examine whether the common-slope, proportional-odds restriction is plausible; systematic mismatch in cumulative-category predictions is a warning.
- Multinomial: verify that the categories are mutually exclusive and collectively exhaustive for the recorded outcome. Compare observed and predicted category frequencies; rare categories can make estimates unstable.
- Counts: check the exposure units, zeros, and conditional variability. Counts systematically more variable than the fitted Poisson distribution allows, or persistent excess zeros, signal that the chosen count model may be inadequate.
- Event times: confirm event indicators, time units, and censoring records. Compare predicted survival patterns with observed event and censoring patterns, and assess whether the assumed censoring mechanism is defensible.
For all response types, inspect whether fitted predictions and simulation-based summaries reproduce the features relevant to the scientific question. Persistent discrepancies can arise from an unsuitable likelihood, an incomplete predictor structure, an inadequate random-effects distribution, or dependence not represented by the model. A response-model diagnostic does not by itself identify which component is responsible.
| Recorded response | Typical likelihood | Key structure to check |
|---|---|---|
| One yes/no outcome | Bernoulli | One binary observation per modeled unit |
| Successes out of known trials | Binomial | Trial denominator and conditional trial assumptions |
| Ordered grade | Ordinal cumulative model | Meaningful order and link assumptions |
| Unordered category | Multinomial | Mutually exclusive categories and category probabilities |
| Event count over exposure | Poisson or another count likelihood | Count dispersion and exposure offset |
| Time to event, possibly censored | Event-time likelihood | Event indicator, time scale, and censoring contribution |
Common Errors and Limits
The most common denominator error is to use the number of subjects as the binomial denominator when the recorded data represent a different number of trials. The most common count error is to omit exposure or to estimate an exposure coefficient when proportionality was intended. The most common survival error is to treat a censored time as an event. Each mistake changes the probability model, not merely the presentation of the results.
A second pitfall is confusing conditional and population-marginal interpretations. A logistic coefficient with a random effect describes a change in conditional log odds at a fixed random-effect value. Averaging predicted probabilities over the random-effects distribution generally does not preserve that same logistic form or coefficient interpretation. Likewise, predicted rates conditional on a subject effect need not equal population-average rates. State which prediction or effect is being reported.
No likelihood is universally best. Ordinal models retain ordering but may impose restrictive cumulative relationships; multinomial models avoid that ordering assumption but estimate more category-specific parameters. Poisson models are simple but constrain conditional variance to equal the mean. Event-time models depend on assumptions about the event process and observation scheme. All mixed-effects models additionally depend on the chosen random-effects distribution and the adequacy of the modeled dependence structure.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A subject has four independent opportunities for a binary response, with the number of successes recorded. What likelihood family and denominator are appropriate under a common conditional probability?
- A three-level response has a defensible mild-to-severe order. What distinguishes an ordinal model from treating the levels as a nominal outcome?
- A count of adverse events is recorded over 5 patient-years. If the modeled rate is 0.6 events per patient-year, what is the Poisson mean under a proportional-exposure offset?
- A subject is event-free when follow-up ends at 8 months. What is the right-censoring contribution, and what does it say about the event time?
- Why must repeated responses be multiplied conditionally before integrating over a shared subject random effect?
Solutions. (1) Use a binomial likelihood with denominator \(m=4\); the observed value is the number of successes out of those four opportunities, assuming the trials share the conditional probability and satisfy the model’s conditional-independence assumption. (2) An ordinal model uses the ordering, often through cumulative probabilities, whereas a nominal model specifies category comparisons without imposing an order. (3) The mean is \(5(0.6)=3\) events. (4) The contribution is \(S(8\text{ months})\); it records that the event time exceeds 8 months, not that an event occurred at that time. (5) The same random effect applies to all of the subject’s occasions, so the correct contribution is \(\int\prod_j f(y_{ij}\mid b_i)p(b_i)\,db_i\); integrating each occasion separately would represent a different random-effect structure.