Why Deviance Dispersion Needs Care in Small GLMMs
A deviance-to-degrees-of-freedom ratio is often used as a quick check of whether a fitted model leaves more variation unexplained than expected. In a small or sparse generalized linear mixed model (GLMM), however, the ratio is not automatically calibrated: expected counts may be low, random effects are estimated, and there may be no uniquely appropriate residual degrees of freedom. A ratio above one can be a useful warning, but a large-sample rule of thumb does not by itself tell us how unusual the observed value is under the fitted model.
The goal here is to define a deviance-based summary precisely, calculate it, and compare it with a reference distribution obtained by simulating from the fitted GLMM and refitting it. The result is a model-based diagnostic, not a universal test of overdispersion. Its interpretation depends on the model, the definition of deviance, and the simulation procedure.
Model and Deviance Being Assessed
Write a GLMM for observation \(i\) in cluster \(j\) as \(Y_{ij}\mid b_j\) having a specified exponential-family distribution with conditional mean \(\mu_{ij}\), where \(g(\mu_{ij})=x_{ij}^{\mathsf T}\beta+z_{ij}^{\mathsf T}b_j\). The random effects are modeled as \(b_j\sim N(0,\Sigma)\), independently across clusters, and observations are conditionally independent given the random effects and fixed covariates. The chosen response distribution, link, random-effects structure, and any offset are part of the model being checked.
A common conditional diagnostic uses fitted means \(\hat{\mu}_{ij}\), obtained using the fitted fixed effects and estimated random effects, to calculate observation-level deviance contributions. For Poisson observations, the contribution is \(d(y,\hat{\mu})=2\{y\log(y/\hat{\mu})-(y-\hat{\mu})\}\), with \(0\log(0/\hat{\mu})\) defined as zero. For a grouped binomial observation with \(y\) successes from \(m\) trials and fitted success count \(\hat{\mu}\), it is
again taking a term with zero observed count to be zero. Summing the contributions gives a conditional residual deviance \(D\). These formulas compare fitted and saturated response likelihoods for the stated conditional distributions. They do not make every choice of GLMM deviance interchangeable: software may use different likelihood conventions, and a diagnostic calculated from conditional fitted means must be compared with simulations calculated in the same way.
A working denominator is sometimes set to the number of response contributions minus an approximate number of fitted parameters. In a GLMM, estimating variance components, integrating over random effects, and using estimated conditional random effects complicate that count. Thus \(D/df_{\mathrm{work}}\) is a compact summary, not a claim that \(D\) has an exact chi-square distribution with that many degrees of freedom.
Why the Large-Sample Rule Can Fail
The familiar chi-square motivation for deviance is a local approximation. For example, in a Poisson model let \(y=\mu+\delta\), with \(\delta\) small relative to \(\mu\). Expanding \(\log(1+\delta/\mu)\) about zero gives
The right-hand side is the squared Pearson residual for a Poisson observation. With adequately large expected counts and regular estimation, sums of such contributions can be approximated by chi-square distributions, with degrees of freedom adjusted for fitted parameters. The approximation is less reliable when counts are sparse: the Taylor expansion may be poor, the distribution is discrete and asymmetric, and fitting random effects changes the distribution of the diagnostic. The approximation does not become exact simply because a software package reports a degrees-of-freedom value.
Worked Example: A Sparse Poisson Deviance Ratio
Consider a hypothetical four-observation Poisson GLMM diagnostic. At the fitted conditional means, the observed counts are \(0,1,4,0\), and the corresponding fitted means are \(0.5,1,2,0.5\). These means are positive, as required by the Poisson model. Suppose the analyst uses \(df_{\mathrm{work}}=2\) for this illustration.
For the first observation, \(d(0,0.5)=2(0.5)=1.000\). For the second, \(d(1,1)=0\). For the third, \(d(4,2)=2[4\log(4/2)-(4-2)]=2[4\log(2)-2]=1.5452\). For the fourth, \(d(0,0.5)=1.000\). Therefore,
The ratio exceeds one, but that fact alone does not establish that the fitted model is inadequate. The observations are sparse, and the denominator is a working approximation. A simulation reference, using this same deviance definition and denominator, is needed to judge whether a statistic at least this large is unusual under the fitted model.
Building a Simulated Reference Distribution
A parametric simulation asks what values of the chosen statistic could arise if data were generated under the fitted GLMM and analyzed by the same fitting and diagnostic procedure. For an unconditional reference, each simulated data set is generated using the fitted random-effects distribution and then the conditional response distribution. The model is refitted to each simulated data set; the same statistic, including the same working denominator rule, is then recalculated. Refitting matters because the observed statistic was calculated after estimating model parameters.
State the deviance definition, whether it uses conditional fitted means or another explicitly defined quantity, and the working degrees of freedom.
Keep the design, offsets, and cluster structure fixed. Generate random effects and responses according to the stated fitted GLMM.
Apply the same estimation procedure to each simulated data set, then calculate its deviance and dispersion summary in the same way as for the observed data.
For an upper-tail check, count how many simulated statistics are at least as large as the observed statistic. Inspect the simulated distribution as well as the resulting tail estimate.
With \(B\) successful simulations and observed statistic \(T_{\mathrm{obs}}\), a common finite-simulation estimate of the upper-tail probability is
where \(I(\cdot)\) equals one when its condition is true and zero otherwise. The added one avoids reporting a zero tail estimate merely because no simulated statistic exceeded the observed one. This is a Monte Carlo estimate under a plug-in fitted model, not generally an exact test: nuisance parameters were estimated, and the result depends on whether the model is a reasonable description of the data.
Worked Example: Reading a Simulated Tail Estimate
Return to the hypothetical Poisson diagnostic above, with \(T_{\mathrm{obs}}=D/df_{\mathrm{work}}=1.7726\). Suppose \(B=9{,}999\) simulated data sets were generated and refitted successfully, and 710 produced a ratio at least \(1.7726\). Then
Under this fitted-model simulation procedure, about 7.11% of the simulated ratios were at least as large as the observed ratio, allowing for the stated finite-simulation calculation. This is not the probability that the model is true. It also does not prove the model is adequate; it says that this particular upper-tail discrepancy is not especially rare at a conventional 5% cutoff. A useful report would include the statistic, the simulation design, the number of replicates, and the estimated tail probability.
Binomial Calculation and the Value of Matching the Reference
Worked Example: Deviance for a Sparse Binomial Cell
Suppose a hypothetical grouped-binomial observation has \(m=5\) trials, \(y=4\) successes, and fitted success count \(\hat{\mu}=3\). The fitted success probability is \(3/5=0.6\), strictly between zero and one, so both fitted success and failure counts are positive. Its deviance contribution is
The intermediate sum is \(4\log(4/3)+\log(1/2)=0.4575811092\ldots\), which rounds to \(0.4575811\); the contribution rounds to \(0.9152\). The success and failure counts are both included: omitting the failure term would give the wrong deviance.
Now suppose a second hypothetical cell has \(m=4\), \(y=0\), and fitted success count \(\hat{\mu}=1\). Its fitted failure count is \(3\), so
The total is \(D=0.9151622\ldots+2.3014566\ldots=3.2166\). If a stated working convention uses \(df_{\mathrm{work}}=1\), the ratio is \(3.2166\). In sparse binomial data, this ratio alone is not a calibrated measure of evidence. A simulated reference should preserve the two trial sizes and the fitted GLMM structure, simulate responses under that model, refit it, and recompute the same ratio.
Diagnostics, Alternatives, and Failure Modes
The simulated distribution should be examined, not reduced automatically to one tail probability. Check that the observed statistic is calculated from the same likelihood contributions and denominator convention as the simulated statistics. Confirm that simulations respect the original covariates, offsets, trial sizes, and clusters. Verify that refits converge and that the estimated parameters are plausible. If many simulated fits fail, silently discarding them can distort the reference distribution; investigate the failures, report how many occurred, and do not interpret a tail estimate from an unstable set of refits as reliable.
The Pearson dispersion statistic is a related alternative: it sums squared residuals standardized by their model-based variances, then may divide by a working residual degrees of freedom. The earlier material on Pearson dispersion statistics explains why that summary also has limits. Deviance and Pearson summaries emphasize discrepancies differently, so a concern appearing in both is informative, but neither is a substitute for checking residual patterns and model fit. In small samples, either statistic can be compared with a simulated reference if its definition is held consistent across the observed and simulated data.
The simulation itself has limitations. A parametric reference is only as credible as the fitted model used to generate the data; it cannot validate assumptions simply by reproducing data from those assumptions. The result can also depend on the chosen conditional or marginal diagnostic, the random-effects distribution, and the fitting algorithm. If the question concerns a specific feature such as unusually many zeros or a particular cluster pattern, a single total deviance ratio may not be sensitive to it. Interpret it alongside appropriately targeted residual displays rather than treating it as a complete diagnostic.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- For a Poisson observation with \(y=0\) and fitted mean \(\hat{\mu}=0.4\), calculate its deviance contribution.
- A grouped binomial cell has \(m=6\), \(y=2\), and fitted success count \(\hat{\mu}=3\). Write its deviance contribution, including both success and failure terms.
- In 1,999 successful simulations, 38 statistics are at least as large as the observed value. Calculate the finite-simulation upper-tail estimate.
- Why should each simulated data set be refitted before calculating the deviance dispersion summary?
- True or false: if the observed deviance ratio is greater than one, this by itself demonstrates that the GLMM is overdispersed.
Solutions. (1) \(d=2(0.4)=0.8000\). (2) \(d=2[2\log(2/3)+4\log(4/3)]\); the first term accounts for successes and the second for failures. (3) \(\hat{p}_{\mathrm{sim}}=(1+38)/(1{,}999+1)=39/2{,}000=0.0195\). (4) The observed statistic follows parameter estimation, so refitting simulated data makes the reference reflect the same estimation step. (5) False: the ratio is a summary with an approximate denominator; sparsity and random-effects estimation can invalidate simple rules of thumb, so compare it with a suitable reference and other diagnostics.