Residual and Overdispersion Diagnostics for Abundance Models

Why Diagnose Abundance Models?

A fitted abundance model summarizes how observed cell counts relate to covariates, but a plausible coefficient or small p-value does not establish that the model describes the data adequately. In single-cell studies, diagnostics are especially important because many cells may be measured for each donor while the number of independent biological replicates remains much smaller. A model that treats donors as independent replicates can still fit poorly if its variance assumptions, mean structure, or dependence assumptions are wrong.

This tutorial focuses on residuals, fitted proportions, dispersion, and donor influence in binomial and multinomial abundance models. These checks are not automatic pass-or-fail tests. They are ways to locate systematic departures from a model and decide what needs investigation.

Models and Diagnostic Quantities

For a binomial abundance analysis, let \(Y_i\) be the number of cells of a specified type among \(n_i\) captured cells from donor \(i\). A binomial regression specifies \(Y_i\mid n_i\sim\operatorname{Binomial}(n_i,p_i)\), with a regression model for \(p_i\), often \(\operatorname{logit}(p_i)=\mathbf{x}_i^\top\boldsymbol{\beta}\). Conditional on the model covariates, the binomial assumption says the count has mean \(n_i p_i\) and variance \(n_i p_i(1-p_i)\). It also assumes donors are independent; the binomial variance is appropriate only if within-donor cell dependence does not add extra variability beyond that model.

For a multinomial abundance analysis, let \(Y_{ik}\) be the count of cell type \(k\) for donor \(i\), with \(K\) cell types and total \(n_i=\sum_{k=1}^K Y_{ik}\). The model is \( (Y_{i1},\ldots,Y_{iK})\mid n_i\sim\operatorname{Multinomial}(n_i,\boldsymbol{p}_i)\), where the probabilities sum to one. It has expected count \(E_{ik}=n_i p_{ik}\) and variance \(n_i p_{ik}(1-p_{ik})\) for each component, with covariance \(-n_i p_{ik}p_{i\ell}\) between distinct components \(k\) and \(\ell\). Thus, component residuals are not independent: an excess in one part of a fixed total must be balanced by deficits elsewhere.

A fitted proportion \(\hat p_i\) or \(\hat p_{ik}\) should be considered alongside the observed proportion \(Y_i/n_i\) or \(Y_{ik}/n_i\). A useful diagnostic plot places observed proportions against fitted proportions, with donors identified and, where relevant, grouped or colored by condition or batch. A systematic trend in residuals across a covariate suggests that the mean structure may be inadequate. A few large residuals may instead indicate unusual donors, sparse categories, or data-quality problems.

Dispersion: Is the Variation Larger Than the Model Allows?

If the binomial or multinomial variance is appropriate, a Pearson goodness-of-fit statistic is the sum of squared Pearson residuals. Dividing it by the residual degrees of freedom gives a dispersion estimate. For grouped binomial observations with \(D\) donor-level counts and \(q\) fitted regression parameters, a common estimate is \(\hat\phi=X^2/(D-q)\), when \(D>q\). For multinomial counts, the denominator is the number of independent count contrasts minus the number of fitted parameters; with \(D\) donors, a common composition, and \(K\) categories, it is \((D-1)(K-1)\). For more complex models, the appropriate residual degrees of freedom depend on the model and design.

Under adequate fit and suitable large-sample conditions, \(X^2\) is approximately chi-square distributed with the residual degrees of freedom. The ratio \(\hat\phi\) is then often near one. A ratio substantially above one is a warning that variation exceeds the fitted model’s expectation; a ratio below one can reflect less variability than expected or simply sampling variation. Neither ratio should be treated as a universal cutoff. The chi-square approximation is unreliable when expected counts are very small, and a few unusual donors can dominate the statistic.

Overdispersion can arise when donors differ in unmodeled ways, when cells within a donor are correlated, or when the mean model omits important predictors. It can also arise from technical variation or misspecified cell-type probabilities. Earlier material on beta-binomial and Dirichlet-multinomial models provides alternatives that explicitly allow extra-binomial or extra-multinomial variability. A larger standard error from an overdispersion adjustment may address uncertainty, but it does not repair a wrong mean structure.

Multinomial Residual Patterns

For multinomial models, inspect residuals by both donor and cell type. One cell type may be persistently underpredicted, or donors may show coordinated shifts among several types. Because proportions sum to one, such shifts are linked: an overprediction of one category can appear alongside underpredictions in others. Looking only at a single component can therefore miss the composition-wide pattern.

A signed deviance residual can complement a Pearson residual. For a multinomial cell, it is the sign of \(Y_{ik}-E_{ik}\) multiplied by the square root of the cellwise deviance contribution \(2\{Y_{ik}\log(Y_{ik}/E_{ik})-(Y_{ik}-E_{ik})\}\), with \(Y_{ik}\log(Y_{ik}/E_{ik})\) defined as zero when \(Y_{ik}=0\). Deviance and Pearson residuals emphasize discrepancies differently, especially in sparse counts. If many expected counts are small, neither their usual approximations nor a simple dispersion ratio should be trusted without further assessment.

Influential Donors and Model Stability

A donor is influential when its inclusion substantially changes a fitted quantity of interest, such as a condition coefficient. Influence is not identical to having a large residual: a donor can be unusual but have little effect on an estimate, while a donor with substantial leverage in the design can affect the estimate even with a moderate residual. In abundance studies, the donor—not each cell—is generally the unit to remove for a leave-one-donor-out diagnostic.

For a coefficient \(\hat\beta_j\), define the deletion change as the full-data estimate minus the estimate after excluding donor \(i\):

Inspect both the size and direction of \(\Delta_{ij}\), preferably relative to the coefficient’s uncertainty and the range of changes across donors. A large change suggests sensitivity to that donor; it does not establish that the donor is erroneous. Check the donor’s counts, covariates, sequencing and annotation records, and representation in the design before deciding how to proceed. Do not remove a donor merely because deletion makes a result less statistically significant.

Diagnostics, Alternatives, and Limits

A practical diagnostic sequence is to check fitted proportions and residuals first, then assess aggregate dispersion, then investigate donors whose residuals or deletion changes stand out. Compare diagnostics across conditions and relevant covariates. If residuals systematically vary with a predictor, consider whether the mean model needs a different functional form or an omitted term. If residual variation is broadly larger than expected, consider an overdispersed model or an uncertainty method that respects donor-level replication. If a handful of categories have very small fitted counts, treat chi-square-based summaries cautiously.

The main alternatives address different concerns. A beta-binomial model can allow extra-binomial variability for a cell-type count; a Dirichlet-multinomial model can allow extra-multinomial variation across a composition; and a logistic-normal model can represent correlated variation in transformed proportions. These are not interchangeable fixes. A robust or resampling-based uncertainty analysis may reduce reliance on a particular variance assumption, but it does not correct a systematically wrong fitted mean. Model choice should reflect the data structure and scientific question, not just whichever option produces a preferred test result.

Diagnostics also have limits. With few donors, residual and dispersion estimates are noisy. With many cell types, rare categories can produce unstable residuals. A good-looking residual plot cannot establish that donors are representative, that annotation is correct, or that observed relative abundances equal absolute cell numbers. Nor does an unusual donor automatically invalidate a result. These checks identify where the model deserves scrutiny; they do not replace study-design judgment.