Residual Diagnostics for Mixed-Model Integration

Why Diagnose Residuals After Integration?

A mixed-model integration estimates batch-associated shifts while accounting for measured cell state. The adjusted values are useful only to the extent that the fitted model represents the structure in the data. A model can leave systematic batch differences in its residuals, fit different batches with unequal accuracy, or mistake a biological pattern for a batch effect. Residual diagnostics help expose these problems before adjusted values are interpreted further.

The examples here concern a linear mixed model fitted to a transformed expression measurement, such as log-normalized expression for one gene. They do not assume that a Gaussian model is appropriate for raw counts. Diagnostics should be interpreted on the scale on which the model was fitted, and should be repeated across genes or features of interest rather than inferred from one illustrative feature.

For a random-intercept batch model, \(z_i\) identifies the batch of cell \(i\), and each batch has a gene-specific intercept deviation. More complex models can include random slopes or additional grouping levels. Residuals are defined relative to a particular fitted model, so a diagnostic is meaningful only when its fixed effects, random effects, response scale, and grouping structure are clear.

Residuals and Their Patterns

The conditional residual is the observed value minus the fitted value including estimated random effects: \(e_{ig}=y_{ig}-x_i^\top\hat\beta_g-z_i^\top\hat b_g\). It asks what remains after the model's estimated batch effects and fixed effects have been applied. A marginal residual omits the fitted random effects: \(e^{(m)}_{ig}=y_{ig}-x_i^\top\hat\beta_g\). It includes batch variation that the random effects are intended to represent, so it is not the right residual for checking whether that variation remains after conditional adjustment.

Under a correctly specified conditional mean, conditional residuals should not show a systematic mean pattern against fitted values, cell state, or other included predictors. Their spread should also be compatible with the assumed residual variance. These are expectations, not strict rules for a finite data set: fitting the model constrains residuals, observations may have different leverage, and estimated random effects are shrunk toward zero. Consequently, a raw residual's size is not directly comparable across all cells without considering its uncertainty.

Useful displays include conditional residuals against fitted values, against continuous state covariates, and grouped by batch and state. Look for curvature, trends, clusters, or a change in spread. A residual trend with state can indicate a missing nonlinear term, an omitted state-by-batch interaction, or an imperfect state representation. A residual shift across batches can indicate incomplete batch correction or a misspecified batch structure. Neither pattern alone identifies the cause: imbalance in cell-state composition, unmodeled sample structure, or measurement artifacts can produce similar evidence.

Batch-Wise Residual Variance

A model with one residual variance \(\sigma_g^2\) assumes a common residual spread for the observations it models, subject to any explicitly modeled variance structure. To assess that assumption, calculate or plot conditional residual spread by batch. For \(n_b\) residuals in batch \(b\), a descriptive sample variance is \(s_{e,b}^2=\sum_{i\in b}(e_{ig}-\bar e_{g,b})^2/(n_b-1)\), where \(\bar e_{g,b}\) is the mean conditional residual in that batch. Large differences can indicate unequal noise, unmodeled cell-state composition, outliers, or other misspecification.

This variance comparison is not a formal test by itself. Small batches yield unstable variance estimates, and cells from the same donor, sample, or library may be dependent. Treating many correlated cells as independent can make apparent evidence much stronger than it is. When batch sizes differ, compare distributions and uncertainty as well as point estimates; consider grouping or resampling at the independent experimental-unit level where appropriate. Also compare like with like: if batches contain different proportions of cell states, a pooled within-batch variance can reflect that composition difference rather than a difference in residual noise.

Uncertainty in Fitted Batch Effects

Estimated random effects are not exact batch corrections. They are predictions that combine the observed data with the model's population distribution for batch effects. In a standard random-intercept model, this partial pooling shrinks noisy batch estimates toward the population mean, zero under the usual centered parameterization. A small fitted effect may therefore reflect either genuinely little evidence for a shift or limited information about that batch.

For a fitted effect, report its estimate together with an uncertainty measure supplied by the fitted model. Conditional prediction uncertainty for a random effect is not identical to the uncertainty of a fixed-effect coefficient; the precise calculation depends on the model and software. An approximate normal interval can be useful as a diagnostic summary when the model's approximation is reasonable, but it should not be read as a definitive test that a batch effect is present or absent. Variance-component estimation, model selection, and possible misspecification can add uncertainty not reflected in a simple interval.

What Residual Diagnostics Can and Cannot Establish

A useful justification for these checks follows from the model's conditional mean. If the model is correctly specified, then before estimating parameters, the error term has conditional expectation zero given the modeled predictors and random effects. Thus, a persistent residual trend against a predictor is evidence that the conditional mean may not capture some structure. Similarly, if the model assumes a common residual variance, persistent differences in residual spread across groups are evidence against that variance assumption. These implications motivate the diagnostics; they do not turn visual patterns into formal hypothesis tests.

In practice, parameters and random effects are estimated from the same data being checked. This makes fitted residuals constrained and can make their variability differ across cells. A useful standardized residual divides a conditional residual by an appropriate estimate of its own standard deviation, \(r_{ig}=e_{ig}/SE(e_{ig})\), when the fitting procedure provides a suitable residual standard error. This scaling can help identify unusually large residuals, but it does not make residuals independent or remove the need to examine their patterns. A universal cutoff should not replace inspection of the model and data.

Residual diagnostics complement rather than replace other integration approaches and checks. Anchor-based corrections estimate shifts from matched cell relationships; mixed-model residuals instead assess what remains after a specified hierarchical model has been fitted. For mixed models, conditional residuals assess the post-adjustment fit, while marginal residuals show variation before incorporating fitted batch effects. Neither residual plots nor fitted-effect intervals alone quantify overall batch mixing or biological conservation. They also cannot prove that a biological signal has been preserved: a model may remove a real state-specific difference if that difference is confounded with batch.

Common Pitfall and Limitations

A common mistake is to see a fitted batch effect near zero and conclude that batch correction is complete. The flawed reasoning is: “The estimated random effect is small, so the adjusted data have no batch pattern.” That conclusion ignores shrinkage and uncertainty, and it says nothing about residual differences in spread, state-dependent shifts, or omitted grouping structure. The correction is to assess conditional residuals across batch and state, examine uncertainty in fitted effects, and interpret those checks in light of the model's assumptions.

Residual diagnostics can reveal incompatibility between the fitted model and observed data, but they rarely identify one unique repair. If batches have little overlap in biological states, the data may not distinguish a batch shift from a true biological difference. A visually clean residual plot cannot resolve that identification problem. Conversely, a few large residuals do not by themselves show that integration failed; they may be isolated observations or reflect the chosen expression scale. Diagnostics should guide investigation, not substitute for biological knowledge or a separate evaluation of what the integration is meant to preserve.