Residual Diagnostics For Factor Models

Why Inspect Residuals?

A factor model summarizes a collection of measurements using a smaller set of latent dimensions. After fitting it, the reconstructed data will rarely match every observed value. The differences between the observations and their reconstructions are residuals. Their patterns help answer a practical question: have the fitted factors captured the systematic structure that matters for this analysis, or is recognizable structure still left behind?

In multi-omics data, a residual pattern can be specific to one feature, shared by a group of features in one view, or present across subjects or views. For example, residuals may be unusually variable for a few features, correlated across features, or associated with a measured technical variable. These patterns do not by themselves identify the cause. They indicate that the fitted reconstruction or its error assumptions warrant closer examination.

Definition: For view \(m\), let \(X^{(m)}\) be the \(n \times p_m\) matrix analyzed by the factor model, with subjects in rows and features in columns. Let \(\hat{X}^{(m)}\) be its fitted reconstruction on the same analysis scale. The residual matrix is \(E^{(m)} = X^{(m)} - \hat{X}^{(m)}\); its entry \(e_{ij}^{(m)}\) is the observed value minus the reconstructed value for subject \(i\) and feature \(j\). A standardized residual divides this difference by an appropriate feature-specific residual standard deviation, when such a scale is defined.

The analysis scale matters. If a view was centered or standardized before fitting, residuals refer to that transformed scale, not automatically to the original measurement units. For count, binary, or other non-Gaussian observations, the meaning and expected variability of residuals depend on the observation model; a raw difference need not have constant variance or an approximately normal distribution.

Assumptions and What Residuals Can Show

Residual plots are most directly interpretable when the fitted model specifies a meaningful conditional mean and a suitable error scale. Under a simple continuous-data model, one might write \(X^{(m)} = \hat{X}^{(m)} + \varepsilon^{(m)}\), where the remaining errors have mean zero conditional on the fitted structure and, for standardized residual interpretation, have a correctly specified variance. Some factor models also assume errors are independent across features or subjects, or follow a particular distribution. Those are model assumptions to assess, not properties guaranteed by computing residuals.

In-sample residuals are calculated using factors and loadings estimated from the same observations. Estimation can make them look smaller or more structured than residuals for new subjects. Moreover, fitted factor scores and loadings are estimated quantities, so residuals are not generally independent observations with a known reference distribution. Graphical patterns are useful diagnostics, but a residual histogram or correlation alone is not a formal test of model adequacy.

Useful checks include the distribution and scale of residuals within each view, correlations between residual features, and plots of residuals against fitted values, subjects, or measured variables that might reveal a trend. For multi-omics comparisons, inspect views separately as well as jointly: the numerical scale and noise structure can differ substantially from one view to another.

Worked Example: Checking Residual Spread for One Feature

Suppose a hypothetical continuous feature was standardized before fitting a factor model. For five subjects, its residuals are \(-1.2, -0.4, 0.1, 0.3, 1.2\), in standardized measurement units. Their mean is zero, and the sum of squared residuals is

$$ (-1.2)^2 + (-0.4)^2 + 0.1^2 + 0.3^2 + 1.2^2 = 1.44 + 0.16 + 0.01 + 0.09 + 1.44 = 3.14. $$

The root mean squared residual is \(\sqrt{3.14/5} = 0.792\) standardized units. If the aim is to describe the sample standard deviation after estimating the residual mean, it is \(\sqrt{3.14/(5-1)} = 0.886\) standardized units. The denominators answer different descriptive questions; neither should be treated as a universal estimate of the model's error scale without considering how the model was fitted.

These five values show no obvious extreme residual, but that is weak evidence: five observations cannot establish a distributional assumption. In a real diagnostic, compare the feature's residual spread with other features in the same view and inspect a residual plot or quantile plot. A much larger spread could indicate a poor noise-scale assumption or a feature whose variation is not captured well.

Residual Correlations and Remaining Structure

If the model assumes residual independence across features, a correlation matrix of residual columns can expose departures from that assumption. Strong residual correlation means that subjects with a positive residual for one feature tend also to have a positive or negative residual for another. It may suggest an omitted shared pattern, an unmodeled technical effect, or dependence that the assumed error model does not represent. It does not, on its own, reveal which explanation is correct.

A short derivation clarifies why omitted factors can leave correlated residuals. Suppose the true data contain an additional subject-level latent component \(U\) with feature loadings \(A\), but the fitted reconstruction omits it. For a given subject, write the remaining residual vector as \(e = A u + \varepsilon\), where \(u\) is that subject's omitted latent score and \(\varepsilon\) is the remaining error. If \(u\) and \(\varepsilon\) are uncorrelated, the residual covariance is

\operatorname{Cov}(e) = A \operatorname{Cov}(u) A^T + \operatorname{Cov}(\varepsilon).

This follows by expanding \(\operatorname{Cov}(Au+\varepsilon)\): the two cross-covariance terms are zero under the stated uncorrelatedness assumption. If errors are also uncorrelated across features, their covariance is diagonal. The omitted component can nevertheless create off-diagonal residual covariance wherever its loadings connect features. Thus, correlated residuals can be evidence of remaining shared structure, subject to sampling variability and the effects of fitting the model.

Worked Example: Calculating a Residual Correlation

In a hypothetical analysis, residuals for two features across five subjects are \(a=(-2,-1,0,1,2)\) and \(b=(-2,0,2,0,0)\). Both residual means are zero. The sample correlation is the sum of cross-products divided by the square root of the two sums of squared deviations:

r_{ab} = \frac{\sum_{i=1}^{5} a_i b_i} {\sqrt{\sum_{i=1}^{5}a_i^2 \sum_{i=1}^{5}b_i^2}} = \frac{4}{\sqrt{10 \times 8}} = 0.447.

Here the cross-product sum is \(4 + 0 + 0 + 0 + 0 = 4\); the sums of squares are \(10\) and \(8\). The residuals have a moderate positive sample correlation in these five subjects. This calculation is descriptive, not a significance test, and the small sample makes the estimate imprecise. A larger analysis should examine whether the pattern is stable and whether it concentrates in a biologically or technically coherent feature group.

Feature-Wise Patterns and Plots

Residual summaries can hide structure if they are reported only as one number for the entire model. Examine residual distributions by feature and view, and use plots that preserve the subject or feature ordering when that ordering is meaningful. A heat map can reveal blocks of positive and negative residuals; a plot against fitted values can reveal nonlinearity or changing spread; and plots against recorded covariates can reveal residual association with a variable not represented adequately in the fitted model.

A residual trend against a measured covariate is not proof that the covariate caused the pattern. It may reflect omitted structure, confounding, a poor transformation, or a feature-specific issue. Earlier material on confounding between latent factors and covariates is relevant when interpreting such associations. The diagnostic question is narrower: does residual structure remain associated with a measured variable, and is that consistent with the model's intended assumptions?

Worked Example: Detecting a Trend Against Fitted Values

For a hypothetical feature, suppose five fitted values are \(1, 2, 3, 4, 5\) units, with residuals \(-0.8, -0.4, 0, 0.4, 0.8\) units. The residual mean is zero. The slope from regressing residuals on fitted values is the sum of centered cross-products divided by the sum of centered fitted-value squares:

\frac{(-2)(-0.8)+(-1)(-0.4)+(0)(0)+(1)(0.4)+(2)(0.8)} {(-2)^2+(-1)^2+0^2+1^2+2^2} = \frac{4.0}{10} = 0.400.

The residuals increase by \(0.400\) units for each one-unit increase in the fitted value in this small example. The pattern is perfectly linear for these five points: the residual equals \(0.4\) times the fitted value minus \(1.2\). Such a trend is inconsistent with a simple interpretation of residuals as having no systematic mean pattern across the fitted range. It could point to a scale or mean-structure problem, but five constructed values cannot determine the cause.

A Practical Diagnostic Sequence

Confirm the analysis scale.
Record any centering, scaling, transformation, and view-specific residual definition. Compare residuals only on interpretable, compatible scales.

Summarize residual size by view and feature.
Inspect means, spreads, extreme values, and missingness. Investigate features with unusually large residual variation rather than relying only on a pooled summary.

Inspect distributions and dependence.
Use histograms or quantile plots for distributional shape, and examine residual correlations or heat maps for remaining feature dependence.

Look for systematic patterns.
Plot residuals against fitted values, subject order when meaningful, and relevant measured variables. Assess whether apparent patterns are coherent and reproducible rather than isolated fluctuations.

Interpret within the model's assumptions.
Decide which assumption or representation the pattern challenges, and state what the diagnostic cannot establish. Do not infer a cause from a residual association alone.

Alternatives, Failure Modes, and Limits

Residual diagnostics complement, but do not replace, other summaries. Variance decomposition describes how variation is allocated by factors and views; it does not show whether the leftover variation has a systematic pattern. A single overall reconstruction error can be small while a subset of features has large errors or correlated residuals. Conversely, a visually striking pattern in a small sample may be unstable. A residual heat map is useful for locating structure, whereas a scalar error is more compact; neither alone answers every adequacy question.

Formal normality or correlation tests are possible under specific sampling and model assumptions, but in high-dimensional settings they can be sensitive to sample size, multiple comparisons, and factor estimation. Treat plots and summaries as diagnostic evidence, not automatic decision rules. Posterior predictive checks, which assess replicated data under a fitted generative model, are a distinct approach; here the focus is on residuals from the fitted reconstruction itself.

Pooling unlike views. Raw residual magnitudes from views with different units or noise scales are not directly comparable. Standardize only using a scale appropriate to each view's model.
Demanding normal residuals universally. Normality is not a requirement for every factor model or diagnostic. Judge distributions against the assumed observation model and the purpose of the analysis.
Calling every residual correlation a missing biological factor. Dependence may arise from technical structure, feature processing, sampling noise, or misspecified errors. The correlation identifies a pattern, not its cause.
Reading an in-sample plot as an independent validation. The same data helped estimate the reconstruction. Residual diagnostics do not establish predictive performance for new subjects.

Key takeaway: Residual diagnostics ask whether the fitted reconstruction leaves structured discrepancies on the analysis scale. Interpret distributional shape, feature correlations, and feature-wise trends in light of the view-specific model assumptions; treat each as evidence to investigate, not as a causal conclusion or a guarantee of adequacy.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

If an observed value is \(2.7\) and its fitted reconstruction is \(3.1\), what is the residual?
Why might raw residual sizes from two omics views be an invalid direct comparison?
What does positive residual correlation between two features establish, and what does it not establish?
A residual-versus-fitted plot shows increasing residuals as fitted values rise. Name one model concern this could raise, and one reason the plot does not identify its cause.
Why should an in-sample residual diagnostic not be described as independent validation on new subjects?

Solutions. (1) The residual is \(2.7 - 3.1 = -0.4\), in the same units as the analyzed values. (2) The views may use different units, transformations, or noise scales, so equal raw differences need not represent equal discrepancies. (3) It establishes an observed positive association between the residual columns in the analyzed subjects; it does not prove an omitted biological factor or identify a cause. (4) It may raise concern about an inadequate mean or scale representation; the pattern could also reflect other omitted structure or data features, so the plot alone cannot distinguish explanations. (5) The observations were used to estimate the factor reconstruction, so the resulting residuals are not an independent assessment of performance on new subjects.