Within-Cluster Residual Patterns

Why Look Within Clusters?

A cluster random effect accounts for shared features of observations in the same cluster. It does not guarantee that the remaining observations follow the order or covariate structure implied by the model. For example, after accounting for a patient-specific random intercept, residuals might still tend to rise over successive visits, or to be systematically higher when a time-varying exposure is high. Such patterns can point to structure not represented by the fitted model.

This tutorial focuses on residuals within clusters: their arrangement in a meaningful observation order and their relationship to covariates that vary within a cluster. The residual may be a suitable conditional, marginal, or simulation-based residual introduced in earlier material. A plot or summary is interpretable only if the residual definition is clear and used consistently. In particular, do not compare values calculated on different residual scales as though they were interchangeable.

Definition: A within-cluster residual pattern is a systematic arrangement or association among residuals from observations belonging to the same cluster, such as a trend across visit order or an association with a covariate after separating within-cluster variation from between-cluster variation. It is a diagnostic description, not by itself proof of a particular model defect.

What the Random Effect Does—and Does Not—Represent

For orientation, consider a GLMM with cluster \(i\), observation \(j\), and a random intercept \(b_i\):

g(\mu_{ij}) = \mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta} + b_i, \qquad b_i \sim N(0,\sigma_b^2).

Here \(\mu_{ij}\) is the conditional mean given the random effect, \(g\) is the link function, and \(\mathbf{x}_{ij}\) contains the modeled covariates. In the usual formulation, observations are conditionally independent given \(b_i\) and the covariates. The shared random intercept induces within-cluster dependence: observations in the same cluster are linked through the same \(b_i\). In a Gaussian identity-link model, this produces the familiar exchangeable covariance contribution, \(\sigma_b^2\), for any two distinct observations in a cluster.

That shared contribution is not an adjacency-specific or serial dependence model. It does not say that observations at nearby visits have additional association beyond the common cluster effect, nor does it represent an unmodeled rise, fall, or oscillation over visit order. With non-Gaussian outcomes, the observed-scale covariance also depends on the link and covariates, so it need not be a constant for every pair. The important diagnostic distinction is between dependence due to a shared cluster effect and a residual pattern tied to a particular within-cluster order or covariate.

Key takeaway: A random intercept allows observations in a cluster to be associated through a shared effect. A residual sequence that varies systematically with order may indicate additional structure; it does not show that the random intercept failed to account for all within-cluster dependence.

Residual Sequences and Adjacent Products

When observations have a meaningful order, such as visits by time, inspect residuals against that order separately by cluster or in a display that preserves cluster identity. A line joining residuals can make a trend, run, or alternation visible, but the line is a visual aid rather than a fitted model. A numerical summary can complement the plot. For standardized residuals \(r_{ij}\), one simple lag-one summary is the sum of adjacent products:

Q_1 = \sum_i \sum_{j=1}^{m_i-1} r_{ij}r_{i,j+1},

where \(m_i\) is the number of ordered observations in cluster \(i\). Positive \(Q_1\) means the net sum of adjacent residual products is positive; it does not mean every adjacent product is positive. Negative products can occur in a sequence whose total is positive. The magnitude also depends on the number of pairs and the residual scale, so \(Q_1\) has no universal cutoff. For irregular visit times, adjacency may not have a consistent meaning; use the actual ordering and gaps in time when interpreting a pattern.

Worked Example: Summarizing a Short Residual Sequence

Suppose a hypothetical cluster has five ordered visits and standardized residuals \(0.8,\ 0.4,\ -0.2,\ 0.7,\ 1.1\). Calculate the cluster’s lag-one sum and describe what it says.

There are four adjacent pairs, so

Q_{1i} = (0.8)(0.4) + (0.4)(-0.2) + (-0.2)(0.7) + (0.7)(1.1) = 0.32 - 0.08 - 0.14 + 0.77 = 0.87.

The sum is positive: the net adjacent-product contribution is positive for this observed sequence. Two of the four products are negative, so it would be wrong to say that every adjacent pair has positive residuals or positive product. The value \(0.87\) alone does not establish unusual serial structure. It must be judged against an appropriate reference distribution, for example one obtained by simulating data under the fitted GLMM and recalculating the same statistic.

Patterns Against Within-Cluster Covariates

A residual can also be examined against a covariate that changes within a cluster: visit time, dose, temperature, or a time-varying clinical measure. A key distinction is whether the question concerns differences within clusters or differences between clusters. Let \(w_{ij}\) be a covariate and let \(\bar{w}_i\) be its mean in cluster \(i\). The centered value \(w_{ij}-\bar{w}_i\) describes how observation \(j\) differs from its own cluster average. Relating residuals to this centered value focuses the diagnostic on within-cluster departures, rather than letting a between-cluster association dominate.

A descriptive within-cluster residual slope can be calculated by regressing residuals on the centered covariate through the origin. This is a summary of the pattern, not automatically a valid inferential test: residuals are generated using estimated model parameters, and observations within clusters are not generally independent. Model-based simulation can provide a more appropriate reference.

Worked Example: Residuals and a Time-Varying Covariate

In a hypothetical data set, three clusters each have four ordered measurements of a covariate. After subtracting each cluster’s covariate mean, the centered values are \(-1.5,-0.5,0.5,1.5\) in each cluster. Suppose the standardized residuals are \((-0.6,-0.2,0.3,0.7)\), \((-0.4,-0.1,0.2,0.5)\), and \((-0.5,-0.2,0.1,0.4)\), respectively. Calculate the pooled descriptive within-cluster slope.

The slope is the sum of centered-covariate times residual divided by the sum of squared centered covariates. For the first cluster, the numerator contribution is \((-1.5)(-0.6)+(-0.5)(-0.2)+(0.5)(0.3)+(1.5)(0.7)=2.2\). For the second it is \(0.6+0.05+0.10+0.75=1.5\). For the third it is \(0.75+0.10+0.05+0.60=1.5\). The total numerator is \(5.2\). The denominator is \(3[(-1.5)^2+(-0.5)^2+(0.5)^2+(1.5)^2]=3(5)=15\). Thus

\widehat{\gamma}_{r,w} = \frac{\sum_i\sum_j (w_{ij}-\bar{w}_i)r_{ij}} {\sum_i\sum_j (w_{ij}-\bar{w}_i)^2} = \frac{5.2}{15} = 0.347.

The positive descriptive slope indicates that, in these observations, residuals tend to be higher when the covariate is above its cluster mean. Its units are standardized-residual units per covariate unit. It is not a coefficient estimate for the GLMM’s outcome mean, and it does not establish that the covariate effect is statistically significant or causal. A simulation reference that repeats the full diagnostic calculation is needed to judge how unusual this slope would be under the fitted model.

Use a Model-Based Reference

Visual inspection is useful for locating a pattern, but a small number of clusters or sparse outcomes can make apparent runs common by chance. A principled check compares the observed plot or statistic with what the fitted model would generate. A typical procedure is to simulate outcomes under the fitted GLMM, calculate residuals using the same definition and fitting workflow, and recompute the same within-cluster summary. Refitting each simulated data set can be important when the diagnostic depends on estimated fixed effects or random effects; otherwise, the reference may omit variability introduced by estimation.

The simulation scheme must match the question. Simulating new random effects assesses behavior for new clusters under the model. Holding estimated cluster effects fixed instead asks a conditional question about observations given those fitted effects. These are different references, not interchangeable technical choices. For a residual-pattern check, state which is being used and use it consistently.

Worked Example: Interpreting a Simulated Reference Interval

Suppose a hypothetical analysis uses a pooled lag-one statistic \(Q_1\) across all ordered clusters. The observed statistic is \(0.42\). In 1,000 simulations under the fitted model, the 25th and 975th ordered simulated statistics are \(-0.35\) and \(0.38\), respectively. What is the diagnostic conclusion?

The observed value \(0.42\) exceeds the stated upper reference value \(0.38\) by \(0.04\). Thus it falls just outside this central 95% simulation interval and is a modest flag for more positive net adjacent products than the simulations typically showed. This is not a formal p-value, nor evidence that the null model has a 5% probability of being true. It is a discrepancy between the observed diagnostic and a model-based reference. The next step is to inspect the sequence plots and check sensitivity to residual definition, cluster composition, and the fitting procedure before changing the model.

Reading the Plot Carefully

Patterns can suggest different explanations, but they do not identify one uniquely. A gradual increase across visits may reflect a missing time effect, a nonlinear trajectory, changing exposure, or an unsuitable mean structure. Alternating residuals can suggest a cycle or measurement process. A single unusual visit can dominate a statistic without representing a repeated pattern. Examine cluster-specific plots as well as pooled summaries: one cluster with a strong pattern can be hidden by many unremarkable clusters, while pooled summaries can also make several weak patterns easier to see.

An observation order is meaningful only when it represents a scientific or design order. Sorting observations by outcome or arbitrary row number and then interpreting a sequence is not a valid serial diagnostic. Likewise, a residual trend against time is not automatically evidence for serial correlation: a misspecified mean trend can create ordered residuals even when there is no additional dependence between neighboring observations.

For a within-cluster covariate check, use the centered covariate if the target is within-cluster behavior. If both within- and between-cluster associations matter, they should be distinguished rather than combined into one slope. A cloud of residuals around zero without an obvious pattern is reassuring, but it does not prove that all model assumptions hold. Sparse binary or count outcomes can yield many tied residual values or limited visual resolution; simulation-based residuals and model-based displays may be more informative than raw response differences.

Alternatives, Limits, and Common Pitfalls

Approach	Question it addresses	Main caution
Cluster-aggregated residual checks	Are clusters, as a whole, systematically over- or under-predicted?	Aggregation can conceal ordering or covariate patterns within a cluster.
Within-cluster residual plots and summaries	Do residuals vary systematically over meaningful order or within-cluster covariates?	Patterns are exploratory and can be unstable with few observations.
Model-based simulation checks	Is the observed diagnostic unusual under the fitted model and specified simulation scheme?	Conclusions depend on the residual and simulation procedure being appropriate.

A plot is not a substitute for checking the model’s assumptions, and a residual pattern alone does not select a unique replacement. Depending on the scientific design, follow-up analysis may consider a missing within-cluster mean effect, a more flexible time trend, random slopes, or an additional serial dependence structure. These alternatives represent different scientific and statistical assumptions; they should not be added solely to remove an inconvenient-looking plot. Sparse data, few clusters, irregular observation times, and estimated random effects can all make the diagnostic less decisive.

Pitfall: “The random intercept handles all within-cluster dependence.” It induces shared-effect dependence, but does not represent additional adjacency-specific or serial dependence beyond that shared effect. Inspect whether the residuals retain structure tied to order.
Pitfall: “A positive lag-one sum means every adjacent product is positive.” It means only that the sum of those products is positive. Individual products can be negative, as in the first worked example.
Pitfall: treating a descriptive slope or plot as a test. Residuals depend on fitted parameters and cluster structure. Use a suitable model-based reference before making a stronger claim.
Pitfall: confusing within- and between-cluster association. Center the covariate within cluster when the question concerns deviations from each cluster’s own average.

A Practical Sequence for the Check

Define a meaningful order or covariate.
Confirm that observation order represents time, sequence, or another scientifically interpretable structure.

Choose and label the residual.
State whether residuals are conditional, marginal, or simulation-based, and retain that scale throughout the comparison.

Plot by cluster and summarize.
Look for trends, runs, alternation, or covariate association; use a statistic such as \(Q_1\) only as a compact summary.

Compare with a model-based reference.
Reproduce the same residual calculation and diagnostic in simulated data, using a simulation scheme suited to the question.

Investigate, do not over-interpret.
Check whether a mean-structure issue, influential cluster, or additional dependence is plausible before considering a model change.

Check Your Understanding

Try each question, then compare your reasoning with the solutions below.

A cluster has residuals \((-0.5, 0.2, 0.6)\) in visit order. Calculate its lag-one sum and interpret its sign.
Why might a within-cluster covariate be centered by subtracting its cluster mean before relating it to residuals?
A positive lag-one sum is observed. Does this show that every adjacent residual pair has a positive product? Explain.
A residual plot rises across visits even though the model contains a random intercept. Give one possible interpretation and one reason not to conclude immediately that serial correlation is present.

Solutions. (1) \(Q_{1i}=(-0.5)(0.2)+(0.2)(0.6)=-0.10+0.12=0.02\). It is slightly positive, meaning the net sum of the two adjacent products is positive; it does not by itself indicate an unusual pattern. (2) Centering isolates whether residuals tend to be higher or lower when a covariate is above that cluster’s own mean, reducing confusion with differences between cluster means. (3) No. The sum can be positive even if one or more products are negative; only the net sum is positive. (4) The fitted mean may omit a time trend, which could produce a rising residual pattern. A plot alone cannot distinguish that from additional serial dependence, and the apparent pattern should be checked against an appropriate model-based reference.