Residual Autocorrelation and Serial Dependence

Why Residual Order Matters

A PD residual is not just a discrepancy between an observed response and its model prediction; it is also located at a particular time and, in longitudinal data, within a particular subject. If residuals at nearby times tend to have similar signs or magnitudes, treating them as unrelated can misrepresent the information in the data. That pattern is called serial dependence. Conversely, a sequence of residuals that rises or falls over time may indicate that the model’s mean time course is wrong rather than that its errors are correlated.

The distinction matters because a residual diagnostic alone does not identify the cause of a pattern. A lag plot or autocorrelation summary describes ordering; scientific interpretation requires checking whether the fitted mean response is adequate over time and whether the residuals being compared are appropriate for the question. The earlier material on residuals versus time provides the basic time-ordered display; here the focus is dependence across ordered observations.

Definition: For one subject, let $e_j$ be the chosen residual at the $j$th observation in time order, for $j=1,\ldots,m$. A lag-$k$ pair consists of residuals $e_j$ and $e_{j-k}$, where both observations exist. Serial dependence means that the residual distribution at one time is not independent of residuals at earlier times. Lag $k$ refers to $k$ observation steps, not necessarily a fixed elapsed-time interval.

Lag Plots and Sample Autocorrelation

A lag-1 plot places $e_j$ on one axis and $e_{j-1}$ on the other for consecutive observations from the same subject. A cloud without a clear pattern is compatible with weak lag-1 association; an upward-sloping cloud suggests positive association, while a downward-sloping cloud suggests negative association. Plotting several lags can reveal patterns that a lag-1 display misses. These visual assessments are descriptive, not proof of independence.

One common sample autocorrelation at lag $k$ centers the residuals around their within-sequence mean $\bar e$, then compares lagged cross-products with the total sum of squares:

\hat{\rho}_k = \frac{\sum_{j=k+1}^{m}(e_j-\bar e)(e_{j-k}-\bar e)} {\sum_{j=1}^{m}(e_j-\bar e)^2}.

This convention uses the full-series denominator at every lag, so estimates are not exactly the usual sample correlation of two separately formed vectors. Other software conventions exist; interpretation should match the definition used. Values near 1 indicate positive serial association at that lag, values near −1 indicate negative association, and values near 0 indicate little linear association. A small estimate does not rule out nonlinear dependence or dependence at another lag.

Worked Example: Calculating a Lag-1 Autocorrelation

Suppose the standardized residuals for one hypothetical subject, in chronological order, are $-1.5,-1.0,-0.5,0.5,1.0,1.5$. Their mean is $\bar e=0$. Calculate the lag-1 autocorrelation using the definition above.

The lag-1 numerator is the sum of products of adjacent residuals: \[ (-1.0)(-1.5)+(-0.5)(-1.0)+(0.5)(-0.5)+(1.0)(0.5)+(1.5)(1.0) =1.5+0.5-0.25+0.5+1.5=3.75. \]

The denominator is the sum of squared residuals: \[ (-1.5)^2+(-1.0)^2+(-0.5)^2+(0.5)^2+(1.0)^2+(1.5)^2 =2.25+1+0.25+0.25+1+2.25=7.00. \]

Therefore $\hat{\rho}_1=3.75/7.00=0.536$. This is a positive lag-1 pattern in this short sequence. It is not, by itself, evidence that the data-generating errors follow a particular correlation model: the ordered rise in these residuals could also arise because the fitted mean time course failed to capture a trend.

Runs in the Signs of Residuals

A complementary diagnostic discards residual magnitude and considers only whether each residual is positive or negative. A run is a maximal consecutive sequence of the same sign. For example, the signs $+,+,-,-,-,+,-$ contain four runs: $++$, $---$, $+$, and $-$. Too few runs relative to the number expected under random ordering can indicate clustering of signs; unusually many can indicate frequent alternation.

Definition: Conditional on $n_+$ positive and $n_-$ negative residuals, let $N=n_++n_-$ and let $R$ be the number of sign runs. Under the runs-test null, all orderings of those fixed signs are equally likely. For $n_+>0$ and $n_->0$, $$ E(R)=1+\frac{2n_+n_-}{N}, \qquad \operatorname{Var}(R)= \frac{2n_+n_-(2n_+n_- -N)}{N^2(N-1)}. $$

The conditional null is a random-ordering assumption for the signs. It does not establish that residual magnitudes are independent, and it is not automatically appropriate when the sequence has been selected or grouped in a way that makes sign order nonexchangeable. Exact runs-test probabilities are preferable for small samples. A normal approximation may be considered for sufficiently large counts, but its quality depends on the sign counts and the discrete distribution; a continuity correction may also be relevant.

Worked Example: Runs Test With Five Residuals of Each Sign

For a hypothetical subject, suppose ten consecutive PD residuals contain five positive and five negative values, in the sign order $+,+,-,-,+,+,-,-,+,-$. The runs are $++$, $--$, $++$, $--$, $+$, and $-$, so the observed statistic is $R=6$. Under the fixed-count random-ordering null, $n_+=n_-=5$ and $N=10$.

The expected number of runs is \[ E(R)=1+\frac{2(5)(5)}{10}=1+5=6. \] The variance is \[ \operatorname{Var}(R)= \frac{2(5)(5)\{2(5)(5)-10\}}{10^2(10-1)} =\frac{50(40)}{900} =\frac{2000}{900} =\frac{20}{9}\approx2.222, \] so $SD(R)=\sqrt{20/9}\approx1.491$. The observed run count equals its null expectation; the standardized deviation is $z=(6-6)/1.491=0$.

For these counts, an exact two-sided test can be calculated from the 252 equally likely sign orderings, since $\binom{10}{5}=252$. There are 72 orderings with six runs, so the exact probability of six runs is $72/252=0.286$; this is not itself a two-sided p-value. For a two-sided test based on distance from the expectation, the outcomes at least as far from 6 as the observed $R=6$ include every possible outcome, so the p-value is 1. This example provides no evidence against random sign ordering.

To see why approximations can mislead with small samples, consider instead an observed $R=4$, with the same five positive and five negative signs. The variance remains $20/9$, and the uncorrected normal statistic is \[ z=\frac{4-6}{\sqrt{20/9}}\approx-1.342, \] giving an approximate two-sided normal p-value of about 0.180. The exact two-sided p-value is $P(R\leq4)+P(R\geq8)=84/252=0.333$. The difference is substantial: with these small counts, use the exact result rather than relying on the normal approximation. Neither result would reject at $\alpha=0.05$.

Separating Dependence From Mean-Time-Course Error

Residual correlation and mean misspecification are different features of a model. A residual sequence with an unmodeled upward or downward trend can show positive adjacent products and a large autocorrelation, even if the unexplained errors themselves are not serially correlated. A genuinely correlated error process, by contrast, can produce residuals that cluster in sign or magnitude after the mean time course has been adequately represented. Both features can occur together.

Use the residual-versus-time display alongside lag diagnostics. Look for sustained curvature, a systematic rise or fall, a misplaced peak, or a repeated phase-specific pattern; such structure suggests that the expected response may be inadequately modeled. Then examine lag plots and autocorrelation summaries for residuals after accounting for that mean structure. The aim is not to declare one explanation from a single plot, but to compare plausible explanations and assess whether a proposed change improves the relevant diagnostics and predictions.

Worked Example: A Trend Can Mimic Serial Dependence

Consider two hypothetical sequences of six standardized residuals, each observed at equally spaced times. Sequence A is $-1.2,-0.7,-0.2,0.3,0.8,1.0$; Sequence B is $0.9,-0.8,0.7,-0.9,0.8,-0.7$. Sequence A changes gradually from mostly negative to mostly positive, while Sequence B alternates signs.

For Sequence A, the ordered pattern itself is a warning that the fitted mean response may have left a time trend in the residuals. A positive lag-1 autocorrelation in this setting cannot be interpreted uniquely as correlated errors. The next diagnostic step is to inspect the time-course fit and consider whether a scientifically defensible mean structure is missing, then re-evaluate residual dependence.

For Sequence B, the frequent sign changes suggest negative short-lag association or an overly flexible mean that is tracking noise; either possibility merits investigation. The sign pattern alone cannot select between them. In both sequences, a lag plot and autocorrelation estimates summarize temporal structure, while the residual-versus-time plot helps expose systematic mean-pattern errors. These six values are illustrative, not a formal test or a real study.

Practical Conditions, Diagnostics, and Alternatives

Before interpreting serial diagnostics, check that the residual definition is suitable. Variance-standardized or otherwise appropriately standardized residuals are often more comparable over time when the PD error variance changes with response level. Conditional residuals describe deviations conditional on subject-specific estimates; population-level residuals answer a different question. Mixing residual types or pooling subjects without preserving their within-subject order can create patterns that do not represent a within-subject serial process.

Respect subject boundaries. Do not form a lag pair between the last observation of one subject and the first observation of another. With multiple subjects, summarize within-subject patterns or use an analysis that explicitly accounts for subject structure.
Check the time scale. A lag of one observation can represent different elapsed times under irregular sampling. Ordinary lag-$k$ autocorrelation then has no single time-gap interpretation. In particular, do not treat unevenly spaced measurements as though each lag were a fixed duration.
Account for model fitting. Residuals are computed after estimating model parameters, so their dependence and reference distributions need not match those of independent, unestimated errors. Rules of thumb such as bands of approximately $\pm1.96/\sqrt{m}$ for an autocorrelation plot are only rough guides under idealized conditions, not universal tests for fitted PD residuals.
Inspect multiple views. A lag plot emphasizes pairwise association, the autocorrelation function displays association at selected lags, and the runs test uses only signs. A pattern visible in one may be weak or absent in another; none replaces assessment of the mean time course.

A key alternative to diagnosing dependence is to revise a scientifically justified mean model when residuals show systematic time structure. Another is to represent residual correlation explicitly when the mean appears adequate but residuals remain associated in time. These approaches address different model components, and adding a correlation structure should not be used simply to conceal an incorrect expected time course. The distinction also matters for uncertainty: dependence can affect the information in repeated measurements, so calculations that assume independent errors may give misleading precision.

Common pitfall: “The lag-1 autocorrelation is positive, so the PD errors must be positively correlated.” That conclusion is too strong. A missing trend, an unmodeled delayed response, or another mean-time-course defect can also leave adjacent residuals with similar signs. Inspect the time-ordered residual pattern and model predictions before attributing the pattern to the error process.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What observations form lag-1 pairs for one subject, and why should pairs not cross subject boundaries?
A lag-1 estimate is positive and the residual-versus-time plot shows a smooth rise from negative to positive values. Give two plausible explanations and a diagnostic next step.
For $n_+=4$ and $n_-=6$, calculate $N$ and the expected number of runs under the fixed-count random-ordering null.
Why can an ordinary lag-1 autocorrelation be difficult to interpret when observation intervals are highly irregular?
What information is lost when a runs test is applied to residual signs rather than residual values?

Solutions. (1) Lag-1 pairs are $(e_2,e_1),(e_3,e_2),\ldots,(e_m,e_{m-1})$ within that subject. Crossing a subject boundary would pair measurements that are not consecutive observations from the same longitudinal process. (2) A correlated error process is one possibility; a misspecified mean time course leaving an unmodeled trend is another. Inspect the time-course fit and residual pattern, then reassess dependence after considering justified mean structure. (3) $N=4+6=10$, and $E(R)=1+2(4)(6)/10=1+4.8=5.8$ runs. (4) Lag 1 means one observation step, but the corresponding elapsed time varies, so the estimated association does not refer to a consistent time separation. (5) The runs test ignores residual magnitudes; it cannot distinguish small from large deviations or describe their linear association.