Residuals in Observation Order for Independence

Why Put Residuals in Observation Order?

Many regression diagnostic plots arrange residuals against fitted values or a predictor. Those plots help assess whether the mean relationship or residual spread changes with the predictors. A different question is whether observations that are close together in the order of data collection tend to have related errors. For example, consecutive days, successive production runs, or repeated measurements on the same process may share conditions that the regression model does not include.

To investigate this question visually, plot each residual against its position in the actual data-collection sequence. The horizontal axis is the observation number or another meaningful collection-order index, and the vertical axis is the residual. Look for extended runs on one side of zero, repeated cycles, gradual shifts, or clusters. These patterns can suggest that the independence assumption deserves further investigation; they do not establish that the errors are dependent.

Definition: For observation \(i\), the residual is \(e_i=y_i-\hat{y}_i\), the observed response minus its fitted value. An observation-order residual plot displays \(e_i\) against the order in which the observations were collected, with a horizontal reference line at zero. The order must reflect the data-collection sequence relevant to the question, not an arbitrary sorting of the rows.

Model Assumptions and What the Plot Can Show

Consider the multiple linear regression model \(Y=X\beta+\varepsilon\), where \(Y\) is the vector of responses, \(X\) is the design matrix, \(\beta\) is the coefficient vector, and \(\varepsilon\) is the vector of errors. For the usual independence assumption, conditional on the predictors, the errors have mean zero and are uncorrelated across observations. In the common constant-variance formulation, \(\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I\). Normality is a separate assumption used for certain distributional results; a plot in observation order does not assess normality.

An orderly pattern in residual signs or levels may indicate that the model has not captured some sequential structure. The cause might be dependence, an omitted predictor that changes over time, a mean relationship that changes over the sequence, or another feature of the data-collection process. The plot is a diagnostic clue, not a way to identify the cause by itself.

It is important to distinguish a pattern in the location of residuals from a pattern in their spread. Many consecutive residuals above zero followed by many below zero is a sign or level pattern. Residuals that are centered around zero throughout but become much more widely scattered later show a possible change in spread. Either can occur without the other. A change in spread is not, by itself, evidence that successive errors are dependent.

Reading Runs, Cycles, and Clusters

A run is a consecutive group of residuals with the same sign. A zero residual, if one occurs, lies exactly on the reference line and has no positive or negative sign; state how it is handled rather than quietly assigning it a sign. For a quick visual description, count the groups of consecutive positive and negative residuals. A small number of long runs can indicate prolonged periods above or below the fitted relationship. A sequence that repeatedly moves above and below zero may suggest a cycle or a systematic alternation.

Run counts are descriptive in this tutorial. They are not, on their own, a formal significance test, and there is no universal cutoff that makes a particular count proof of dependence. Interpretation depends on the sample size, the collection process, the fitted model, and other evidence. In particular, do not treat an alternating sequence as proof of independence: a structured alternation is itself a pattern to investigate.

Worked Example: Counting Sign Runs Correctly

A hypothetical pricing study records eight residuals in the order that eight policy files were reviewed. Their signs, in sequence, are \(+,-,-,+,+,-,-,+\). Describe the sign pattern and count its runs. Do not infer a formal test result from the count.

Method. Read the sequence from left to right and begin a new run each time the sign changes. The first \(+\) starts run 1. The second and third observations are both negative, so together they form run 2. The fourth and fifth are positive, forming run 3. The sixth and seventh are negative, forming run 4. The final positive observation forms run 5.

+\ \big|\ --\ \big|\ ++\ \big|\ --\ \big|\ + \quad\Longrightarrow\quad 5\text{ runs}

There are five runs, not six: each grouped block is one run. The sequence changes sign four times, and the number of runs is one more than the number of sign changes, giving \(4+1=5\). The alternating groups are not all single observations, so calling every observation a separate run would be incorrect.

Interpretation and trap avoided. This sequence contains stretches of like-signed residuals, which could motivate checking whether the review sequence is associated with an omitted feature. The count alone does not show that errors are dependent. A common mistake is to count the observations or sign transitions rather than the consecutive same-sign groups.

Why Residuals Are Not the Errors

The plotted quantities are fitted residuals, not the unobserved errors themselves. This distinction matters because fitting the regression model imposes relationships among residuals. Under ordinary least squares with a full-column-rank design matrix, define the projection matrix \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\). Then \(\hat{Y}=HY\), and the residual vector is \(e=Y-\hat{Y}=(I-H)Y\). Since \(Y=X\beta+\varepsilon\) and \((I-H)X=0\), it follows that \(e=(I-H)\varepsilon\).

If the errors have conditional covariance \(\sigma^2I\), the residual covariance is therefore \(\operatorname{Var}(e\mid X)=\sigma^2(I-H)(I-H)^\mathsf{T}\). The matrix \(H\) is symmetric and idempotent, so \(I-H\) is also symmetric and idempotent. Hence \(\operatorname{Var}(e\mid X)=\sigma^2(I-H)\), which generally is not a diagonal matrix.

\begin{aligned} e&=(I-H)Y=(I-H)(X\beta+\varepsilon)\\ &=(I-H)X\beta+(I-H)\varepsilon=(I-H)\varepsilon,\\ \operatorname{Var}(e\mid X) &=(I-H)\operatorname{Var}(\varepsilon\mid X)(I-H)^\mathsf{T}\\ &=\sigma^2(I-H)(I-H)^\mathsf{T} =\sigma^2(I-H). \end{aligned}

This derivation explains a useful caution: residuals can have relationships induced by fitting even when the model errors are uncorrelated. For example, when the regression includes an intercept, the residuals sum to zero. Thus, a visible feature in residuals should be interpreted in context rather than treated as a direct observation of the error sequence. The plot is still useful for spotting sustained or repeated patterns that may be too structured to ignore.

Worked Example: Recognizing a Repeated Cycle

A hypothetical laboratory fits a regression to account for measured calibration settings. The twelve residuals, in the order samples were processed, are \(1,2,1,-1,-2,-1,1,2,1,-1,-2,-1\). Describe the pattern and calculate the number of sign runs.

Method. Separate the sequence into its first and second six-observation blocks. Each block has signs \(+,+,+,-,-,-\), so within each block the signs make one positive run followed by one negative run. Across all twelve observations, the sign groups are \(+++,---,+++,---\).

\underbrace{(+,+,+)}_{\text{run 1}}, \underbrace{(-,-,-)}_{\text{run 2}}, \underbrace{(+,+,+)}_{\text{run 3}}, \underbrace{(-,-,-)}_{\text{run 4}} \quad\Longrightarrow\quad 4\text{ runs}

The values also repeat in the same six-observation pattern. Each block sums to \(1+2+1-1-2-1=0\), so the full sequence sums to \(0+0=0\); that overall balance does not remove the visible cycle. The repeated upward and downward pattern is a reason to examine the process sequence and consider whether relevant structure is missing from the fitted model.

Interpretation and trap avoided. The repeated cycle is not simply a long stretch of residuals on one side of zero. It is a recurring sequence of positive and negative residuals. Do not conclude that the errors are independent because positive and negative values balance, or label this as changing variance: the magnitudes repeat too, rather than steadily widening or narrowing. The pattern suggests investigation, not a definitive diagnosis.

Separating Location Patterns from Changing Spread

An order plot can display both the sign and the magnitude of residuals. To focus on location, inspect whether residuals stay on one side of zero or display repeated structure. To focus on spread, compare the typical distance from zero across portions of the sequence. A later segment with much larger absolute residuals may indicate a variance change, but isolated large values can also arise by chance. A pattern in magnitude can therefore call for follow-up without establishing either a cause or dependence.

The following example computes a simple descriptive comparison. The average squared residual is a measure of observed residual magnitude for a segment; it is not automatically an unbiased estimate of the error variance for that segment, particularly because the residuals come from a fitted regression.

Worked Example: Large Residuals Late in the Sequence

A hypothetical equipment-monitoring model produces eight residuals in collection order: \(-0.2,0.1,0.3,-0.2,-1.8,2.1,-2.2,1.9\). Compare the first four and last four residuals using their mean squared residuals, and distinguish the pattern from a run of like signs.

Step 1: Compare the first four magnitudes. Their squared residuals are \(0.04,0.01,0.09,0.04\). Their sum is \(0.04+0.01+0.09+0.04=0.18\), and their mean is \(0.18/4=0.045\).

Step 2: Compare the last four magnitudes. Their squared residuals are \(3.24,4.41,4.84,3.61\). Their sum is \(3.24+4.41+4.84+3.61=16.10\), and their mean is \(16.10/4=4.025\).

\frac{\text{mean squared residual, last four}} {\text{mean squared residual, first four}} =\frac{4.025}{0.045}\approx 89.4

The last-four mean squared residual is about \(89.4\) times the first-four value, rounded to one decimal place. The signs are \(-,+,+,-,-,+,-,+\), which form six runs: \(-\), \(++\), \(--\), \(+\), \(-\), \(+\). Thus, the conspicuous feature here is the much larger late-sequence spread, not a long sequence of residuals with one sign.

Interpretation and trap avoided. This magnitude difference is a reason to investigate whether residual spread changes across the collection sequence. It does not by itself prove nonconstant error variance, and it does not demonstrate dependence. Do not confuse alternating or mixed signs with constant variance: signs describe location relative to zero, while squared magnitudes describe distance from zero.

Practical Reading Strategy and Common Traps

A disciplined reading of an observation-order plot helps keep its conclusions appropriately limited. First confirm what the horizontal order represents. If rows were sorted by response, predictor, or file label after collection, their displayed order may have no meaningful connection to the process that generated the observations. If observations are repeated measurements on the same unit, the sequence should preserve that unit’s actual measurement order; observations from different units may require separate consideration.

Verify the order.
Identify the actual collection sequence, such as time, production order, or the order of repeated measurements. An arbitrary row order cannot support a meaningful sequential interpretation.

Locate the zero line.
Check whether residuals fluctuate around zero, remain mostly above or below it for stretches, or show repeated changes in sign.

Inspect magnitude separately.
Compare how far residuals lie from zero across the sequence. A change in spread is a different visual feature from a run in signs.

Describe, then qualify.
State the observed pattern and explain that it may motivate further investigation. Do not claim that a plot alone proves dependence, independence, or a particular cause.

A frequent mistake is to reorder residuals from smallest to largest before making this plot. That destroys the collection sequence and can conceal cycles or clusters. Another is to treat every departure from zero as evidence of dependence; residuals naturally vary, and their covariance is affected by fitting. Finally, a plot with no obvious structure does not certify the independence assumption. It only means that this visual check has not revealed a clear order-related pattern.

Key takeaway: An observation-order residual plot is a visual check for sequential structure. Runs, cycles, or clusters can be warning signs, while a change in vertical spread raises a separate question. Neither feature alone proves a violation of an assumption; interpret the plot alongside the collection process and other diagnostics.

Check Your Understanding

For each problem, use the stated collection order and keep sign patterns distinct from changes in residual magnitude.

A hypothetical sequence of residual signs is \(+,+,-,-,-,+,-\). Count the runs. (Answer: 4 runs: \(++\), \(---\), \(+\), \(-\).)
Six ordered residuals are \(0.4,-0.5,0.2,-0.3,0.6,-0.4\). How many sign runs are present, and what does the sequence show descriptively? (Answer: 6 runs; the signs alternate at every observation, a pattern to investigate rather than proof of independence.)
Twelve residuals have signs \(+,+,+,-,-,-,+,+,+,-,-,-\). Count the runs and identify the broad pattern. (Answer: 4 runs; two repeated positive-then-negative cycles.)
The first four residuals are \(-0.2,0.2,-0.1,0.1\), and the last four are \(-1,1.2,-1.1,0.9\). Calculate the mean squared residual for each segment and their ratio, last segment to first. (Answer: First \(0.025\); last \(1.115\); ratio \(1.115/0.025=44.6\). This indicates much greater observed late-sequence magnitude, not proof of dependence.)
A data set is sorted by predictor value before a residual-order plot is made, although observations were collected over time. Is the resulting plot suitable for assessing time-order patterns? Explain. (Answer: No. Restore the actual collection order; sorting by predictor does not preserve the time sequence needed to inspect time-order patterns.)

What You'll Learn