Independent Errors Across Observations

Why Independence Across Observations Matters

A regression model describes a response for each observation as a systematic part plus an error. In a simple linear regression, write the response for observation $i$ as $Y_i$, its predictor as $X_i$, and its error as $\varepsilon_i$:

Y_i=\beta_0+\beta_1X_i+\varepsilon_i.

The error is the part of the response not accounted for by the model. Earlier material on zero conditional means and constant conditional variance addresses each error separately: its conditional expectation and its conditional variability given the predictor. Independence across observations is a different question. It asks whether knowing one observation's error provides information about another observation's error.

For example, measurements on different people may be affected by a shared clinic, assessor, or time period. Repeated measurements on one person may share persistent characteristics. Such structures can make errors move together even when the regression's average response is correctly specified for each predictor value.

Definition: For fixed predictor values $x_1,\ldots,x_n$, the errors $\varepsilon_1,\ldots,\varepsilon_n$ are independent across observations if their joint conditional distribution factors into the product of their individual conditional distributions. In particular, for all sets $A_1,\ldots,A_n$, $$ P(\varepsilon_1\in A_1,\ldots,\varepsilon_n\in A_n\mid X_1=x_1,\ldots,X_n=x_n) =\prod_{i=1}^n P(\varepsilon_i\in A_i\mid X_1=x_1,\ldots,X_n=x_n). $$

When predictors are treated as fixed, this is simply independence of the error terms under those fixed predictor values. When predictors are random, the conditional formulation makes the scope explicit: the errors are independent given the full predictor vector. This is not the same as requiring the errors to have a particular distribution. In particular, independence alone does not say that errors are normally distributed.

Independence Is Not the Same as a Zero Conditional Mean

The zero conditional mean condition is an observation-by-observation statement:

E[\varepsilon_i\mid X_i]=0.

It says that, at each predictor value, the error averages to zero. It does not say how errors for different observations relate to each other. Even if the stronger condition $E[\varepsilon_i\mid X_1,\ldots,X_n]=0$ holds for every $i$, the errors may still be dependent. They may, for instance, share a random disturbance that averages to zero across possible samples.

Independence has a useful implication when the relevant conditional second moments are finite: errors have zero conditional covariance. The reverse implication does not generally hold. Errors can have zero covariance and still be dependent, as a later example demonstrates.

Theorem: If $\varepsilon_i$ and $\varepsilon_j$ are conditionally independent given the predictor vector, and their conditional expectations and second moments are finite, then $$ \operatorname{Cov}(\varepsilon_i,\varepsilon_j\mid X_1,\ldots,X_n)=0 \qquad (i\ne j). $$ Zero conditional covariance by itself does not establish conditional independence.

Why Independence Implies Zero Covariance

The justification follows from the definition of conditional covariance. Conditional independence lets the conditional expectation of a product factor into a product of conditional expectations. Subtracting that product leaves zero:

\begin{aligned} \operatorname{Cov}(\varepsilon_i,\varepsilon_j\mid \mathbf X) &=E[\varepsilon_i\varepsilon_j\mid\mathbf X] -E[\varepsilon_i\mid\mathbf X]E[\varepsilon_j\mid\mathbf X]\\ &=E[\varepsilon_i\mid\mathbf X]E[\varepsilon_j\mid\mathbf X] -E[\varepsilon_i\mid\mathbf X]E[\varepsilon_j\mid\mathbf X]\\ &=0, \end{aligned}

Here $\mathbf X=(X_1,\ldots,X_n)$. The factorization in the second line is justified by conditional independence, and finite second moments ensure the covariance is defined. The proof also shows the limitation of the result: knowing that the covariance is zero only gives equality of two expectations. It does not give the full factorization of joint probabilities required for independence.

Worked Examples

Worked Example: Checking Independence for Two Fixed Predictors

A hypothetical model has $\beta_0=3$, $\beta_1=2$, and fixed predictor values $X_1=1$ and $X_2=2$. Suppose $\varepsilon_1$ takes values $-1$ and $1$, each with probability $1/2$, and $\varepsilon_2$ takes values $-2$ and $2$, each with probability $1/2$. The two errors are generated independently. Find the conditional means and variances, and check independence using the event $Y_1=6,\;Y_2=9$.

Method. First obtain each response's possible values from the regression equation. Then compare the joint event probability with the product of the marginal event probabilities. The stated independent generation is the model assumption being checked numerically.

For observation 1, the fitted part is $3+2(1)=5$. Therefore $Y_1=5+\varepsilon_1$, so $Y_1$ is 4 or 6 with probability $1/2$ each. The error mean and variance are

E[\varepsilon_1\mid\mathbf X]=\frac{-1+1}{2}=0,\qquad \operatorname{Var}(\varepsilon_1\mid\mathbf X)=\frac{(-1)^2+1^2}{2}=1.

For observation 2, the fitted part is $3+2(2)=7$. Thus $Y_2=7+\varepsilon_2$, so $Y_2$ is 5 or 9 with probability $1/2$ each. Its error moments are

E[\varepsilon_2\mid\mathbf X]=\frac{-2+2}{2}=0,\qquad \operatorname{Var}(\varepsilon_2\mid\mathbf X)=\frac{(-2)^2+2^2}{2}=4.

The event $Y_1=6$ is the event $\varepsilon_1=1$, with probability $1/2$. The event $Y_2=9$ is $\varepsilon_2=2$, also with probability $1/2$. Independence gives

P(Y_1=6,Y_2=9\mid\mathbf X)=\frac12\cdot\frac12=\frac14.

The product of the marginal probabilities is also $1/4$, as required. The calculation is consistent with independence; the independent generation assumption establishes independence for all events, not just this one pair. Check: the response values 6 and 9 equal their fitted parts 5 and 7 plus the selected errors 1 and 2. The common mistake to avoid is treating equal error means of zero as evidence of independence; here independence is supplied by the joint generation assumption.

Worked Example: Detecting a Shared Cluster Shock

Suppose two hypothetical observations in the same clinic have errors $\varepsilon_i=U+V_i$, for $i=1,2$, where $U,V_1,V_2$ are mutually independent and each is equally likely to be $-1$ or $1$. The predictor values are fixed. Find each error's mean and variance, their covariance, and compare $P(\varepsilon_1=2,\varepsilon_2=2)$ with the product of the marginal probabilities.

Method. Use the shared-shock representation to calculate moments and the relevant joint probability. A shared random component is a warning that observations may not be independent.

Each component has mean zero and variance one. Therefore

E[\varepsilon_i\mid\mathbf X]=E[U]+E[V_i]=0,\qquad \operatorname{Var}(\varepsilon_i\mid\mathbf X)=\operatorname{Var}(U)+\operatorname{Var}(V_i)=1+1=2.

The covariance includes the common $U$, but the observation-specific components $V_1$ and $V_2$ have zero covariance with each other and with $U$:

\operatorname{Cov}(\varepsilon_1,\varepsilon_2\mid\mathbf X) =\operatorname{Cov}(U+V_1,U+V_2)=\operatorname{Var}(U)=1.

For one error to equal 2, both its components must equal 1. Thus $P(\varepsilon_i=2)=1/4$, and the product of the marginal probabilities is $1/16$. For both errors to equal 2, the three independent components $U,V_1,V_2$ must all equal 1, so

P(\varepsilon_1=2,\varepsilon_2=2\mid\mathbf X)=\left(\frac12\right)^3=\frac18\ne\frac1{16}.

The errors are dependent: the joint probability fails to factor, and their covariance is positive. Both errors nevertheless have conditional mean zero. Check: each error takes values $-2,0,2$, with probabilities $1/4,1/2,1/4$, giving mean zero and variance $2$. The common mistake to avoid is concluding that zero conditional means make errors independent; the shared clinic shock affects both observations.

Worked Example: Zero Covariance Does Not Prove Independence

For two observations with fixed predictors, let $U$ take values $-1,0,1$, each with probability $1/3$, and define $\varepsilon_1=U$ and $\varepsilon_2=U^2-2/3$. Calculate both error means, their covariance, and a probability comparison that determines whether they are independent.

Method. Compute the moments directly from the three equally likely values of $U$, then test factorization for a particular pair of events. This guards against mistaking zero covariance for independence.

The first error has mean $E[\varepsilon_1]=(-1+0+1)/3=0$. The second takes value $1/3$ when $U=-1$ or $1$, and value $-2/3$ when $U=0$. Hence

E[\varepsilon_2]=\frac13\left(\frac13\right)+\frac13\left(-\frac23\right)+\frac13\left(\frac13\right)=0.

Since both means are zero, the covariance equals the expected product:

\operatorname{Cov}(\varepsilon_1,\varepsilon_2) =E[U(U^2-2/3)] =E[U^3]-\frac23E[U] =0-\frac23(0)=0.

Now compare the events $\varepsilon_1=0$ and $\varepsilon_2=-2/3$. Both occur exactly when $U=0$. Their individual probabilities are $1/3$, so the product is $1/9$, but their joint probability is $1/3$. Since

P(\varepsilon_1=0,\varepsilon_2=-2/3)=\frac13\ne P(\varepsilon_1=0)P(\varepsilon_2=-2/3)=\frac19,

the errors are dependent despite having zero covariance and zero means. Check: $\varepsilon_2$'s values and probabilities give the mean $2/9-2/9=0$; the probability comparison directly contradicts independence. The common mistake to avoid is using zero covariance as a substitute for the full independence condition.

Repeated and Clustered Observations

Dependence is plausible whenever observations share a source of variation. Repeated measurements on the same person may share a persistent personal effect. Measurements taken in one clinic, school, workplace, or time period may share a group-level disturbance. In these settings, an error for one observation can help predict another error even after conditioning on the predictors in the regression.

The shared-shock model in the second example illustrates why this matters. For $n$ observations in one cluster, suppose $\varepsilon_i=U+V_i$, where $U$ is a common, mean-zero shock with variance 1 and the $V_i$ are mutually independent, mean-zero errors with variance 1, independent of $U$. The average error is

\bar{\varepsilon}=U+\frac1n\sum_{i=1}^n V_i.

The two terms are independent, so the variance of their sum is the sum of their variances. The average of the $V_i$ has variance $1/n$, giving

\operatorname{Var}(\bar{\varepsilon})=1+\frac1n.

If instead all $n$ errors were independent and each had variance 2, the average would have variance $2/n$. The common component prevents the average error's variance from shrinking to zero as $n$ grows: $1+1/n$ approaches 1. This calculation shows why the number of observations alone does not describe how much independent information a clustered sample contains.

Common Mistakes and Exam Checks

When assessing the independence assumption, identify what is being conditioned on and what could connect observations. Ask whether responses are repeated on the same unit, whether observations share a cluster, or whether the data have a time order that could leave neighboring errors related. Then separate the relevant claims:

Zero conditional mean: each error averages to zero at the specified predictor values. This is not a statement of independence across observations.
Zero covariance: a pair of errors has no linear association as measured by covariance. This does not rule out other forms of dependence.
Independence: the joint conditional distribution factors for all relevant events. It is a stronger condition than zero covariance.

A common incorrect argument is: “The errors have conditional mean zero, so their covariance is zero; therefore they are independent.” The first step does not follow: zero means alone do not force zero covariance. Even when separate assumptions establish zero covariance, the conclusion of independence still does not follow. For example, the third worked example has zero means and zero covariance but dependent errors. Independence is an assumption about the joint behavior of observations, not a consequence of the regression line or of a zero average error.

Key takeaway: Independence across observations requires the joint conditional distribution of the errors to factor. It is distinct from zero conditional means, implies zero conditional covariance when moments exist, and is not implied by zero covariance. Repeated or clustered observations deserve particular scrutiny for shared sources of error.

Check Your Understanding

For each problem, identify what the stated information does and does not establish about independence across observations.

For fixed predictors, $\varepsilon_1$ and $\varepsilon_2$ are generated independently, each taking values $-3$ and $3$ with equal probability. Find their means, variances, and $P(\varepsilon_1=3,\varepsilon_2=-3)$. (Answer: both means are 0, both variances are 9, and the joint probability is $1/4$, the product of the marginal probabilities.)
A clinic's two errors are $\varepsilon_1=U+V_1$ and $\varepsilon_2=U+V_2$, where $U,V_1,V_2$ are mutually independent, mean zero, and each has variance 4. Find the covariance between the errors. (Answer: $\operatorname{Cov}(\varepsilon_1,\varepsilon_2)=4$, due to the shared $U$.)
Suppose $E[\varepsilon_1\mid\mathbf X]=E[\varepsilon_2\mid\mathbf X]=0$. Does this establish independence? State the reason. (Answer: No; zero conditional means describe each error separately and do not determine their joint conditional distribution.)
For fixed predictors, let $U$ be equally likely to be $-1,0,1$, and set $\varepsilon_1=U$, $\varepsilon_2=U^2-2/3$. Find $P(\varepsilon_1=0,\varepsilon_2=-2/3)$ and decide whether the errors are independent. (Answer: the joint probability is $1/3$, whereas the product of the marginal probabilities is $1/9$; they are dependent.)
A study records each participant's response on four successive days. The fitted regression has zero conditional mean for each day's error, but all four measurements for a participant may share a persistent unobserved participant effect. Is independence across the four errors guaranteed? (Answer: No; the shared participant effect can make the errors dependent despite their individual zero conditional means.)

What You'll Learn