Independent Cases With Correlated Responses

Why Independent Cases Can Have Correlated Responses

In multivariate regression, each observational case contributes a vector of outcomes. A patient, for example, might have both systolic and diastolic blood pressure recorded. The two measurements on one patient may be associated even when that patient’s errors are independent of the errors for every other patient. These are different kinds of dependence: within-case response dependence concerns outcomes on the same case, while dependence across cases concerns outcomes belonging to different cases.

The standard model combines independent cases with a common covariance matrix for the outcomes within each case. Keeping the two indices—the case and the response—clear makes the model’s assumptions and implications easier to read.

Definition: Let $Y$ be an $n\times p$ response matrix, $X$ a fixed $n\times q$ design matrix, and $B$ a $q\times p$ coefficient matrix. The multivariate linear model is $$ Y=XB+E. $$ Write $y_i$ and $\varepsilon_i$ as $p$-dimensional column vectors containing the responses and errors for case $i$, and let $x_i^T$ be row $i$ of $X$. Then $$ y_i=B^T x_i+\varepsilon_i,\qquad i=1,\ldots,n. $$ The independent-case, common-covariance assumptions are $E[\varepsilon_i\mid X]=0$, $\operatorname{Cov}(\varepsilon_i\mid X)=\Sigma$, and independence of $\varepsilon_1,\ldots,\varepsilon_n$, where $\Sigma$ is a positive-definite $p\times p$ matrix. In the Gaussian version, $E\mid X\sim MN_{n,p}(0,I_n,\Sigma)$, equivalently the error vectors are independent with $\varepsilon_i\mid X\sim N_p(0,\Sigma)$.

Here “independent cases” refers to the error vectors across rows, conditional on the fixed design $X$. It does not mean that the $p$ responses within one row are independent. The diagonal entries $\Sigma_{aa}$ are the error variances for response $a$; the off-diagonal entries $\Sigma_{ab}$ describe error covariances between responses $a$ and $b$ on the same case.

In matrix-normal notation, the row covariance is $I_n$, the $n\times n$ identity matrix, and the column covariance is $\Sigma$. In entrywise form, the assumptions imply $$ \operatorname{Cov}(E_{ia},E_{jb}\mid X)= \begin{cases} \Sigma_{ab}, & i=j,\\ 0, & i\ne j. \end{cases} $$ Thus the response label $a$ and case label $i$ play distinct roles. The mean $B^T x_i$ can vary from case to case, while the error covariance matrix is common across cases.

Reading the Covariance Structure

For the observed response vectors, the conditional mean and covariance are $$ E[y_i\mid X]=B^T x_i,\qquad \operatorname{Cov}(y_i\mid X)=\Sigma. $$ For two different cases $i\ne j$, the independent-case assumption gives $$ \operatorname{Cov}(y_i,y_j\mid X)=0_{p\times p}. $$ The zero matrix here says that every response on case $i$ has zero covariance with every response on case $j$. Under the Gaussian model, the vectors $y_i$ and $y_j$ are independent as well. Without joint Gaussianity, zero covariance alone does not establish independence; in this model independence across cases is an explicit assumption.

Worked Example: Covariance Between Two Responses

Suppose each case has two responses and the within-case error covariance matrix is $$ \Sigma=\begin{pmatrix}4&3\\3&9\end{pmatrix}. $$ The response-specific error standard deviations are $2$ and $3$, so the within-case error correlation is $$ \frac{3}{\sqrt{4}\sqrt{9}}=\frac{3}{6}=0.5. $$ The responses therefore have positively correlated errors on the same case.

For case $i$, let $D_i=\varepsilon_{i1}-\varepsilon_{i2}$. Its variance is $$ \operatorname{Var}(D_i\mid X) = \operatorname{Var}(\varepsilon_{i1}\mid X) +\operatorname{Var}(\varepsilon_{i2}\mid X) -2\operatorname{Cov}(\varepsilon_{i1},\varepsilon_{i2}\mid X) =4+9-2(3)=7. $$ For a different case $j$, independence gives $$ \operatorname{Cov}(\varepsilon_{i1},\varepsilon_{j2}\mid X)=0 \qquad (i\ne j). $$ The same response pair can therefore be correlated within one case and uncorrelated across cases. The correlation $0.5$ does not indicate that the cases are dependent.

Worked Example: Different Means Do Not Imply Dependent Cases

Take two responses, an intercept and a treatment indicator, with $$ B=\begin{pmatrix}10&20\\2&-1\end{pmatrix}, \qquad \Sigma=\begin{pmatrix}4&1.2\\1.2&9\end{pmatrix}. $$ The rows of $X$ are $(1,0)$ for a control case and $(1,1)$ for a treated case. The model gives the control mean vector $$ B^T\begin{pmatrix}1\\0\end{pmatrix} =\begin{pmatrix}10\\20\end{pmatrix} $$ and the treated mean vector $$ B^T\begin{pmatrix}1\\1\end{pmatrix} =\begin{pmatrix}12\\19\end{pmatrix}. $$

Within either case, the response error correlation is $$ \frac{1.2}{\sqrt{4}\sqrt{9}}=\frac{1.2}{6}=0.2. $$ The means differ by treatment, but the error vectors for a control case and a treated case are independent under the model. In particular, the conditional covariance between any response on the control case and any response on the treated case is zero. Different conditional means are not evidence of cross-case dependence: the model allows the mean to depend on $x_i$ while keeping the case errors independent.

Two Useful Consequences of the Model

The entrywise covariance pattern follows directly from the independent-case assumptions. The Gaussian specification adds a further useful characterization: for a jointly Gaussian error matrix, the block-diagonal covariance across cases implies row independence. These statements formalize why within-case covariance and cross-case dependence must be interpreted separately.

Result: Under the independent-case assumptions, for any response indices $a,b$ and case indices $i,j$, $$ \operatorname{Cov}(Y_{ia},Y_{jb}\mid X)=\mathbf{1}\{i=j\}\Sigma_{ab}, $$ where $\mathbf{1}\{i=j\}$ equals $1$ when $i=j$ and $0$ otherwise.

Proof. Since $X$ is fixed, $Y_{ia}=(XB)_{ia}+E_{ia}$, and $(XB)_{ia}$ is a constant conditional on $X$. Adding or subtracting constants does not change covariance, so $$ \operatorname{Cov}(Y_{ia},Y_{jb}\mid X) =\operatorname{Cov}(E_{ia},E_{jb}\mid X). $$ If $i=j$, the two entries belong to the same error vector, whose covariance matrix is $\Sigma$; hence the covariance is $\Sigma_{ab}$. If $i\ne j$, they belong to independent error vectors, so their covariance is zero. Combining the cases proves the formula. $\square$

Result: Suppose the stacked error vector $(\varepsilon_1^T,\ldots,\varepsilon_n^T)^T$ is jointly multivariate normal and its covariance matrix has diagonal blocks $\Sigma$ and zero off-diagonal blocks. Then the error vectors $\varepsilon_1,\ldots,\varepsilon_n$ are mutually independent.

Proof. Let $z=(\varepsilon_1^T,\ldots,\varepsilon_n^T)^T$. Its covariance matrix is block diagonal, with one block $\Sigma$ for each case. Because $\Sigma$ is positive definite, this full covariance matrix is positive definite and has determinant $|\Sigma|^n$. The multivariate normal density at $z$ is therefore $$ (2\pi)^{-np/2}|\Sigma|^{-n/2} \exp\left\{-\frac12\sum_{i=1}^n\varepsilon_i^T\Sigma^{-1}\varepsilon_i\right\}. $$ This expression equals the product over $i$ of the $N_p(0,\Sigma)$ density evaluated at $\varepsilon_i$. A joint density that factors into the marginal densities establishes mutual independence. $\square$

The Gaussian condition matters in the second result. For a general joint distribution, a block-diagonal covariance matrix establishes zero cross-covariances, but not necessarily independence. The standard matrix-normal model includes Gaussianity, so the density factorization applies directly.

Worked Example: Spotting Dependence Across Cases

Consider three cases with two responses per case. Let the within-case covariance be $$ \Sigma=\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}. $$ Under independent cases, the covariance matrix of the six error values, ordered by case, has the block form $$ \begin{pmatrix} \Sigma&0&0\\ 0&\Sigma&0\\ 0&0&\Sigma \end{pmatrix}. $$ The off-diagonal $0.6$ inside each block represents correlated responses within a case; the zero blocks represent no covariance across cases.

Now consider instead the block covariance $$ \begin{pmatrix} \Sigma&0.4\Sigma&0\\ 0.4\Sigma&\Sigma&0.4\Sigma\\ 0&0.4\Sigma&\Sigma \end{pmatrix}. $$ This is a valid positive-definite covariance matrix, but it does not describe independent cases. To verify validity, the case-level matrix $$ R=\begin{pmatrix}1&0.4&0\\0.4&1&0.4\\0&0.4&1\end{pmatrix} $$ has eigenvalues $1,\ 1+0.4\sqrt{2},\ 1-0.4\sqrt{2}$, all positive because $0.4\sqrt{2}<1$. The response covariance $\Sigma$ has eigenvalues $1.6$ and $0.4$, also positive. The displayed block covariance has eigenvalues equal to products of these positive eigenvalues, so it is positive definite. Yet its adjacent off-diagonal blocks are nonzero: for example, $\operatorname{Cov}(\varepsilon_{11},\varepsilon_{21})=0.4$. This is dependence across cases, which the independent-case model excludes.

Common Mistake: Treating All Correlation as the Same

A frequent error is to see a nonzero off-diagonal element in $\Sigma$ and conclude that observational cases are dependent. That conclusion confuses response indices with case indices. An entry $\Sigma_{ab}\ne0$ concerns responses $a$ and $b$ on the same case. Dependence across cases instead appears in a covariance such as $\operatorname{Cov}(\varepsilon_{ia},\varepsilon_{jb}\mid X)$ with $i\ne j$; the independent-case model sets every such covariance to zero.

The reverse mistake is also possible: observing zero covariance across cases does not by itself prove independence unless a suitable distributional condition, such as joint Gaussianity, is in force. In the standard model, independence is assumed directly, and Gaussianity supplies the factorization result above.

Key takeaway: Read the covariance using both indices. The response covariance $\Sigma$ allows outcomes from one case to be correlated; the identity row covariance encodes independent cases. Neither feature implies the other.

Check Your Understanding

Use the case index and response index separately when answering.

In the model $Y=XB+E$, what does a nonzero $\Sigma_{12}$ say about responses 1 and 2?
Under independent cases, what is $\operatorname{Cov}(Y_{i1},Y_{j2}\mid X)$ when $i\ne j$?
Why can two cases have different expected response vectors without being dependent?
What additional distributional condition lets zero cross-case covariance imply independence in the Gaussian result?
In a covariance matrix ordered by cases, which blocks describe within-case response covariance, and which blocks reveal cross-case dependence?

Independent Cases With Correlated Responses

What You'll Learn