Why Independence Understates Regression Uncertainty

Why the Number of Rows Can Mislead

A regression data set may contain hundreds of rows but much less than hundreds of independent pieces of information. For example, several patients may be treated at the same clinic, or several measurements may come from the same person. Observations from the same group can resemble one another because they share circumstances that are not fully captured by the predictors.

This matters because the usual regression standard error calculations rely on a model for how errors vary across observations. If errors from the same group tend to move together, treating every row as independent can make estimates seem more precise than they are. The fitted regression line may still be reasonable, but the uncertainty attached to its coefficients can be understated.

A Model for Shared Cluster Variation

To make the issue precise, suppose observation \(i\) belongs to cluster \(j\), and consider the regression model

Here \(Y_{ij}\) is the outcome, \(x_{ij}\) is the predictor, and \(\beta_0\) and \(\beta_1\) are the intercept and slope. The term \(U_j\) is a shared cluster effect: it shifts outcomes in cluster \(j\) together. The individual error \(\varepsilon_{ij}\) is the remaining observation-specific variation.

The assumptions imply that the total error \(U_j+\varepsilon_{ij}\) has variance \(\tau^2+\sigma^2\) for each observation. But two observations in the same cluster share \(U_j\), so their errors have covariance \(\tau^2\). The independence assumption used in ordinary regression is therefore violated when \(\tau^2>0\) and a cluster contains multiple observations.

Independence-based regression formulas treat the observations’ errors as separate contributions to uncertainty. A shared cluster effect does not behave that way: seeing another observation from the same cluster gives another outcome, but not another independent realization of the cluster-level conditions.

Worked Example: A Mean From Patients at Several Clinics

How Dependence Enters a Regression Slope

The same reasoning applies to regression coefficients, but the impact depends on how the predictor varies. For a simple regression with an intercept, define \(\bar{x}\) as the mean predictor value and \(S_{xx}=\sum_j\sum_i(x_{ij}-\bar{x})^2\). The ordinary least-squares slope can be written as

Substituting the model shows that the estimation error in the slope is a weighted sum of the cluster effects and individual errors. For each cluster, let \(W_j=\sum_i(x_{ij}-\bar{x})\). Then

This expression also gives a short derivation of the slope variance. Independent cluster effects contribute \(\tau^2\sum_jW_j^2\) to the variance of the numerator. Independent individual errors contribute \(\sigma^2\sum_j\sum_i(x_{ij}-\bar{x})^2=\sigma^2S_{xx}\). Dividing by \(S_{xx}^2\) gives

By contrast, an independence calculation using the marginal error variance \(\tau^2+\sigma^2\) would give \((\tau^2+\sigma^2)/S_{xx}\). That calculation effectively treats each cluster effect as if it were a separate error for every observation. When the predictor is constant within each cluster, the observations in a cluster all contribute to \(W_j\), so the shared effect can substantially increase the slope’s variance relative to this independence calculation.

Dependence Does Not Affect Every Slope in the Same Way

Positive dependence within clusters is an important warning, but it does not mean that every coefficient’s ordinary standard error must be too small by the same amount. The formula for the slope includes \(\sum_jW_j^2\), which depends on the alignment between the predictor and cluster membership. A shared shift matters for a coefficient when its observations contribute weights that add up within clusters.

What Ordinary Regression Standard Errors Assume

In the usual simple linear regression model, the conditional mean is linear in the predictor, the errors have mean zero, and their variance is constant. The conventional standard error for the slope also relies on errors being independent across observations. With independent errors of variance \(\sigma^2\), the slope variance is \(\sigma^2/S_{xx}\), and its standard error is estimated using the residual variation.

When the errors share cluster effects, the observations are still allowed to have the same conditional mean relationship, and the least-squares slope can still be unbiased under the stated mean-zero and predictor-independence assumptions. What fails is the independence-based uncertainty calculation. Consequently, tests and confidence intervals that use the ordinary standard error may have poorer calibration than their stated levels suggest. A small ordinary standard error is not evidence that the observations truly supplied that much independent information.

Common Mistake: Counting Rows as Independent Evidence

A common mistake is to reason, “There are 200 observations, so the regression has 200 independent pieces of information.” This is not justified when, for example, those rows are 20 classrooms with 10 students each and classroom conditions contribute to students’ outcomes. The correction is to identify how the data were collected, ask which observations share possible sources of variation, and consider how the predictor varies relative to those groups.

A second mistake is to assume that clustering changes the fitted coefficient automatically. Dependence primarily changes how uncertainty is calculated; with errors that have mean zero conditional on the predictors, ordinary least squares may still estimate the conditional mean relationship without systematic bias. The coefficient and its standard error answer different questions: the first describes the fitted association, while the second quantifies its sampling variability under a specified error structure.