Dependence and the CLT

Why Dependence Matters

A sample may contain many measurements without containing just as many independent pieces of information. For example, measurements from the same patient may resemble one another, and patients treated at the same clinic may share conditions that make their outcomes similar. If those relationships are ignored, calculations based on independent observations can give the wrong standard error and therefore misleading probabilities or intervals.

The central limit theorem (CLT) is often used to approximate the distribution of a sample mean by a normal distribution. In its familiar independent, identically distributed setting, the observations have a common mean and finite variance, and the standard error of their mean is $\sigma/\sqrt{n}$. Dependence changes the variance calculation underlying that standard error. It may also change which units should be counted as independent when applying a CLT.

Definition: Two random variables $X_i$ and $X_j$ are independent if knowing the value of one does not change the probability distribution of the other. Their covariance is $$ \operatorname{Cov}(X_i,X_j)=E[(X_i-E[X_i])(X_j-E[X_j])]. $$ If both variables have positive standard deviations, their correlation is $$ \operatorname{Corr}(X_i,X_j)=\frac{\operatorname{Cov}(X_i,X_j)}{\operatorname{SD}(X_i)\operatorname{SD}(X_j)}. $$ Positive correlation means that values above their respective means tend to occur together, as do values below them. Negative correlation means that a high value of one tends to accompany a low value of the other.

Variance of a Mean When Observations Are Dependent

For $n$ observations $X_1,\ldots,X_n$, the sample mean is $\bar{X}=(X_1+\cdots+X_n)/n$. The variance of a sum includes both the individual variances and the covariances between pairs. Dividing the sum by $n$ divides its variance by $n^2$, so

\operatorname{Var}(\bar{X}) = \frac{1}{n^2} \left[ \sum_{i=1}^{n}\operatorname{Var}(X_i) + 2\sum_{i<j}\operatorname{Cov}(X_i,X_j) \right].

The factor 2 appears because each distinct pair contributes twice when the variance of a sum is expanded: once as $\operatorname{Cov}(X_i,X_j)$ and once as $\operatorname{Cov}(X_j,X_i)$. If the observations are independent, all those covariances are zero. If they are positively dependent, the covariance terms are positive and the mean usually varies more than the independent-observation formula predicts. Negative covariances can instead reduce its variance.

A particularly useful model has $C$ independent clusters, each with $m$ observations, so $n=Cm$. Suppose each observation has mean $\mu$ and variance $\sigma^2$, and any two observations in the same cluster have correlation $\rho$. Observations in different clusters are independent. The common within-cluster covariance is then $\rho\sigma^2$. Each cluster contains $m(m-1)/2$ distinct pairs, giving

\begin{aligned} \operatorname{Var}(\bar{X}) &=\frac{1}{(Cm)^2}\left[Cm\sigma^2+ 2C\frac{m(m-1)}{2}\rho\sigma^2\right]\\ &=\frac{\sigma^2}{Cm}\left[1+(m-1)\rho\right]. \end{aligned}

The quantity $1+(m-1)\rho$ is the variance inflation factor relative to treating all $Cm$ observations as independent. When $\rho>0$, the standard error is multiplied by the square root of this factor. This calculation depends on the stated equal-size, equal-variance, common-correlation model; it is not a universal correction for every dependent data set.

Worked Example: Two Repeated Readings per Person

Suppose 25 people each provide two readings of the same measurement. In a hypothetical model, every reading has mean 50 units and standard deviation 10 units, and the correlation between a person's two readings is 0.60. Different people are independent. Find the variance and standard error of the mean of all 50 readings, and compare them with the independent-observation calculation.

There are $C=25$ independent people and $m=2$ readings per person. Thus $n=50$, $\sigma^2=10^2=100$, and $\rho=0.60$. Applying the clustered formula,

\operatorname{Var}(\bar{X}) =\frac{100}{50}[1+(2-1)(0.60)] =2(1.60)=3.20\ \text{units}^2.

The standard error is $\sqrt{3.20}=1.789$ units. If the 50 readings were incorrectly treated as independent, the calculation would be $\sqrt{100/50}=1.414$ units. The smaller naïve result misses the similarity between readings from the same person. In this model, the actual standard error is $\sqrt{1.60}=1.265$ times the naïve one.

The CLT at the Cluster Level

Within-cluster dependence does not automatically rule out a CLT. It does mean that the individual observations may not be the right units to treat as independent. With a fixed cluster size $m$, define the average within cluster $c$ as $A_c=(X_{c1}+\cdots+X_{cm})/m$. If the clusters are independent and identically distributed and the cluster averages have finite variance, the ordinary CLT can be applied to the $C$ independent cluster averages as $C$ grows. With equal cluster sizes, the overall mean of all observations is exactly the mean of those cluster averages.

This gives a normal approximation centered at $\mu$, with variance $\operatorname{Var}(A_c)/C$, when the number of independent clusters is sufficiently large for the approximation to be useful. The amount of within-cluster dependence affects $\operatorname{Var}(A_c)$, and hence the standard error. A large total number of observations is not by itself enough: 1,000 readings from a small number of highly similar clusters may offer much less independent information than 1,000 independent observations.

Key result: Under the equal-size cluster model, if clusters are independent and identically distributed and have finite variance, the CLT can be applied to the cluster averages as the number of clusters $C$ grows. The variance of the overall mean is $\operatorname{Var}(A_c)/C$; it is not generally $\sigma^2/(Cm)$.

Worked Example: Clinic Clusters and a Tail Probability

Imagine a hypothetical data set of five patient measurements from each of 20 independent clinics. Each patient measurement has population standard deviation 12 units, and two measurements from the same clinic have correlation 0.25. Suppose the clinic clusters follow the equal-size model. Use a normal approximation to estimate the probability that the overall mean exceeds its population mean by more than 3 units. Compare it with the calculation that incorrectly treats all 100 patients as independent.

Here $C=20$, $m=5$, $n=100$, $\sigma^2=144$, and $\rho=0.25$. The variance inflation factor is $1+(5-1)(0.25)=2$. Therefore,

\operatorname{Var}(\bar{X}) =\frac{144}{100}(2)=2.88\ \text{units}^2, \qquad SE=\sqrt{2.88}=1.697\ \text{units}.

Under the cluster-level normal approximation, the standardized threshold is $z=(3-0)/1.697=1.768$, so the estimated upper-tail probability is approximately $P(Z>1.768)=0.0386$, where $Z$ is standard normal. The independent-observation calculation would use $SE=\sqrt{144/100}=1.20$, giving $z=3/1.20=2.50$ and an upper-tail probability of about 0.00621. The naïve calculation substantially understates the probability of a deviation this large. These probabilities are normal approximations; their accuracy depends in part on whether 20 independent clusters are enough for the cluster-level approximation.

Dependence Does Not Always Increase Variance

Positive correlation is common in clustered and repeated measurements, but dependence alone does not imply that the variance must increase. Negative covariances contribute negative terms to the variance-of-a-mean formula. As a result, an appropriate mean can be more precise than the independent-observation calculation suggests. The direction and size of the effect depend on the covariance pattern, not just on whether observations are dependent.

Worked Example: Negatively Correlated Paired Measurements

Suppose 20 independent pairs of measurements are collected. Each of the 40 measurements has mean 8 units and standard deviation 4 units, and the two measurements within each pair have correlation $-0.40$. Find the variance and standard error of the mean of all 40 measurements, then compare with the independent calculation.

Here $C=20$, $m=2$, $n=40$, $\sigma^2=16$, and $\rho=-0.40$. The variance inflation factor is $1+(2-1)(-0.40)=0.60$, so

\operatorname{Var}(\bar{X}) =\frac{16}{40}(0.60)=0.24\ \text{units}^2, \qquad SE=\sqrt{0.24}=0.490\ \text{units}.

Treating the measurements as independent would give $SE=\sqrt{16/40}=0.632$ units. Here the independent formula overstates the standard error. The pairwise correlation is mathematically permissible for pairs, and the example illustrates why the covariance pattern must be considered rather than assuming that all dependence has the same effect.

Checking the Model Before Using a CLT

A normal approximation should follow an explicit model of how the observations were generated. For clustered data, a useful first check is to identify the units that can reasonably be regarded as independent. Then consider whether the clusters have comparable sizes and whether the assumed within-cluster dependence is plausible. The simple formula above requires equal cluster sizes, common marginal variance, and a common correlation within clusters. If those assumptions do not fit, the formula may not describe the variance correctly.

Identify the independent units.
Determine whether the independent units are individuals, households, clinics, or some other clusters. Repeated observations on one unit are not automatically independent units.

Describe dependence within units.
State which observations may be correlated and why. Use a covariance model appropriate to the setting rather than silently setting all covariances to zero.

Calculate the variance at the correct level.
Include covariance contributions, or work with cluster averages when clusters are independent and equally sized.

Judge the normal approximation.
For a cluster-level CLT, assess the number and distribution of independent clusters. Many observations within only a few clusters do not substitute for many independent clusters.

Common Mistake: Counting Measurements as Independent Replicates

A common mistake is to use the total number of recorded measurements as $n$ in $SE=\sigma/\sqrt{n}$ simply because that is how many rows appear in a data set. That calculation assumes the observations are independent (as well as having the stated common variance). When observations within a cluster are positively correlated, it omits positive covariance terms and understates the standard error. Inferences based on that standard error can then make ordinary sampling variation look unusually small.

The correction is not to discard repeated or clustered measurements. Instead, describe the dependence, calculate the variance that follows from that model, and base any CLT argument on the independent units when appropriate. Dependence is also not proof that a normal approximation fails: it is a reason to check the approximation and standard error at the level justified by the sampling structure.

Key takeaway: The CLT concerns how sums or averages vary under a sampling model. If observations are dependent, covariance affects that variation. Count independent clusters—not merely recorded measurements—when deciding what the CLT and its standard error can support.

Check Your Understanding

Try these questions without looking back.

Why does the variance of a sum contain covariance terms, and what happens to those terms when the observations are independent?
There are 12 independent clusters of four observations each, with within-cluster correlation 0.30. What is the variance inflation factor under the equal-size cluster model?
Explain why 800 repeated readings from a small number of people may not justify treating $n=800$ as the number of independent observations for a CLT.
Can dependence reduce the variance of a mean? Explain using the sign of covariance terms.
State the assumptions needed to use the simple variance inflation formula for equally sized clusters.

What You'll Learn