Deriving Design Effects from Intraclass Correlation

Why Clustered Observations Carry Overlapping Information

Suppose a survey records outcomes from several people in each clinic, or a study measures students within the same schools. The total number of observations is not necessarily the amount of information those observations would provide if they were independent. People in one clinic or students in one school may resemble each other, so adding another observation within the same group can contribute less new information than adding an observation from a different group.

The design effect gives a simple way to quantify this loss for a particular setting: estimating a grand mean from equally sized clusters with a common within-cluster correlation. It compares the variance of that mean under clustering with the variance it would have under independence, holding the total sample size and marginal outcome variance fixed.

Definition: Suppose there are $K$ clusters, each containing $m$ observations, for a total of $N=Km$ observations. If any two distinct observations in the same cluster have correlation $\rho$, and observations from different clusters are independent, the equal-cluster-size design effect for the grand mean is $$ \mathrm{DEFF}=1+(m-1)\rho. $$ It is the ratio of the variance of the clustered grand mean to the variance of the grand mean under independence, given the same $N$ and marginal variance.

Here $\rho$ is the intraclass correlation: the correlation between two distinct outcomes from the same cluster. The formula uses the same $\rho$ for every within-cluster pair and assumes equal cluster sizes. The number $m$ is the size of each cluster, not the number of clusters. The design effect is a variance ratio: it is not itself a multiplier for the standard error.

Deriving the Design Effect

We derive the result for a grand mean, the average of all $N$ outcomes. State the conditions first: every outcome has variance $\sigma^2$; distinct outcomes within a cluster have covariance $\rho\sigma^2$; distinct clusters are independent; and all $K$ clusters have size $m$. These are the equal-size, common-correlation conditions that make the calculation work.

Write $Y_{ij}$ for the outcome of observation $j$ in cluster $i$, where $i=1,\ldots,K$ and $j=1,\ldots,m$. The grand mean is $\bar{Y}=(Km)^{-1}\sum_i\sum_j Y_{ij}$. The variance of a sum is the sum of all variances and covariances. There are $Km$ variance terms, and within each of the $K$ clusters there are $m(m-1)$ ordered covariance terms. Between-cluster covariances are zero by independence.

\begin{aligned} \operatorname{Var}(\bar{Y}) &=\frac{1}{(Km)^2} \left[K m\sigma^2+K m(m-1)\rho\sigma^2\right]\\ &=\frac{\sigma^2}{Km}\left[1+(m-1)\rho\right]. \end{aligned}

If those same $N=Km$ observations were independent with variance $\sigma^2$, the grand mean would have variance $\sigma^2/N=\sigma^2/(Km)$. Dividing the clustered variance by this independent-observation variance gives the design effect:

\frac{\operatorname{Var}(\bar{Y})} {\sigma^2/N} = \frac{\frac{\sigma^2}{Km}[1+(m-1)\rho]} {\frac{\sigma^2}{Km}} =1+(m-1)\rho.

This derivation also explains why the multiplier depends on both cluster size and correlation. Each observation has $m-1$ other observations in its cluster with which it shares correlation $\rho$. A larger cluster or a stronger positive correlation generally means more overlap in the information.

From Variance Inflation to Standard Error and Effective Sample Size

A design effect of 1.5 means the variance is 1.5 times the independent-observation variance. Since standard error is the square root of variance, the corresponding standard-error multiplier is $\sqrt{1.5}$, not 1.5. Keeping variance and standard error multipliers distinct is essential when describing the size of the effect.

A related summary is the effective sample size, $n_{\mathrm{eff}}$. For this calculation, it means the size of an independent sample with the same marginal variance that would give the same variance for the mean. Set the independent-sample variance $\sigma^2/n_{\mathrm{eff}}$ equal to the clustered variance and solve:

\frac{\sigma^2}{n_{\mathrm{eff}}} =\frac{\sigma^2}{N}\mathrm{DEFF} \quad\Longrightarrow\quad n_{\mathrm{eff}}=\frac{N}{\mathrm{DEFF}} =\frac{Km}{1+(m-1)\rho}.

This effective sample size is a variance-equivalence summary, not a literal count of observations in the data. It does not alter the actual sample size, and it should not be read as a direct statement about the number of clusters. For the grand mean under these assumptions, it expresses how much an independent sample would need to contain to have the same variance.

Worked Example: Calculating the Design Effect

A hypothetical survey records one outcome from each of 8 people in each of 25 clinics. Suppose the within-clinic intraclass correlation is $\rho=0.10$, and the assumptions above are reasonable for the outcome. Find the design effect and the standard-error multiplier relative to treating all observations as independent.

There are $K=25$ clusters of size $m=8$, so $N=25(8)=200$. Substitute $m=8$ and $\rho=0.10$:

\mathrm{DEFF}=1+(m-1)\rho =1+(8-1)(0.10) =1+0.70 =1.70.

Thus the variance of the grand mean is 1.70 times its independent-observation variance. The standard-error multiplier is the square root:

\sqrt{\mathrm{DEFF}}=\sqrt{1.70}=1.304.

Under this model, clustering increases the standard error of the mean by about 30.4% relative to the independent-observation calculation. That is a statement about uncertainty under the specified assumptions; it is not a claim that the estimated mean itself changes.

Worked Example: Finding the Variance-Equivalent Effective Sample Size

Imagine a hypothetical data set with 30 clinics, 10 people per clinic, and an outcome with within-clinic correlation $\rho=0.08$. Find its total sample size, design effect, and effective sample size for the grand mean.

The total number of observations is $N=30(10)=300$. The design effect is

\mathrm{DEFF}=1+(10-1)(0.08) =1+0.72 =1.72.

The effective sample size is therefore

n_{\mathrm{eff}}=\frac{N}{\mathrm{DEFF}} =\frac{300}{1.72} =174.4.

For estimating this grand mean, the clustered sample of 300 has the same variance, under the stated model, as an independent sample of about 174 observations with the same marginal outcome variance. The value 174.4 is a variance-equivalence calculation; the data still contain 300 measured people in 30 clinics.

Worked Example: Translating Variance Inflation into Standard Error

In a hypothetical study, 24 schools contribute 15 students each. Suppose the intraclass correlation for a measured outcome is $\rho=0.12$, and the assumptions for the equal-size grand-mean calculation are met. For illustration, suppose an independence-based calculation gives a standard error of 0.211 outcome units. Find the design effect and the corresponding approximate standard error.

Here $m=15$, so

\mathrm{DEFF}=1+(15-1)(0.12) =1+1.68 =2.68.

The standard error is multiplied by $\sqrt{2.68}=1.637$. Thus the clustered standard error is approximately

SE_{\mathrm{clustered}} \approx 0.211\sqrt{2.68} =0.211(1.637) =0.345 \text{ outcome units}.

The variance is about 2.68 times the independent calculation, while the standard error is about 1.64 times as large. These are different comparisons, both relative to the same independence-based calculation.

Interpreting the Formula Carefully

Several checks help make the design effect intuitive. If $\rho=0$, then $\mathrm{DEFF}=1$: within-cluster membership creates no covariance under this model, so the variance equals the independent-observation variance. If $\rho>0$ and $m>1$, then $\mathrm{DEFF}>1$: positive similarity increases the variance. If $\rho=1$, then $\mathrm{DEFF}=m$, and $n_{\mathrm{eff}}=Km/m=K$. In that limiting case, outcomes within a cluster are perfectly correlated, so each cluster contributes the same outcome repeated $m$ times rather than $m$ independent pieces of information about the mean.

The design effect is specific to the estimand and structure used in its derivation. It describes the variance of the grand mean when all observations enter equally and the equal-size, common-correlation assumptions hold. It is not automatically an exact variance multiplier for every regression coefficient. Regression coefficients can depend on predictor variation, the pattern of predictor values across clusters, and how observations contribute to the estimate. A simple design-effect calculation may sometimes be used as an approximation, but one should not silently treat this mean-based formula as a general correction for regression uncertainty.

Common Pitfall

A common error is to multiply a standard error by the design effect itself. For example, if $\mathrm{DEFF}=1.70$, saying that the standard error is 1.70 times as large confuses a variance ratio with a standard-error ratio. The correct standard-error multiplier is $\sqrt{1.70}=1.304$. Another error is to assume that a large total $N$ guarantees a small variance: with positive within-cluster correlation, observations in the same group partly repeat information, so the variance can be substantially larger than the independence calculation suggests.

Key takeaway: For $K$ equal-sized clusters of size $m$, with common within-cluster correlation $\rho$ and independent clusters, the grand-mean design effect is $1+(m-1)\rho$. It multiplies variance; its square root multiplies standard error; and dividing the total sample size by it gives a variance-equivalent effective sample size.

Check Your Understanding

Use the equal-cluster-size grand-mean model and show the calculation where appropriate.

There are 12 observations in each cluster and an intraclass correlation of 0.05. What is the design effect?
If a design effect is 2.25, by what factor is the standard error multiplied?
A study has 40 clusters of 6 observations each and $\rho=0.10$. Calculate $N$, the design effect, and $n_{\mathrm{eff}}$.
In one sentence, explain what an effective sample size of 150 means when the actual data set contains 240 observations.
Why should the equal-size design-effect formula not automatically be applied as an exact correction to any regression coefficient?