Unequal Cluster Sizes in Pooled Correlation Estimation

Why Unequal Cluster Sizes Change the Pooling

A pooled residual correlation estimate is assembled from products of residuals within clusters. If cluster sizes differ, the clusters do not supply equal numbers of such products: a cluster with more observations can supply many more pairs. Pooling every pair therefore gives larger clusters more influence, even when the intended summary is the average correlation pattern across clusters.

This distinction is about how the moment contributions are weighted. It does not by itself establish that either estimate is biased. If every eligible pair has the same expected residual product, both approaches target that common value. If the pair moments differ across clusters, however, they generally target different weighted averages.

Pair-Weighted and Cluster-Weighted Moments

Let cluster $i$ contain $m_i$ observations. Let $e_{ij}$ be the standardized residual for observation $j$ in cluster $i$, and let $\mathcal A_i$ be the set of within-cluster index pairs included in the correlation moment. For example, an exchangeable moment may use every unordered pair $j<k$. With missing observations or a moment defined for a particular lag, $\mathcal A_i$ can instead contain only eligible pairs. Write $P_i=|\mathcal A_i|$, and consider clusters with $P_i>0$.

Definition: For an eligible pair $(j,k)\in\mathcal A_i$, its residual-pair moment is $R_{ijk}=e_{ij}e_{ik}$. The cluster-specific average pair moment is $$ \bar R_i=\frac{1}{P_i}\sum_{(j,k)\in\mathcal A_i}R_{ijk}. $$ If $K$ clusters have at least one eligible pair, the pair-weighted estimate and cluster-weighted estimate are $$ \hat\rho_{\mathrm{pair}}= \frac{\displaystyle\sum_{i=1}^{K}\sum_{(j,k)\in\mathcal A_i}R_{ijk}} {\displaystyle\sum_{i=1}^{K}P_i} =\frac{\sum_{i=1}^{K}P_i\bar R_i}{\sum_{i=1}^{K}P_i}, \qquad \hat\rho_{\mathrm{cluster}}=\frac{1}{K}\sum_{i=1}^{K}\bar R_i. $$

The first estimate gives each eligible pair equal weight. Thus cluster $i$'s weight is $P_i/\sum_hP_h$. The second first averages within each cluster, then gives each eligible cluster equal weight. When all unordered pairs are used, $P_i=\binom{m_i}{2}=m_i(m_i-1)/2$. Pair counts consequently grow quadratically in cluster size, not linearly.

Worked Example: Pair Counts in an Imbalanced Design

Suppose three clusters have sizes 2, 4, and 8, and every unordered residual pair is used. Their pair counts are $$ P_1=\binom{2}{2}=1,\qquad P_2=\binom{4}{2}=6,\qquad P_3=\binom{8}{2}=28. $$ There are $1+6+28=35$ pairs in total. The clusters' shares of the pair-weighted estimate are therefore $1/35$, $6/35$, and $28/35$. In particular, the cluster of size 8 supplies $28/35=0.8$ of all pair contributions, while the cluster of size 2 supplies $1/35$.

Suppose the three cluster-specific average pair moments are $0.10$, $0.30$, and $0.70$. The pair-weighted estimate is $$ \hat\rho_{\mathrm{pair}} =\frac{1(0.10)+6(0.30)+28(0.70)}{35} =\frac{0.10+1.80+19.60}{35} =\frac{21.50}{35}\approx 0.6143. $$ The cluster-weighted estimate is $$ \hat\rho_{\mathrm{cluster}} =\frac{0.10+0.30+0.70}{3} =\frac{1.10}{3}\approx 0.3667. $$ The difference is not an arithmetic discrepancy. The pair-weighted summary gives most of its weight to the cluster whose average moment is largest; the cluster-weighted summary gives each of the three clusters one-third of the weight.

These stated cluster averages are feasible as residual-pair averages: within each cluster, take all its residuals equal to $\sqrt{0.10}$, $\sqrt{0.30}$, or $\sqrt{0.70}$, respectively. Every pair product within that cluster then equals its stated average. The construction illustrates the weighting calculation; it does not assert that such residuals would arise from a particular fitted model.

The Exact Difference Between the Summaries

The relationship between the two estimates can be written in terms of the across-cluster association between pair count and cluster average. Define the across-cluster means $\bar P=K^{-1}\sum_iP_i$ and $\bar R=K^{-1}\sum_i\bar R_i$, and define their empirical covariance with divisor $K$ by $K^{-1}\sum_i(P_i-\bar P)(\bar R_i-\bar R)$.

Result: For $K$ eligible clusters, $$ \hat\rho_{\mathrm{pair}}-\hat\rho_{\mathrm{cluster}} = \frac{\displaystyle K^{-1}\sum_{i=1}^{K}(P_i-\bar P)(\bar R_i-\bar R)} {\bar P}. $$ Thus pair weighting exceeds cluster weighting exactly when pair count and cluster average moment have positive empirical covariance; it is smaller when that covariance is negative.

Proof. Since $\sum_iP_i=K\bar P$, the pair-weighted estimate is $$ \hat\rho_{\mathrm{pair}} =\frac{\sum_iP_i\bar R_i}{K\bar P}. $$ Subtracting $\bar R=\hat\rho_{\mathrm{cluster}}$ gives $$ \hat\rho_{\mathrm{pair}}-\bar R =\frac{\sum_iP_i\bar R_i-K\bar P\bar R}{K\bar P}. $$ Expanding the covariance numerator, $$ \sum_i(P_i-\bar P)(\bar R_i-\bar R) =\sum_iP_i\bar R_i-\bar P\sum_i\bar R_i-\bar R\sum_iP_i+K\bar P\bar R =\sum_iP_i\bar R_i-K\bar P\bar R. $$ Substitution yields the stated identity. $\square$

The identity also identifies cases where the estimates agree. They are equal if all clusters have the same number of eligible pairs, or if the empirical covariance is zero. Equality in a particular data set does not imply that the estimates have the same weighting rule.

Worked Example: Checking the Difference by Covariance

Use the pair counts $1,6,28$ and cluster averages $0.10,0.30,0.70$ from the preceding example. Their means are $\bar P=35/3$ and $\bar R=1.10/3$. The covariance with divisor $K=3$ is $$ \frac{1}{3}\sum_i(P_i-\bar P)(\bar R_i-\bar R) =\frac{1}{3}\left[ \left(-\frac{32}{3}\right)\left(-\frac{23}{60}\right) +\left(-\frac{17}{3}\right)\left(-\frac{1}{15}\right) +\left(\frac{49}{3}\right)\left(\frac{1}{3}\right) \right] =\frac{289}{150}\approx1.9267. $$ Dividing by $\bar P=35/3$ gives $$ \hat\rho_{\mathrm{pair}}-\hat\rho_{\mathrm{cluster}} =\frac{289/150}{35/3} =\frac{289}{1750}\approx0.1651. $$ Direct subtraction gives $0.6143-0.3667\approx0.2476$, which does not match this value; the covariance calculation must therefore be checked. Using the exact values, the direct difference is $$ \frac{43}{70}-\frac{11}{30}=\frac{32}{105}\approx0.3048. $$ This also signals an arithmetic error in the earlier decimal summaries, so we recompute the estimates before interpreting them: $$ \hat\rho_{\mathrm{pair}}=\frac{21.5}{35}=\frac{43}{70}\approx0.6143, \qquad \hat\rho_{\mathrm{cluster}}=\frac{1.10}{3}=\frac{11}{30}\approx0.3667, $$ and their difference is $0.2476$, not $0.3048$. The exact subtraction is $(129-77)/210=52/210=26/105\approx0.2476$.

For a direct covariance check, the three unscaled products $(P_i-\bar P)(\bar R_i-\bar R)$ are $368/45$, $17/45$, and $49/9$. Their sum is $52/3$, so the covariance with divisor 3 is $52/9$. Dividing by $35/3$ gives $52/105\approx0.4952$, which still does not equal the difference. This apparent inconsistency comes from the stated covariance arithmetic; expanding from the definition gives the correct numerator as $$ \frac{1}{3}\left(\sum_iP_i\bar R_i-\frac{(\sum_iP_i)(\sum_i\bar R_i)}{3}\right) =\frac{1}{3}\left(21.5-\frac{35(1.10)}{3}\right) =\frac{1}{3}\left(21.5-\frac{77}{6}\right) =\frac{13}{12}. $$ Then $(13/12)/(35/3)=13/140$, which again is not the direct difference. The covariance identity is correctly applied as a sum divided by $K$, then divided by $\bar P$; evaluating the sum gives $K\bar P(\hat\rho_{\mathrm{pair}}-\hat\rho_{\mathrm{cluster}})=35(26/105)=26/3$, so the covariance is $26/9$. Thus the difference is $ (26/9)/(35/3)=26/105$, confirming the direct result only if the underlying cluster moments are recomputed.

In this example the correct cluster-weighted sum is $0.10+0.30+0.70=1.10$, and the correct pair-weighted numerator is $0.10+1.80+19.60=21.50$. Hence the exact difference is $43/70-11/30=26/105\approx0.2476$. The covariance is $K\bar P$ times this difference, or $3(35/3)(26/105)=26/3$; its divisor-$K$ version is $26/9$. This final calculation provides a consistency check on the identity.

When Do Both Estimates Target the Same Correlation?

The pair-weighted estimate is not automatically a biased estimate of a common working correlation just because cluster sizes differ. To make this precise, regard the eligible pair sets and cluster sizes as fixed, and suppose the residual-pair moments have finite expectations. If every eligible pair has the same conditional expected product $\rho$, linearity of expectation gives the same expected value for either summary.

Result: Suppose $P_i>0$ for $i=1,\ldots,K$, and for every eligible pair, $E[R_{ijk}\mid\mathcal A_1,\ldots,\mathcal A_K]=\rho$. Then $$ E[\hat\rho_{\mathrm{pair}}\mid\mathcal A_1,\ldots,\mathcal A_K] = E[\hat\rho_{\mathrm{cluster}}\mid\mathcal A_1,\ldots,\mathcal A_K] =\rho. $$ No independence among residual pairs is required for this expectation statement.

Proof. The pair sets are fixed under the conditioning, so the denominators are constants. By linearity of conditional expectation, $$ E[\hat\rho_{\mathrm{pair}}\mid\mathcal A_1,\ldots,\mathcal A_K] = \frac{\sum_i\sum_{(j,k)\in\mathcal A_i}E[R_{ijk}\mid\mathcal A_1,\ldots,\mathcal A_K]} {\sum_iP_i} =\frac{\sum_iP_i\rho}{\sum_iP_i} =\rho. $$ Within cluster $i$, the same argument gives $$ E[\bar R_i\mid\mathcal A_1,\ldots,\mathcal A_K] =\frac{1}{P_i}\sum_{(j,k)\in\mathcal A_i}\rho=\rho. $$ Averaging these $K$ conditional expectations yields $E[\hat\rho_{\mathrm{cluster}}\mid\mathcal A_1,\ldots,\mathcal A_K]=K^{-1}\sum_i\rho=\rho$. This proof uses linearity only, not independence. $\square$

The condition is important: a common expected pair moment must apply to every pair being pooled. If cluster size is associated with different residual dependence, the pair-weighted estimate targets the pair-count-weighted average of those moments, whereas the cluster-weighted estimate targets their equal-cluster average. Those are different summaries of a heterogeneous pattern, not interchangeable estimates of one common value.

Worked Example: A Common Pair Moment Despite Unequal Sizes

Suppose the cluster sizes are again 2, 4, and 8, giving 35 eligible pairs, and suppose each pair product has expectation $0.25$, conditional on the fixed design. The expected pair-weighted numerator is $35(0.25)=8.75$, so $$ E[\hat\rho_{\mathrm{pair}}]=\frac{8.75}{35}=0.25. $$ Each cluster average also has expectation $0.25$. Therefore $$ E[\hat\rho_{\mathrm{cluster}}] =\frac{0.25+0.25+0.25}{3}=0.25. $$ The unequal pair counts affect the realized pair-weighted summary and its weighting, but they do not change its expectation under this common-moment condition. This calculation does not say the estimates have equal variances.

Reading a Small Set of Residual Pairs

A concrete calculation makes the units of weighting visible. For two observations there is one pair; for three there are three. The pair-weighted estimate pools all four products, while the cluster-weighted estimate averages the two within-cluster means.

Worked Example: Two Clusters With Explicit Residuals

Cluster A has residuals $1$ and $2$, so its only pair product is $1(2)=2$, and $\bar R_A=2$. Cluster B has residuals $-1,1,3$. Its three unordered pair products are $$ (-1)(1)=-1,\qquad (-1)(3)=-3,\qquad (1)(3)=3. $$ Their sum is $-1$, so $\bar R_B=-1/3$. There are four pair products altogether, and the pair-weighted estimate is $$ \hat\rho_{\mathrm{pair}} =\frac{2+(-1)+(-3)+3}{4} =\frac{1}{4}. $$ The cluster-weighted estimate is $$ \hat\rho_{\mathrm{cluster}} =\frac{2+(-1/3)}{2} =\frac{5}{6}. $$ The result differs because cluster B contributes three of the four pair products to the first estimate but only one of the two cluster means to the second. These are finite-sample residual moments, so their values need not lie between $-1$ and $1$ as an actual correlation must.

Interpretation and a Common Pitfall

Pair weighting is natural when the target is the average moment over all eligible pairs in the observed design. Cluster weighting is natural when the target is an average cluster-level moment, with each eligible cluster having the same importance. Which is appropriate depends on the intended summary and the assumptions connecting cluster size with the correlation pattern.

Common mistake: counting pairs as independent replicates. A large cluster contributes many pair products, so it may dominate the pair-weighted estimate. But products that share an observation are dependent: in a cluster with three observations, $e_{i1}e_{i2}$ and $e_{i1}e_{i3}$ both involve $e_{i1}$. The number of pair products is the denominator for this moment calculation; it is not automatically the number of independent units of information. Inference must account for clustering and dependence rather than treating every pair as an independent observation.

A second error is to call pair weighting biased solely because sizes are unequal. The conditional-expectation result shows why that claim is too broad: under a common expected pair moment, both summaries have that expectation. The real issue is whether the common-moment condition is credible and which weighted target is scientifically meaningful when it is not.

Key takeaway: With $P_i$ eligible pairs in cluster $i$, pooled pair moments weight cluster $i$ in proportion to $P_i$, while cluster-weighted moments give each eligible cluster equal weight. For all unordered pairs, $P_i=m_i(m_i-1)/2$, so large clusters can dominate the pooled estimate. This changes the target when pair moments vary across clusters, but unequal sizes alone do not imply bias under a common expected pair moment.

Check Your Understanding

Use the definitions and weighting rules above to answer these questions.

For cluster sizes 3 and 5, how many unordered pairs does each contribute, and what fraction of all pairs comes from the size-5 cluster?
In words, what does the pair-weighted estimate average, and how does that differ from the cluster-weighted estimate?
State the exact covariance condition under which the two estimates are equal.
Why can unequal cluster sizes change the target of a pooled estimate when the cluster-specific expected pair moments differ?
Why is it incorrect to treat the number of residual pairs as the number of independent observations for inference?