Estimating Unstructured Correlations with Missing Visits

Why Missing Visits Require Pairwise Estimation

An unstructured working correlation assigns a separate correlation to each pair of scheduled visits. When some clusters miss visits, not every cluster contributes information to every pair. The estimate for visits 1 and 2 may therefore use many clusters, while the estimate for visits 1 and 3 uses only a few. Keeping these pair-specific sample sizes visible is essential: an unstructured estimate is assembled from separate pieces of evidence, not from one common sample.

The quantities being combined are standardized residual products. For cluster $i$ and visit $j$, let $Y_{ij}$ be the response, $\hat\mu_{ij}$ its fitted marginal mean, and $\hat\phi$ the estimated dispersion. With variance function $V$, the Pearson residual is

\hat r_{ij}=\frac{Y_{ij}-\hat\mu_{ij}}{\sqrt{\hat\phi V(\hat\mu_{ij})}}.

This tutorial focuses on estimating the pairwise working correlations from these residuals when visits are missing. To describe the moment calculation precisely, first consider the idealized case where the true means and scale are known, so the standardized residual $r_{ij}$ is defined using them. The practical estimates replace these residuals with fitted residuals.

Definition: For scheduled visits $j$ and $k$, let $R_{i,jk}=1$ if cluster $i$ has observed responses at both visits, and $R_{i,jk}=0$ otherwise. Define the available cluster set $\mathcal O_{jk}=\{i:R_{i,jk}=1\}$ and its size $m_{jk}=\sum_i R_{i,jk}$. The pairwise residual-moment estimate is $$ \hat\rho_{jk}=\frac{1}{m_{jk}}\sum_{i\in\mathcal O_{jk}}\hat r_{ij}\hat r_{ik},\qquad m_{jk}>0. $$

The estimate is symmetric: $\hat\rho_{jk}=\hat\rho_{kj}$. The diagonal of the working correlation matrix is set to 1, rather than estimated by averaging squared residuals. If $m_{jk}=0$, the data provide no estimate for that pair. A software routine must flag or handle that absence; it cannot calculate the displayed average.

Calculating Each Pair Separately

Worked Example: Unequal Pair-Specific Sample Sizes

Suppose there are three scheduled visits. For visits 1 and 2, four clusters have both visits observed, and their residual products are $0.4,-0.2,0.1,0.5$. For visits 1 and 3, two clusters contribute products $0.6,-0.2$. For visits 2 and 3, three clusters contribute products $0.3,0.0,0.6$. Calculate the three estimates and record their sample sizes.

For the first pair, $m_{12}=4$, and the product sum is $0.4-0.2+0.1+0.5=0.8$. Thus $\hat\rho_{12}=0.8/4=0.20$.

For the second pair, $m_{13}=2$, and the product sum is $0.6-0.2=0.4$. Thus $\hat\rho_{13}=0.4/2=0.20$.

For the third pair, $m_{23}=3$, and the product sum is $0.3+0.0+0.6=0.9$. Thus $\hat\rho_{23}=0.9/3=0.30$.

Putting the estimates on the off-diagonal and setting the diagonal to 1 gives

\hat R= \begin{pmatrix} 1&0.20&0.20\\ 0.20&1&0.30\\ 0.20&0.30&1 \end{pmatrix}, \qquad (m_{12},m_{13},m_{23})=(4,2,3).

The pair 1–3 estimate rests on fewer clusters than the other two. Reporting only the three point estimates would hide that difference in information.

The formula uses the average residual product among clusters observed at both visits. It does not mean that every cluster must be complete, nor that an unavailable residual should be filled in with zero. A missing visit contributes no product to that pair’s sum and does not change another pair’s denominator.

When Does the Pairwise Average Estimate a Correlation?

The moment interpretation depends on how the observed pairs relate to their residuals. Write $Z_{i,jk}=r_{ij}r_{ik}$, and suppose clusters are independent and identically distributed. For the idealized residuals, assume $E[Z_{i,jk}]=\rho_{jk}$, where $\rho_{jk}$ is the target working residual correlation. If pair availability is independent of $Z_{i,jk}$, then conditioning on being available does not alter that moment.

Theorem: Fix a visit pair $j,k$. Suppose clusters are independent and identically distributed, $E[|Z_{i,jk}|]<\infty$, $P(R_{i,jk}=1)=q_{jk}>0$, and $R_{i,jk}$ is independent of $Z_{i,jk}$. If $E[Z_{i,jk}]=\rho_{jk}$, then the pairwise average based on the true standardized residuals converges in probability to $\rho_{jk}$ as the number of clusters tends to infinity.

Proof. Write $n$ for the number of clusters and express the available-pair estimate as

\tilde\rho_{jk} = \frac{n^{-1}\sum_{i=1}^{n} R_{i,jk}Z_{i,jk}} {n^{-1}\sum_{i=1}^{n} R_{i,jk}},

whenever the denominator is positive. By independence of $R_{i,jk}$ and $Z_{i,jk}$, $E[R_{i,jk}Z_{i,jk}]=q_{jk}\rho_{jk}$, while $E[R_{i,jk}]=q_{jk}$. The iid clusters and finite absolute first moment allow the weak law of large numbers to be applied to both sums. Therefore the numerator converges in probability to $q_{jk}\rho_{jk}$, and the denominator converges in probability to $q_{jk}>0$. The continuous mapping theorem for the ratio gives $\tilde\rho_{jk}\xrightarrow{p}\rho_{jk}$. The denominator is positive with probability tending to 1, since its normalized value converges to a positive limit. $\square$

This result concerns the idealized residual products and a particular availability condition. In practice, the means and scale are estimated, so fitted residuals replace true residuals; their finite-sample products need not have exactly the same expectation. The pair-specific denominator remains $m_{jk}$, and interpretation still depends on the observation process and adequacy of the fitted marginal mean and variance.

Worked Example: Selective Observation Can Change the Target

Let $X$ and $Y$ be independent random variables that each equal $1$ or $-1$ with probability $1/2$. They have mean 0, variance 1, and correlation $E[XY]=0$. Suppose a pair is observed only when $XY=1$.

In the full population, the products $XY$ are $1$ and $-1$ with equal probability, so their mean is $0$, the target correlation. Among observed pairs, however, the observation rule guarantees $XY=1$. Consequently,

E[XY\mid \text{pair observed}]=1,

and the average product among available pairs targets 1, not 0. The issue is not a mistake in the arithmetic: availability depends on the residual product, so the observed pairs do not represent the full pairwise residual moment.

Pairwise Validity Is Not Matrix Validity

Each pairwise estimate is calculated separately. Even if each off-diagonal value lies between $-1$ and 1, the assembled matrix need not be a valid correlation matrix. A correlation matrix must be positive semidefinite: for every vector $a$, $a^\mathsf{T}Ra\geq 0$. For three visits, a necessary condition is that its determinant be nonnegative.

Worked Example: Plausible Pairs, Invalid Three-Visit Matrix

Suppose sparse, pair-specific observation produces estimates $\hat\rho_{12}=0.9$, $\hat\rho_{13}=0.9$, and $\hat\rho_{23}=-0.9$. Each value is individually within the allowable range for a correlation. The assembled matrix is

\hat R= \begin{pmatrix} 1&0.9&0.9\\ 0.9&1&-0.9\\ 0.9&-0.9&1 \end{pmatrix}.

Its determinant is

\det(\hat R) =1+2(0.9)(0.9)(-0.9)-0.9^2-0.9^2-(-0.9)^2 =1-1.458-2.43 =-2.888.

A real symmetric positive semidefinite matrix has nonnegative eigenvalues and hence a nonnegative determinant, which is their product. This negative determinant proves that the assembled matrix is not positive semidefinite. Thus all three pair estimates can be individually plausible while the joint matrix is invalid.

Sparse pair coverage makes this problem easier to encounter because different entries may be supported by different sets of clusters. Pairwise calculation alone imposes no guarantee that all estimates can coexist as correlations among the same multivariate response. If an estimated matrix is intended for use as a working correlation matrix, its joint validity must be checked; counting on each entry to be between $-1$ and 1 is not enough.

Common Mistakes and Practical Checks

Using one denominator for every pair. The denominator is $m_{jk}$, the number of clusters observed at both visits $j$ and $k$, not the total number of clusters and not the count available for another pair.
Treating missing residuals as zero. A cluster missing either visit is excluded from that pair’s product average. Replacing an unobserved residual with zero changes the moment being calculated.
Assuming availability is harmless. If pair observation depends on the residual product, the available-pair average can estimate a selected moment instead of the target moment.
Assuming separate valid-looking entries make a valid matrix. Check the assembled matrix as a whole; elementwise bounds do not imply positive semidefiniteness.

A useful report gives the pairwise estimates alongside their $m_{jk}$ values. This makes thinly supported pairs visible and helps distinguish a numerical estimate from strong evidence about a correlation. In a fixed visit schedule, the number of distinct off-diagonal pairs grows quickly as visits are added, so missing visits can leave some entries especially weakly supported or not estimable at all.

Check Your Understanding

Use the definitions and arguments in this tutorial to answer the following questions.

For a specified visit pair, what belongs in the available cluster set and in the denominator of its residual-moment estimate?
State the availability condition used in the consistency result and explain why it matters.
If no cluster has both visits in a pair, what can and cannot be calculated for that pair?
Why does the pair-specific estimate in the selective-observation example differ from the population correlation?
Why do off-diagonal estimates within $[-1,1]$ not guarantee a valid three-visit correlation matrix?