Boundary Estimates and Working-Correlation Failures

Why a Boundary Estimate Needs a Diagnosis

In GEE, the working correlation affects the within-cluster covariance used to weight the estimating equations. For cluster \(i\), this covariance is commonly written \(V_i=A_i^{1/2}R_i(\alpha)A_i^{1/2}\), where \(A_i\) contains the marginal variances and \(R_i(\alpha)\) is the working correlation matrix. The inverse of \(V_i\) is needed in the usual GEE calculation. Consequently, a correlation estimate that is singular, or so close to singularity that its inverse is numerically unstable, can disrupt estimation even when the mean model is otherwise well specified.

A boundary estimate is not automatically an error. It may be the constrained estimate closest to the data's residual moments, or it may reflect observed dependence that is genuinely very strong. But a boundary value can also result from incompatible pairwise estimates, sparse information, missing-data patterns, or a poorly behaved iterative update. The practical task is to distinguish these cases rather than treating every extreme estimate as evidence of perfect dependence.

Definition: An estimate is on the admissible boundary when it satisfies the model's correlation constraints but lies on the edge of the allowed parameter space. An estimate is inadmissible if it violates those constraints. A working correlation matrix is singular if it has no inverse, and is positive definite if \(x^\mathsf{T}Rx>0\) for every nonzero vector \(x\). In standard GEE calculations that require \(R^{-1}\), positive definiteness, not merely nonnegative definiteness, is needed.

What the Boundaries Mean

For an exchangeable correlation matrix of cluster size \(m\), every diagonal entry is 1 and every off-diagonal entry is \(\rho\). Its eigenvalues are \(1-\rho\), with multiplicity \(m-1\), and \(1+(m-1)\rho\), with multiplicity 1. Thus the positive-definite range is \(-1/(m-1)<\rho<1\) for \(m>1\). At either endpoint the matrix is positive semidefinite but singular; outside that interval it is not a valid correlation matrix.

For an AR(1) correlation matrix with equally spaced visits, the entries are \(\rho^{|j-k|}\). For a cluster with at least two visits, it is positive definite when \(|\rho|<1\). At \(\rho=1\), all visits are perfectly correlated and the matrix is singular. At \(\rho=-1\), the alternating-sign matrix is positive semidefinite and singular for every cluster size greater than one. More generally, do not assume that a boundary value from one structure has the same meaning for another: check the structure's admissible parameter space and matrix.

An unstructured estimate has a separate issue: each pairwise correlation can be within \([-1,1]\), while the collection of estimates fails to define a positive-definite matrix. Pairwise admissibility is not enough. The matrix as a whole must be checked, including its eigenvalues or a Cholesky factorization.

Worked Example: An Exchangeable Moment Estimate at the Boundary

Suppose a residual-product moment calculation for clusters of size three gives three unconstrained pair contributions, \(1.10\), \(0.90\), and \(1.12\). The pooled exchangeable moment is their average:

\hat{\rho}_{\mathrm{unc}}=\frac{1.10+0.90+1.12}{3} =\frac{3.12}{3}=1.04.

The exchangeable positive-definite range for \(m=3\) is \(-1/2<\rho<1\). Thus \(1.04\) is inadmissible, not a valid correlation estimate. If the estimation procedure imposes the positive-semidefinite closure, the closest allowed upper value is \(\rho=1\). The corresponding matrix is

R(1)= \begin{pmatrix} 1&1&1\\ 1&1&1\\ 1&1&1 \end{pmatrix}.

This matrix has rank 1 and is singular, so it cannot be inverted in the usual GEE weighting step. Simply reporting the constrained value as if it were an ordinary interior estimate hides the computational problem. Also note the distinction between this pooled moment and a Pearson sample correlation: a Pearson correlation is bounded by 1, but a residual-product moment used in an estimating procedure need not be. The value 1.04 signals that the unconstrained moment update must be handled; by itself, it does not show that the underlying responses are perfectly correlated.

Checking a Small Unstructured Matrix

For three visits, write the candidate unstructured correlation matrix as

R= \begin{pmatrix} 1&a&b\\ a&1&c\\ b&c&1 \end{pmatrix}.

Its determinant is \(1+2abc-a^2-b^2-c^2\). A negative determinant rules out positive definiteness. Even when the determinant is positive, check the pairwise minors and, in larger matrices, the full matrix: one scalar check does not replace a positive-definiteness check in general.

Theorem: The three-visit matrix above is positive definite if and only if \(|a|<1\), \(|b|<1\), \(|c|<1\), and \(1+2abc-a^2-b^2-c^2>0\).

Proof. A positive-definite matrix has positive principal minors. In particular, its \(2\)-by-\(2\) principal minors are \(1-a^2\), \(1-b^2\), and \(1-c^2\), and its determinant is positive. This proves necessity. For sufficiency, pivot on the upper-left entry, which is 1. The remaining Schur complement is

S= \begin{pmatrix} 1-a^2&c-ab\\ c-ab&1-b^2 \end{pmatrix}.

The assumptions imply \(1-a^2>0\), and direct expansion gives \(\det(S)=\det(R)>0\). A symmetric \(2\)-by-\(2\) matrix with positive first diagonal entry and positive determinant is positive definite. Therefore \(S\), and hence \(R\), is positive definite. \(\square\)

Worked Example: Valid Pairwise Estimates, Invalid Matrix

Suppose pairwise estimation with different available subsets of visits gives \(a=0.90\), \(b=0.90\), and \(c=-0.90\). Every estimate is individually within \([-1,1]\). But the determinant is

\det(R) =1+2(0.90)(0.90)(-0.90) -\left(0.90^2+0.90^2+(-0.90)^2\right) =1-1.458-2.43=-2.888.

Because the determinant is negative, these three numbers cannot be used together as a valid positive-definite correlation matrix. Different pairwise subsets can produce estimates that are individually plausible but mutually incompatible. The conclusion is not that one of the pairwise correlations must be false; it is that their assembled matrix is unsuitable for direct inversion.

One transparent repair is to shrink the off-diagonal entries toward zero by a common factor \(\lambda\). Taking \(\lambda=0.5\) gives \(a=b=0.45\) and \(c=-0.45\). The determinant becomes

1+2(0.45)(0.45)(-0.45) -\left(0.45^2+0.45^2+(-0.45)^2\right) =1-0.18225-0.6075=0.21025.

All three \(2\)-by-\(2\) principal minors are \(1-0.45^2=0.7975>0\). The theorem therefore confirms that the shrunken matrix is positive definite. This repair makes the matrix usable; it does not prove that the shrunken correlations are the true dependence parameters.

Why Iterative Updates Can Fail

Correlation estimation in GEE often alternates between estimating the mean parameters and updating the working correlation from residual information. An update that is valid when each pair is considered separately may produce an invalid matrix when assembled. Even valid matrices can become nearly singular, and an iteration that crosses a boundary may cause an inverse to fail or produce extreme numerical values. Common warning signs include a non-positive Cholesky pivot, a rapidly growing condition number, large changes in estimates between iterations, or convergence that depends strongly on the starting value.

A useful diagnostic is to check, after every update, whether the proposed parameter is admissible and whether each resulting \(R_i\) is positive definite. If an update is invalid, damping can sometimes keep it in a valid region. But damping is a numerical strategy, not a general guarantee: the damped matrix must still be checked. The following calculation shows both the failure and a checked damped update.

Worked Example: An Unstable Three-Visit Update

Suppose an iteration proposes the unstructured correlations \(a=0.98\), \(b=0.97\), and \(c=-0.96\). All are individually within the correlation bounds, but the determinant is

\begin{aligned} 2abc&=2(0.98)(0.97)(-0.96)=-1.825152,\\ a^2+b^2+c^2&=0.9604+0.9409+0.9216=2.8229,\\ \det(R)&=1-1.825152-2.8229=-3.648052. \end{aligned}

The proposed update is not a valid correlation matrix. Suppose the previous valid iterate was the exchangeable matrix with all off-diagonal entries \(0.20\). A damped update with step fraction \(0.10\) gives

a_{\mathrm{new}}=0.90(0.20)+0.10(0.98)=0.278,\qquad b_{\mathrm{new}}=0.90(0.20)+0.10(0.97)=0.277,\qquad c_{\mathrm{new}}=0.90(0.20)+0.10(-0.96)=0.084.

For this damped matrix, the squared terms sum to \(0.278^2+0.277^2+0.084^2=0.161069\), and the product term is \(2(0.278)(0.277)(0.084)=0.012937008\). Hence

\det(R_{\mathrm{new}}) =1+0.012937008-0.161069 =0.851868008.

The three pairwise minors are \(1-0.278^2=0.922716\), \(1-0.277^2=0.923271\), and \(1-0.084^2=0.992944\), all positive. The three-visit criterion confirms that this particular damped update is positive definite. This does not establish that every damped update will be valid or that the iteration will converge; it demonstrates why validation must follow the update rather than be assumed.

Interior Shrinkage and What It Guarantees

A general matrix fact explains one controlled way to move a valid positive-semidefinite correlation matrix away from singularity. Let \(R\) be symmetric and positive semidefinite with diagonal entries 1, and define \(R_\lambda=(1-\lambda)I+\lambda R\), where \(0\leq\lambda<1\). Then \(R_\lambda\) remains a correlation matrix and is positive definite.

Proof. The diagonal of \(R_\lambda\) is \((1-\lambda)+\lambda=1\), so it remains a correlation matrix if it is positive semidefinite. By the spectral theorem, \(R\) has real eigenvalues \(\mu_j\geq0\). The eigenvalues of \(R_\lambda\) are \(1-\lambda+\lambda\mu_j\), each of which is at least \(1-\lambda>0\). Thus \(R_\lambda\) is positive definite. \(\square\)

This result applies when the starting matrix is already positive semidefinite. It does not say that shrinking an arbitrary invalid matrix by any chosen factor will fix it. For an invalid matrix, check the eigenvalues or use a justified procedure to find a factor that makes the result positive definite. In applied work, also report what constraint or repair was used, since it changes the working weights.

Boundary Estimates: Dependence or Estimation Failure?

A high correlation estimate can reflect strong dependence, but a boundary or failed update alone cannot settle that question. Compare the estimate with the residual data and the estimation process. For example, near-perfect alignment across many well-measured pairs is different evidence from incompatible pairwise estimates computed on small, differing subsets. Similarly, a stable estimate close to a boundary is different from an update that repeatedly jumps outside the admissible set.

Inspect the data support. Check how many clusters and observed pairs contribute to each estimate, and whether missing visits make pairwise subsets differ substantially.
Inspect the whole matrix. Record its minimum eigenvalue or attempt a Cholesky factorization; individual entries do not establish validity.
Inspect the iteration. Track parameter changes and matrix diagnostics across iterations and starting values. A single invalid proposal is evidence of a computational problem, not proof of perfect dependence.
Assess near-singularity. A matrix may technically be positive definite yet have a very small eigenvalue. Its inverse can then be highly sensitive to small changes, making computation unstable.
Use a defensible alternative. Depending on the study, this may mean enforcing an interior constraint, using a simpler working-correlation structure, or using a documented regularization step. Recheck the resulting matrices.

Common pitfall: “Every estimated pairwise correlation lies between \(-1\) and \(1\), so the unstructured matrix must be valid.” This is wrong because pairwise bounds do not ensure joint positive definiteness. Check the matrix itself. Conversely, a constrained boundary estimate is not automatically evidence that the responses are perfectly dependent; investigate residual support, missingness, and iteration stability before interpreting it substantively.

Check Your Understanding

Use the matrix conditions and diagnostic principles from this tutorial.

For an exchangeable working correlation with cluster size \(m=4\), give the positive-definite range for \(\rho\) and identify both boundary values.
Why can three pairwise correlations, each within \([-1,1]\), still fail to define a valid unstructured correlation matrix?
For \(a=0.8\), \(b=0.8\), and \(c=-0.8\), calculate the determinant of the three-visit correlation matrix and determine whether it is positive definite.
What does a failed Cholesky factorization suggest, and what does it not establish about the underlying responses?
Why does shrinking a positive-semidefinite matrix toward the identity by a factor \(\lambda<1\) yield a positive-definite matrix?