Correlation Estimates at Parameter Bounds

Why a Correlation at a Bound Needs Investigation

A random-effects correlation near $+1$ or $-1$ can look like a strong scientific finding: two individual-level deviations appear to move almost perfectly together. It can also be a warning that the model cannot reliably separate the corresponding variance and covariance components. The numerical estimate alone does not distinguish these explanations.

The distinction matters in a pharmacometric model because the estimated covariance structure affects how individual deviations are represented and can influence other parameter estimates. A correlation at its allowed limit is not automatically a failed fit, nor is it automatically evidence of near-perfect biological dependence. Interpretation requires checking the covariance matrix, the amount of information about both random-effects components, and whether the result is stable under reasonable estimation checks.

Definition: Let the vector of individual random effects be $\eta=(\eta_1,\ldots,\eta_q)^\mathsf{T}$, with mean zero and covariance matrix $\Omega$. The variance of $\eta_j$ is $\omega_j^2$, and the covariance of $\eta_j$ and $\eta_k$ is $\omega_{jk}$. When both variances are positive, their correlation is $$ \rho_{jk}=\frac{\omega_{jk}}{\omega_j\omega_k}, \qquad \omega_{jk}=\rho_{jk}\omega_j\omega_k. $$

The usual random-effects model assumes that $\Omega$ is positive semidefinite: for every vector $a$, $a^\mathsf{T}\Omega a\geq 0$. This is required for $\Omega$ to be a valid covariance matrix. For a pair of random effects, it implies $-1\leq\rho_{jk}\leq1$. The bound is a mathematical constraint, not a recommended target for an estimate.

How the Bound Arises—and Why It Can Be Misleading

For two random effects, the covariance matrix and its determinant are

\Omega= \begin{pmatrix} \omega_1^2 & \omega_{12}\\ \omega_{12} & \omega_2^2 \end{pmatrix}, \qquad \det(\Omega)=\omega_1^2\omega_2^2-\omega_{12}^2.

A positive-semidefinite matrix must have a nonnegative determinant. Substituting $\omega_{12}=\rho_{12}\omega_1\omega_2$, when both standard deviations are positive, gives the bound directly:

\det(\Omega)=\omega_1^2\omega_2^2(1-\rho_{12}^2)\geq0 \quad\Longrightarrow\quad |\rho_{12}|\leq1.

At $|\rho_{12}|=1$, the determinant is zero. The two random effects then have a singular covariance matrix: their deviations are perfectly linearly related under the model. Estimates close to that limit describe a nearly singular matrix, which can be difficult to estimate reliably.

There is a second issue. The covariance depends on the correlation and both standard deviations. If one variance is close to zero, the covariance can be small even when the reported correlation is near its limit. Moreover, the covariance changes only slowly with the correlation when either standard deviation is small:

\frac{\partial\omega_{12}}{\partial\rho_{12}}=\omega_1\omega_2.

Thus, if $\omega_2$ is very small, a substantial change in $\rho_{12}$ can correspond to a small change in covariance. The data may then provide little information to distinguish one correlation value from another. If a variance is exactly zero, the corresponding correlation is undefined by its formula; a software-reported value at that point should not be interpreted as an estimated dependence.

Worked Example: A Near-Bound Estimate With Supporting Information

Suppose a hypothetical population pharmacokinetic fit estimates a clearance random-effect variance of $0.0400$, a volume random-effect variance of $0.0900$, and their correlation as $0.930$. The standard deviations are $0.200$ and $0.300$, respectively. Calculate the covariance and determinant, then consider what the estimate does and does not establish.

Covariance. The units of the two standard deviations are the units of the respective random effects; the covariance has their product as its units. The calculation is

\widehat{\omega}_{12}=0.930(0.200)(0.300)=0.0558.

Determinant. The estimated two-effect covariance matrix has determinant

$$ (0.0400)(0.0900)-(0.0558)^2 =0.003600-0.003114 =0.000486. $$

Equivalently, $0.0400(0.0900)(1-0.930^2)=0.000486$, which is positive. The matrix is therefore not singular, although the correlation is close enough to its upper bound to warrant scrutiny.

Assume, for this hypothetical example, that the estimate remains between $0.91$ and $0.94$ across reasonable starting values, the two variance estimates remain well away from zero, and the fit has no warning signs in its curvature or covariance-step diagnostics. Also assume the estimated standard error for the correlation is $0.04$. Together, these checks support interpreting the fit as showing positive association between the two random effects. They do not establish that the population correlation is exactly $0.930$, or that it is biologically causal. The estimate is still close to a constraint, so its uncertainty and sensitivity should be reported rather than hidden.

Distinguishing Dependence From Instability

The key question is not simply whether the reported value is close to a bound. Ask whether the model has enough information to estimate the two variances and their covariance separately, and whether the evidence for the estimated correlation persists when estimation is checked. The earlier material on Hessian and curvature diagnostics, weak directions, and practical nonidentifiability provides relevant checks; here those checks are applied specifically to a random-effects correlation.

Check that both random-effects variances are estimable.
A correlation is meaningful only when both variances are positive and supported by the data. A near-zero variance can leave its correlation poorly determined even if the optimizer reports a precise-looking number.

Inspect the covariance, not just the correlation.
Calculate $\widehat{\omega}_{jk}=\widehat{\rho}_{jk}\widehat{\omega}_j\widehat{\omega}_k$. A near-bound correlation paired with a negligible covariance and a very small variance has a different practical interpretation from a near-bound correlation with substantial, well-supported variances.

Check parameter information and fit stability.
Review standard errors when available, covariance-step results, curvature diagnostics, and variation across reasonable starting values. A very large or unavailable standard error, a weak Hessian direction, or materially different correlations across fits points to instability rather than a well-resolved dependence estimate.

Check the complete covariance structure.
For three or more random effects, pairwise correlations must be jointly compatible with a positive-semidefinite matrix. Valid pairwise values considered separately do not guarantee a valid overall covariance matrix.

Worked Example: A Near-Bound Correlation With a Weak Component

Consider a hypothetical fit with variances $0.0400$ and $0.000400$, and a reported correlation of $0.990$. Calculate the covariance and determinant. Then suppose alternative reasonable starting values produce correlations of $0.990$, $0.720$, and $-0.880$, while the objective-function values differ by less than $0.05$.

The standard deviations are $\sqrt{0.0400}=0.200$ and $\sqrt{0.000400}=0.0200$. The covariance at the reported estimate is

\widehat{\omega}_{12}=0.990(0.200)(0.0200)=0.00396.

The determinant is

$$ (0.0400)(0.000400)-(0.00396)^2 =0.0000160-0.0000156816 =0.0000003184. $$

This is positive but very small, consistent with a matrix close to singular. More importantly, the second variance is small, and the correlation changes markedly among fits with nearly indistinguishable objective-function values. These are signs that the data do not pin down the correlation well. The value $0.990$ should not be reported as evidence of nearly perfect dependence without qualification. The small variance and instability indicate that the apparent boundary estimate may reflect weak information about that component.

The covariance calculation also shows why the correlation alone can mislead: its near-bound value is paired with a covariance of only $0.00396$ in the product units of the effects. That number is not, by itself, proof that the dependence is practically unimportant; its scale and model context matter. It is a prompt to assess the component’s information and the consequences of including it.

Several Correlations Must Form One Valid Matrix

In a model with three random effects, each pairwise correlation must lie between $-1$ and $1$, but that condition is not sufficient. For a correlation matrix with correlations $a=\rho_{12}$, $b=\rho_{13}$, and $c=\rho_{23}$, positive semidefiniteness also requires

\det(R)=1+2abc-a^2-b^2-c^2\geq0.

This constraint matters when several correlations are estimated near their bounds. Even if each estimate is individually allowed, the combined matrix can be nearly singular or invalid. A nearly singular matrix suggests that some combination of random effects has little independent variation under the fitted structure; it can also make estimation numerically delicate.

Worked Example: Checking Three Pairwise Correlations

Suppose a hypothetical three-effect model reports correlations $\rho_{12}=0.980$, $\rho_{13}=0.960$, and $\rho_{23}=0.900$. Each value is individually within the allowed range. Check whether the corresponding correlation matrix is positive semidefinite using its determinant.

Substitution gives

\begin{aligned} \det(R) &=1+2(0.980)(0.960)(0.900) -(0.980)^2-(0.960)^2-(0.900)^2\\ &=1+1.69344-0.9604-0.9216-0.8100\\ &=0.00144. \end{aligned}

The determinant is positive but close to zero, so the matrix is near singular. The three correlations are jointly permissible, but their combination signals little margin from a singular covariance structure. This calculation does not identify which correlation is poorly estimated. That diagnosis still requires checking the associated variances, parameter information, and fit stability.

Diagnostics, Alternatives, and Limits

A supported correlation near a bound should be reproducible: reasonable changes to starting values or estimation settings should lead to similar estimates and comparable fit diagnostics. Its associated variances should also be estimable, and the covariance-step and curvature results should not indicate a poorly determined direction. Even with these checks, a near-bound estimate remains sensitive to the model’s assumptions and should be described as an estimated association among random effects, not as proof of a mechanism.

Warning signs include correlations that change substantially across starts while objective-function values barely change; very small associated variances; large, unavailable, or implausible uncertainty estimates; covariance-step failure; and weak curvature in a direction involving the covariance parameters. A covariance matrix that is invalid or nearly singular is another warning. These clues are cumulative: any one deserves attention, but none alone proves the scientific absence of dependence.

A common alternative is a diagonal random-effects covariance matrix, which sets the correlations to zero. This can be useful as a simpler comparison or as a deliberate modeling choice when the data do not support a more complex structure. It is not a diagnostic cure: a diagonal structure assumes no correlations rather than showing that the full-model correlations were spurious. Conversely, retaining a full covariance structure solely because the optimizer returned a value is not evidence that every correlation is informed by the data.

The evidence has limits. Estimated standard errors based on local curvature can be unreliable near constraints or with weak directions. Multiple successful runs do not guarantee that the model is well identified, and a stable estimate does not guarantee that the assumed random-effects distribution or structural model is correct. Interpret the estimate alongside its covariance, the variance magnitudes, model diagnostics, and the scientific context. Do not treat the numerical proximity to $+1$ or $-1$ as a stand-alone conclusion.

Key takeaway: A near-bound random-effects correlation is interpretable only in the context of its variances, covariance, complete positive-semidefinite matrix, and supporting parameter information. Stable estimation with informative variances is more consistent with supported dependence; a weak component, flat information, or sensitivity across fits points to instability.

Check Your Understanding

For each question, state the reasoning that supports your answer.

A fit estimates two positive random-effects standard deviations of $0.10$ and $0.20$, with correlation $-0.95$. What is the estimated covariance?
Why can a correlation be poorly determined when one associated variance is close to zero, even if the optimizer converges?
A three-effect model reports three individually valid pairwise correlations, but its correlation-matrix determinant is negative. What does this indicate?
Two fits have nearly identical objective-function values, but their estimates of one random-effects correlation are $0.98$ and $0.15$. What should be investigated before interpreting the larger estimate?
Does fitting a diagonal covariance matrix demonstrate that the full model’s near-bound correlation was false? Explain.

Solutions. (1) The covariance is $(-0.95)(0.10)(0.20)=-0.0190$, in the product units of the two random effects. (2) Since $\omega_{12}=\rho_{12}\omega_1\omega_2$, a small standard deviation makes changes in $\rho_{12}$ produce small changes in covariance. The likelihood may therefore contain little information to distinguish correlation values. (3) The estimated correlations do not form a positive-semidefinite correlation matrix, so the combined covariance structure is invalid; pairwise bounds alone are insufficient. (4) Check whether both variances are supported, inspect standard errors and curvature or covariance-step diagnostics, and assess sensitivity to reasonable starting values. Similar objective-function values with very different correlations suggest weak information about the estimate. (5) No. The diagonal model imposes zero correlations. It is a simpler alternative, not proof that a correlation estimated in the full model was false.