Why Correlation Identifiability Needs Its Own Check
A model can describe correlated random effects and produce a positive-definite \(\Omega\) matrix, yet the observations may contain little information about the correlation. This is not merely a numerical inconvenience. When changes in the correlation can be offset by changes in random-effect variances or other model parameters, many parameter values explain the data almost equally well. The resulting estimate may depend strongly on starting values, constraints, or small changes in the data.
Identifiability asks whether distinct parameter values imply distinct distributions for the observations under the specified model and sampling design. For a correlated pair of random effects, the issue is whether the observations distinguish their covariance from the effects of their separate variances, fixed effects, and residual error. A successful optimizer run does not establish that they do.
For a positive-definite \(2\times2\) covariance matrix, \(|\rho|<1\). This mathematical constraint, also represented through Cholesky factors, ensures a valid covariance matrix; it does not ensure that the data identify \(\rho\). In an NLME model, random effects are latent, so information about them comes indirectly through the observation model and the timing and type of measurements.
How the Sampling Design Carries Information
Consider two subject-level random effects, \(\eta_{1i}\) and \(\eta_{2i}\), for subject \(i\). The model specifies their variances and covariance through \(\Omega\). At a given observation time, the predicted measurement responds to those effects according to the model. If the measurement responds to both effects in essentially the same way at every sampled time, the data may reveal only a combined effect, not how much variability belongs to each component or how they covary.
By contrast, observations at times or conditions where the two effects have distinguishable influences can provide separate information. Repeated measurements can also reveal whether subjects with a high response associated with one effect tend to have a high or low response associated with the other. This reasoning depends on the model being adequate: misspecified structural equations or residual error can imitate patterns attributed to random effects. Independent subjects and a correctly specified observation and residual model are also essential assumptions when interpreting the likelihood.
Worked Example: One Combined Response Cannot Separate Two Effects
Suppose a simplified model gives one response per subject: \(Y_i=\eta_{1i}+\eta_{2i}+\epsilon_i\), where the two random effects have mean zero, variances \(\omega_{11}\) and \(\omega_{22}\), covariance \(\omega_{12}\), and independent residual error with known variance \(0.01\). Assume subjects are independent and the random effects are bivariate normal. The response variance is \(\operatorname{Var}(Y_i)=\omega_{11}+\omega_{22}+2\omega_{12}+0.01\).
Imagine the data indicate a response variance of \(0.18\). Then the random-effects contribution must be \(0.18-0.01=0.17\). One valid covariance specification is \(\omega_{11}=0.04\), \(\omega_{22}=0.09\), and \(\omega_{12}=0.02\): \(0.04+0.09+2(0.02)=0.17\). Its correlation is \(0.02/\sqrt{0.04(0.09)}=0.333\).
A different valid specification is \(\omega_{11}=0.04\), \(\omega_{22}=0.07\), and \(\omega_{12}=0.03\). It gives the same contribution, \(0.04+0.07+2(0.03)=0.17\), but its correlation is \(0.03/\sqrt{0.04(0.07)}=0.567\). Both covariance matrices are positive definite because \(0.02^2<0.04(0.09)\) and \(0.03^2<0.04(0.07)\).
With this one-response design, the response distribution depends on the three covariance parameters through only their sum \(\omega_{11}+\omega_{22}+2\omega_{12}\). Even with many subjects, the likelihood cannot distinguish these two specifications using that response variance. The example demonstrates structural confounding in the simplified model, not a universal claim that one measurement per subject always prevents correlation estimation.
A useful diagnostic question is therefore parameter-specific: which features of the observation distribution change when \(\rho\) changes, after allowing other parameters to adjust? Information about a variance does not automatically imply information about its associated correlation. Correlation can be especially weakly informed when one random-effect variance is near zero, because \(\omega_{12}\) must then also be small for \(\Omega\) to remain valid, while the ratio defining \(\rho\) can still vary substantially.
Worked Example: Distinguishable Sampling Signatures
Suppose a hypothetical study measures each of many independent subjects twice. Under a simplified model, the first measurement is \(Y_{i1}=\eta_{1i}+\epsilon_{i1}\) and the second is \(Y_{i2}=\eta_{2i}+\epsilon_{i2}\). Assume zero-mean bivariate-normal random effects, independent residual errors, and known residual variances. The design makes the first measurement sensitive only to \(\eta_{1i}\) and the second only to \(\eta_{2i}\).
The covariance of the two measurements is \(\operatorname{Cov}(Y_{i1},Y_{i2})=\operatorname{Cov}(\eta_{1i},\eta_{2i})=\omega_{12}\), because the residual errors are independent of each other and of the random effects. The marginal variances are \(\operatorname{Var}(Y_{i1})=\omega_{11}+\operatorname{Var}(\epsilon_{i1})\) and \(\operatorname{Var}(Y_{i2})=\omega_{22}+\operatorname{Var}(\epsilon_{i2})\).
For example, suppose the population quantities estimated from the paired data are \(0.05\) for the first measurement variance, \(0.10\) for the second, and \(0.03\) for their covariance. If the known residual variances are \(0.01\) and \(0.02\), then \(\omega_{11}=0.05-0.01=0.04\), \(\omega_{22}=0.10-0.02=0.08\), and \(\omega_{12}=0.03\). The implied correlation is \(0.03/\sqrt{0.04(0.08)}=0.530\).
The arithmetic shows how this particular design can separate the covariance components under its assumptions. Real NLME observations are often more complicated: both effects may influence each measurement, residual variances may need estimating, and model parameters may be uncertain. Then the same design argument is a guide to where information may come from, not proof that the fitted correlation will be precise.
Likelihood Geometry and Parameter-Specific Information
The likelihood summarizes how well each parameter setting explains the observed data. If changing \(\rho\) causes a substantial likelihood decrease and that change cannot be offset by adjusting other parameters, the likelihood is curved in the correlation direction. If the likelihood remains nearly unchanged along a path that changes \(\rho\) and other parameters together, it has a flat ridge. Such a ridge signals weak or absent information about the correlation.
The local geometry can be expressed with the information matrix. Write \(\psi\) for all parameters other than \(\rho\), including variances and fixed-effect parameters. The expected information matrix is the expected negative second derivative of the log likelihood; the observed version uses the fitted data. Partition either matrix into blocks for \(\rho\) and \(\psi\):
The diagonal entry \(I_{\rho\rho}\) describes curvature if the other parameters were held fixed. In practice, they are estimated too. The information about \(\rho\) after allowing nuisance parameters to adjust is the Schur complement \(I_{\rho\cdot\psi}=I_{\rho\rho}-I_{\rho\psi}I_{\psi\psi}^{-1}I_{\psi\rho}\), assuming \(I_{\psi\psi}\) is invertible. A small value means that much of the apparent curvature in \(\rho\) can be absorbed by moving the other parameters.
To see why, approximate the negative log likelihood near its maximum by a quadratic. For a proposed change \(\delta\rho\), choose the nuisance-parameter change \(\delta\psi\) that minimizes
Differentiating this quadratic with respect to \(\delta\psi\) gives \(I_{\psi\rho}\delta\rho+I_{\psi\psi}\delta\psi=0\), hence \(\delta\psi=-I_{\psi\psi}^{-1}I_{\psi\rho}\delta\rho\). Substituting this value into the quadratic leaves \(\tfrac{1}{2}I_{\rho\cdot\psi}(\delta\rho)^2\). Thus the Schur complement is the local curvature in \(\rho\) after the other parameters have adapted. If it is near zero, a standard-error calculation based on an ill-conditioned Hessian will be unstable or very large.
Worked Example: Reading a Flat Profile Likelihood
Suppose a hypothetical NLME fit profiles \(\rho\): at each fixed correlation, it re-optimizes all remaining parameters. The fitted values of \(-2\) times the log likelihood are \(99.80\) at \(\rho=0\), \(99.84\) at \(\rho=-0.4\), \(99.84\) at \(\rho=0.4\), \(100.05\) at \(\rho=-0.8\), and \(100.05\) at \(\rho=0.8\). The profile minimum on this grid is \(99.80\).
For each grid point, the increase relative to the minimum is calculated by subtraction. At \(\rho=0.4\), for example, the increase is \(99.84-99.80=0.04\); at \(\rho=0.8\), it is \(100.05-99.80=0.25\). All five grid points are close in likelihood. Under regular interior conditions, a likelihood-ratio cutoff of about \(3.84\) is commonly used for a nominal 95% profile interval for one parameter. Each increase here is far below that cutoff, so the grid does not show a well-localized correlation estimate.
This coarse grid does not establish the endpoints of an interval: the profile must be evaluated more finely, and its behavior near the admissible boundaries checked. It does show why reporting the optimizer's value alone would be misleading. If the profile remains flat, the data have weak practical information about \(\rho\), even if the algorithm returns a single numerical estimate.
Diagnosing Failure and Choosing an Alternative
Useful checks focus on the likelihood and the model's parameterization, not just whether the software converged. Examine the profile likelihood for \(\rho\), the estimated standard error and covariance matrix of the estimates, and whether repeated fits from different starting values reach similar likelihoods and parameter values. A nearly singular Hessian, a large correlation between parameter estimates, or estimates that change sharply with small modeling or data changes are warnings. None alone proves structural nonidentifiability, but together they motivate closer investigation.
Compare the design's sensitivity to each random effect across the actual sampling times. If the effects have similar signatures throughout the observed window, the likelihood may contain little information to separate them. Ask whether residual error, an unestimated variance, or a fixed-effect parameter could explain the same pattern. Profile likelihoods are useful because they directly show the likelihood cost of changing the parameter while re-optimizing the others; a Hessian is a local approximation and can be unreliable for a broad, asymmetric, or boundary-limited profile.
When correlation information is inadequate, reasonable alternatives depend on the scientific question. A diagonal \(\Omega\) fixes the correlation to zero; this is a modeling restriction, not a discovery that the true correlation is zero. A simpler random-effects structure can avoid estimating an unsupported covariance, while new measurements at times with different sensitivities may improve identification. A correlation can also be fixed to a scientifically justified value and sensitivity analyses reported. Merely changing to a Cholesky parameterization ensures a valid covariance matrix, but cannot add information absent from the observations.
- Structural failure: different parameter values imply the same observation distribution under the design. More subjects of the same design may not resolve this.
- Practical weakness: the distribution changes with the parameter, but the available data are not precise enough to distinguish those changes. More informative sampling or more data may help.
- Numerical difficulty: the likelihood may be informative, but optimization or approximation fails to find or represent its shape. Check starts, convergence, and the profile before concluding that the parameter is unidentified.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- In one sentence, distinguish structural from practical identifiability of a random-effect correlation.
- If every observation depends only on \(\eta_1+\eta_2\), which combination of covariance terms determines the random-effects contribution to response variance?
- In the paired-measurement example, why does the cross-measurement covariance equal \(\omega_{12}\)?
- A profile likelihood is nearly flat across a wide range of correlations, but the optimizer reports \(\hat{\rho}=0.25\). What is the appropriate interpretation?
Solutions. (1) Structural identifiability concerns whether distinct correlations imply distinct observation distributions in principle; practical identifiability concerns whether the finite data estimate the correlation precisely enough to be useful. (2) The contribution is \(\omega_{11}+\omega_{22}+2\omega_{12}\), so one response of that form does not separate the components. (3) Each measurement contains a different random effect and independent residual error, so the only cross-measurement covariance is \(\operatorname{Cov}(\eta_1,\eta_2)=\omega_{12}\). (4) The reported value is a numerical optimizer result, not evidence of a precise estimate; the flat profile indicates weak practical information, and uncertainty should be assessed from the profile and reported cautiously.