Why Random-Effect Variances Need Informative Designs
A population design can be informative about a typical response while revealing little about how much subjects differ from one another. A random-effect variance is learned through the variation it induces in the observed data. Whether that variation can be detected depends on how strongly each subject-level response changes with the relevant random effect, and on how that induced variation compares with residual variability.
The earlier material on the Gaussian approximation to the subject-level likelihood represents a subject's repeated observations by a multivariate normal distribution. This tutorial isolates the covariance contribution to information about random-effect variances in that approximation. It treats residual covariance as specified, rather than deriving information about residual-error parameters, and focuses on variances rather than random-effect correlations.
Suppose subject \(i\) has \(n_i\) observations and a vector of random effects \(\eta_i\sim N(0,\Omega)\). Under a first-order Taylor approximation of the individual response around zero random effects, write \(J_i\) for the \(n_i\)-by-\(q\) sensitivity matrix: its \(k\)th column is the derivative of the subject's predicted observations with respect to random effect \(k\). If the random effects are independent, \(\Omega=\operatorname{diag}(\omega_1,\ldots,\omega_q)\), where each \(\omega_k\) is a variance. With residual covariance \(R_i\), the approximate observation covariance is $$ \Sigma_i=J_i\Omega J_i^T+R_i. $$
This expression assumes the local linearization is adequate for the design and parameter values being considered, the random effects have mean zero and the stated covariance, and \(R_i\) is positive definite. The Gaussian Fisher formula also presumes differentiable model quantities and a nonsingular \(\Sigma_i\). These are approximations and regularity conditions, not guarantees that the underlying nonlinear mixed-effects likelihood is exactly Gaussian.
Covariance Derivatives and the Variance-Information Formula
Let \(j_{ik}\) denote column \(k\) of \(J_i\). Since changing \(\omega_k\) changes only the \(k\)th diagonal element of \(\Omega\), differentiation gives $$ \frac{\partial\Sigma_i}{\partial\omega_k} =J_i\frac{\partial\Omega}{\partial\omega_k}J_i^T =j_{ik}j_{ik}^T. $$
Thus the derivative is an outer product of the subject-level sensitivity vector with itself. This rank-one form explains the role of sensitivity directly: if \(j_{ik}=0\), the observations do not change to first order with random effect \(k\), and the approximate covariance contains no information about its variance. A large sensitivity can create a larger covariance change, but the resulting information also depends on the total covariance \(\Sigma_i\), including residual noise and the effects of other random effects.
If the approximate mean \(m_i\) does not depend on \(\omega_k\), its mean-derivative term is zero. In the first-order approximation around zero random effects, the mean is often the response at zero random effects and hence does not depend on the variances. The covariance contribution for two variance parameters is then $$ \mathcal I_{i,\omega_k\omega_l} =\frac{1}{2}\operatorname{tr} \left( \Sigma_i^{-1}j_{ik}j_{ik}^T \Sigma_i^{-1}j_{il}j_{il}^T \right) =\frac{1}{2} \left(j_{ik}^T\Sigma_i^{-1}j_{il}\right)^2. $$
The last equality follows from the trace identity \(\operatorname{tr}(uv^T)=v^Tu\). In particular, the information for one variance is $$ \mathcal I_{i,\omega_k\omega_k} =\frac{1}{2}\left(j_{ik}^T\Sigma_i^{-1}j_{ik}\right)^2. $$ For independent subjects, expected information adds across subjects: \(\mathcal I_{\omega_k\omega_l}=\sum_i\mathcal I_{i,\omega_k\omega_l}\). The formula is local: it uses the covariance and sensitivities at the parameter values and design under evaluation.
Worked Example: One Random-Effect Variance With Two Observations
Consider a hypothetical subject with two observations and one random effect. The sensitivity vector is \(j=(2,1)^T\), the random-effect variance is \(\omega=0.25\), and residual covariance is \(R=0.04I\). Assume the first-order Gaussian model and known residual covariance. Find the approximate information about \(\omega\).
The covariance and its derivative are $$ \Sigma=0.25 \begin{pmatrix}4&2\\2&1\end{pmatrix} +0.04\begin{pmatrix}1&0\\0&1\end{pmatrix} = \begin{pmatrix}1.04&0.50\\0.50&0.29\end{pmatrix}, \qquad \frac{\partial\Sigma}{\partial\omega} =jj^T= \begin{pmatrix}4&2\\2&1\end{pmatrix}. $$
The determinant of \(\Sigma\) is \(1.04(0.29)-(0.50)^2=0.0516\), so $$ \Sigma^{-1} =\frac{1}{0.0516} \begin{pmatrix}0.29&-0.50\\-0.50&1.04\end{pmatrix}. $$
Using the rank-one formula, $$ j^T\Sigma^{-1}j =\frac{1}{0.0516}\left(4(0.29)-4(0.50)+1.04\right) =\frac{0.20}{0.0516} \approx 3.876, \qquad \mathcal I_{\omega\omega} =\frac{1}{2}(3.876)^2\approx 7.51. $$
The approximate information is \(7.51\) inverse variance-squared units (if the random effect is dimensionless, this is inverse units of \(\omega^2\)). The substantial sensitivity in both observations makes the variance affect their covariance, while the residual variance limits how clearly that effect can be resolved. This is information for one subject under the specified local model, not a standard error by itself.
How Sampling Design Changes Variance Information
For a single random effect, the scalar quantity \(j_i^T\Sigma_i^{-1}j_i\) measures sensitivity after accounting for the covariance of the observations. The inverse covariance downweights directions that are already noisy or strongly variable. Consequently, counting observations alone does not determine variance information: their sensitivity pattern and covariance matter.
Worked Example: Comparing One and Two Measurements
Compare two hypothetical designs for one subject with one random effect of variance \(\omega=0.50\). Each measurement has residual variance \(0.25\), residual errors are independent, and the sensitivity of each measurement to the random effect is \(1\). Design A takes one measurement; Design B takes two.
For Design A, \(j=(1)\) and \(\Sigma=0.25+0.50=0.75\). Therefore $$ \mathcal I_A=\frac{1}{2}\left(\frac{1}{0.75}\right)^2\approx 0.8889. $$
For Design B, \(j=(1,1)^T\) and $$ \Sigma=0.50 \begin{pmatrix}1&1\\1&1\end{pmatrix} +0.25I = \begin{pmatrix}0.75&0.50\\0.50&0.75\end{pmatrix}. $$
The determinant is \(0.75^2-0.50^2=0.3125\), and $$ \Sigma^{-1}=\frac{1}{0.3125} \begin{pmatrix}0.75&-0.50\\-0.50&0.75\end{pmatrix}, \qquad j^T\Sigma^{-1}j=1.6. $$ Hence \(\mathcal I_B=\frac{1}{2}(1.6)^2=1.28\). The second measurement raises information from \(0.8889\) to \(1.28\), but does not double it: both measurements share the same random effect, so the induced random-effect variation is correlated across them. The comparison holds the random-effect variance, residual variance, and per-measurement sensitivity fixed.
A design with measurements at times where the response is insensitive to a particular random effect may contribute little to that variance, even if it contributes information about fixed effects. Conversely, measurements with complementary sensitivities can reveal variation patterns more clearly. These conclusions concern the local covariance model; practical designs must also assess whether the first-order approximation remains credible over the relevant parameter range.
Several Variances and What the Information Matrix Shows
For multiple independent random effects, the diagonal entries of the variance-information matrix describe information about each variance. The off-diagonal entries describe overlap between the covariance changes generated by distinct variance parameters. A nonzero off-diagonal entry does not mean the random effects themselves are correlated; it means the observed covariance patterns induced by their variances are not orthogonal under the \(\Sigma_i^{-1}\) weighting.
Worked Example: Overlapping Sensitivity Patterns
Consider a hypothetical subject with two observations, residual covariance \(R=I\), two independent random effects, and variances \(\omega_1=\omega_2=0.50\). Let their sensitivity vectors be \(j_1=(1,0)^T\) and \(j_2=(1,1)^T\). The approximate covariance is $$ \Sigma=I+0.50j_1j_1^T+0.50j_2j_2^T =\begin{pmatrix}2&0.50\\0.50&1.50\end{pmatrix}, \qquad \Sigma^{-1}=\frac{1}{2.75}\begin{pmatrix}1.50&-0.50\\-0.50&2.00\end{pmatrix}. $$
The weighted sensitivity products are $$ j_1^T\Sigma^{-1}j_1=\frac{1.50}{2.75}\approx0.5455,\quad j_2^T\Sigma^{-1}j_2=\frac{2.50}{2.75}\approx0.9091,\quad j_1^T\Sigma^{-1}j_2=\frac{1.00}{2.75}\approx0.3636. $$
Applying \(\mathcal I_{\omega_k\omega_l}=\frac{1}{2}(j_k^T\Sigma^{-1}j_l)^2\) gives $$ \mathcal I= \begin{pmatrix} 0.1488&0.0661\\ 0.0661&0.4132 \end{pmatrix}. $$
Both variances affect the first observation, so their covariance changes overlap and the off-diagonal information is positive. The second variance has the larger diagonal entry in this example because its sensitivity extends to both observations. The matrix is not diagonal, so judging precision for each variance from its diagonal alone can be misleading when the other variance is also unknown.
Derivation, Assumptions, and Checks
The covariance part of the information formula can be justified by differentiating the Gaussian log likelihood. For one subject with mean \(m\) and covariance \(\Sigma\), its log likelihood, omitting a constant, is $$ \ell=-\frac{1}{2}\log|\Sigma|-\frac{1}{2}(y-m)^T\Sigma^{-1}(y-m). $$
For a covariance parameter \(\theta_a\) with mean fixed, set \(\Sigma_a=\partial\Sigma/\partial\theta_a\). Differentiating and using \(\partial\log|\Sigma|/\partial\theta_a=\operatorname{tr}(\Sigma^{-1}\Sigma_a)\) and \(\partial\Sigma^{-1}/\partial\theta_a=-\Sigma^{-1}\Sigma_a\Sigma^{-1}\) yields the score $$ \frac{\partial\ell}{\partial\theta_a} =-\frac{1}{2}\operatorname{tr}(\Sigma^{-1}\Sigma_a) +\frac{1}{2}(y-m)^T\Sigma^{-1}\Sigma_a\Sigma^{-1}(y-m). $$
Under the model, \(y-m\sim N(0,\Sigma)\). The expected score is zero. Taking the expected product of two such scores, using the second and fourth moments of a multivariate normal vector, gives $$ \mathcal I_{ab} =\frac{1}{2}\operatorname{tr}(\Sigma^{-1}\Sigma_a\Sigma^{-1}\Sigma_b). $$ This is the covariance term used above. If the mean also depends on the parameters, the Gaussian information additionally contains the mean-derivative term stated in the definition; it must not be discarded unless those derivatives are zero.
For independent subjects, log likelihoods add, so their scores add. Independence and zero expected scores make cross-subject expected score products zero, and total expected information is the sum of subject-level information matrices. In a design with differing subject schedules, calculate the appropriate \(J_i\), \(R_i\), and \(\Sigma_i\) for each schedule before summing.
State the random-effect covariance, sensitivity matrix, residual covariance, and parameter values. Verify that the resulting \(\Sigma_i\) is symmetric positive definite.
For diagonal \(\Omega\), use \(\partial\Sigma_i/\partial\omega_k=j_{ik}j_{ik}^T\). If the model uses a different covariance structure, derive its derivative rather than reusing this expression unchanged.
Evaluate the trace formula or the rank-one expression, include any nonzero mean-derivative term, and sum across independent subjects. Check parameter units and whether the information matrix is well-conditioned.
Diagnostics, Alternatives, and Limitations
Several checks help detect a calculation or design that is going wrong. First, the derivative should have the correct units: \(\Sigma_i\) has units of observation squared, and its derivative with respect to a variance has units of observation squared per variance unit. Second, a numerical finite-difference check can compare \(j_{ik}j_{ik}^T\) with \(\{\Sigma_i(\omega_k+h)-\Sigma_i(\omega_k-h)\}/(2h)\) for a small \(h\) that keeps both variances admissible. Third, the covariance matrix and the resulting information matrix should be symmetric; small numerical discrepancies can occur, but substantial asymmetry usually signals an implementation error.
Very small sensitivities, large residual variability, or nearly indistinguishable covariance patterns can make variance information weak. A variance near zero is a special caution: variance parameters are constrained to be nonnegative, and at the boundary the usual interior, approximately normal large-sample interpretation of standard errors and confidence procedures may fail. A local Fisher-information calculation can still describe curvature under its approximation, but it does not by itself resolve boundary behavior or establish reliable finite-sample uncertainty.
The expected Fisher information is a model-based, local summary. Alternatives include the observed information evaluated at particular data, profile likelihood for a variance, or simulation-based assessment using the nonlinear marginal likelihood. These approaches answer related but not identical questions: expected information averages over possible data under the model, while observed information and profiles use the realized data. They can be useful when the Gaussian approximation is doubtful or when a variance is close to zero, but they do not make an uninformative design informative.
A common mistake is to conclude that doubling the sensitivity doubles the variance information. If the sensitivity vector is multiplied by \(c\), its outer product derivative is multiplied by \(c^2\), but \(\Sigma_i\) changes too because it includes the random-effect contribution. The correct calculation updates both the covariance and its derivative before applying the information formula. Another mistake is to treat a large diagonal information entry as a guarantee of precise estimation; correlated information across unknown variance parameters can make their joint estimation less stable.
Check Your Understanding
Use the local Gaussian covariance model and show the calculation or reasoning for each answer.
- If \(j=(0,0)^T\), what is the covariance derivative with respect to the corresponding random-effect variance, and what is its covariance information?
- For \(j=(1)\), residual variance \(R=1\), and random-effect variance \(\omega=1\), calculate the single-subject covariance information for \(\omega\).
- Why is a nonzero off-diagonal entry in variance-parameter information not evidence that the random effects are correlated?
- Name two conditions or features that can make the approximate variance information unreliable or weak.
Solutions. (1) The derivative is \(jj^T=0\), so the rank-one information is \(\frac{1}{2}(j^T\Sigma^{-1}j)^2=0\). (2) Here \(\Sigma=1+1=2\) and \(\partial\Sigma/\partial\omega=1\), giving \(\mathcal I_{\omega\omega}=\frac{1}{2}(1/2)^2=0.125\), in inverse variance-squared units. (3) The off-diagonal information measures overlap between the covariance patterns induced by two variance parameters; the random effects can still be independent. (4) Examples include a poor first-order approximation, near-zero sensitivity, large residual variability, nearly overlapping covariance patterns, a nearly singular covariance matrix, or a variance at the nonnegative boundary.