Why a Boundary Variance Deserves Investigation
A random-effect variance estimate of zero is a valid result: it says that the fitted model finds no detectable between-individual variation in that random-effect direction. A near-zero estimate says the fitted variability is small on the model’s scale. Neither result, by itself, proves that the population variance is truly zero or that the model has failed. The same boundary can arise because the data support little heterogeneity, because the study contains little information about that heterogeneity, or because the specified random-effects structure asks the data to estimate too many poorly distinguished quantities.
The diagnostic task is therefore not to “fix” every boundary estimate. It is to examine how much information the observations contain about the random effect, whether the covariance structure is estimable from the design, and whether the boundary conclusion is stable under defensible model checks. As emphasized in the earlier material on convergence messages versus model adequacy, an optimizer’s success message does not establish that every parameter is well supported by the data.
For a simple model, let \(b_i\) be an individual’s random effect, with \(b_i\sim N(0,\tau^2)\), and let observations follow \(y_{ij}=f(t_{ij},\theta,b_i)+\epsilon_{ij}\). Here \(\tau^2\) is the population variance of the random effect; its units are the square of the random effect’s units. When \(b_i\) is a random effect on a log-parameter scale, \(\tau\) is on that log scale and should not be confused with a variance in the original parameter’s units.
A variance estimate near zero should be judged relative to the model scale, residual variability, observation schedule, number of individuals, and the way the random effect changes predicted observations. A numerical value alone is not a useful diagnosis.
How Individual Information Limits Variance Estimation
A random-effect variance is informed by differences among individuals, but those differences must be distinguishable from residual noise and fixed-effect patterns. Repeated observations can help estimate an individual’s response pattern; more individuals provide more information about the population distribution of those patterns. Many records from only a few individuals do not automatically provide strong information about a population variance.
A simple random-intercept model makes this distinction explicit. Suppose \(y_{ij}=\mu+b_i+\epsilon_{ij}\), where \(b_i\sim N(0,\tau^2)\), \(\epsilon_{ij}\sim N(0,\sigma^2)\), and the random effects and residual errors are mutually independent. For \(m\) observations per individual, the individual mean has variance \(\tau^2+\sigma^2/m\). Thus, the observed spread of individual means includes both genuine between-individual variability and sampling noise in each mean.
This expression gives a useful diagnostic intuition: if the observed spread in individual means is no larger than the amount expected from residual noise, the data may provide little support for a positive \(\tau^2\). This is not a general-purpose estimator for nonlinear mixed-effects models; it is a transparent approximation that helps explain why an estimate can land at zero.
Worked Diagnostics
Worked Example: Individual Means Show Little Evidence of a Random Intercept
Suppose a hypothetical study has eight individuals, each measured twice. For illustration, assume the fitted residual variance is \(\sigma^2=0.0900\) concentration-units squared, and the eight individual means are 5.02, 4.94, 5.01, 4.96, 5.04, 4.98, 5.00, and 5.05 concentration units. The sample mean of these eight values is 5.000 concentration units.
The sum of squared deviations from 5.000 is \(0.020^2+(-0.060)^2+0.010^2+(-0.040)^2+0.040^2+(-0.020)^2+0^2+0.050^2=0.0102\). Therefore, the sample variance of the individual means is \(s_{\bar{y}}^2=0.0102/7=0.001457\) concentration-units squared.
Under the stated balanced random-intercept assumptions, the residual contribution to the variance of an individual mean is \(\sigma^2/m=0.0900/2=0.0450\) concentration-units squared. The method-of-moments contrast is therefore \(\hat{\tau}^2_{\mathrm{mom}}=s_{\bar{y}}^2-\sigma^2/m=0.001457-0.0450=-0.04354\) concentration-units squared.
A variance cannot be negative. If this simple contrast is constrained to the permissible range, its estimate is zero. The interpretation is not that the population variance has been proven to be zero. Rather, the observed dispersion of these means is smaller than the residual sampling variation expected under the assumptions, so this small data set supplies little evidence for a positive random-intercept variance. The calculation also warns against treating a raw spread in individual means as direct evidence of heterogeneity.
For a general NLME model, the marginal covariance of an individual’s observations has the form \(V_i=Z_i\Omega Z_i^\mathsf{T}+R_i\) in a local linear representation. Here \(Z_i\) describes how the random effects affect that individual’s predictions, \(\Omega\) is their covariance matrix, and \(R_i\) is the residual covariance. If a column of \(Z_i\) is small or poorly distinguished from other model effects, changes in the corresponding variance may have little observable effect. The variance can then be weakly informed even when the optimizer has converged.
Worked Example: A Random-Slope Variance With a Narrow Observation Range
Consider a hypothetical model with a random slope \(b_{1i}\) for centered time \(x\), where \(b_{1i}\) is measured in log-concentration units per hour. Suppose the fitted slope standard deviation is \(\tau_1=0.20\) log-concentration units per hour, so \(\tau_1^2=0.0400\) (log-concentration units per hour) squared. The study observes each individual only between \(x=-0.5\) and \(x=0.5\) hours.
At the edge of this range, the random-slope contribution to the individual linear predictor has standard deviation \(|x|\tau_1=0.5(0.20)=0.100\) log-concentration units. If the residual standard deviation on the same modeled scale is 0.300, the random-slope signal at the edge is one third of the residual standard deviation. Near \(x=0\), its contribution is smaller still: at \(x=0.1\), it is \(0.1(0.20)=0.0200\) log-concentration units.
This calculation does not establish that the slope variance is unimportant or zero. It identifies a design limitation: the observed time range gives the random slope little leverage, especially near the centered time. A diagnostic next step is to inspect the time coverage and predicted sensitivity across individuals, then determine whether the fitted variance is stable under a simpler, scientifically defensible random-effects structure. If all individuals have essentially the same narrow range, the data may not distinguish a small slope variance from residual variation.
Checking Covariance Structure and Data Support
A boundary variance may appear in a model that asks the data to estimate several random effects at once. The concern is not merely the number of parameters: it is whether the observation schedules and model sensitivities contain distinct information about each direction of variability. For example, if two random effects change predictions in very similar ways over the available times, their separate contributions may be difficult to distinguish. A variance can then be driven toward zero as the fitted covariance structure allocates variability elsewhere.
Inspect the structure in context. Identify which observations are most sensitive to the random effect; check whether those observations exist for a sufficient number of individuals; and compare the estimated variability with residual variation on the same prediction scale. Examine whether the variance remains near zero when fitting a simpler covariance structure that preserves the scientifically important random effects. A variance that is near zero in one complex model but appreciably positive in a simpler, well-supported model is a warning that the estimate depends on structure, not a direct finding of no heterogeneity.
For an individual random effect that contributes little to predictions over the observed design, a boundary estimate may be a reasonable summary of limited data support. If the variance is near zero across reasonable fits, individual-level information is adequate for the relevant predictions, and the fit is otherwise stable, the data may support omitting that random effect for the present analysis. “Support” here is conditional on the model, design, and population being studied; it is not proof that biological variation is absent.
Worked Example: Comparing a Near-Boundary Fit With a Zero-Variance Fit
In a hypothetical NLME analysis, compare a model fixing one random-effect variance at zero with the otherwise corresponding model that estimates it. Assume both use the same likelihood, observations, residual structure, and treatment of all other parameters. The fixed-zero model has maximized log likelihood \(\ell_0=-102.40\). The estimated-variance model has \(\ell_1=-102.31\), with the variance estimate small but strictly positive, \(\hat{\tau}^2=0.0040\) on the model’s squared log-scale units.
The likelihood-ratio statistic is \(2(\ell_1-\ell_0)=2[-102.31-(-102.40)]=2(0.0900)=0.180\). The estimated-variance model has the larger maximized log likelihood, as it must when it includes the fixed-zero model as a special case. The estimate is near, but not exactly at, the boundary; consequently, these two likelihoods need not be identical.
For a single variance component tested at zero, a common large-sample reference distribution under suitable regularity conditions is a 50:50 mixture of a point mass at zero and a \(\chi^2_1\) distribution. Under that approximation, the upper-tail p-value for a positive statistic \(0.180\) is \(\tfrac12P(\chi^2_1\geq0.180)\), approximately \(0.336\). This is not evidence against the zero-variance model at a conventional 0.05 level. It is also not the probability that the null hypothesis is true, nor proof that the variance is exactly zero.
The mixture approximation requires a correctly specified comparison and suitable large-sample conditions; it may be unreliable with few individuals, weak identification, or more complicated boundary constraints. A parametric bootstrap under the zero-variance model can be considered when the reference approximation is doubtful. If the estimated variance were exactly zero and the models differed only by allowing this variance, their maximized likelihoods would be equal and the likelihood-ratio statistic would be zero.
A Practical Diagnostic Sequence
Record whether the estimated variance is exactly zero, a small positive value, or merely displayed as zero after rounding. Check the parameter scale and units; a small variance on a log scale has a different interpretation from a small variance on the original scale.
Ask how many individuals contribute information, how observations vary within each individual, and whether the sampling times or covariates reveal the random-effect direction. Assess the predicted variability over the actual design, not only the numerical variance.
Determine whether other random effects or fixed effects can produce similar changes in predictions. Fit a simpler scientifically defensible structure as a diagnostic comparison, without assuming that simplification is automatically correct.
Compare estimates, likelihoods, and relevant predictions across reasonable specifications. If the boundary persists and prediction or inference is not materially altered, report it transparently; if conclusions change substantially, describe the dependence on model structure and uncertainty.
A likelihood comparison is one tool, not a universal decision rule. Since zero is a boundary value, the usual reference distribution for a one-parameter likelihood-ratio test does not generally apply unchanged. In addition, a small p-value or a positive variance estimate does not establish that the random-effects structure is scientifically adequate. Conversely, a non-significant comparison does not demonstrate that the variance is zero. The evidence should be read alongside design information, uncertainty, and the purpose of the model.
Alternatives, Limitations, and Common Pitfalls
| Approach | What it can show | Main limitation |
|---|---|---|
| Inspect design and predicted variability | Whether the random effect has observable leverage in the sampled individuals | Does not by itself quantify uncertainty in the variance |
| Compare with a model fixing the variance at zero | Whether allowing the component improves the maximized likelihood | Boundary reference distributions and finite-sample behavior need care |
| Fit a simpler covariance structure | Whether the boundary depends on asking the data to estimate an overly rich structure | A simpler model may omit real heterogeneity or change interpretation |
| Use Bayesian posterior summaries | How uncertainty about small variance values depends on likelihood and prior information | Posterior conclusions can be sensitive to the prior, especially with weak data support |
A common mistake is to say, “The estimate is zero, so individuals do not vary,” and delete the random effect without further checks. This confuses a constrained estimate with proof about the population. The correction is to ask whether the design can reveal that variation, whether competing model components obscure it, and whether the boundary is stable under reasonable checks. Another mistake is to treat a rounded near-zero estimate as exactly zero when computing a likelihood comparison. Use the actual fitted value and likelihoods; if the estimate is exactly at zero and the nested models otherwise coincide, the likelihood-ratio statistic is zero.
Boundary diagnoses have limits. The observed data cannot resolve a variance direction that the design barely measures. Comparisons across model structures can also change the meaning of other parameters, so a better likelihood alone is not a complete scientific argument. Report the estimated variance and its scale, the model structure, the nature of the comparison, and any material dependence of conclusions on the structure. Avoid describing a boundary as a convergence failure unless there is separate evidence of an optimization problem.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- In a balanced random-intercept model with \(m=4\), \(\sigma^2=0.16\), and observed variance of individual means \(s_{\bar{y}}^2=0.030\), calculate the simple method-of-moments contrast for \(\tau^2\). What does a constrained zero imply?
- Why can many observations per individual still provide weak information about a population random-effect variance if there are very few individuals?
- A model estimates a variance as a small positive value. Is its maximized log likelihood required to equal that of the corresponding model fixing the variance at zero? Explain.
- Give one reason a random-slope variance may be near zero when the underlying population has slope heterogeneity.
- What is the appropriate interpretation of a non-significant likelihood-ratio comparison of a variance fixed at zero against a model estimating it?
Solutions. (1) The residual contribution is \(0.16/4=0.040\), so the contrast is \(0.030-0.040=-0.010\) in the squared units of the response. Constraining the variance to be nonnegative gives zero; this indicates little support from this contrast for positive variance, not proof of no population heterogeneity. (2) The population variance describes variation across individuals, so the number and diversity of individuals matter; repeated records chiefly improve information about each person’s pattern and do not replace independent individuals. (3) No. If the estimate is strictly positive and the model includes the zero-variance model, its maximized log likelihood can be greater. If the estimate is exactly zero and the models otherwise coincide, the maximized likelihoods are equal. (4) A narrow or poorly placed covariate/time range may make predicted differences from individual slopes small relative to residual variation. (5) The data do not show a sufficiently large likelihood improvement under that comparison and approximation; this does not prove the variance is zero or establish that the model is adequate.