Boundary Tests for Random-Effect Variances

Why a Zero Variance Needs Special Treatment

A likelihood-ratio test compares a smaller model with a nested larger model. For many nested models, the statistic is compared with a chi-square distribution whose degrees of freedom equal the number of added parameters. Testing a random-effect variance equal to zero is a special case: a variance cannot be negative, so zero is the edge of its allowable parameter space. The usual chi-square result assumes the null parameter is in the interior, and that assumption fails here.

This matters in pharmacometrics because a random-effect variance describes between-subject variability in a model parameter. If its estimate is zero, the fitted model is at the boundary, not at an ordinary interior estimate. For a single tested variance component, the asymptotic likelihood-ratio reference distribution is a mixture: half its probability is at zero, and the other half follows a chi-square distribution with one degree of freedom. This tutorial develops that result and shows how to use it.

Definition: Let \(M_0\) be an NLME model in which one specified random-effect variance is fixed at zero, and let \(M_1\) be the nested model in which that variance is estimated subject to being nonnegative. The likelihood-ratio statistic is \(T=2\{\ell_1-\ell_0\}\), where \(\ell_1\) and \(\ell_0\) are the maximized log likelihoods under \(M_1\) and \(M_0\), respectively. Under the standard one-variance boundary conditions, the null reference distribution is \(\tfrac12\chi^2_0+\tfrac12\chi^2_1\), where \(\chi^2_0\) denotes a point mass at zero.

Conditions for the Mixture Reference Distribution

The mixture is not a universal adjustment for every comparison involving random effects. It applies in the basic case of testing one scalar variance component while the rest of the model is held comparable. In particular, the fixed-effects structure and other variance components are the same in the two models; only the tested variance is added in \(M_1\). The tested variance is constrained to be nonnegative, and its null value is zero.

The usual asymptotic justification also requires a correctly specified, identifiable model apart from the boundary restriction, nuisance parameters that remain identifiable and are in the interior under the null, and enough independent subjects for the likelihood approximation to be reliable. The models must be fitted to the same data using the same likelihood basis, and the reported likelihoods should represent the relevant maxima. A single-variance test is not the same as testing a set of covariance parameters together; the reference distribution for more complicated restrictions need not be this simple mixture.

For this comparison, \(T\) should be nonnegative because the larger model contains the smaller one: the alternative can reproduce the null by setting the tested variance to zero. A noticeably negative computed statistic therefore indicates a fitting, likelihood-reporting, or model-comparability problem, rather than evidence for a negative test statistic.

Why the Null Distribution Is a Mixture

Under an ordinary interior null, a locally scaled estimate of a parameter is approximately normal and can move in either direction. Here, the unconstrained local estimate is still approximately normal, but the fitted variance cannot be negative. Locally, the constrained estimate is therefore the larger of zero and that normal estimate. The likelihood-ratio statistic is approximately the squared constrained estimate on the standardized scale.

Z\sim N(0,1),\qquad T\ \dot{\sim}\ \{\max(0,Z)\}^2

If \(Z\leq 0\), which occurs with probability one half for a standard normal variable, the constrained estimate is zero and so is \(T\). If \(Z>0\), the statistic is \(Z^2\). The sign of a standard normal variable is independent of its squared magnitude, and \(Z^2\) has a \(\chi^2_1\) distribution. Thus the positive part has one-half the probability and follows \(\chi^2_1\); the other half is concentrated at zero.

P(T=0)=\tfrac12,\qquad P(T\geq t)=\tfrac12 P(\chi^2_1\geq t)\quad (t>0)

For an observed positive statistic, the mixture p-value is therefore half the ordinary one-degree-of-freedom chi-square upper-tail probability. At exactly zero, the upper-tail probability including the point mass is one. Because the mixture puts only half its mass in the continuous part, the 5% critical value is the 90th percentile of \(\chi^2_1\), about 2.7055, rather than the 95th percentile, about 3.841. More generally, for a conventional significance level \(\alpha<0.5\), the critical value \(c\) solves \(\tfrac12P(\chi^2_1\geq c)=\alpha\).

Calculating and Interpreting the Test

Worked Example: A Small Statistic Below the 5% Threshold

Suppose a hypothetical analysis compares two models fitted to the same data with the same fixed-effects structure. The only difference is that the alternative estimates one nonnegative random-effect variance. The maximized log likelihoods are \(\ell_0=-120.00\) and \(\ell_1=-119.60\). Assume the one-variance boundary conditions are appropriate.

Calculate the statistic. Substituting the log likelihoods gives \(T=2\{(-119.60)-(-120.00)\}=2(0.40)=0.80\). This is positive, as required by the nesting of the models.

Calculate the p-value. For \(T=0.80>0\), use the mixture upper tail:

p_{\mathrm{mix}}=\tfrac12 P(\chi^2_1\geq 0.80)\approx 0.1855

The tail probability is evaluated before rounding the final product. At \(\alpha=0.05\), the p-value exceeds the significance level, and \(0.80<2.7055\), the 5% mixture critical value. The test does not reject \(H_0\). The result does not establish that the variance is exactly zero; it indicates that this likelihood comparison does not provide sufficient evidence against a zero variance at the 5% level.

Worked Example: A Statistic Above the Mixture Critical Value

In a second hypothetical comparison, the null model has maximized log likelihood \(\ell_0=-118.20\), and the alternative with one additional random-effect variance has \(\ell_1=-116.28\). The fixed-effects structure and all other model components are unchanged. Assume the fits have converged to the relevant maxima and the mixture conditions are reasonable.

Calculate the statistic.

T=2\{(-116.28)-(-118.20)\}=2(1.92)=3.84

Calculate the mixture p-value. The one-degree-of-freedom chi-square upper tail at \(3.84\) is approximately \(0.0500\), so \(p_{\mathrm{mix}}=\tfrac12P(\chi^2_1\geq3.84)\approx0.0250\). Equivalently, \(3.84>2.7055\), the 5% mixture critical value.

At the 5% level, reject \(H_0\) in favor of the model allowing the variance to be positive. This gives evidence that including this random-effect variance improves the likelihood beyond what would typically be expected under the zero-variance null and the stated asymptotic reference distribution. It does not by itself quantify the scientific importance of the variability or establish that the model is otherwise adequate.

Worked Example: The Alternative Estimate Is at Zero

A third hypothetical analysis reports a constrained estimate of the tested variance equal to zero. The maximized log likelihoods are both \(-75.40\): the larger model has not achieved a higher likelihood than the null model after fitting.

Calculate the statistic. \(T=2\{(-75.40)-(-75.40)\}=0\). The observed statistic is at the point mass of the mixture. Its upper-tail probability, including probability at the observed value, is \(P(T\geq0)=1\). There is no evidence against the zero-variance null from this test.

A zero estimate can be a legitimate boundary maximum, but it can also arise from poor numerical optimization. The conclusion is defensible only after checking that the optimizer has converged, that the likelihood is evaluated consistently, and that alternative starting values do not reveal a better fit with a positive variance.

Checking the Fit and Choosing an Approach

The mixture test is an asymptotic likelihood result, not a repair for unreliable model fitting. Before interpreting it, check that the two models are genuinely nested, use the same observations and likelihood basis, and differ only in the specified variance. Confirm that the alternative variance estimate is nonnegative and that both fits have converged. Refit from plausible starting values, particularly when the alternative estimate is near zero or the likelihood surface is shallow. A suspiciously small improvement, a negative computed \(T\), or a result that changes materially across starting values calls for investigation.

The design also affects how persuasive the approximation is. The relevant asymptotic information comes primarily from independent subjects, not simply from a large number of repeated records from a few subjects. Sparse subject-level information, a small number of subjects, strong model misspecification, or a difficult likelihood approximation can make the nominal mixture p-value poorly calibrated. The mixture handles the boundary in the specified simple test; it does not resolve identifiability or general model inadequacy.

The main alternatives have different roles. Comparing with an ordinary \(\chi^2_1\) distribution ignores the boundary and, for positive statistics in this setting, yields twice the mixture p-value; it is therefore more conservative than the mixture test. A parametric bootstrap can estimate a finite-sample null distribution by simulating data under the fitted null model and repeating both fits for each simulated data set. That approach can be useful when the asymptotic approximation is doubtful, but it depends on the null model and simulation setup and can be computationally demanding. Profile-likelihood methods address the shape of the likelihood for a variance parameter and its uncertainty; they are not a substitute for stating the correct reference distribution when conducting this likelihood-ratio test.

Common pitfall: Using the usual \(\chi^2_1\) p-value because exactly one parameter was added treats the null as an interior point. The tested variance is instead constrained at zero under \(H_0\), so the standard one-parameter chi-square argument does not apply. For the single-variance boundary case, use \(p_{\mathrm{mix}}=\tfrac12P(\chi^2_1\geq T)\) when \(T>0\), and report the assumptions supporting that reference distribution.

The half-and-half mixture is a focused result: one variance component, one boundary restriction, and otherwise suitably regular nuisance parameters. If the tested structure includes additional covariance restrictions or parameters that are not identified under the null, the reference distribution requires separate justification. The practical lesson is to identify the exact restriction first, then select the reference distribution that corresponds to that restriction—not simply the number of parameters appearing in the larger model.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Why does testing a variance of zero violate the usual interior-parameter condition for a standard chi-square likelihood-ratio test?
A single-variance comparison gives \(T=1.00\). Write the mixture p-value in terms of a chi-square tail and state whether it is below 0.05.
For the one-variance mixture test at \(\alpha=0.05\), why is the critical value about 2.7055 rather than 3.841?
A fitted statistic is negative even though the larger model is intended to contain the smaller model. Give one appropriate response.
Name one situation in which a parametric bootstrap may be worth considering instead of relying only on the asymptotic mixture.

Solutions. (1) A variance is restricted to be nonnegative, so zero is an endpoint of its allowable values, not an interior point with permissible local changes in both directions. (2) \(p_{\mathrm{mix}}=\tfrac12P(\chi^2_1\geq1.00)\approx\tfrac12(0.3173)=0.1587\), so it is not below 0.05. (3) The mixture upper tail is half the \(\chi^2_1\) tail; to make it 0.05, the chi-square upper tail must be 0.10, giving the 90th percentile, about 2.7055. (4) Check nesting, data and likelihood comparability, convergence, and whether the fits reached the relevant maxima; a negative statistic should not be interpreted as evidence against the null. (5) A small number of independent subjects or another reason to doubt the asymptotic approximation can motivate simulation under the null, provided the bootstrap model and fitting procedure are appropriate.