Boundary Likelihood-Ratio Tests For Variance Components

Why a Zero Variance Is a Special Null Hypothesis

A population pharmacokinetic model may include a subject-specific random effect to represent between-subject variation in a parameter such as clearance. A likelihood-ratio test can ask whether that source of variation is needed. The special feature is that a variance cannot be negative: testing whether it equals zero places the null hypothesis at the edge of the allowed parameter space, not in its interior. That boundary changes the usual large-sample reference distribution of the test statistic.

The effect is practical. For one suitably regular variance component, the null reference distribution is not an ordinary chi-square distribution with one degree of freedom. It is a mixture: half of its probability is concentrated at zero, and half follows a chi-square distribution with one degree of freedom. Using the ordinary chi-square reference instead gives different p-values and critical values.

Definition: Let $\omega^2\geq 0$ be a single random-effect variance component. The boundary likelihood-ratio test compares $H_0:\omega^2=0$ with $H_a:\omega^2>0$. If $\ell_0$ and $\ell_1$ are the maximized log likelihoods under the null and alternative models, respectively, the likelihood-ratio statistic is $D=2(\ell_1-\ell_0)$. When the regularity conditions below hold, its asymptotic null distribution is a 50:50 mixture of a point mass at zero and $\chi^2_1$.

The alternative model must contain the null model as a special case. Both models should use the same observations, likelihood basis, fixed-effect structure, and residual-error structure, differing only in whether the variance component is fixed at zero or estimated subject to being nonnegative. The likelihoods should be maximized under their respective constraints. In population models, the marginal likelihood integrates over individual random effects; the comparison is between these marginal likelihoods, not between likelihoods conditional on estimated individual effects.

Conditions for the Mixture Reference

The 50:50 result is an asymptotic result, not a universal rule for every model containing random effects. Its simplest form applies when one scalar variance component is tested and the remaining nuisance parameters are identifiable under the null, lie in the interior of their allowed ranges, and are estimated regularly. The likelihood must be correctly specified and sufficiently smooth near the null, and the information for the tested component must be positive. The observations must provide enough independent information for the large-sample approximation to be relevant.

The null model sets the random-effect variance to zero, so that random effect disappears. Other model parameters must still have a well-defined interpretation under that null. For example, if a correlation parameter is defined only through the tested random effect, it is not identifiable when the variance is zero. In that case the simple one-component mixture cannot automatically be assumed. Likewise, testing several variance components at once, or testing a variance while also estimating covariance terms constrained by positive semidefiniteness, can produce more complicated reference distributions.

The variance parameterization makes the boundary explicit: $\omega^2$ is restricted to $[0,\infty)$, and zero is its lower endpoint. A standard likelihood-ratio test for a scalar parameter strictly inside its parameter space has a $\chi^2_1$ limit under regularity conditions. That standard interior-parameter result does not apply unchanged here.

Why the Distribution Is a Mixture

A short local-likelihood argument shows where the mixture comes from. Near the null, after accounting for regular nuisance parameters, the log likelihood can be approximated in a standardized local coordinate $h$ by a quadratic expression. Because the variance cannot be negative, the local coordinate is constrained to $h\geq 0$. Under the null, the standardized score $Z$ is approximately $N(0,1)$.

\ell(h)-\ell(0)\approx hZ-\frac{h^2}{2},\qquad h\geq 0.

Without the constraint, the quadratic is maximized at $h=Z$. With the constraint, its maximum occurs at $h=\max(0,Z)$: if $Z$ is negative, increasing $h$ from zero lowers the approximation, so the constrained maximum stays at zero; if $Z$ is positive, the unconstrained maximum is permitted. Twice the gain in log likelihood is therefore approximately $\max(0,Z)^2$.

D\ \xrightarrow{d}\ \max(0,Z)^2,\qquad Z\sim N(0,1).

A standard normal variable is negative with probability one half and positive with probability one half. When it is negative, $\max(0,Z)^2=0$, giving the point mass at zero. When it is positive, its squared value has the same distribution as $Z^2$ without conditioning on the sign, because the normal distribution is symmetric; $Z^2$ follows $\chi^2_1$. Thus the limiting distribution is exactly the stated half-at-zero, half-$\chi^2_1$ mixture.

Theorem: Under the stated one-component regularity conditions, for an observed $D>0$, the asymptotic boundary-test p-value is $$ p_{\mathrm{mix}}=\frac{1}{2}P(\chi^2_1\geq D). $$ Equivalently, if $Z\sim N(0,1)$, then $p_{\mathrm{mix}}=P(Z\geq\sqrt{D})$. At significance level $\alpha$, the positive critical value $c$ satisfies $P(\chi^2_1\geq c)=2\alpha$.

For $\alpha=0.05$, the critical value is approximately $2.7055$, the 90th percentile of $\chi^2_1$. Thus a positive statistic greater than $2.7055$ is significant at the 5% level under this mixture calibration. The usual $\chi^2_1$ 5% critical value, approximately $3.841$, is not the appropriate cutoff for this particular boundary test.

Worked Calculations

Worked Example: Testing a Clearance Random-Effect Variance

Hypothetical data. A model estimates a subject-level random effect on clearance. The null model fixes its variance at zero and has maximized log likelihood $\ell_0=-812.40$. The alternative estimates the variance subject to $\omega^2\geq0$ and has $\ell_1=-810.95$. Assume the models are nested, differ only in this one variance component, and the other parameters remain identifiable under the null.

First calculate the likelihood-ratio statistic:

D=2(\ell_1-\ell_0)=2[-810.95-(-812.40)]=2(1.45)=2.90.

Since $D>0$, use the mixture tail. Taking the square root gives $\sqrt{2.90}\approx1.703$, so

p_{\mathrm{mix}}=P(Z\geq1.703)\approx0.0443.

Equivalently, the ordinary $\chi^2_1$ upper-tail probability is about $0.0886$, and half of it is about $0.0443$. At $\alpha=0.05$, $D=2.90>2.7055$, so the null is rejected using the boundary reference. The result is evidence that including this clearance variance improves the model under the stated assumptions; it does not establish that the random effect is the only explanation for the observed variability.

Worked Example: A Statistic Just Below the 5% Cutoff

Hypothetical data. A second one-component comparison has maximized log likelihoods $\ell_0=-501.20$ and $\ell_1=-499.85$. Suppose again that the null and alternative are nested and all nuisance parameters satisfy the required regularity conditions.

The statistic is

$$ D=2[-499.85-(-501.20)]=2(1.35)=2.70. $$

Here $\sqrt{2.70}\approx1.6432$. Therefore,

p_{\mathrm{mix}}=P(Z\geq1.6432)\approx0.05017.

The ordinary $\chi^2_1$ tail is approximately $0.1003$; halving it gives approximately $0.05015$ when using that rounded tail value. The more direct normal-tail calculation gives about $0.05017$. Since $2.70<2.7055$, the result is just above the 5% cutoff and is not significant at $\alpha=0.05$. It is better to report the statistic and p-value than to describe this result as a decisive distinction between models.

Worked Example: A Stronger Likelihood Improvement

Hypothetical data. Consider a test of one occasion-level random-effect variance, with $\ell_0=-326.70$ and $\ell_1=-324.10$. Assume the occasion effect is the sole added variance component and the models meet the conditions for the mixture reference.

The likelihood-ratio statistic is

$$ D=2[-324.10-(-326.70)]=2(2.60)=5.20. $$

The equivalent normal threshold is $\sqrt{5.20}\approx2.280$, giving

p_{\mathrm{mix}}=P(Z\geq2.280)\approx0.0113.

The null is rejected at the 5% level, since $5.20>2.7055$. The corresponding ordinary $\chi^2_1$ tail is approximately $0.0226$, twice the mixture p-value. This difference illustrates why the reference distribution must match the boundary hypothesis.

Diagnostics, Alternatives, and Limitations

Before interpreting a result, check the model comparison and the fit itself. The alternative log likelihood should not be materially lower than the null log likelihood, because the alternative includes the null parameter value. A negative $D$ beyond numerical tolerance can indicate failed optimization, inconsistent likelihood calculations, or models that are not truly nested as fitted. A variance estimate pinned at zero is compatible with the boundary null, but it is not by itself proof that the model and its optimization are sound.

Check the nesting. Confirm that the models use the same data and likelihood definition, and that setting the added variance to zero recovers the null model.
Check optimization near zero. Review convergence messages, gradients, and likelihood profiles. Variance parameters often have flat or asymmetric likelihoods near the boundary, making a single optimizer result less reassuring.
Check identifiability. Ask whether all nuisance parameters have a meaningful, estimable value under the null. A correlation involving a random effect that vanishes at $\omega^2=0$ may not be identifiable there.
Check information in the design. Sparse sampling or limited repeated observations can make a variance component weakly identified. A theoretical asymptotic reference may then approximate the finite-sample distribution poorly.

The ordinary $\chi^2_1$ test is a tempting alternative, but it treats the tested parameter as if it could move in either direction from an interior null value. That is not the geometry of a nonnegative variance. The 50:50 mixture is appropriate for the simple one-component case described here. For small samples or complicated models, a parametric bootstrap can be a useful alternative calibration: simulate data under the fitted null model, refit both models to each simulated data set using the same constraints, and compare the observed statistic with the simulated null statistics. The bootstrap must reproduce the design and fitting procedure; it does not remove problems of model misspecification or non-identifiability.

There is no single mixture rule that applies to every more complex test. With multiple variance components, the allowed parameter changes form a higher-dimensional constrained region, and the limiting distribution may be a weighted mixture of chi-square distributions whose weights depend on the model structure. If a nuisance parameter exists only under the alternative, even that description may not apply. Such cases require a result specific to the tested structure or a carefully designed simulation-based calibration.

Common pitfall: “The test has one added variance, so use $\chi^2_1$ and its 3.841 cutoff at 5%.” The correction is to check whether the null fixes that variance at its boundary of zero. When the one-component mixture conditions hold, use $p=\frac12P(\chi^2_1\geq D)$ for $D>0$, or the 2.7055 cutoff at 5%. Do not apply this rule automatically to multi-component or non-identifiable tests.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Why is $H_0:\omega^2=0$ a boundary hypothesis when $\omega^2$ is a variance?
A valid one-component test gives $D=3.00$. Calculate the mixture p-value approximately and decide whether to reject at $\alpha=0.05$.
What is the approximate 5% critical value for the one-component mixture test, and why does it differ from the usual $\chi^2_1$ critical value?
Why can the 50:50 mixture not automatically be used when two variance components are tested together?

Solutions. (1) A variance is restricted to be nonnegative, so zero is the smallest permitted value; the null lies at the edge rather than in the interior. (2) $\sqrt{3.00}=1.732$, so $p_{\mathrm{mix}}=P(Z\geq1.732)\approx0.0416$. Since $3.00>2.7055$ and $0.0416<0.05$, reject the null at the 5% level, assuming the conditions hold. (3) The cutoff is approximately $2.7055$, chosen so the upper tail of $\chi^2_1$ is $0.10$; the mixture assigns only half that tail probability to positive statistics, so this yields a 0.05 test. (4) The joint parameter space has more than one constrained direction, and the limiting distribution depends on its geometry and model structure. A single 50:50 point-mass/chi-square rule is therefore not guaranteed.