Testing a Variance Component at Zero

Why a Zero Variance Requires Special Treatment

In a genetic linear mixed model, a variance component measures the amount of variation assigned to a particular random effect. For example, a genetic effect \(u\) may be modeled as \(u\sim N(0,\sigma_g^2 K)\), where \(K\) is a specified genetic relationship matrix. Testing whether this component is absent means testing \(H_0:\sigma_g^2=0\) against \(H_a:\sigma_g^2>0\).

The difficulty is not that a likelihood-ratio test cannot compare these models. The difficulty is that the null value lies at the edge of the allowable parameter space: a variance cannot be negative. The usual likelihood-ratio reference distribution, a chi-square distribution with one degree of freedom, relies on regularity conditions that include an interior null parameter. At zero, that condition fails. Under appropriate assumptions, the resulting reference distribution is instead a mixture: half its probability is at zero and half follows a chi-square distribution with one degree of freedom.

The Model and the Likelihood-Ratio Statistic

Consider the Gaussian mixed model

where \(y\) is the vector of measured outcomes, \(X\beta\) describes fixed effects, \(K\) is a known genetic relationship matrix, \(\sigma_g^2\geq 0\) is the genetic variance component being tested, and \(\sigma_e^2>0\) is the residual variance. The test compares a null model with \(\sigma_g^2=0\) to an alternative model in which \(\sigma_g^2\) is estimated subject to \(\sigma_g^2\geq 0\).

Let \(\ell_0\) and \(\ell_1\) be the maximized log likelihoods under the null and alternative models, using the same likelihood convention. The likelihood-ratio statistic is

Because the null model is included in the alternative parameter space, maximizing under the alternative cannot give a lower likelihood, so \(T\geq 0\). The test statistic is often written as \(\mathrm{LR}\) as well. For a restricted maximum likelihood (REML) comparison, the fixed-effects design must be the same in both models; the same boundary issue and mixture reference apply under the corresponding regularity conditions.

Why the Reference Distribution Is a Mixture

The key feature is the constraint \(\sigma_g^2\geq 0\). A local likelihood approximation makes clear how this constraint changes the usual chi-square result. Write \(\theta\) for the tested variance parameter after accounting for nuisance parameters, so that \(\theta\geq 0\) and the null is \(\theta=0\). Near the null, the log likelihood, after profiling out nuisance parameters, has the approximate form

where \(U\) is the efficient score at the null and \(I_{\mathrm{eff}}>0\) is the corresponding information. Under the null, \(U/\sqrt{I_{\mathrm{eff}}}\) is approximately standard normal. Set \(h=\theta\sqrt{I_{\mathrm{eff}}}\) and \(Z=U/\sqrt{I_{\mathrm{eff}}}\). Since \(\theta\geq 0\), the local parameter must satisfy \(h\geq 0\), and the likelihood increment is approximately \(hZ-h^2/2\).

If \(Z\leq 0\), this expression is maximized over \(h\geq 0\) at \(h=0\), giving no likelihood improvement. If \(Z>0\), its maximum occurs at \(h=Z\), giving an increment of \(Z^2/2\). Multiplying the maximized increment by two gives the limiting likelihood-ratio statistic \(\{\max(Z,0)\}^2\). Since a standard normal variable is negative with probability one half and positive with probability one half, this yields the mixture distribution.

Here \(\chi_0^2\) denotes a point mass at zero, not an ordinary continuous chi-square distribution. The other component, \(\chi_1^2\), is a chi-square distribution with one degree of freedom. Thus the limiting statistic is zero with probability one half; with probability one half, it follows the \(\chi_1^2\) distribution. In finite samples this is generally an approximation, not a guarantee of an exact distribution.

where \(\Phi\) is the standard normal cumulative distribution function. The equality follows because a \(\chi_1^2\) variable has the same distribution as the square of a standard normal variable. At \(T=0\), the mixture upper-tail probability is 1, including the mass at zero.

Worked Likelihood-Ratio Tests

Critical Values and the Testing Decision

The mixture distribution also changes critical values. For a test of size \(\alpha<0.5\), choose a threshold \(c\) whose mixture upper tail is \(\alpha\). For \(c>0\), that tail is one half of the \(\chi_1^2\) upper tail, so \(c\) is the \(1-2\alpha\) quantile of \(\chi_1^2\). For example, at \(\alpha=0.05\), the critical value is approximately 2.706, rather than the ordinary \(\chi_1^2\) critical value of 3.841.

Interpretation and Common Pitfall

The boundary mixture is not a generic correction for every likelihood-ratio test. It is the standard asymptotic reference in the particular setting of one identifiable, nonnegative variance component tested at zero with otherwise regular nuisance parameters. If the model has several tested variance components, or if a component cannot be identified under the null, this simple mixture need not apply. Those situations require their own analysis rather than an automatic substitution of the half-\(\chi_1^2\) rule.

A common mistake is to say, “The null has one restriction, so use \(\chi_1^2\).” That reasoning treats the tested parameter as if small deviations in both directions from zero were possible. But a variance is constrained to be nonnegative: when the score points toward a negative value, the best allowable fit stays at zero. The resulting probability mass at zero is why the standard interior-parameter reference does not apply. Another mistake is to interpret a nonsignificant result as acceptance of \(H_0\); a failure to reject can reflect limited information as well as a genuinely small component.