Likelihood-Ratio Tests at a Variance Boundary

Why a Zero QTL Variance Requires Special Calibration

A likelihood-ratio test compares the best fit under a null hypothesis with the best fit under a broader alternative. For many regular tests, the statistic has an asymptotic chi-square distribution under the null. A test of a QTL variance component is different: a variance cannot be negative, so the null value of zero lies at the edge of the permitted parameter space. That boundary changes the reference distribution and, consequently, the p-value.

The distinction matters in family QTL analysis because evidence for a locus is often assessed by asking whether adding a QTL covariance component improves the fit beyond polygenic and residual variation. The test is about improvement in fit, not proof that a locus causes a trait or that a particular biological mechanism has been established.

Model and Hypotheses

Consider a quantitative phenotype vector \(y\) from a set of relatives. At a specified locus, suppose the Gaussian model is

Here \(X\beta\) describes fixed effects, \(I\) is the identity matrix, and \(K_P\) and \(K_Q\) are the polygenic and QTL covariance matrices, respectively. The matrices are treated as known for this test. The parameters \(\sigma_e^2\), \(\sigma_P^2\), and \(\sigma_Q^2\) are variance components, constrained to be nonnegative. In particular, \(\sigma_Q^2\) measures the contribution of the specified QTL covariance structure to the phenotypic covariance.

The null model retains the fixed effects, polygenic component, and residual component; it removes only the QTL component. For a likelihood-ratio comparison, both fits must use the same likelihood framework and the same observations. If maximum likelihood is used, compare the two maximum-likelihood fits; do not compare an ML likelihood in one model with a restricted likelihood in the other.

Why Zero Is a Boundary

The admissible values of \(\sigma_Q^2\) are \(0\) and positive numbers. There are no allowed values below zero. Thus, under \(H_0\), the parameter is not in the interior of its parameter space: it is at the boundary. In an ordinary one-parameter likelihood-ratio test, local departures from the null can be considered in both directions. Here, departures are possible only toward positive QTL variance.

The usual one-degree-of-freedom chi-square result does not account for this one-sided constraint. Under suitable regularity conditions, the likelihood-ratio statistic instead has the asymptotic reference distribution

This result assumes that the likelihood is sufficiently smooth near the null, the relevant information is nonsingular after accounting for nuisance parameters, and the QTL covariance component is identifiable separately from the other covariance components. The families must provide an appropriate independent sampling basis for the asymptotic approximation. The result is not a universal correction for every variance-component test: testing multiple boundary components, or testing a locus parameter that is undefined when \(\sigma_Q^2=0\), can require a different reference distribution.

Deriving the Mixture

The boundary effect can be seen from a local quadratic approximation to the profile log-likelihood, after nuisance parameters have been accounted for. Write \(h\) for a local value of the nonnegative QTL variance parameter, \(S\) for its efficient score at the null, and \(I\) for the corresponding positive efficient information. Near zero, the log-likelihood difference is approximately

Without the constraint, the quadratic is maximized at \(h=S/I\). With the constraint \(h\geq 0\), the maximizing value is \(\hat h=\max(0,S/I)\). Substituting this constrained maximizer into twice the log-likelihood difference gives

Under the null and the stated regularity conditions, \(Z\) is asymptotically standard normal. By symmetry, \(P(Z\leq 0)=0.5\); on that half of the cases, the constrained estimate is zero and \(T=0\). When \(Z>0\), which also has probability one-half, \(Z^2\) has a chi-square distribution with one degree of freedom. This gives the stated mixture.

For an observed positive statistic \(t\), its upper-tail mixture p-value is therefore

The factor of one-half is essential: the other half of the limiting null distribution is concentrated at zero, below any positive observed statistic. At significance level \(\alpha\), a positive statistic is compared with the upper-tail cutoff satisfying \(P(\chi_1^2\geq c)=2\alpha\). For \(\alpha=0.05\), this gives \(c\approx 2.706\), not the ordinary one-degree-of-freedom 5% cutoff of about 3.841.

Worked Tests

Common Mistake and Practical Checks

A frequent error is to calculate the ordinary chi-square p-value \(P(\chi_1^2\geq T)\) and stop there. For a positive statistic in this one-component boundary setting, that tail is twice the mixture p-value. For example, at \(T=3.80\), the ordinary tail is about \(0.05125\), while the appropriate mixture p-value is \(0.02563\). The correction follows from the point mass at zero; it is not an arbitrary adjustment for a small sample.

The mixture approximation is asymptotic, not an exact finite-sample result. Before applying it, check that the comparison concerns one nonnegative variance component, that the null sets that component to zero, and that the other relevant parameters remain identifiable and regular under the null. In particular, do not automatically use this mixture when several variance components are tested jointly or when a parameter such as an unknown QTL location exists only under the alternative. Those settings alter the null problem.