Nuisance Parameters in Boundary Likelihood-Ratio Tests

Why Nuisance Parameters Matter at a Boundary

A variance-component test asks whether a variance is zero, even though variances cannot be negative. The likelihood-ratio statistic therefore does not generally have the ordinary chi-square reference distribution. In a GLMM, the likelihood also involves nuisance parameters: for example, fixed-effect coefficients and other covariance or dispersion parameters. These parameters are usually estimated, not known, and their treatment can affect both the test statistic and its calibration.

The key distinction is whether the nuisance parameters are regular and identifiable under the null. If the same nuisance parameters are present under both hypotheses, are identifiable when the tested variance is zero, and lie in the interior of their allowed parameter space, profiling them changes the information used in the test but does not change the familiar one-component asymptotic mixture. If nuisance parameters become unidentified under the null, that conclusion need not hold.

Definition: Let $\lambda\geq 0$ be a variance component to test, and let $\eta$ collect nuisance parameters. The null hypothesis is $H_0:\lambda=0$, and the alternative allows $\lambda>0$. The profiled likelihood-ratio statistic is $$ T=2\left\{\sup_{\eta,\,\lambda\geq 0}\ell(\eta,\lambda)-\sup_{\eta}\ell(\eta,0)\right\}, $$ where $\ell$ is the log likelihood. Nuisance parameters are profiled by maximizing over them separately under the alternative and the null.

In a GLMM, the likelihood in this definition is ordinarily the marginal likelihood, with random effects integrated out. The asymptotic argument below presumes that the amount of independent information grows, typically through an increasing number of independent clusters. A large number of observations within only a few clusters does not automatically supply the same asymptotic justification.

Regular Nuisance Parameters and the Local Likelihood

Suppose $\eta$ is a finite-dimensional nuisance vector that is identifiable under $H_0$, has a true value in the interior of its parameter space, and is estimated under both hypotheses. Assume the likelihood is sufficiently smooth near the null, the nuisance information is nonsingular, and the usual local asymptotic normal approximation holds. The only boundary in this local comparison is then $\lambda\geq 0$. These conditions are substantive: they are not guaranteed merely because software returns estimates for both models.

To see the role of estimating $\eta$, write the local displacement of the nuisance parameter as $h_\eta/\sqrt{m}$ and that of the tested parameter as $h_\lambda/\sqrt{m}$, where $m$ indexes the increasing information. Let $U_\eta,U_\lambda$ denote the local score components and partition the per-information-unit Fisher information as $I_{\eta\eta}, I_{\eta\lambda}, I_{\lambda\eta}, I_{\lambda\lambda}$. The local log-likelihood, up to terms that do not depend on these displacements, is

Q(h_\eta,h_\lambda) = U_\eta^{\mathsf T}h_\eta+U_\lambda h_\lambda -\frac{1}{2} \left( h_\eta^{\mathsf T}I_{\eta\eta}h_\eta +2h_\eta^{\mathsf T}I_{\eta\lambda}h_\lambda +I_{\lambda\lambda}h_\lambda^2 \right), \qquad h_\lambda\geq 0.

Under the null, $h_\lambda=0$, and maximizing over $h_\eta$ gives the nuisance estimate’s local adjustment. Under the alternative, $h_\eta$ is maximized again for every candidate $h_\lambda$. Subtracting the null maximum is equivalent to profiling $h_\eta$ out of the quadratic likelihood. Completing the square gives

Q_{\mathrm{prof}}(h_\lambda)-Q_{\mathrm{prof}}(0) = U_{\mathrm{eff}}h_\lambda-\frac{1}{2}I_{\mathrm{eff}}h_\lambda^2, \qquad U_{\mathrm{eff}}=U_\lambda-I_{\lambda\eta}I_{\eta\eta}^{-1}U_\eta, \qquad I_{\mathrm{eff}}=I_{\lambda\lambda} -I_{\lambda\eta}I_{\eta\eta}^{-1}I_{\eta\lambda}.

Here $U_{\mathrm{eff}}$ is the score for $\lambda$ after accounting for the nuisance score, and $I_{\mathrm{eff}}$ is the corresponding efficient information. Nuisance estimation therefore changes the relevant score and information: using the unadjusted $U_\lambda$ and $I_{\lambda\lambda}$ would generally describe a different, conditional calculation.

Theorem: Under the regularity conditions above, with one scalar variance component constrained by $\lambda\geq0$ and otherwise regular nuisance parameters, the null limit of the profiled likelihood-ratio statistic is $$ T\ \xrightarrow{d}\ \{\max(0,Z)\}^2 \ \sim\ \tfrac{1}{2}\chi_0^2+\tfrac{1}{2}\chi_1^2, \qquad Z\sim N(0,1). $$ Here $\chi_0^2$ denotes a point mass at zero. Regular nuisance estimation affects the efficient score and information, but after standardization it does not change the one-half mixture weights.

The reason is direct. Under $H_0$, $U_{\mathrm{eff}}/\sqrt{I_{\mathrm{eff}}}$ converges to a standard normal variable $Z$. Maximizing the profiled quadratic over $h_\lambda\geq0$ gives $h_\lambda=0$ if $U_{\mathrm{eff}}\leq0$, and $h_\lambda=U_{\mathrm{eff}}/I_{\mathrm{eff}}$ if $U_{\mathrm{eff}}>0$. Twice the maximized log-likelihood gain is therefore $\{\max(0,Z)\}^2$. Symmetry gives probability one-half to $Z\leq0$, producing the atom at zero; conditional on a positive score, the squared standardized score has the $\chi_1^2$ component.

Worked Calculations

Worked Example: Profiling a Regular Nuisance Score

Consider a hypothetical local likelihood calculation with one nuisance parameter $\eta$. Suppose the score vector and information matrix at the null are $U_\eta=1$, $U_\lambda=2$, and $I_{\eta\eta}=4$, $I_{\eta\lambda}=I_{\lambda\eta}=1$, $I_{\lambda\lambda}=2$. The matrix is positive definite, since its determinant is $4(2)-1^2=7>0$, and the nuisance information is positive.

First compute the efficient score and information:

U_{\mathrm{eff}}=2-\frac{1}{4}(1)=1.75, \qquad I_{\mathrm{eff}}=2-\frac{1}{4}(1)(1)=1.75.

Because $U_{\mathrm{eff}}>0$, the constrained local maximum has $h_\lambda=U_{\mathrm{eff}}/I_{\mathrm{eff}}=1$. The resulting likelihood-ratio statistic is

T_{\mathrm{local}} =2\left\{1.75(1)-\tfrac{1}{2}(1.75)(1)^2\right\} =1.75.

This calculation shows why nuisance adjustment belongs inside the likelihood comparison. The statistic uses the efficient score and information, not the score and information for $\lambda$ as if $\eta$ were known.

Worked Example: A Mixture-Reference P-Value

Suppose a hypothetical GLMM comparison tests one random-intercept variance, $\lambda$, while fixed effects and a dispersion parameter are identifiable under the null and estimated in both models. Assume the likelihood conditions for the one-component mixture apply. The maximized log likelihood is $-41.925$ under the alternative and $-42.800$ under the null.

The observed statistic is

T=2\{-41.925-(-42.800)\}=2(0.875)=1.750.

For a positive observed statistic, the upper-tail probability under the half-at-zero, half-$\chi_1^2$ law is

p=\tfrac{1}{2}P(\chi_1^2\geq1.750) =P(Z\geq\sqrt{1.750}) =1-\Phi(1.323)\approx0.0929.

At significance level $\alpha=0.05$, this p-value does not provide sufficient evidence to reject $H_0$. It is not the ordinary $\chi_1^2$ upper tail, which would be approximately $0.186$; the point mass at zero halves that upper tail for positive values of $T$. The p-value is a tail probability calculated under the null reference law, not the probability that the null hypothesis is true.

When the Mixture Calibration Can Fail

A nuisance parameter that exists only under the alternative, or that is not identifiable under the null, violates the regular argument. In that situation, there is no single well-defined null value for that nuisance parameter around which the joint local quadratic expansion above can be applied. Maximizing over it under the alternative may add a random search over possible directions or configurations, and the reference distribution can differ from the simple half-$\chi_1^2$ mixture.

A familiar variance-component example is a random slope whose variance is tested while its correlation with a random intercept is also estimated. When the random-slope variance is zero, that correlation does not affect the random-effects distribution and is not identified by the null model. Treating it as an ordinary, common, interior nuisance parameter and applying the one-component result without checking the model structure is not justified. The exact calibration then depends on the specific geometry and assumptions; it cannot be inferred from the fact that the tested variance is nonnegative alone.

Worked Example: An Unidentified Correlation Under the Null

Consider a hypothetical random-intercept and random-slope GLMM. The test is $H_0:\tau^2=0$, where $\tau^2$ is the random-slope variance. Under the alternative, the fitted covariance structure also includes a correlation $\rho$ between the random intercept and slope. Under the null, the random slope has zero variance, so changing $\rho$ does not change the model distribution. Thus $\rho$ is not identified under $H_0$.

Suppose the maximized log likelihoods are $-120.4$ under $H_0$ and $-117.9$ under the alternative. The observed statistic is

T=2\{-117.9-(-120.4)\}=2(2.5)=5.00.

If one incorrectly used the one-component mixture, the computed tail area would be $1-\Phi(\sqrt{5.00})=1-\Phi(2.236)\approx0.0127$. That arithmetic is correct for the assumed mixture, but the assumption is not justified here: $\rho$ is unidentified under the null. Consequently, $0.0127$ is not established as a valid p-value for this comparison. A suitable calibration must account for the actual null behavior of the fitted model; the statistic alone does not supply that calibration.

Practical Checks and Alternative Approaches

Before using a boundary reference law, check the model and the fitting procedure, not just the reported likelihoods. Confirm that the models are nested, that the alternative allows the tested variance to increase from zero, and that both likelihoods are maximized over the nuisance parameters appropriate to each model. Also check whether every nuisance parameter used under the alternative is identifiable under the null and whether any other parameters are themselves at boundaries or weakly identified. A singular fit, an unstable covariance estimate, or a nuisance estimate that changes dramatically between fits can be a warning that the regular approximation deserves scrutiny.

The main alternatives answer different questions or rely on different assumptions:

Ordinary $\chi_1^2$ likelihood-ratio calibration ignores the one-sided boundary and is not the standard asymptotic calibration for a single regular variance component at zero.
Wald tests use an estimated variance and its standard error as though the parameter were locally unrestricted. At a boundary, that approximation can be unreliable; earlier material on Wald tests for variance components discusses this failure.
Holding nuisance parameters fixed does not perform the stated likelihood-ratio comparison when those parameters are estimated from the data. It omits their adjustment and generally changes the statistic and its information scale.
Simulation-based calibration can be considered when the regular mixture conditions are doubtful, but its validity depends on correctly representing the null model and the data-generating structure. The mixture result should not be treated as a substitute for checking those conditions.

The half-at-zero, half-$\chi_1^2$ law is an asymptotic result, not a universal finite-sample guarantee. It can be a poor approximation with limited independent clusters, complex covariance constraints, or nuisance parameters that are weakly identified. In particular, a software convergence message does not establish the identifiability and regularity conditions required for the reference law.

Key takeaway: With one tested variance component and regular nuisance parameters shared by the null and alternative, profile the nuisance parameters in both models and use the half-at-zero, half-$\chi_1^2$ asymptotic reference law. If nuisance parameters are unidentified under the null or additional boundary constraints are active, do not assume that calibration applies.

Check Your Understanding

Answer each question using the conditions and calculations in this tutorial. Brief solutions follow.

In a regular one-component test, why does the efficient score subtract a term involving the nuisance score?
A regular boundary test has observed $T=0.64$. Express its half-mixture p-value using the standard normal distribution.
Why can estimating nuisance parameters under both hypotheses leave the one-component mixture weights unchanged while still affecting the observed test statistic?
A nuisance correlation is present in the alternative model but has no effect on the model under the null. Is the ordinary half-mixture calibration justified by the regular-nuisance argument? Explain.

Solutions. (1) Profiling lets the nuisance parameter adjust with the tested parameter. The term $I_{\lambda\eta}I_{\eta\eta}^{-1}U_\eta$ removes the part of the tested-parameter score explained by the nuisance score, leaving the efficient score. (2) Since $0.64>0$, $p=\tfrac12P(\chi_1^2\geq0.64)=1-\Phi(0.8)\approx0.2119$. (3) Profiling changes the score and information to their efficient versions, which can change the realized statistic. Under the stated regularity conditions, standardizing those quantities still gives a standard normal score, and the one-sided constraint still yields one-half probability at zero. (4) No. The correlation is unidentified under the null, so the regular common-nuisance derivation does not apply; a reference law must be justified for the particular model rather than presumed to be the simple mixture.