Boundary Parameters and Nonregular Deviance Tests

Why a Nested Test May Not Have a Chi-Square Null Distribution

A likelihood-ratio test for nested models is often calibrated using a chi-square distribution, as in Wilks' theorem. That calibration depends on regularity conditions, including that the null parameter lies in the interior of the alternative parameter space. If a parameter is constrained to be nonnegative and the null sets it to zero, the null is at the boundary. The likelihood can then fluctuate toward the alternative on only one side, changing the limiting distribution of the test statistic.

Other failures are possible even when a parameter value does not look like a boundary value. For example, a parameter may cease to be identifiable under the null, or the maximum-likelihood estimate may fail to exist as a finite value. In such cases, neither the usual chi-square reference distribution nor the simple boundary mixture derived below should be assumed automatically.

Definition: A null parameter value is on the boundary of the alternative parameter space if every neighborhood of that value contains both parameter values allowed by the alternative and values excluded from it. For example, zero is a boundary point of \([0,\infty)\), but an interior point of \(\mathbb{R}\). A likelihood-ratio test is nonregular when one or more regularity conditions needed for its standard asymptotic calibration fail.

The One-Parameter Boundary Limit

Consider testing \(H_0:\theta=0\) against \(H_1:\theta\geq 0\), where \(\theta\) is one scalar parameter. Suppose all nuisance parameters are identifiable and are interior under the null. Under standard smoothness and large-sample conditions, after accounting for nuisance parameters, the local log-likelihood has a quadratic approximation. The efficient score, divided by the square root of its information, converges under the null to \(Z\sim N(0,1)\). The constraint on the local parameter is that it be nonnegative.

For the result below, assume the null is true; the parameter space is locally a half-line at the null; the log-likelihood is sufficiently smooth for a local asymptotic quadratic expansion; the efficient information is finite and positive; nuisance parameters are identifiable and interior; and the relevant maximum-likelihood estimates exist and are localized near the null. These assumptions exclude, among other things, nuisance parameters that appear only under the alternative.

Theorem: Under the stated regularity conditions, the likelihood-ratio statistic for one scalar parameter constrained to be nonnegative, tested at the boundary value zero, converges in distribution under \(H_0\) to \(\tfrac12\chi_0^2+\tfrac12\chi_1^2\). Here \(\chi_0^2\) denotes a point mass at zero, and \(\chi_1^2\) is a chi-square distribution with one degree of freedom.

The result follows directly from maximizing the local quadratic likelihood over a half-line. Let \(U\) be the efficient score and \(I>0\) its information. The local quadratic log-likelihood difference at displacement \(h\geq 0\) is \(hU-\tfrac12 h^2I\). Write \(Z=U/\sqrt I\) and \(t=h\sqrt I\). Then twice the maximized log-likelihood difference is twice the maximum of \(tZ-\tfrac12t^2\) over \(t\geq0\).

\widehat t=\max(Z,0), \qquad T=2\max_{t\geq0}\left(tZ-\frac{t^2}{2}\right) =\{\max(Z,0)\}^2.

If \(Z\leq0\), the quadratic is maximized at \(t=0\), so \(T=0\). If \(Z>0\), it is maximized at \(t=Z\), giving \(T=Z^2\). Since a standard normal variable is nonpositive with probability \(1/2\), \(T\) has probability \(1/2\) at zero. For \(x>0\),

P(T\geq x)=P(Z\geq\sqrt{x}) =\frac12 P(\chi_1^2\geq x).

Thus the positive part of the limiting distribution has half the usual one-degree-of-freedom chi-square tail probability, and the remaining probability is concentrated at zero. This derivation also shows why the result depends on having just one constrained direction and otherwise regular nuisance parameters.

Worked Example: A Quadratic Likelihood at a Boundary

Suppose the local log-likelihood difference for a constrained parameter \(h\geq0\) is \(\ell(h)-\ell(0)=3h-2h^2\). Find the likelihood-ratio statistic and its asymptotic boundary p-value.

Differentiate the expression: \(d\{\ell(h)-\ell(0)\}/dh=3-4h\). Its unconstrained maximizer is \(h=3/4\), which satisfies the constraint. The log-likelihood gain is \(3(3/4)-2(3/4)^2=9/4-18/16=9/8\). Therefore \(T=2(9/8)=9/4=2.25\).

The boundary-mixture p-value is \(\tfrac12P(\chi_1^2\geq2.25)\). Since \(\sqrt{2.25}=1.5\), the chi-square tail is \(P(|Z|\geq1.5)=2\{1-\Phi(1.5)\}\), approximately \(0.1336\). The mixture p-value is half of this, approximately \(0.0668\). Using an ordinary \(\chi_1^2\) reference distribution would therefore double the asymptotic tail probability in this example.

If the score term had instead been nonpositive, the constrained maximum would have occurred at \(h=0\), giving \(T=0\). This is the source of the point mass in the limiting distribution.

Deviance Differences and the Scale of the Likelihood

Boundary calibration concerns the null distribution of a likelihood-ratio statistic, not an unscaled deviance difference by itself. Define \(T=2\{\ell_{\text{alt}}-\ell_{\text{null}}\}\), using maximized log-likelihoods from the same likelihood. If the deviance is defined as twice the log-likelihood gap from the same saturated model, then its null-minus-alternative difference is exactly \(T\). In exponential-family GLMs, however, the reported unit deviance may omit the dispersion scale.

Key result: For a common fixed-dispersion likelihood whose log-likelihood has dispersion factor \(\phi\), the likelihood-ratio statistic is the unit-deviance difference divided by \(\phi\): \(T=(D_{\text{null}}-D_{\text{alt}})/\phi\). When \(\phi=1\), as in the usual Poisson and binomial likelihoods, this is the deviance difference. If dispersion is estimated, calculate the likelihood ratio from the appropriate full likelihood; do not assume it equals a deviance difference divided by an estimated dispersion.

To see the scaling, suppose the log-likelihood can be written as a term independent of the fitted mean plus a mean-dependent term divided by \(\phi\). The saturated-model log-likelihood is common to both nested fits. The null-minus-alternative difference in unit deviances is twice the difference in their mean-dependent terms, while the corresponding log-likelihood difference is that same quantity divided by \(2\phi\). Multiplying the log-likelihood difference by two gives \(T=(D_{\text{null}}-D_{\text{alt}})/\phi\), as claimed. This identity requires a common likelihood and fixed dispersion; it is not a license to substitute a deviance difference for every likelihood-ratio statistic.

Worked Example: A Constrained Poisson Regression Term

Let \(Y_i\) be independent Poisson responses with \(\log E[Y_i]=\alpha+\beta x_i\), and suppose the scientific model constrains \(\beta\geq0\). Test \(H_0:\beta=0\). Assume the fitted null and alternative have unit deviances \(D_{\text{null}}=8.7\) and \(D_{\text{alt}}=5.5\), respectively.

The null value is a boundary point because the alternative permits only \(\beta\geq0\). For the Poisson likelihood, \(\phi=1\), so the likelihood-ratio statistic is \(T=8.7-5.5=3.2\). If the one-parameter boundary conditions hold, its asymptotic p-value is \(\tfrac12P(\chi_1^2\geq3.2)\).

Since \(\sqrt{3.2}\approx1.789\), this tail is \(\tfrac12\{2[1-\Phi(1.789)]\}=1-\Phi(1.789)\approx0.0368\). The ordinary \(\chi_1^2\) tail is approximately \(0.0736\), twice as large. This calculation is an asymptotic calibration: whether it is accurate at a particular sample size is a separate question.

If this were instead a fixed-dispersion model with \(\phi=2\) and the same unit-deviance difference of \(3.2\), the likelihood-ratio statistic would be \(3.2/2=1.6\), not \(3.2\). The boundary mixture would then be applied to \(1.6\). The scale conversion must be made before choosing a reference distribution.

Boundary Does Not Always Mean a Simple Mixture

A boundary parameter alone does not guarantee the half-at-zero, half-\(\chi_1^2\) result. The theorem requires regular nuisance parameters and a locally one-dimensional constrained direction. With several inequality constraints, the limit may be a weighted mixture of chi-square distributions, with weights determined by the local geometry and information. If a nuisance parameter is not identified under the null, the local quadratic argument above no longer applies in its stated form.

Worked Example: Zero Inflation at the Null Boundary

For a zero-inflated Poisson model, let \(\lambda>0\) be the Poisson mean and let \(0\leq\pi\leq1\) be the probability of a structural zero. The probability mass function is

P(Y=y)=(1-\pi)\frac{e^{-\lambda}\lambda^y}{y!} +\pi\,\mathbf{1}\{y=0\}, \qquad y=0,1,2,\ldots .

At \(\pi=0\), this reduces to an ordinary Poisson distribution. The null value is on the boundary of the allowed range for \(\pi\). For example, with \(\lambda=2\) and \(\pi=0.1\), \(P(Y=0)=0.9e^{-2}+0.1\approx0.2218\) and \(P(Y=1)=0.9(2e^{-2})\approx0.2436\). At the null \(\pi=0\), these probabilities are \(e^{-2}\approx0.1353\) and \(2e^{-2}\approx0.2707\).

For testing \(\pi=0\), the simple boundary mixture is a candidate asymptotic calibration only if \(\lambda\) remains identifiable and interior and the other regularity conditions hold. It should not be inferred merely from the fact that \(\pi\) is nonnegative; the fitted likelihood and the parameterization must support the required regular local approximation.

Worked Example: A Mixture Component Unidentified Under the Null

Consider a two-component Poisson mixture with mass function

P(Y=y)=(1-\pi)\frac{e^{-\lambda}\lambda^y}{y!} +\pi\frac{e^{-(\lambda+\kappa)}(\lambda+\kappa)^y}{y!}, \qquad \lambda>0,\ \kappa>0,\ 0\leq\pi\leq1.

Test the single-component null \(H_0:\pi=0\). At that null, the mass function is \(e^{-\lambda}\lambda^y/y!\) for every \(\kappa>0\); changing \(\kappa\) changes no null probability. Thus \(\kappa\) is not identified under \(H_0\). The null is also on the boundary in \(\pi\), but this is not the regular one-parameter situation of the theorem: the nuisance parameter \(\kappa\) exists only in the alternative component and is unidentified under the null.

Consequently, it is not justified to assign either an ordinary \(\chi_1^2\) reference distribution or the simple half-at-zero mixture by counting one tested parameter. The null distribution requires methods appropriate to this nonidentifiable mixture problem; the boundary calculation alone does not supply it.

Practical Diagnosis and Common Pitfalls

Before using a chi-square reference distribution for a nested deviance test, check the parameter space and the likelihood, not just the difference in parameter counts. Ask whether the null value is interior, whether nuisance parameters remain identified under the null, and whether the fitted estimates are finite. A coefficient that is unrestricted over the real line has an interior null value at zero; it is not a boundary test unless the model actually imposes a constraint such as nonnegativity.

Mistake: treating every one-parameter test as \(\chi_1^2\). A parameter constrained to be nonnegative and tested at zero can have the half-at-zero, half-\(\chi_1^2\) limit under the stated conditions.
Mistake: applying that mixture to every boundary problem. Multiple constraints or nuisance parameters unidentified under the null can produce different, more complicated limits.
Mistake: equating any deviance difference with the likelihood ratio. Establish the dispersion convention and use a common likelihood. Estimated-dispersion comparisons require the appropriate full likelihood calculation.
Mistake: assuming asymptotic validity guarantees a good finite-sample approximation. A formally appropriate limit can still be inaccurate with limited information; the test's assumptions and sample size both matter.

The practical lesson is to treat ordinary Wilks calibration as conditional, not automatic. When the null is an interior point and the regularity conditions hold, the usual chi-square result is appropriate. When a single regular parameter is tested at a one-sided boundary, the specific mixture derived here may apply. When identification, existence, or other regularity conditions fail, the reference distribution must be established for that problem rather than guessed from the number of parameters.

Check Your Understanding

For each question, identify the relevant parameter-space or regularity issue before choosing a calibration.

A coefficient is unrestricted over the real line and is tested at zero. Is zero a boundary point of the alternative parameter space? Explain.
Under the one-parameter boundary conditions, what is the limiting null distribution of the likelihood-ratio statistic, and why does it have an atom at zero?
A fixed-dispersion model has unit-deviance difference \(D_{\text{null}}-D_{\text{alt}}=5\) and known \(\phi=2\). What is the likelihood-ratio statistic?
In a mixture model, a component-specific mean has no effect on the distribution under the null mixing weight. Which regularity condition fails, and why is the simple boundary mixture not guaranteed?
Why should a deviance difference not automatically be used as a likelihood-ratio statistic when dispersion is estimated?