Boundary Parameters And Nonstandard Likelihood Inference

Why a Boundary Changes Likelihood Inference

Likelihood methods often rely on a local approximation in which the parameter can vary a little in either direction around its true value. A variance component cannot do that at zero: negative variance is not allowed. When the true component is zero, the parameter sits at the edge of its permissible values, and the usual likelihood approximations can change. This matters in practice because fitting a variance-component model can produce an estimate exactly equal to zero, and testing whether a component is needed puts the null hypothesis at that same edge.

The distinction is between an interior estimate, which is strictly inside the parameter space, and a boundary estimate, which lies on its edge. A boundary estimate is not automatically a computational failure, nor is it proof that the corresponding population component is exactly zero. It is an estimate obtained under a constraint, and its inferential consequences depend on whether the true value is at the boundary or in the interior.

Definition: A parameter value is an interior point of its parameter space if sufficiently small changes in every direction remain permissible. It is a boundary point if it is permissible but arbitrarily small changes in at least one direction are not. For a variance component \(\tau\) constrained by \(\tau\geq 0\), every value \(\tau>0\) is interior and \(\tau=0\) is a boundary point.

A Variance-Component Model

Consider a random-intercept model for \(J\) independent clusters, with \(m\) observations in each cluster:

Y_{ij}=x_{ij}^{\mathsf T}\beta+u_j+\varepsilon_{ij}, \qquad u_j\sim N(0,\tau), \qquad \varepsilon_{ij}\sim N(0,\sigma^2).

Here \(i=1,\ldots,m\) indexes observations within cluster \(j=1,\ldots,J\); \(\beta\) is a vector of regression coefficients; \(\tau\geq 0\) is the variance of the cluster-level random intercept; and \(\sigma^2>0\) is the residual variance. Assume the random intercepts are independent across clusters, the residuals are independent across observations, and the two sets of random variables are independent. The regression specification and the Gaussian distributions are assumed correct. With at least two observations per cluster, the shared covariance induced by \(u_j\) can distinguish \(\tau\) from \(\sigma^2\); the component is then identifiable under suitable design conditions.

When \(\tau=0\), the random intercept has no between-cluster variation, and the model reduces to one without that random effect. To test for this component, compare \(H_0:\tau=0\) with \(H_a:\tau>0\). The null is not an interior point of the allowed parameter space: it is exactly its boundary. For the standard one-component result below, assume that the other parameters are identifiable and regular under the null, that the information for \(\tau\) after accounting for those nuisance parameters is positive, and that the number of independent clusters grows. These conditions rule out cases where the component cannot be distinguished from other model features.

Worked Example: Constraining a Negative Unrestricted Estimate

Suppose a hypothetical fitted model has a profile log-likelihood for \(\tau\) that is locally represented by

\ell_p(\tau)=C-\frac{(\tau+0.12)^2}{2(0.08)^2}, \qquad \tau\geq 0,

where \(\tau\) is measured in squared outcome units and \(C\) is a constant. Without the constraint, this quadratic is maximized at \(\tau=-0.12\), because its squared term is zero there. That value is impermissible for a variance component. Over the allowed values \(\tau\geq 0\), the squared distance \((\tau+0.12)^2\) is smallest at \(\tau=0\). Thus the constrained maximum-likelihood estimate is \(\hat{\tau}=0\).

This is a boundary estimate: the likelihood is still increasing as \(\tau\) moves toward zero from the unrestricted peak, but the peak itself lies outside the allowed parameter space. The estimate does not mean that the data establish a population variance of exactly zero. It says that, among permissible values, zero gives the largest likelihood in this fit.

Why the Usual Likelihood-Ratio Limit Changes

For a regular scalar parameter at an interior null value, the likelihood-ratio statistic has the familiar asymptotic chi-square distribution with one degree of freedom. At a boundary, the local likelihood is instead maximized over a half-line. The maximizing value is therefore either the unrestricted local estimate, if it points into the allowed region, or the boundary itself, if it points out of that region.

The key geometric step can be seen in the one-parameter local approximation. Under \(H_0:\tau=0\), let \(Z\) denote the standardized, efficient score for moving in the direction of increasing \(\tau\). Under the regularity conditions stated above, \(Z\) is asymptotically standard normal. In the local quadratic likelihood, the unrestricted standardized maximizer is \(Z\), but the parameter is constrained to be nonnegative. The constrained maximizer is therefore \(\max(0,Z)\). For the likelihood-ratio statistic \(Q=2\{\ell(\hat{\tau})-\ell(0)\}\), the quadratic improvement is the square of that constrained maximizer:

Q \ \overset{d}{\longrightarrow}\ [\max(0,Z)]^2.

If \(Z\leq 0\), the local likelihood is maximized at the boundary, so \(Q=0\). If \(Z>0\), then \(Q=Z^2\). A standard normal variable is positive with probability one half, and its square has a chi-square distribution with one degree of freedom. This gives the boundary limit:

Q \ \overset{d}{\longrightarrow}\ \frac{1}{2}\,\delta_0+\frac{1}{2}\,\chi^2_1,

where \(\delta_0\) denotes a point mass at zero. In words, the limiting distribution places probability one half at zero and uses a chi-square distribution with one degree of freedom for the other half. The atom at zero represents samples whose local likelihood improvement points outside the permissible region, so the constrained optimum remains at the null value.

Theorem: For a single identifiable variance component tested at zero, under regularity of the nuisance parameters and the one-sided local quadratic approximation, the likelihood-ratio statistic has the asymptotic null distribution \(\frac{1}{2}\delta_0+\frac{1}{2}\chi^2_1\), rather than an ordinary \(\chi^2_1\) distribution.

Worked Example: Calibrating a Boundary Likelihood-Ratio Test

Suppose a hypothetical model comparison for \(H_0:\tau=0\) against \(H_a:\tau>0\) gives \(Q=2.25\). Use the one-component boundary limit to calculate the approximate p-value.

For a positive observed statistic \(q\), the mixture tail probability is one half of the \(\chi^2_1\) upper-tail probability:

P(Q\geq q)\approx \frac{1}{2}P(\chi^2_1\geq q).

Since a \(\chi^2_1\) variable is the square of a standard normal variable, \(P(\chi^2_1\geq 2.25)=P(|Z|\geq 1.5)=2[1-\Phi(1.5)]\). Using \(\Phi(1.5)\approx 0.9332\),

p\approx \frac{1}{2}\{2(1-0.9332)\}=0.0668.

At the 5% significance level, this p-value is not small enough to reject \(H_0\). Under the stated model and the boundary approximation, the result is not strong evidence that the population cluster-level variance is positive. For comparison, treating \(Q\) as an ordinary \(\chi^2_1\) statistic would give \(p\approx 0.1336\), twice the boundary-calibrated value. The reference distribution must match the boundary problem.

Interior Estimates and Ordinary Approximations

If the true variance component is strictly positive and stays away from zero as the sample grows, the parameter is locally free to move in either direction. Under the usual likelihood regularity conditions, the constrained maximum is then asymptotically interior, the estimator is approximately normal, and the ordinary chi-square likelihood-ratio approximation can apply. A Wald interval based on an estimated standard error can also be reasonable in that interior regime.

This is a statement about the true parameter and the asymptotic regime, not a guarantee based only on seeing a positive estimate in one sample. When the true value is zero, estimates can accumulate at zero, and a symmetric normal approximation does not represent that pile-up correctly. When the true value is positive but very close to zero, finite-sample behavior can still be strongly affected by the boundary.

Worked Example: An Interior Profile Estimate

Suppose a second hypothetical fit produces the local profile log-likelihood

\ell_p(\tau)=C-\frac{(\tau-0.24)^2}{2(0.08)^2}, \qquad \tau\geq 0.

Its maximum occurs at \(\hat{\tau}=0.24\), which is inside the allowed parameter space. The quadratic curvature corresponds to an approximate standard error of \(0.08\) squared outcome units. If the true component is positive and the model is in a regime where interior likelihood approximations are appropriate, the Wald statistic for testing \(\tau=0\) is

z=\frac{0.24-0}{0.08}=3.00.

A two-sided normal-based 95% interval is

0.24\pm 1.96(0.08) =[0.0832,\ 0.3968].

Both endpoints are in squared outcome units, the units of a variance component. This interval illustrates the interior approximation, but it should not be used as a general recipe when the true value may be zero: the normal approximation can extend into impermissible negative values, and truncating such an interval does not by itself fix the boundary distribution.

Practical Consequences and a Common Pitfall

A variance-component estimate at zero has several practical implications. It is a legitimate constrained maximum, but it may make a conventional standard-error summary or symmetric Wald interval uninformative. For a test of a single component equal to zero, the likelihood-ratio reference distribution must account for the boundary. Other boundary problems can have different limits: with multiple constrained components, the mixture weights need not be one half, and parameters that are not identified under the null can invalidate the simple result. The one-component mixture should therefore be used only when its assumptions fit the model.

Common mistake: “The estimate is zero, so the component is certainly absent; alternatively, the likelihood-ratio statistic can be compared with an ordinary \(\chi^2_1\) because one parameter was tested.” Both claims overlook the constraint. A zero estimate can arise because the unrestricted likelihood peak falls below zero, while the model only permits nonnegative values. And the null is a boundary point, so ordinary interior-point calibration does not generally apply. State the model and constraint, check the relevant regularity conditions, and then use the boundary reference distribution when appropriate.

Key takeaway: A nonnegative variance component has a boundary at zero. Under the standard one-component conditions, testing that boundary yields a likelihood-ratio limit of \(\frac{1}{2}\delta_0+\frac{1}{2}\chi^2_1\). Strictly positive interior values can support ordinary likelihood approximations, but boundary estimates require boundary-aware interpretation.

Check Your Understanding

Answer each question by referring to the constraint and the relevant asymptotic conditions.

For a variance component constrained to be nonnegative, which values are interior points and which value is the boundary?
Why can a constrained maximum-likelihood estimate equal zero even when the unrestricted local likelihood peak is negative?
What are the point mass and continuous components in the one-component boundary likelihood-ratio limit?
A boundary likelihood-ratio statistic is \(Q=1.00\). Write the mixture-tail formula for its approximate p-value in terms of a \(\chi^2_1\) tail probability.
Why does observing a positive estimate in one sample not, by itself, establish that ordinary interior asymptotics are appropriate?