Finite-Sample Calibration of Boundary Tests

Why Check a Boundary Test in Finite Samples?

A likelihood-ratio test for a random-effect variance is often calibrated using an asymptotic mixture distribution. For one variance component tested at zero under suitable conditions, the reference law is a half-at-zero, half-\(\chi^2_1\) mixture. That result describes a limiting distribution; it does not guarantee that the test has its nominal error rate for every finite data set. The gap can matter when the number of independent clusters is small, even if each cluster contributes many observations.

Here, finite-sample calibration means how closely a test’s actual null rejection probability in the specified finite design matches its nominal significance level. If a test is used at \(\alpha=0.05\), good calibration means it rejects about 5% of repeated data sets generated under the null. A test that rejects substantially more often is anti-conservative; one that rejects substantially less often is conservative.

The asymptotic mixture law is motivated by the geometry of a variance parameter constrained to be nonnegative. Its accuracy depends on how well the finite data approximate the limiting conditions behind that law. In a generalized linear mixed model (GLMM), information about between-cluster variation is strongly tied to the number of independent clusters. More observations within the same few clusters do not necessarily supply the same information as more independent clusters.

What the Asymptotic Reference Assumes

Consider a null model with a random-intercept variance \(\tau^2=0\), compared with a model allowing \(\tau^2\geq 0\). Let \(\ell_0\) and \(\ell_1\) be the maximized log likelihoods under the null and alternative models, respectively. The likelihood-ratio statistic is \(T=2(\ell_1-\ell_0)\). For a single variance component, under the standard asymptotic conditions, its null reference law is

Here \(\chi^2_0\) denotes a point mass at zero. For a positive threshold \(c\), this mixture has upper-tail probability \(\tfrac{1}{2}P(\chi^2_1\geq c)\). Thus its 5% critical value is approximately 2.706: the corresponding \(\chi^2_1\) upper-tail probability is 0.10. This reference is not an ordinary \(\chi^2_1\) law, nor is it a guarantee of exact finite-sample size.

The approximation relies on a correctly specified likelihood and the relevant asymptotic regularity conditions. The tested variance must be on the boundary as described, the compared models must be nested and fitted consistently, and nuisance parameters must behave regularly enough for the limiting result to apply. The particular mixture may also depend on the geometry of the tested parameters and on nuisance parameters; the half-and-half law should not be substituted automatically for every boundary test.

With few clusters, finite-sample likelihoods can be asymmetric or irregular, estimates may pile up at zero, and nuisance parameters may be weakly informed. The statistic’s null distribution can then differ appreciably from its limiting mixture. There is no universal finite-sample correction based only on the cluster count: the link, outcome distribution, cluster-size pattern, fixed-effects design, and nuisance-parameter values can all matter.

Using Null Simulations to Assess Calibration

A simulation study for calibration generates repeated data sets under a specified null model and design, fits the null and alternative models to each, and applies the same test rule each time. It then estimates the null rejection probability by the fraction of simulated tests that reject. This is evidence about the test under the specified conditions; it does not establish calibration for every GLMM or every nuisance-parameter setting.

If \(R\) of \(M\) independent simulated null data sets reject, the estimated rejection rate is \(\hat{q}=R/M\). Because \(R\) is binomial with rejection probability \(q\), the Monte Carlo standard error of \(\hat{q}\) is estimated by

This standard error measures simulation noise, not uncertainty in the model parameters and not the sampling standard error of an estimate from the original study. An approximate 95% interval for the simulation rejection rate is \(\hat{q}\pm1.96SE(\hat{q})\), though an exact or binomial-based interval can be preferable when \(M\) is small or the rate is near zero or one.

Check More Than One Threshold

A single estimated rejection rate is informative at its chosen \(\alpha\), but assessing calibration across several nominal levels can show whether the discrepancy is confined to one part of the tail. For each chosen \(\alpha\), apply the corresponding mixture critical value to every simulated statistic and estimate the rejection rate. The observed rates should be interpreted alongside their Monte Carlo uncertainty.

A useful graphical check plots empirical rejection rates against nominal levels and includes the diagonal line where the two are equal. A curve consistently above the diagonal suggests anti-conservative behavior at those levels; a curve below suggests conservatism. Monte Carlo intervals or error bars matter: small departures from the diagonal may be simulation noise, especially when the number of replicates is limited.

Why Cluster Count and Design Matter

Under the null of zero random-intercept variance, the data may look like independent observations conditional on the fixed effects. Yet the alternative model estimates a between-cluster variance using the pattern of outcomes across clusters. With only a few clusters, that pattern is based on few independent units. The likelihood can consequently have finite-sample behavior unlike the limiting shape that leads to the mixture law.

Diagnostics, Alternatives, and Limitations

Before attributing a discrepancy to the asymptotic reference law, check that the simulation and fitting process implement the intended test. Verify that the null data-generating model, fixed design, cluster sizes, and nuisance-parameter values are recorded; that the null and alternative models differ only as intended; and that the statistic is calculated as \(2(\ell_1-\ell_0)\). Review convergence warnings, boundary estimates, and whether the same likelihood convention is used for both fits. Failed fits or inconsistent model comparisons can distort rejection rates independently of the reference distribution.

Compare the distribution of simulated statistics with the mixture reference, not only the average statistic. In particular, examine how often the fitted variance is zero, how often \(T=0\) or is very small, and how often statistics exceed the mixture critical values. A markedly different mass near zero or an excess of large values can help explain a poor rejection rate. Also consider repeating the assessment at plausible nuisance-parameter settings: a single simulation setting may not represent the range relevant to an application.

The main asymptotic alternative is the appropriate chi-bar-square mixture. It is principled when its assumptions and mixture weights apply, but it may be inaccurate at the finite sample size of interest. An ordinary \(\chi^2_1\) reference is not a generic repair: for the single-variance case it ignores the boundary mixture and has a different upper tail. Finite-sample simulation can diagnose the approximation, but its evidence is conditional on the specified model and design. It cannot protect against a misspecified random-effects distribution, an incorrect link, omitted dependence, or a simulation design that fails to match the intended analysis.

Common Pitfall

Wrong reasoning: “The test uses the correct half-and-half mixture, so its type I error rate must be exactly 5% in my data.” The mixture result is asymptotic; the finite-sample null distribution can depart from it, especially with few independent clusters or weakly informed nuisance parameters.

Correction: Treat the nominal level as the target, not a guarantee. Check the actual finite-sample rejection rate in relevant null simulations, report the number of replicates and Monte Carlo uncertainty, and investigate fitting failures before concluding that the reference law is poorly calibrated.