Why the Number of Clusters Matters
A GLMM can contain many observations but only a few independent units of information about a random-effect variance. If patients are grouped in clinics, for example, measurements from the same clinic share a cluster-level effect. Adding patients within a clinic may improve information about that clinic’s outcomes, but it does not create additional independent clinics. For random-effect tests, the number and arrangement of clusters can therefore matter more than the total observation count.
This issue is especially important when testing whether a variance component is zero. Zero lies on the boundary of the allowed parameter space, so the usual regular large-sample reference distribution for a likelihood-ratio statistic need not apply. A boundary-adjusted mixture distribution is often the relevant asymptotic reference under particular conditions. But “asymptotic” refers to an increasing-information limit, commonly with the number of independent clusters growing. With few clusters, that approximation can be poor even when the total number of observations looks large.
The comparison must be specified correctly before interpreting \(D\). The models should have the same fixed-effects structure when the target is a random-effect variance, and should differ in the variance component being tested. Use the same likelihood criterion for both fits. For a restricted likelihood comparison, the fixed-effects structure must be the same. The null model must also be fitted with the tested variance constrained to zero; a fit that allows a negative variance is not the constrained null model.
What the Mixture Reference Does—and Does Not—Say
For a single variance component, under suitable regularity conditions and with sufficiently many independent clusters, the limiting null distribution is often a mixture with probability one-half at zero and probability one-half from a chi-square distribution with one degree of freedom. This is not the ordinary chi-square distribution. It reflects the fact that the variance cannot go below zero, so the likelihood-ratio statistic has a point mass at zero in the limit.
The mixture result requires more than merely fitting a GLMM. It presumes a correctly specified model, independent clusters, an identifiable variance component under the alternative, and nuisance parameters that behave regularly under the null. The simple one-half/one-half mixture does not automatically apply to every random-effect test; multiple components, correlations, or nuisance parameters that are not identified under the null can change the reference distribution. Even where the correct asymptotic mixture is known, a small \(m\) can make its finite-sample accuracy uncertain.
A Short Derivation of the Mixture Tail
The notation \(\chi^2_0\) here represents a point mass at zero. For a positive observed statistic \(d\), that point mass contributes no probability to the event that the statistic is at least \(d\). Only the chi-square component contributes, and it has mixture weight one-half. Thus,
At \(d=0\), the upper-tail probability includes the mass at zero as well as the chi-square component, giving one. The mixture adjustment is therefore not a rule to halve every p-value in every boundary problem. It follows from this particular limiting distribution and applies to positive statistics under its assumptions.
Worked Example: Reading a Mixture-Reference p-Value
Suppose a hypothetical study has \(m=6\) independent clinics and compares two otherwise matching models for a single random-effect variance. The observed likelihood-ratio statistic is \(D=2.40\). Assume the one-half/one-half mixture is the selected asymptotic reference for this test. Calculate its p-value and describe what the small number of clinics changes.
For a one-degree-of-freedom chi-square variable, the upper-tail probability at 2.40 is approximately 0.121. Therefore,
This is the asymptotic mixture-reference p-value, approximately 0.061. At \(\alpha=0.05\), this calculation does not reject the null. It does not establish that the variance is zero, and it does not mean there is a 6.1% probability that the null hypothesis is true. It measures how often a statistic at least this large would occur under the assumed null reference distribution.
There are only six independent clinics, so the calculation’s main limitation is not its arithmetic: it is the uncertainty about how well the limiting mixture approximates the finite-sample null distribution for this design. The p-value should be reported as an asymptotic result, not as a precisely calibrated finite-sample probability.
Why Few Clusters Can Weaken Calibration
Likelihood-based asymptotic results rely on the accumulation of information across independent units. When \(m\) is small, the realized cluster effects and cluster-level covariate patterns can have an unusually large influence on the fitted variance. The null distribution may consequently depend more strongly on the design and nuisance-parameter values than the simple limiting reference suggests. Unequal cluster sizes, sparse binary outcomes within clusters, and covariates that vary little between clusters can make that concern more acute.
A large total sample size does not remove this problem by itself. For example, 500 observations collected as 100 observations in each of five clinics still represent only five independent clinic-level units. The observations within each clinic can help estimate within-clinic response patterns, but they do not provide 500 independent replications of the clinic-level random effect.
Worked Example: Same Total, Different Cluster Information
Consider two hypothetical studies with 600 binary observations. Study A has 6 independent clinics with 100 observations per clinic. Study B has 60 independent clinics with 10 observations per clinic. Suppose both test a clinic-level random-intercept variance using otherwise comparable models.
Study A: \(m=6\) independent clusters and \(n=6(100)=600\) observations. There are six separate clinic-level units contributing information about between-clinic variation.
Study B: \(m=60\) independent clusters and \(n=60(10)=600\) observations. There are sixty separate clinic-level units. The same total number of observations is distributed across ten times as many independent clusters.
The arithmetic confirms equal totals, but not equal information about the random-intercept variance. Study B generally offers a more credible setting for a large-\(m\) approximation because the number of independent clusters is much larger. This comparison is not a guarantee that Study B has greater precision in every respect: outcomes, covariates, cluster-size balance, and the fitted model also matter. It shows why reporting only \(n\) can conceal the feature most relevant to this test.
Calibration Options and Their Limits
When the cluster count is small, treat the asymptotic mixture as a reference to scrutinize rather than an automatic final answer. Finite-sample calibration methods and parametric bootstrap likelihood-ratio tests are relevant alternatives. A null parametric bootstrap simulates data from the fitted null model, refits the null and alternative to each simulated data set under the same constraints, and compares the simulated statistics with the observed statistic. This approximates a null distribution tailored to the fitted model and design.
The bootstrap is not an exact repair. With few clusters, estimates of nuisance parameters under the null can themselves be unstable, and a parametric bootstrap inherits the assumptions of the fitted model. A small number of bootstrap replicates also adds Monte Carlo error. The comparison below summarizes the main trade-offs.
| Approach | Potential value | Important limitation with few clusters |
|---|---|---|
| Asymptotic mixture reference | Fast and interpretable when its boundary conditions apply and the large-cluster approximation is adequate | May be poorly calibrated in the finite sample; the number of clusters is not large merely because \(n\) is large |
| Null parametric bootstrap | Uses the fitted null model and observed design to approximate a finite-sample reference | Depends on model and null nuisance estimates; simulation error and model misspecification remain |
| Wald test for a variance | Can appear computationally simple | Ordinary symmetric standard-error reasoning is inappropriate at a boundary and may give misleading results |
Worked Example: Comparing Bootstrap and Mixture Calibration
Return to the hypothetical statistic \(D=2.40\). Suppose a null parametric bootstrap with \(B=999\) simulated data sets produces 43 statistics at least as large as 2.40. Using the plus-one estimate,
The bootstrap estimate is 0.044, compared with the mixture-reference value of about 0.061. At a nominal 0.05 level, the two calculations fall on opposite sides of the threshold. The bootstrap result is not proof that the asymptotic result is wrong: both are approximations, and their difference signals that the conclusion is calibration-sensitive.
The bootstrap estimate also has Monte Carlo uncertainty. A rough Monte Carlo standard error is
That uncertainty is large enough that 0.044 should not be treated as a sharply determined value. More replicates can reduce simulation error, but cannot eliminate uncertainty from having few observed clusters, unstable nuisance estimates, or a misspecified model. A responsible report would state the observed statistic, both calibration methods, the bootstrap settings, and the sensitivity of the conclusion.
Diagnostics, Pitfalls, and Reporting
Before interpreting a boundary-test p-value, check that the independent-cluster count is explicit and that the tested models differ only as intended. Verify that the variance constraint was enforced, the likelihood fits converged, and the estimated model is not being driven by one or two clusters. Review cluster sizes, outcome sparsity, and the distribution of relevant covariates across clusters. These checks do not prove the reference distribution is accurate, but they can reveal settings in which uncritical asymptotic inference is especially risky.
If a null parametric bootstrap is used, verify that simulated data are generated under the constrained null and that each replicate repeats the same fitting and testing procedure. Check for failed or boundary fits and report how they were handled. A bootstrap that silently omits difficult replicates may change the reference distribution. Ordinary resampling of individual observations is not a substitute: it breaks the within-cluster structure. Resampling whole clusters can also be unstable when there are very few clusters.
A common mistake is to argue, “There are hundreds of observations, so the chi-square mixture must be reliable.” The correction is to distinguish \(n\), the total observation count, from \(m\), the number of independent clusters, and assess the approximation with the cluster-level design in view. Another mistake is to interpret a non-significant test as evidence that the random-effect variance is exactly zero. A small-cluster test may have weak information or imperfect calibration; failure to reject does not establish the null.
In reporting, give the estimated variance and its units squared, the number of independent clusters, the likelihood-ratio statistic, the reference distribution, and the p-value. If results differ under an available finite-sample calibration, explain that sensitivity instead of selecting whichever p-value supports a preferred conclusion. The test addresses evidence against a zero-variance null under the specified model; it does not by itself establish the practical importance of the random effect or validate the model.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A study has 800 observations from 8 independent wards. Which count is most directly relevant to the large-cluster approximation for a ward-level variance test, and why?
- Under a valid one-half/one-half mixture reference, an observed statistic is \(d=0\). What is the upper-tail probability, and why is it not one-half?
- A random-effect test has a mixture-reference p-value of 0.03 and a null-bootstrap estimate of 0.08. What is a defensible interpretation?
- Why is resampling individual observations generally not an adequate bootstrap strategy for clustered data?
Solutions. (1) The relevant count is \(m=8\), because the ward-level effects are replicated across eight independent wards; 800 observations do not represent 800 independent ward effects. (2) The upper-tail probability is 1: at \(d=0\), all of the point mass at zero is included, along with the continuous component. (3) The inference is calibration-sensitive. Report both methods and their assumptions, investigate the model and bootstrap fits, and avoid presenting either p-value as definitive. (4) Individual resampling breaks the dependence structure within wards and therefore does not reproduce the clustered null distribution being tested.