Testing a Treatment Effect with Unequal Cluster Sizes

Why Unequal Cluster Sizes Matter

In a clustered study, participants within the same clinic, school, household, or other group may have correlated outcomes. A generalized linear mixed model (GLMM) represents this dependence through random effects. If clusters also have different numbers of observations, the treatment test must account for both the correlation and the uneven information contributed by clusters. Unequal sizes do not, by themselves, invalidate a test or change its degrees of freedom; they can affect precision, the influence of individual clusters, and how well large-sample approximations work.

This tutorial uses a binary outcome and a logistic random-intercept GLMM to make the calculations concrete. The same testing logic applies to other GLMM response distributions when the relevant regularity conditions hold. The treatment effect discussed here is a model-based association; a causal interpretation requires an appropriate treatment-assignment design and assumptions.

Definition: For clusters $i=1,\ldots,K$, let $m_i$ be the number of observations in cluster $i$, $Y_{ij}$ the binary outcome for observation $j$, $T_{ij}$ a treatment indicator, and $X_{ij}$ a vector of covariates. A logistic random-intercept GLMM is $$ Y_{ij}\mid b_i \sim \operatorname{Bernoulli}(p_{ij}),\qquad \operatorname{logit}(p_{ij})=\beta_0+\beta_T T_{ij}+\boldsymbol{\gamma}^{\mathsf T}X_{ij}+b_i,\qquad b_i\sim N(0,\tau^2). $$ Conditional on the random effects and covariates, outcomes are assumed independent. The random effects are independent across clusters and, under the usual model, independent of the covariates and treatment assignment. The treatment null hypothesis for a one-coefficient test is $H_0:\beta_T=0$; a two-sided alternative is $H_a:\beta_T\ne 0$.

The coefficient $\beta_T$ is a conditional log odds ratio: holding covariates and the cluster random intercept fixed, a one-unit increase in $T$ changes the log odds of the outcome by $\beta_T$. For a binary treatment coded 0 and 1, the corresponding conditional odds ratio is $\exp(\beta_T)$. It is not generally the same as a population-averaged odds ratio.

The cluster sizes $m_i$ may vary. The model and tests do not require them to be equal, but they do require a defensible account of the outcome dependence and the treatment design. In a cluster-randomized study, for example, treatment is constant within each cluster and the number of independently assigned clusters—not the total number of people—is central to the amount of independent treatment information.

Two Tests for a Single Treatment Coefficient

A Wald test uses the estimated coefficient and its estimated standard error. Under $H_0$, if the estimator is approximately normal with standard error estimated consistently, the ratio of the estimate to its standard error is approximately standard normal. Squaring it gives a one-degree-of-freedom chi-square statistic:

Z_W=\frac{\hat{\beta}_T}{SE(\hat{\beta}_T)},\qquad W=Z_W^2\ \dot{\sim}\ \chi^2_1.

The likelihood-ratio test compares the maximized likelihood under the alternative model with that under the null model. Both models must use the same outcome data and random-effects structure; the null model imposes $\beta_T=0$, while the alternative estimates $\beta_T$. Under regularity conditions, the statistic has an approximate chi-square distribution with degrees of freedom equal to the number of restrictions—in this case, one.

G^2=2\{\ell_{\text{alternative}}-\ell_{\text{null}}\}\ \dot{\sim}\ \chi^2_1.

Here $\ell$ denotes the maximized log likelihood. The likelihoods must be comparable marginal likelihoods for the GLMM, with nuisance parameters re-estimated in each model. The reference distributions are approximations: they rely on an adequate amount of independent information, an identifiable model, and a correctly specified likelihood. The treatment coefficient must be a regular, identifiable fixed-effect parameter. A one-coefficient test has one degree of freedom whether the observed cluster sizes are equal or unequal.

Worked Example: A Wald Test with Unequal Cluster Sizes

Suppose a hypothetical cluster-randomized trial has 40 clinics, 20 assigned to each treatment condition. Clinic sizes range from 8 to 31 patients. A logistic random-intercept GLMM, including prespecified baseline covariates, estimates the treatment coefficient as $\hat{\beta}_T=-0.42$, with $SE(\hat{\beta}_T)=0.18$. Assume the model converged, the random-intercept specification is plausible, and there are enough independent clinics for the large-sample approximation to be used cautiously.

For $H_0:\beta_T=0$ against $H_a:\beta_T\ne0$, the Wald statistic is $$ Z_W=\frac{-0.42}{0.18}=-2.3333,\qquad W=\left(\frac{-0.42}{0.18}\right)^2=5.444. $$

Using the approximate $\chi^2_1$ reference distribution gives a two-sided p-value of about 0.0196. At $\alpha=0.05$, this is evidence against a zero treatment coefficient under the fitted model. The estimated conditional odds ratio is $$ \exp(-0.42)=0.6570. $$

Thus, conditional on the included covariates and the clinic random intercept, the estimated odds of the outcome in the treated condition are about 0.657 times the odds in the control condition. This is not a statement that treatment caused a reduction unless the assignment and study design support a causal interpretation. The displayed test statistic uses the unrounded ratio $-0.42/0.18$; squaring the rounded display $-2.333$ instead gives approximately 5.443.

Worked Example: A Likelihood-Ratio Test

In a separate hypothetical study, 32 clinics have unequal sizes, ranging from 6 to 28 patients. The same covariates and random-intercept structure are used in both fitted models. The null model fixes the treatment coefficient at zero and has maximized log likelihood $\ell_{\text{null}}=-186.420$. The alternative model estimates that coefficient and has $\ell_{\text{alternative}}=-183.910$. Suppose the model is identifiable and the number of independent clusters makes a chi-square approximation reasonable for this illustration.

The alternative model has the larger maximized log likelihood, as expected because it includes the null model as a special case. The likelihood-ratio statistic is $$ G^2=2\{ -183.910-(-186.420)\} =2(2.510)=5.020. $$

Against $\chi^2_1$, this gives an approximate p-value of 0.0250. At the 5% level, the result is evidence against $H_0:\beta_T=0$, conditional on the stated model and approximation. The likelihood-ratio statistic tests whether adding the treatment coefficient improves fit; it does not by itself give the direction or size of the treatment effect. Those require the coefficient estimate and its interpretation.

Why the Cluster-Size Pattern Needs Attention

An unequal $m_i$ changes the data available to estimate the model, not the definition of $H_0$. A larger cluster may provide more information than a smaller one, but it does not generally provide a number of independent observations equal to its size: outcomes in that cluster are correlated through $b_i$. Consequently, the precision of $\hat{\beta}_T$ depends on the number of clusters, the size distribution, the degree of within-cluster dependence, and the treatment-assignment pattern.

A simple descriptive calculation shows why pooled rates and equally weighted cluster summaries can tell different stories. Suppose, purely for illustration, two control clusters have sizes 5 and 20 and event counts 0 and 8, while two treated clusters have sizes 5 and 20 and event counts 2 and 12. The pooled control event rate is $8/25=0.320$, and the pooled treated rate is $14/25=0.560$, a difference of $0.240$. Giving each cluster equal weight instead, the mean cluster rates are $(0/5+8/20)/2=0.200$ for control and $(2/5+12/20)/2=0.500$ for treatment, a difference of $0.300$. Neither calculation is a GLMM treatment estimate or test. They illustrate that a summary can implicitly give different weight to clusters, and that the estimand and analysis method matter.

For a GLMM, the marginal likelihood combines the within-cluster observations through an integral over the cluster random effect. A cluster with more observations can affect the likelihood more strongly, but the contribution is not equivalent to treating every patient as independent. If cluster size is related to the random effect, outcome process, or treatment in a way the model fails to represent, the usual analysis may not answer the intended question. This is sometimes called informative cluster size; the independence and design assumptions should be considered rather than presumed.

Worked Example: Translating a Test into a Treatment Interpretation

Consider a hypothetical study of 36 clinics with sizes from 7 to 35. A prespecified logistic random-intercept model estimates $\hat{\beta}_T=0.31$ with standard error 0.14. Assume treatment is coded 1 for the active condition, the model is identifiable and adequately fitted, and the independent-cluster count supports an approximate Wald reference distribution.

The signed Wald statistic and its square are $$ Z_W=\frac{0.31}{0.14}=2.2143,\qquad W=\left(\frac{0.31}{0.14}\right)^2=4.903. $$

The approximate $\chi^2_1$ p-value is 0.0268. At $\alpha=0.05$, the test provides evidence against a zero treatment coefficient. The estimated conditional odds ratio is $\exp(0.31)=1.363$, so the estimated odds are about 36.3% higher in the active condition, conditional on the model covariates and clinic random intercept. This percentage describes odds, not probability; it does not imply that the event probability rises by 36.3 percentage points.

Unequal clinic sizes do not call for changing $df=1$ to a value based on the number of patients. They do call for checking whether a few large clinics dominate the information and whether the model adequately describes the observed dependence. The p-value remains an approximation and should not be treated as exact merely because the arithmetic is straightforward.

Assumptions, Diagnostics, and Alternatives

Before interpreting either test, check the assumptions relevant to the fitted GLMM and its design:

Outcome and link specification: the response distribution and link should be appropriate for the outcome and scientific question.
Dependence structure: the random-effects structure should capture the important clustering. Conditional independence given the included random effects and covariates is part of the model.
Assignment and covariates: treatment coding, allocation level, and adjustment variables must match the design. If treatment is assigned by cluster, the cluster—not each member—is independently assigned.
Cluster-size process: inspect the size distribution and consider whether cluster size could be associated with treatment, latent risk, or outcomes in a way that matters for the target estimand.
Numerical fit and information: verify convergence and inspect whether estimates or standard errors are unstable. Many observations concentrated in very few clusters do not automatically provide strong fixed-effect test approximations.

A Wald test is quick and uses the coefficient estimate and standard error from one fit. It can be sensitive to how well the estimate is approximately normal and to the quality of its estimated standard error. A likelihood-ratio test directly compares the two nested models and can behave differently in finite samples; it requires fitting both models and comparable likelihoods. Neither test is guaranteed to be superior in every small or unbalanced data set.

Other approaches include a score test, which evaluates evidence for the treatment effect from the null fit, and analyses based on cluster-level summaries. Cluster-level analyses can be useful for some designs, but may discard individual-level information and target a different weighting or estimand. A method using cluster-robust standard errors is another possible approach, but it has its own requirements, particularly regarding the number of independent clusters. The choice should follow the design and inferential target, not a desire to obtain the smallest p-value.

Common Pitfalls

Counting patients as independent treatment replications. In a cluster-randomized design, many patients in a few clinics do not equal many independently randomized clinics. Account for the cluster structure and interpret small-cluster approximations cautiously.
Changing degrees of freedom because sizes differ. A test of one treatment coefficient has one restriction and hence a one-degree-of-freedom reference distribution under the stated regularity conditions. Unequal sizes affect information and approximation quality, not the count of restrictions.
Assuming the largest clusters should simply be ignored or weighted equally. Neither equal weighting nor patient-level pooling is automatically the correct model. State the estimand and use a model or design-based analysis suited to it.
Reading an odds ratio as a probability difference. The exponentiated logistic coefficient is a conditional odds ratio. Converting it to a probability change requires a baseline risk and model-based calculation.
Equating a small p-value with a large or causal effect. A p-value measures compatibility with the null model under assumptions; it is not an effect-size measure and does not establish causation.

Key takeaway: For a one-coefficient treatment test, unequal cluster sizes do not alter the one-degree-of-freedom Wald or likelihood-ratio reference test. They do affect the information and can expose design or model problems, so report the treatment estimate, its uncertainty, the cluster structure, and the assumptions alongside the p-value.

Check Your Understanding

Answer each question, then compare with the brief solutions.

A fitted model gives $\hat{\beta}_T=0.24$ and $SE(\hat{\beta}_T)=0.12$. Calculate the Wald statistic and state its approximate degrees of freedom.
A null model has log likelihood $-90.6$, and a nested alternative model has log likelihood $-88.8$. Calculate the likelihood-ratio statistic for one added treatment coefficient.
In a cluster-randomized trial with 15 clinics per treatment arm, does a one-coefficient test have 30 degrees of freedom because there are 30 clinics?
Why can the pooled event-rate difference differ from the difference between equally weighted mean cluster rates?

Solutions. (1) $Z_W=0.24/0.12=2.00$, so $W=2.00^2=4.00$, with an approximate $\chi^2_1$ reference distribution. (2) $G^2=2\{-88.8-(-90.6)\}=2(1.8)=3.60$, with one degree of freedom because one coefficient is added. (3) No. The test has one degree of freedom because it tests one restriction; the 30 clinics are relevant to independent information and approximation quality, not the restriction count. (4) The pooled rate weights observations, while the mean of cluster rates gives clusters equal weight; unequal cluster sizes therefore produce different summaries in general.