Test Disagreement in Small and Sparse Samples

Why Test Statistics Separate in Sparse Data

Wald, score, and likelihood-ratio tests are often introduced as three routes to a similar large-sample conclusion. Their agreement is approximate: it relies on a likelihood that is well behaved near the null and has enough information for a local quadratic approximation to be useful. With only a few successes or events, the likelihood may be asymmetric, the fitted value may be close to a boundary, or the unrestricted estimate may lie far from the null. In these settings, the three statistics can differ substantially.

The point is not that one formula is universally correct and the others are defective. They measure departure from the null in different ways. Wald uses the estimated parameter and curvature at the unrestricted fit; the score test uses the slope and curvature at the null; the likelihood-ratio test compares the maximized likelihoods. Sparse data can make those reference points and approximations matter.

A Binomial Model: Three Different Reference Points

Let \(Y\sim\operatorname{Binomial}(n,p)\), with observed count \(y\), and test \(H_0:p=p_0\), where \(0<p_0<1\). Write \(\hat p=y/n\). For an interior estimate \(0<y<n\), the canonical parameter is the logit \(\eta=\log\{p/(1-p)\}\), with null value \(\eta_0=\log\{p_0/(1-p_0)\}\). The Wald statistic below is specifically the Wald statistic for \(\eta\).

To derive these expressions, ignore the binomial coefficient, which is constant in \(p\), and write \(\ell(p)=y\log p+(n-y)\log(1-p)\). The score in the canonical parameter is \(U(\eta)=y-np\), and the information is \(I(\eta)=np(1-p)\). Thus the score statistic is \(U(\eta_0)^2/I(\eta_0)=n^2(\hat p-p_0)^2/\{np_0(1-p_0)\}\). At the unrestricted maximum, \(p=\hat p\), so the estimated variance of \(\hat\eta\) is \(1/\{n\hat p(1-\hat p)\}\); substituting it into the Wald formula gives \(W_\eta\). Finally, twice the log-likelihood difference between \(p=\hat p\) and \(p=p_0\) gives \(G^2\) above. This derives the three statistics and also shows why the probability-scale and logit-scale Wald statistics need not agree.

What Happens at a Boundary?

When \(y=0\) or \(y=n\), the binomial maximum-likelihood estimate is on the boundary of the probability parameter space. A finite logit estimate no longer exists. The score and likelihood-ratio statistics can still be calculated for an interior null value, but the ordinary canonical-logit Wald statistic cannot be formed from a finite unrestricted estimate and its standard error.

Sparse Poisson Counts and Extreme Fitted Means

The same issue arises in a Poisson model. Let \(Y\sim\operatorname{Poisson}(\mu)\), where \(\mu=E\lambda\) is the expected count for exposure \(E>0\) and rate \(\lambda>0\). To test a specified mean \(\mu_0>0\), use the canonical log-mean parameter \(\theta=\log\mu\). For an observed count \(y>0\), the unrestricted estimate is \(\hat\mu=y\), which may be far from \(\mu_0\) when counts are sparse.

Diagnosing Disagreement

A useful diagnosis asks where each statistic gets its information and how far the fit lies from the null. The Wald statistic relies on the unrestricted estimate and its local estimated variance. When the estimate is near a boundary, its standard error may be unstable or undefined; when curvature at the fit is weak, a large parameter displacement can produce a Wald statistic that poorly represents the shape of the likelihood elsewhere. The score statistic evaluates slope and curvature at the null, so its behavior can differ when the null is far from the maximum. The likelihood-ratio statistic uses the full log-likelihood difference, but its usual chi-square reference is also an asymptotic approximation.

The local equivalence of the three tests under regular large-sample conditions does not imply finite-sample identity. In particular, a small p-value from one statistic and a larger p-value from another should not be resolved by silently choosing the preferred answer. Report the test and parameterization, inspect the fitted and null values, and consider whether the asymptotic approximation is credible for the available data.

Common Mistakes

• Calling every Wald formula “the” Wald statistic. In the binomial example, the canonical-logit statistic is \(1.53745\), whereas the probability-scale statistic is \(2.8125\). State the parameter used for the Wald calculation.
• Treating boundary estimates as ordinary interior fits. With zero binomial successes, the fitted logit is not finite. Do not substitute it into an interior Wald formula and report a finite statistic.
• Assuming a finite statistic guarantees a reliable p-value. Score and likelihood-ratio statistics can be computed at a boundary, but the chi-square calibration remains approximate and may be poor in sparse samples.
• Expecting asymptotic equivalence to mean equal answers in small samples. Equivalence is a large-sample, local result. Sparse counts and large shifts from the null are precisely situations where its approximation can be weak.