Why Finite-Sample Calibration Needs Attention
A likelihood-ratio test often compares its observed statistic with a chi-square distribution. That reference distribution is usually a large-sample approximation, not a guarantee about the statistic’s exact distribution for the data at hand. In a small or unusual sample, the approximation can give a misleading tail probability, even when the likelihoods and the test statistic have been calculated correctly.
The concern is especially relevant when there are few claims, several categories with small expected counts, or parameters that the observed data barely distinguish from one another. These conditions can make the likelihood surface depart substantially from the smooth, well-curved shape that supports the chi-square approximation. The result is not automatically that a test is invalid; rather, its nominal chi-square calibration deserves scrutiny.
The chi-square limit is most persuasive when the null parameter lies in the interior of a well-behaved model, the relevant parameters are identifiable, and the sample contains enough information for the log likelihood to be locally close to a quadratic function. Boundary cases and nuisance parameters absent under the null require special treatment; the ordinary chi-square reference should not be presumed for them. Even when the usual regularity conditions apply, small samples can leave substantial approximation error.
Why the Chi-Square Approximation Arises
The intuition is a local quadratic approximation to the log likelihood. Near the true parameter, a sufficiently regular log likelihood is approximately a constant plus a linear score term minus a quadratic curvature term. Maximizing under the null removes the directions allowed by the null, while maximizing in the larger model can also use the directions released by the restrictions. In a large sample, the standardized score in those \(q\) released directions is approximately multivariate normal, where \(q\) is the number of restrictions.
The likelihood-ratio statistic is then approximately the squared length of that \(q\)-dimensional standardized score. A sum of squares of \(q\) independent standard normal variables has a chi-square distribution with \(q\) degrees of freedom. This is the core reason for the asymptotic calibration.
The approximation depends on the quadratic description being adequate over the range of parameter values that matters. Sparse data can produce a jagged or highly asymmetric likelihood; weak identification can produce a nearly flat direction. In either case, the finite-sample statistic may not behave like the squared length of a standard normal vector. The approximation can be poor in either direction: a chi-square tail probability may be too large or too small.
Worked Example: A Small Binomial Sample
A hypothetical insurer reviews 8 independent policies, each with the same probability \(p\) of producing a specified type of claim. Test \(H_0:p=0.5\) against the larger model \(H_1:0<p<1\). Suppose 7 of the 8 policies produce the claim. Compare the chi-square approximation with the exact null tail probability for the likelihood-ratio statistic.
Method. Under \(H_1\), the maximum-likelihood estimate is \(\hat p=7/8=0.875\). Under \(H_0\), \(p=0.5\). The binomial coefficient is the same in both likelihoods, so it cancels in their ratio. Thus
The two log terms are \(7\log(1.75)=7(0.559616)=3.917311\) and \(\log(0.25)=-1.386294\). Therefore
There is one restriction, so the usual asymptotic reference is \(\chi^2_1\). Its upper-tail probability at 5.062032 is approximately 0.0245. This is a large-sample approximation, not an exact probability for this sample.
Under the null, \(X\), the claim count, has a \(\operatorname{Binomial}(8,0.5)\) distribution. The likelihood-ratio statistic is at least as large as the observed value when \(X\leq 1\) or \(X\geq 7\). The exact probability is
The exact tail is notably larger than the chi-square approximation in this example. The data are still compatible with \(H_0\) under the exact calculation; the key point is that the two calibrations give materially different tail probabilities. Common mistake: treating the chi-square value as an exact finite-sample answer simply because the statistic has the standard likelihood-ratio form. Here the exact binomial distribution is available and provides a direct check.
Sparse Claims and Small Expected Counts
In a multinomial likelihood-ratio goodness-of-fit test, the null specifies category probabilities and the alternative allows each category probability to vary. If the observed counts are \(O_j\) and the null expected counts are \(E_j\), the statistic is
A category with \(O_j=0\) contributes zero, using the limiting convention \(0\log(0/E_j)=0\). That convention makes the statistic computable; it does not make a sparse-data chi-square approximation reliable. When some expected counts are very small, the statistic’s finite-sample distribution can differ from its asymptotic reference. A rule such as “every expected count must be at least 5” is a useful warning heuristic in some settings, not a universal theorem guaranteeing accuracy above the cutoff or failure below it.
Worked Example: Sparse Multinomial Counts
A hypothetical claims study classifies 10 claims into four mutually exclusive severity bands. Under a proposed model, the band probabilities are \(0.7, 0.1, 0.1,\) and \(0.1\). The observed counts are \(8,1,1,\) and \(0\). Calculate the likelihood-ratio statistic and assess whether its asymptotic calibration is convincing.
Expected counts. Multiplying each null probability by 10 gives \(E=(7,1,1,1)\). The total is 10, matching the observed total.
Statistic. The two categories with observed count 1 contribute \(1\log(1/1)=0\), and the category with observed count 0 contributes zero by convention. The first category contributes
Against the saturated multinomial model, the usual asymptotic degrees of freedom are \(4-1=3\), because the four probabilities sum to one. However, three of the four expected counts are only 1. The fact that the statistic is easy to calculate does not establish that a \(\chi^2_3\) tail probability is accurate. The data provide very little information about the three low-probability bands individually.
For this small sample, a cautious analysis would report the sparsity and avoid presenting the asymptotic tail probability as definitive. If the null model is fully specified and the outcome space is manageable, its finite-sample null distribution can be evaluated directly. Alternatively, a simulation under the null can estimate the statistic’s reference distribution, provided the simulation reproduces the sampling scheme and the statistic is recomputed consistently. Common mistake: interpreting the zero cell as a computational error, or dropping it and silently changing the model. It contributes zero to this statistic, but remains relevant to the sparse-sample diagnosis.
Weak Identification: When the Data Barely Pin Down a Parameter
A parameter is weakly identified when plausible changes in its value have little effect on the distribution of the observed data. The likelihood then changes slowly in that parameter’s direction, so the data provide little curvature or information. Weak identification can occur even when the formal model is smooth and the parameter is not on a boundary. In such cases, large-sample approximations may require much more information than the nominal sample size suggests.
One diagnostic is the information remaining for a parameter after accounting for nuisance parameters. In a two-parameter model with parameters \(\alpha\) and \(\beta\), the information for \(\beta\) after allowing \(\alpha\) to vary is the Schur complement \(I_{\beta\beta}-I_{\beta\alpha}^2/I_{\alpha\alpha}\), when the information matrix is nonsingular. A value close to zero indicates that the data have little ability to separate changes in \(\beta\) from changes in \(\alpha\).
Worked Example: A Logistic Slope With Little Covariate Variation
Suppose 20 independent hypothetical policy outcomes \(Y_i\) are modeled as Bernoulli variables with \(\operatorname{logit}(P(Y_i=1))=\alpha+\beta x_i\). Ten policies have \(x_i=0.99\) and ten have \(x_i=1.01\). Assess the information available for the slope \(\beta\) at \(\alpha=\beta=0\), where all fitted probabilities are 0.5.
Method. For a Bernoulli logistic model, the expected information matrix is the sum of \(p_i(1-p_i)(1,x_i)^\mathsf{T}(1,x_i)\). At the stated parameter values, each weight \(p_i(1-p_i)=0.25\). The sums are \(\sum 1=20\), \(\sum x_i=10(0.99)+10(1.01)=20\), and \(\sum x_i^2=10(0.99^2)+10(1.01^2)=10(0.9801)+10(1.0201)=20.002\). Therefore
After allowing the intercept \(\alpha\) to vary, the information for \(\beta\) is
The corresponding local standard-error scale for the slope is approximately \(1/\sqrt{0.0005}=44.721\). This is a diagnostic, not a confidence interval or a likelihood-ratio result. It shows that the narrow range of \(x\) values makes the slope hard to distinguish from an intercept change. A chi-square approximation for a test involving \(\beta\) should therefore be treated cautiously unless there is convincing evidence that the available information is sufficient. Common mistake: deciding that 20 observations must be enough solely from the sample size. Information depends on the design and the model, not just the number of records.
A Cautious Calibration Workflow
When a likelihood-ratio test uses a chi-square reference, the analyst should distinguish the numerical calculation of \(G^2\) from the separate question of calibration. A correctly computed statistic can still have an inaccurate asymptotic tail probability. Use the following checks before placing weight on that probability.
Check that the comparison is nested, the restrictions are correctly counted, and the model conditions support the claimed chi-square limit. Do not substitute the ordinary chi-square reference for a known boundary or nonregular case.
Review claim counts, category frequencies, fitted probabilities, parameter estimates, and likelihood curvature. A large nominal sample can still provide little information about a particular parameter.
When feasible, use an exact calculation or simulate data under the null and recompute the same statistic in each replicate. State the method and its limitations; simulation error and model assumptions still matter.
A parametric simulation should generate observations under the null model, preserve the actual sampling design, and refit the restricted and unrestricted models for each replicate. Comparing the simulated statistics with the observed statistic gives a finite-sample-oriented calibration under the assumed null. It is not automatically a cure: a misspecified null, numerical fitting problems, or a nonregular model can still undermine the result.
For an actuarial conclusion, report what the test establishes and what it does not. A small asymptotic tail probability is not compelling evidence if the approximation is visibly doubtful; a larger one does not prove the null model is correct. Where exact or simulated calibration is unavailable, explain the finite-sample limitations and interpret the chi-square result as approximate. These distinctions are aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.
Check Your Understanding
For each problem, distinguish calculation of the statistic from confidence in its chi-square calibration.
- For 6 independent hypothetical policies, test \(H_0:p=0.5\) against an unrestricted binomial probability after observing 5 claims. Calculate \(G^2\) and the exact null probability of a likelihood-ratio statistic at least this large. (Answer: \(G^2=2.9110\); exact tail probability \(14/64=0.21875\).)
- A multinomial null model assigns probabilities \(0.6,0.2,0.1,0.1\) to four bands, and a sample of 20 has counts \(15,3,2,0\). Give the expected counts and state the usual degrees of freedom against the saturated model. (Answer: expected counts \(12,4,2,2\); 3 degrees of freedom; the small expected counts counsel caution.)
- A hypothetical binomial experiment has \(n=8\), \(p=0.5\) under the null, and observed count \(x=6\). Calculate the LR statistic against an unrestricted \(p\), and the exact null probability of an LR statistic at least that large. (Answer: \(G^2=2.09299\); exact tail probability \(74/256=0.2890625\).)
- In a logistic model with 12 observations at each of \(x=2.98\) and \(x=3.02\), evaluate the efficient information for the slope at fitted probabilities 0.5, allowing an intercept. (Answer: \(I_{\beta\beta\cdot\alpha}=0.25(24)(0.02^2)=0.0024\); this indicates weak slope information.)
- A claim-band likelihood-ratio test has 80 total observations, but two categories have null expected counts 0.8 and 1.2. Is the sample total alone enough to justify an asymptotic chi-square tail probability, and name one possible check. (Answer: No; inspect the sparse cells and, if feasible, calculate or simulate the null distribution while preserving the sampling design.)