Common Chi-Square Goodness-of-Fit Errors

Introduction: Check the Setup Before the Statistic

The chi-square goodness-of-fit statistic is straightforward to calculate once the observed and expected counts are right. The hard part is often checking that those counts correspond to the sample actually observed, that the degrees of freedom reflect how the model was fitted, and that the categories were not selected in a way that depends on the observed results. A polished calculation using an incorrect setup is still an incorrect test.

This tutorial focuses on four errors: using unconditional probabilities for a conditional sample, applying a degrees-of-freedom rule without checking how parameters were estimated, leaving categories with sparse expected counts, and regrouping only after inspecting the observed counts. The earlier material on expected counts and combining sparse categories supplies the basic test framework; here the emphasis is on diagnosing and correcting exam-level traps.

Definition: For $n$ independent observations allocated among $k$ mutually exclusive and exhaustive categories, let $O_j$ be the observed count and $E_j=n p_j$ the expected count under the null model, where $p_j$ is the null probability of category $j$. The Pearson chi-square goodness-of-fit statistic is $$ X^2=\sum_{j=1}^{k}\frac{(O_j-E_j)^2}{E_j}. $$ The expected counts must be based on the distribution of the observations as sampled, and the categories must cover that sample.

First Error: Expected Counts for the Wrong Population

A category probability is not automatically the probability to use in $E_j=np_j$. It must be the probability under the actual sampling scheme. If observations are sampled conditional on exceeding a threshold, use conditional probabilities for categories within that truncated sample. Unconditional probabilities may omit observations that could not have entered the sample, causing the expected counts not to sum to $n$.

Worked Example: Losses Recorded Only Above a Threshold

A hypothetical portfolio records 80 losses, each sampled conditional on $X>5$. Assume the loss model is exponential with rate $\lambda=0.2$ per dollar, so $S_X(x)=e^{-0.2x}$. The categories are $(5,10]$ and $(10,\infty)$, with observed counts 58 and 22. Calculate the chi-square statistic using the correct expected counts. Identify the error in using unconditional probabilities.

Method and model. The exponential model has $P(X>a)=e^{-0.2a}$. Because the sample is conditional on $X>5$, the category probabilities must be conditional on that event. The two stated categories exhaust the sampled losses.

For the first category, $$ P(5<X\leq 10\mid X>5) =\frac{P(5<X\leq10)}{P(X>5)} =\frac{e^{-1}-e^{-2}}{e^{-1}} =1-e^{-1} \approx 0.6321. $$ For the second category, $$ P(X>10\mid X>5) =\frac{e^{-2}}{e^{-1}} =e^{-1} \approx 0.3679. $$ The probabilities sum to $1$, as required. Therefore the expected counts are $$ E_1=80(0.6321206)\approx50.5696,\qquad E_2=80(0.3678794)\approx29.4304. $$ They sum to 80. Substituting $O_1=58$ and $O_2=22$, $$ X^2=\frac{(58-50.5696)^2}{50.5696} +\frac{(22-29.4304)^2}{29.4304} \approx 1.092+1.876=2.968. $$ Thus the statistic is $2.968$, to three decimal places. With the model parameters specified rather than estimated, the usual reference degrees of freedom are $k-1=1$. The calculation is checked by noting that the observed counts sum to 80 and the expected counts also sum to 80.

Common mistake and correction. Using unconditional probabilities gives $P(5<X\leq10)=e^{-1}-e^{-2}\approx0.2325$ and $P(X>10)=e^{-2}\approx0.1353$. Multiplying these by 80 gives expected counts of about 18.60 and 10.83, which sum to only 29.43. That is not a valid expected-count table for 80 observations distributed across the two categories. The correction is to condition on $X>5$, just as the sample was conditioned.

Second Error: Subtracting Parameters Without Checking How They Were Fitted

For a fully specified null model with $k$ categories, the usual chi-square reference distribution has $k-1$ degrees of freedom: the counts sum to $n$, so only $k-1$ count deviations can vary independently. In a regular grouped-data test, estimating $m$ model parameters from the same grouped counts typically gives $k-1-m$ degrees of freedom. The parameter adjustment is not a mechanical subtraction detached from the estimation method.

The usual adjustment relies on the fitted parameters being estimated in a way that corresponds to the information in the tested grouped counts. If parameters instead come from additional raw-data information, a separately estimated model, or a different fitting procedure, the grouped statistic does not automatically have the standard chi-square distribution with $k-1-m$ degrees of freedom. The estimation and calibration must match.

Theorem: In the regular multinomial goodness-of-fit setting, with fixed categories and $m$ identifiable parameters estimated by maximum likelihood from those same grouped counts, the large-sample reference distribution for the Pearson statistic is commonly $\chi^2_{k-1-m}$. This approximation also requires adequate expected counts and regularity conditions. Do not apply the degrees-of-freedom formula automatically when these conditions or the matching grouped-data estimation procedure do not hold.

Worked Example: Parameter Estimated From the Tested Counts

A hypothetical severity classification has three categories. Under the null model their probabilities are $\theta$, $2\theta$, and $1-3\theta$, where $0<\theta<1/3$. In a sample of 100 losses, the observed counts are 24, 32, and 44. Estimate $\theta$ by maximum likelihood from these grouped counts, calculate the statistic, and state the usual degrees of freedom.

Method and estimate. The grouped counts have a multinomial likelihood proportional to $$ L(\theta)=\theta^{24}(2\theta)^{32}(1-3\theta)^{44}. $$ Taking the log and differentiating gives $$ \frac{d}{d\theta}\log L(\theta) =\frac{24+32}{\theta}-\frac{3(44)}{1-3\theta}. $$ Set this equal to zero: $$ \frac{56}{\theta}=\frac{132}{1-3\theta} \quad\Longrightarrow\quad 56(1-3\theta)=132\theta \quad\Longrightarrow\quad \widehat{\theta}=\frac{56}{300}=\frac{14}{75}\approx0.1867. $$ This value is within the allowed interval $0<\theta<1/3$. The fitted probabilities are $0.1867$, $0.3733$, and $0.4400$. Multiplying by $n=100$, the expected counts are 18.6667, 37.3333, and 44.0000, which sum to 100.

Statistic and degrees of freedom. The observed-minus-expected differences are $5.3333$, $-5.3333$, and $0$. Therefore, $$ X^2=\frac{(5.3333)^2}{18.6667} +\frac{(-5.3333)^2}{37.3333} +\frac{0^2}{44} \approx1.5238+0.7619+0=2.2857. $$ The statistic is $2.286$, to three decimal places. Here $k=3$ and $m=1$, so the usual grouped-data reference degrees of freedom are $3-1-1=1$. The estimate and expected counts both use the same three grouped counts, as required for this standard calibration.

Common mistake and correction. It is tempting to subtract one estimated parameter whenever one appears in a problem statement. That is not sufficient. For example, estimating a Poisson parameter from individual claim counts and then testing only a grouped table that includes a $3+$ category does not automatically justify $k-1-1$ degrees of freedom. The raw-data estimate uses information not represented in the grouped counts. Use a calibration justified for that estimation procedure, rather than asserting the standard grouped-data reference distribution without checking.

Third Error: Ignoring Sparse Expected Counts

The chi-square reference distribution is an approximation, and it can be unreliable when expected counts are too small. A common response is to combine categories, often adjacent categories for an ordered severity variable. Combine based on the null model and the meaning of the categories, not to manufacture a desired result. After combining, add both the observed counts and the expected counts in each merged group, then recalculate the statistic and degrees of freedom for the new table.

A frequently used diagnostic is to avoid expected counts below 5, although adequacy is a rule of thumb rather than a universal theorem. The categories, sample size, and desired accuracy matter. Combining categories also changes the test: it removes detail and can reduce sensitivity to departures that occur within a merged group.

Worked Example: Combining an Adjacent Sparse Category

A fully specified hypothetical model gives six ordered categories with expected counts $2,8,20,30,25,15$ in a sample of 100. The observed counts are $0,10,18,32,24,16$. The first expected count is sparse. Combine the first two adjacent categories, then calculate the grouped statistic and its usual degrees of freedom.

Method. Merge the first two categories. Their combined observed count is $0+10=10$, and their combined expected count is $2+8=10$. The five resulting observed counts are $10,18,32,24,16$; the expected counts are $10,20,30,25,15$. Both totals remain 100, and all five expected counts are at least 5.

The Pearson statistic is $$ X^2= \frac{(10-10)^2}{10} +\frac{(18-20)^2}{20} +\frac{(32-30)^2}{30} +\frac{(24-25)^2}{25} +\frac{(16-15)^2}{15}. $$ The five contributions are $0$, $4/20=0.2000$, $4/30=0.1333$, $1/25=0.0400$, and $1/15=0.0667$. Their sum is $0.4400$. The model has no estimated parameters, so the usual degrees of freedom after combining are $k-1=5-1=4$. The counts check: observed $10+18+32+24+16=100$, expected $10+20+30+25+15=100$.

Common mistake and correction. Keeping the original six-category degrees of freedom after merging is incorrect: the test now has five categories. Also, combining categories only because their observed counts look inconvenient is not a safe repair. The sparse expected count is a feature of the null model, so a defensible grouping should be specified from that model and the category structure, preferably before examining the observed counts.

Fourth Error: Regrouping After Seeing the Results

A chi-square test assumes the categories are fixed for the calculation. If an analyst inspects observed counts, tries several groupings, and reports the grouping with the most favorable statistic or conclusion, the reported test no longer has the nominal calibration for a single preselected table. The selection process itself has used the data. This is a form of data-dependent analysis, even if each individual regrouping seems reasonable.

Regrouping to address sparse expected counts is not inherently wrong. The important distinction is whether the grouping is based on the null probabilities and substantive category structure, or chosen in response to the observed pattern. If the grouping is selected after inspection, the ordinary reference distribution does not account for that selection. A sound analysis should prespecify the grouping or use a calibration method that accounts for the selection procedure.

Check the sampling condition.
Determine whether the observations are unconditional, truncated, censored, or otherwise selected. Use probabilities for the observations actually included.

Check the probability table.
Confirm that the categories are mutually exclusive and exhaustive, and that the null probabilities sum to 1. Then verify that the expected counts sum to $n$.

Check estimation and degrees of freedom.
Record how every fitted parameter was estimated. Use the standard $k-1-m$ adjustment only when its grouped-data conditions are appropriate.

Check sparse counts and category choices.
If combining is needed, merge categories coherently, add observed and expected counts, and use the number of categories in the resulting table. Do not conceal data-driven regrouping.

Why These Checks Matter

The chi-square statistic measures discrepancies relative to expected variability. An incorrect expected count changes both the residual $O_j-E_j$ and its divisor $E_j$. An incorrect degrees-of-freedom reference changes the tail probability assigned to an otherwise correctly calculated statistic. A sparse table can make that reference distribution inaccurate, while post hoc category choices can make the nominal probability misleading.

On an exam, show enough of the setup to make the test auditable: state the null category probabilities, show how expected counts follow from the sampling scheme, identify whether parameters were estimated and from what data, and state the final number of categories. A quick check that $\sum_j O_j=\sum_j E_j=n$ catches many errors, but it does not by itself validate the degrees of freedom or the category-selection process.

Key takeaway: A valid chi-square goodness-of-fit calculation requires more than substituting counts into a formula. Match probabilities to the sample actually observed, match the degrees-of-freedom adjustment to the parameter-fitting procedure, and treat sparse-category regrouping as part of the test design rather than as a way to improve the observed result.

Check Your Understanding

For each problem, check the sampling condition and category structure before using a rule or formula.

A hypothetical severity model is exponential with rate $0.1$ per dollar. Fifty losses are sampled conditional on $X>20$, and the categories are $(20,30]$ and $(30,\infty)$, with observed counts 18 and 32. Find the conditional expected counts and Pearson statistic, rounding the statistic to two decimals. (Answer: expected counts approximately 31.61 and 18.39; $X^2\approx15.92$.)
A grouped goodness-of-fit test has four categories, and two parameters are estimated by maximum likelihood from those same grouped counts. Under the regular grouped-data conditions, what is the usual degrees-of-freedom count? (Answer: $4-1-2=1$ degree of freedom.)
A fully specified model gives three category expected counts $3,7,20$, and the observed counts are $4,6,20$. Combine the first two categories. Give the new observed and expected counts, statistic, and usual degrees of freedom. (Answer: observed counts $10,20$; expected counts $10,20$; $X^2=0$; 1 degree of freedom.)
A test uses five grouped categories and one fitted parameter, but the parameter was estimated from the individual loss values, including information not retained in the grouped table. Is $5-1-1=3$ automatically a justified reference degrees of freedom? (Answer: No. The usual grouped-data adjustment is not automatically justified; use a calibration appropriate to the raw-data estimation procedure.)
An analyst combines two adjacent categories only after noticing that their observed counts make the original statistic large, then reports a conventional chi-square p-value for the new table. What is the central calibration problem? (Answer: The grouping depends on the observed data, so the ordinary reference distribution for one prespecified table does not account for the selection.)

What You'll Learn