Common Calculation and Interpretation Errors

Why Small Errors Can Change a Goodness-of-Fit Conclusion

A one-way Pearson chi-square goodness-of-fit test compares observed category counts with the counts expected under a specified probability distribution. The calculation is compact, but it is easy to mix up quantities that have different meanings or units. Treating percentages as counts, comparing counts directly with probabilities, and rounding expected counts prematurely can all change the statistic. Even a correct calculation can be followed by an interpretation that claims more than the test establishes.

The central safeguard is to keep the model, the observed counts, and the expected counts distinct. Then calculate the statistic from unrounded expected counts and interpret rejection as evidence against the specified null distribution—not as proof of a particular explanation.

The Model and the Calculation to Audit

Definition: Suppose $n$ independent observations are classified into $k$ mutually exclusive and exhaustive categories. Under the null hypothesis, the probability of category $i$ is $p_i$, where $\sum_{i=1}^{k}p_i=1$. Let $O_i$ be the observed count in category $i$. Its expected count under the null is $E_i=np_i$. The Pearson goodness-of-fit statistic is $$ X^2=\sum_{i=1}^{k}\frac{(O_i-E_i)^2}{E_i}. $$ When the null probabilities are specified in advance and the chi-square approximation is appropriate, the reference distribution has $df=k-1$. The p-value is the probability, assuming the null model, of obtaining a statistic at least as large as the observed $X^2$.

The count model is multinomial: the total $n$ is fixed, each observation falls in exactly one of the stated categories, and observations are independent with the null category probabilities. The usual chi-square approximation also requires expected counts adequate for that approximation. These are conditions for using the approximate reference distribution; no correction to the formula can compensate for a serious failure of the conditions.

For an audit, check the category definitions and null probabilities, confirm that the observed counts sum to $n$, compute $E_i=np_i$, and use those expected counts in every denominator. If the null probabilities were estimated from the same data rather than specified in advance, the degrees of freedom may need adjustment; do not automatically use $k-1$.

Worked Examples of Calculation Errors

Worked Example: Percentages Are Not Counts

Suppose a hypothetical survey classifies $n=200$ independent responses into three exhaustive categories. Under the null model, the probabilities are $0.50$, $0.30$, and $0.20$. The observed counts are 110, 50, and 40. Check the calculation and the conclusion at $\alpha=0.05$.

The null expected counts are $200(0.50)=100$, $200(0.30)=60$, and $200(0.20)=40$. The observed percentages are 55%, 25%, and 20%, but the Pearson formula uses counts. Thus, $$ X^2=\frac{(110-100)^2}{100} +\frac{(50-60)^2}{60} +\frac{(40-40)^2}{40} =1+\frac{100}{60}+0 =2.6667. $$

There are $k=3$ categories and the probabilities are specified, so $df=3-1=2$. The p-value is approximately 0.2636. Because $0.2636>0.05$, this test does not reject the null model.

A common mistake is to insert 55, 25, and 20—the percentages—as if they were observed counts, while retaining expected counts 100, 60, and 40. That mixes percentages and people, producing a meaningless statistic. Percentages can be used only in a consistent equivalent calculation: if $q_i=O_i/n$, then $n\sum (q_i-p_i)^2/p_i$ equals the count-based statistic. The direct count formula is usually the clearest.

Worked Example: Compare Observed Counts With Expected Counts

In another hypothetical experiment, $n=120$ independent outcomes fall in three categories. The null probabilities are $0.50$, $0.30$, and $0.20$; the observed counts are 66, 30, and 24. Find the statistic.

The probabilities describe the null model, not the expected number of observations. Multiply each probability by the fixed total to obtain expected counts: $120(0.50)=60$, $120(0.30)=36$, and $120(0.20)=24$. The correct calculation is $$ X^2=\frac{(66-60)^2}{60} +\frac{(30-36)^2}{36} +\frac{(24-24)^2}{24} =\frac{36}{60}+\frac{36}{36}+0 =1.600. $$

With $df=2$, the p-value is approximately 0.4493. The data do not provide evidence against this null model at the 0.05 level. Comparing 66 directly with 0.50, or 30 with 0.30, is not a valid alternative: a count and a probability are different quantities. First convert every null probability to an expected count using the same $n$.

Worked Example: Do Not Round Expected Counts Beforehand

Suppose a hypothetical sample of $n=101$ observations has null probabilities $0.25$, $0.25$, and $0.50$. The observed counts are 30, 20, and 51. All expected counts exceed 5, so the usual expected-count guideline for the chi-square approximation is satisfied.

Keep the expected counts unrounded: $25.25$, $25.25$, and $50.50$. Then $$ X^2=\frac{(30-25.25)^2}{25.25} +\frac{(20-25.25)^2}{25.25} +\frac{(51-50.50)^2}{50.50} =0.8936+1.0916+0.0050 =1.990. $$

There are three categories and specified probabilities, giving $df=2$. The p-value is approximately 0.370. If the expected counts are rounded first to 25, 25, and 51, the statistic becomes $$ \frac{(30-25)^2}{25} +\frac{(20-25)^2}{25} +\frac{(51-51)^2}{51} =2.000. $$

Here the rounded result happens to be close, but rounding has changed the calculation without justification. In other data it can change the statistic and p-value more substantially. Keep full precision through the calculation and round only the reported result.

What Rejection Does—and Does Not—Show

For a goodness-of-fit test with specified null probabilities, the alternative hypothesis is that the category probability distribution is not the specified vector: at least one category probability differs. The test statistic measures the overall discrepancy between observed and null-expected counts. It does not, by itself, identify the cause of that discrepancy.

Worked Example: Rejection Is Not Proof of a Specific Alternative

Consider a hypothetical sample of 100 independent outcomes in four exhaustive categories. Under the null, each category has probability 0.25. The observed counts are 40, 20, 20, and 20. Calculate the test and describe its conclusion at $\alpha=0.05$.

Each expected count is $100(0.25)=25$. Therefore, $$ X^2=\frac{(40-25)^2}{25} +\frac{(20-25)^2}{25} +\frac{(20-25)^2}{25} +\frac{(20-25)^2}{25} =9+1+1+1 =12.00. $$

The degrees of freedom are $4-1=3$, and the p-value is approximately 0.00738. Since this is below 0.05, reject the null model of equal probabilities. In this sample, the first category has more observations than its null expected count; however, rejection is not proof that the true probability of that category is exactly 0.40, nor does it establish why the observed pattern occurred. The formal conclusion is evidence that the specified four-category distribution does not fit well.

A Short Justification for Expected Counts

The rule $E_i=np_i$ follows directly from the model. For each of the $n$ observations, define an indicator that equals 1 if the observation is in category $i$, and 0 otherwise. Each indicator has expected value $p_i$. The count $O_i$ is the sum of these $n$ indicators, so by additivity of expectation, $$ E[O_i]=p_i+p_i+\cdots+p_i=np_i. $$

This explains why comparing observed counts with probabilities is an error: the expected value of a count is $np_i$, not $p_i$. It also explains why the total sample size matters to every expected count.

Common Mistakes to Catch Before Reporting

Using percentages as counts. Confirm that each $O_i$ is a number of observations and that $\sum O_i=n$. If using proportions instead, use a mathematically consistent proportion-based formula.
Comparing counts directly with probabilities. Compute $E_i=np_i$ before calculating deviations.
Rounding intermediate values. Retain unrounded expected counts and contributions until the final statistic and p-value are calculated.
Using the wrong degrees of freedom. For specified probabilities across $k$ categories, use $k-1$; if probabilities were estimated from the data, account for that estimation.
Calling a small p-value proof of a story. Rejection is evidence against the specified null distribution, not proof of a particular alternative distribution, mechanism, or cause.

A clear report names the categories and null distribution, gives the statistic and degrees of freedom, reports the p-value, and states the conclusion in context. A large p-value means the observed discrepancy is not unusual enough under the null model to reject it; it does not prove that the null probabilities are exactly true.

Check Your Understanding

For each question, focus on the quantity being compared and the claim supported by the test.

A sample of 80 observations has observed percentages 50% and 50%, while the null probabilities are 0.60 and 0.40. What observed counts and expected counts should enter the Pearson formula?
Why is it incorrect to subtract a null probability such as 0.30 directly from an observed count such as 24?
Expected counts are 18.4, 21.6, and 40.0. Should they be rounded before calculating the chi-square statistic? Explain.
A test rejects a specified probability vector. What does this support, and what does it not establish about the reason for the mismatch?
For $k=5$ categories with probabilities specified before data collection, what degrees of freedom are used for the usual goodness-of-fit test?