Why Pearson's Chi-Square Measures Lack of Fit

Why Measure Lack of Fit?

A goodness-of-fit question asks whether observed category counts are close to the counts predicted by a specified probability model. The raw differences alone are not enough: a difference of 10 counts may be substantial when a category is expected to contain 20 observations, but comparatively modest when it is expected to contain 1,000. Pearson’s chi-square statistic combines the discrepancies across categories while scaling them to reflect the expected counts.

The statistic is designed so that discrepancies in either direction contribute positively. If a category has more observations than expected, or fewer, that difference contributes to the total measure of mismatch. The size of the statistic therefore summarizes the overall departure from the model; it does not, by itself, say whether that departure is unusual under the model.

The Statistic and Its Model

Suppose \(n\) independent observations are each classified into exactly one of \(k\) mutually exclusive and exhaustive categories. A null model specifies the probability \(p_i\) of category \(i\), where every included category has \(p_i>0\) and \(\sum_{i=1}^{k}p_i=1\). Let \(O_i\) be the observed count in category \(i\). Under the null model, the expected count is \(E_i=np_i\). Both sets of counts sum to \(n\).

Definition: Pearson’s chi-square statistic for these counts and this fixed null model is

X^2=\sum_{i=1}^{k}\frac{(O_i-E_i)^2}{E_i}.

Each term is the squared observed-minus-expected difference for one category, divided by that category’s expected count. Because \(E_i=np_i>0\), every term is defined.

The positive-probability requirement matters. A category with null probability zero has expected count zero, so the displayed fraction would be undefined. Such a category is normally left out of the statistic. If an observation is recorded in a category assigned probability zero, that observation is impossible under the stated null model; the model must be reconsidered rather than forcing that count into the formula.

The numerator, \((O_i-E_i)^2\), makes the contribution nonnegative and removes the sign of the discrepancy. The denominator, \(E_i\), puts the squared difference on a scale relative to the count the model predicts. Equivalently, the contribution can be written as the expected count multiplied by the square of the relative discrepancy:

\frac{(O_i-E_i)^2}{E_i} =E_i\left(\frac{O_i-E_i}{E_i}\right)^2.

This form shows that the statistic considers both how large a category’s departure is in relation to its expectation and how much expected count is involved. It is not simply the sum of absolute differences, nor is it the sum of squared differences without scaling.

Worked Example: Calculating the Total Discrepancy

Imagine a hypothetical experiment in which 120 independent rolls of a six-sided die are classified by face. The null model assigns probability \(1/6\) to each face. Suppose the observed counts are 30, 15, 20, 18, 22, and 15. The model’s assumptions are that each roll has the stated category probabilities, rolls are independent, and each roll produces exactly one of the six outcomes.

Each expected count is \(E_i=120(1/6)=20\). The observed counts and expected counts both total 120, as required. Applying the formula gives

\begin{aligned} X^2 &=\frac{(30-20)^2}{20} +\frac{(15-20)^2}{20} +\frac{(20-20)^2}{20} +\frac{(18-20)^2}{20} +\frac{(22-20)^2}{20} +\frac{(15-20)^2}{20}\\ &=\frac{100}{20}+\frac{25}{20}+0+\frac{4}{20}+\frac{4}{20}+\frac{25}{20}\\ &=5+1.25+0+0.20+0.20+1.25\\ &=7.90. \end{aligned}

The value \(X^2=7.90\) is the total scaled discrepancy for these hypothetical counts under the equal-probability model. The calculation says how far the counts are from their expectations on this scale. To decide how surprising that value would be under the model requires a reference distribution and additional testing steps; the statistic alone is not a p-value or a conclusion.

Why Divide by the Expected Count?

Expected counts provide a natural scale for judging differences. A difference of 10 is large relative to an expectation of 10, but small relative to an expectation of 1,000. Squaring makes either direction count as departure, while division by \(E_i\) prevents raw count size from being the only consideration.

For example, if a category is expected to contain 100 observations and has 10 more than expected, its contribution is \(10^2/100=1\). If another category is expected to contain 10 observations and also has 10 more than expected, its contribution is \(10^2/10=10\). The same absolute difference is more consequential on the statistic’s scale when the expected count is smaller.

The denominator should not be described as the exact variance of \(O_i\). In a multinomial model, the marginal variance of a category count is \(np_i(1-p_i)\), not \(np_i\). Pearson’s formula is a particular scaled discrepancy measure; its connection to a variance-based approximation is part of why it has a useful chi-square reference distribution under suitable conditions, but the denominator in the statistic is specifically the expected count \(E_i=np_i\).

Worked Example: Equal Absolute Differences, Different Contributions

Suppose a hypothetical sample of 200 independent observations is classified into four categories. A fixed null model gives probabilities \(0.50, 0.25, 0.15,\) and \(0.10\), respectively. The observed counts are 110, 40, 30, and 20. All null probabilities are positive, the categories are exhaustive and mutually exclusive, and the observations are independent.

The expected counts are \(200(0.50)=100\), \(200(0.25)=50\), \(200(0.15)=30\), and \(200(0.10)=20\). The first two categories differ from expectation by 10 counts in opposite directions; the other two match their expectations. Thus

\begin{aligned} X^2 &=\frac{(110-100)^2}{100} +\frac{(40-50)^2}{50} +\frac{(30-30)^2}{30} +\frac{(20-20)^2}{20}\\ &=\frac{100}{100}+\frac{100}{50}+0+0\\ &=1+2=3. \end{aligned}

The two absolute differences are both 10, but their contributions are not equal: the difference in the category expected to have 50 observations contributes 2, while the difference in the category expected to have 100 contributes 1. Scaling by expectation makes the same raw departure count more heavily when it is large relative to what the model predicts.

A Short Derivation: A Local Quadratic Approximation

Pearson’s statistic also has a mathematical justification as a quadratic approximation to a likelihood-based measure of discrepancy. The derivation explains why squared differences divided by expected counts appear together. For counts \(O_i\) from the fixed model, the likelihood ratio compares the specified probabilities \(p_i\) with the best-fitting category probabilities \(O_i/n\). Twice the log likelihood ratio is

D=2\sum_{i=1}^{k}O_i\log\left(\frac{O_i}{E_i}\right),

where \(E_i=np_i\). A term with \(O_i=0\) is defined by its limiting value, zero. To examine counts near their expectations, write \(O_i=E_i+\delta_i\), so that \(\delta_i=O_i-E_i\). Since both the observed and expected counts sum to \(n\), \(\sum_i\delta_i=0\). For a category with \(O_i>0\), let \(x_i=\delta_i/E_i\). Then \(O_i/E_i=1+x_i\), and

D=2\sum_{i=1}^{k}E_i(1+x_i)\log(1+x_i).

When \(x\) is near zero, the Taylor expansion is

(1+x)\log(1+x)=x+\frac{x^2}{2}-\frac{x^3}{6}+\cdots.

Keeping the linear and quadratic terms gives

\begin{aligned} D &\approx 2\sum_{i=1}^{k}E_i\left(x_i+\frac{x_i^2}{2}\right)\\ &=2\sum_{i=1}^{k}\delta_i+\sum_{i=1}^{k}\frac{\delta_i^2}{E_i}\\ &=\sum_{i=1}^{k}\frac{(O_i-E_i)^2}{E_i}\\ &=X^2, \end{aligned}

The linear term vanishes because \(\sum_i\delta_i=0\). The remaining quadratic term is exactly Pearson’s statistic. This derivation is an approximation when counts are near their expectations; it does not claim that \(X^2\) and \(D\) are identical for every possible set of counts. It shows why Pearson’s statistic is a sensible measure of local departure from a specified model.

Worked Example: Squaring Prevents Cancellation

Consider a hypothetical sample of 100 independent observations in three categories with null probabilities \(0.60, 0.30,\) and \(0.10\). Suppose the observed counts are 66, 27, and 7. The null probabilities are positive and sum to 1; the categories are mutually exclusive and exhaustive.

The expected counts are 60, 30, and 10. The signed differences are \(+6\), \(-3\), and \(-3\), which sum to zero because the observed and expected totals are both 100. Simply adding signed differences would therefore produce zero and hide the departures. Pearson’s statistic squares each difference and scales it:

\begin{aligned} X^2 &=\frac{(66-60)^2}{60} +\frac{(27-30)^2}{30} +\frac{(7-10)^2}{10}\\ &=\frac{36}{60}+\frac{9}{30}+\frac{9}{10}\\ &=0.60+0.30+0.90\\ &=1.80. \end{aligned}

The positive and negative departures all increase the total; none cancels another. Here the third category contributes the most, even though its absolute difference is only 3, because 3 is large relative to its expected count of 10.

What a Large Value Tells You

Every category contributes zero if its observed count equals its expected count. As departures grow relative to expected counts, their squared contributions increase, and so does \(X^2\). Thus, a large value indicates that the observed counts are collectively far from the counts predicted by the null model on Pearson’s scale. A small value indicates that the total scaled discrepancy is small.

This interpretation is about fit, not about which categories are responsible. The single total does not preserve the direction of each category’s discrepancy, since squaring removes the signs. Nor does the total alone identify whether the null model should be rejected. A formal goodness-of-fit test compares the statistic with an appropriate reference distribution and relies on its calibration conditions. Those are separate steps from defining and understanding the statistic.

Common Mistakes

Adding signed differences. The differences sum to zero when the observed and expected totals agree, so positive and negative departures can cancel. Pearson’s statistic squares each difference before adding it.
Using the same raw difference as equally important in every category. A difference must be judged relative to its expected count; the denominator \(E_i\) performs that scaling.
Calling each term an exact squared z-score. The term is a scaled discrepancy, not generally a squared standard score based on the exact marginal variance of a count.
Allowing an included category to have expected count zero. This makes the formula undefined. Use only categories with positive null probabilities, and do not treat an observation in a zero-probability category as an ordinary finite contribution.
Treating \(X^2\) as a p-value or final conclusion. It is a statistic measuring discrepancy. A test conclusion requires an appropriate reference distribution and the relevant conditions.

Key takeaway: Pearson’s chi-square adds \((O_i-E_i)^2/E_i\) across categories. Squaring prevents directional cancellation, and dividing by the expected count scales each discrepancy to the model’s prediction. A larger statistic means greater overall lack of fit on this scale.

Check Your Understanding

Answer each question using the definition and reasoning above.

In your own words, explain why Pearson’s statistic squares \(O_i-E_i\) rather than adding the signed differences.
A category has expected count 25 and observed count 30. What is its contribution to \(X^2\)? Show the calculation.
Two categories each differ from expectation by 8 counts. One has expected count 16 and the other expected count 64. Which contributes more, and by what factor?
Why must every category included in the statistic have a positive null probability?
Does a large value of \(X^2\) by itself give a p-value? Explain what the statistic does tell you and what further ingredient is needed for a test conclusion.