Why Pearson's Chi-Square Statistic Has an Approximate Chi-Square Law

Why a Chi-Square Reference Distribution Appears

A goodness-of-fit test compares observed category counts with the counts expected under a specified probability model. Pearson’s statistic adds up squared discrepancies, each scaled by its expected count. Its chi-square reference distribution is not an exact law for finite samples: it is a large-sample approximation. Understanding how that approximation arises helps explain both the formula and its limits.

The central idea is that, for a large multinomial sample, the standardized count deviations are approximately jointly normal. Their sum is constrained because the category counts must add to the sample size. Squaring and adding the deviations therefore gives a sum of fewer independent squared standard normal quantities than there are categories.

Model and statistic: Suppose $n$ independent observations each fall into one of $k$ mutually exclusive and exhaustive categories. Under the null model, the category probabilities are $p_1,\ldots,p_k$, where each $p_i>0$ and $\sum_{i=1}^{k}p_i=1$. Let $N_i$ be the observed count in category $i$, so $\sum_i N_i=n$. The expected count is $E_i=np_i$, and Pearson’s statistic is $$ Q=\sum_{i=1}^{k}\frac{(N_i-E_i)^2}{E_i}. $$

Under the null model, $N_i$ has mean $np_i$, but the counts are not independent: if one count increases while the total remains fixed, at least one other count must decrease. This dependence is important. It is the reason the reference distribution has $k-1$, rather than $k$, degrees of freedom when all probabilities are specified in advance.

From Multinomial Counts to a Chi-Square Law

For each category, measure the count’s deviation from its expected value in standard-error units: $Z_i=(N_i-np_i)/\sqrt{np_i}$. Pearson’s statistic is exactly the sum of these squared standardized deviations. The multinomial central limit theorem describes their joint behavior as the sample size grows.

Q=\sum_{i=1}^{k} Z_i^2, \qquad Z_i=\frac{N_i-np_i}{\sqrt{np_i}}.

To see why the limit has $k-1$ degrees of freedom, define the vector $W=(Z_1,\ldots,Z_k)^\mathsf{T}$ and the unit vector $a=(\sqrt{p_1},\ldots,\sqrt{p_k})^\mathsf{T}$. The multinomial central limit theorem gives an approximate multivariate normal distribution for $W$, with mean zero and covariance matrix $I-aa^\mathsf{T}$, where $I$ is the $k$-by-$k$ identity matrix. This covariance matrix removes the direction $a$: it has rank $k-1$, not $k$.

Large-sample result: If the probabilities $p_i$ are fixed and positive, and the sample consists of independent observations from the specified multinomial model, then as $n$ grows, $$ Q \ \xrightarrow{d}\ \chi^2_{k-1}. $$ The symbol $\xrightarrow{d}$ means convergence in distribution. Thus the chi-square law is an asymptotic reference distribution, not generally the exact finite-sample distribution of $Q$.

The geometric reason for the result can be made precise. The matrix $I-aa^\mathsf{T}$ is a projection onto the $(k-1)$-dimensional space perpendicular to $a$. An orthogonal change of coordinates expresses the limiting normal vector as $k-1$ independent standard normal coordinates and one coordinate that is zero. Its squared length is therefore the sum of $k-1$ independent squared standard normal variables, which is the definition of a chi-square distribution with $k-1$ degrees of freedom.

Standardize each count.
Under the null, compare $N_i$ with $np_i$ in units of $\sqrt{np_i}$, the approximate standard deviation of that count.

Use the multinomial limit.
For large $n$, the vector of standardized deviations is approximately multivariate normal, but its coordinates are dependent.

Account for the fixed total.
Because the counts sum to $n$, only $k-1$ directions of fluctuation are available.

Square and add.
The squared length in those $k-1$ normal directions has an approximate $\chi^2_{k-1}$ distribution.

Worked Examples

Worked Example: A Fully Specified Three-Category Model

A hypothetical quality-control audit classifies 100 independently selected components into three mutually exclusive categories. Under the specified model, the category probabilities are $(0.20,0.30,0.50)$. The observed counts are $(18,32,50)$. Find Pearson’s statistic and its usual reference degrees of freedom, and explain why the approximation is reasonable.

Method and model. Treat the count vector as multinomial with $n=100$ and fixed probabilities $p_1=0.20$, $p_2=0.30$, and $p_3=0.50$. No parameters are estimated from these observations.

Step 1: Find expected counts. The expected counts are $E_i=np_i$: $100(0.20)=20$, $100(0.30)=30$, and $100(0.50)=50$. Thus the expected vector is $(20,30,50)$.

Step 2: Calculate the statistic. The observed-minus-expected differences are $(-2,2,0)$, so $$ Q=\frac{(-2)^2}{20}+\frac{2^2}{30}+\frac{0^2}{50} =\frac{4}{20}+\frac{4}{30}+0 =0.2000+0.1333 =0.3333. $$ The displayed statistic is rounded to four decimal places.

Step 3: Determine degrees of freedom. All three probabilities are specified, so the degrees of freedom are $k-1=3-1=2$.

Check and interpretation. The observed counts sum to 100, as required, and the differences sum to zero. All expected counts are at least 20, so the large-sample approximation is reasonable by common practical guidance. The reference law is approximately $\chi^2_2$. The common mistake avoided here is misadding the two nonzero contributions: $0.2000+0.1333=0.3333$, not $0.6667$.

Worked Example: Assessing a Four-Category Fit

A hypothetical researcher assigns 160 independently sampled records to one of four categories. The null model specifies probabilities $(0.25,0.25,0.25,0.25)$, and the observed counts are $(36,44,40,40)$. Calculate Pearson’s statistic, state the reference degrees of freedom, and give the 5% decision using the critical value 7.815 for three degrees of freedom.

Method and model. This is a multinomial goodness-of-fit calculation with $n=160$ and four known, equal category probabilities. The test rejects for sufficiently large values of $Q$.

Step 1: Find expected counts. Each expected count is $160(0.25)=40$, giving $(40,40,40,40)$.

Step 2: Calculate the statistic. The deviations are $(-4,4,0,0)$. Therefore, $$ Q=\frac{(-4)^2}{40}+\frac{4^2}{40}+\frac{0^2}{40}+\frac{0^2}{40} =\frac{16}{40}+\frac{16}{40} =0.4000+0.4000 =0.8000. $$

Step 3: Choose degrees of freedom and decide. The probabilities are fully specified, so $df=4-1=3$. Since $0.8000<7.815$, do not reject the null model at the 5% level.

Check and interpretation. The observed counts sum to 160, and each expected count is 40. The approximation is well supported by the expected counts. The common mistake avoided is doubling the statistic: there are two nonzero contributions, each $16/40=0.4$, so their sum is $0.8$, not $1.6$.

When Probabilities Are Estimated

Sometimes the null model contains unknown parameters that are estimated using the same sample. For a regular model with $k$ categories and $q$ independently estimable parameters, the usual reference degrees of freedom are $k-1-q$. The subtraction of one still reflects the fixed total of the counts; estimating $q$ parameters uses a further $q$ directions of variation.

Degrees of freedom with fitted parameters: Under a correctly specified, regular multinomial model, when $q$ identifiable parameters are estimated from the same data by a suitable regular method, Pearson’s statistic is usually compared with $\chi^2_{k-1-q}$. This rule requires regularity conditions, including an interior parameter value and identifiable parameters. It is not a universal rule for irregular models, parameters on boundaries, or nonstandard estimation procedures. Parameters known in advance or estimated from independent external data do not automatically use up degrees of freedom in the same way.

Worked Example: A Model With Two Fitted Parameters

A hypothetical five-category model is fitted to 120 observations. Two model parameters were estimated from those same observations. The fitted expected counts are $(18,22,25,30,25)$, and the observed counts are $(20,20,25,28,27)$. Calculate Pearson’s statistic and the usual reference degrees of freedom. Assume the model is regular and the fitted probabilities are all positive.

Method and model. Use the fitted expected counts in Pearson’s statistic, then subtract the two estimated parameters from the $k-1$ multinomial degrees of freedom.

Step 1: Check the totals. The expected counts sum to $18+22+25+30+25=120$. The observed counts sum to $20+20+25+28+27=120$. Thus the totals agree.

Step 2: Calculate deviations and contributions. The deviations are $(2,-2,0,-2,2)$, so $$ Q=\frac{2^2}{18}+\frac{(-2)^2}{22}+\frac{0^2}{25} +\frac{(-2)^2}{30}+\frac{2^2}{25} $$ $$ =\frac{4}{18}+\frac{4}{22}+0+\frac{4}{30}+\frac{4}{25} \approx0.2222+0.1818+0+0.1333+0.1600 =0.6974. $$ The statistic is rounded to four decimal places.

Step 3: Calculate degrees of freedom. There are $k=5$ categories and $q=2$ fitted parameters, giving $df=k-1-q=5-1-2=2$.

Check and interpretation. The statistic uses each fitted expected count as its denominator, and the displayed contributions sum to approximately $0.6974$. The usual reference distribution is $\chi^2_2$, subject to the stated regularity assumption. The common mistake avoided is subtracting only the two fitted parameters and reporting three degrees of freedom; the fixed-total constraint also removes one degree of freedom.

Conditions, Limits, and Common Traps

The chi-square law is an approximation whose quality depends on the model and the expected counts. The basic setup assumes independent observations, a fixed set of mutually exclusive categories, and a null model that correctly specifies their probabilities. The number of categories is treated as fixed as the sample size grows, and each category probability is positive. Under these conditions, expected counts grow with $n$.

In practical applications, expected-count rules of thumb are used to judge whether the asymptotic approximation is likely to be adequate. They are not the mathematical theorem itself, nor are they guarantees. Very small expected counts can make the finite-sample distribution of the statistic differ noticeably from its chi-square limit. Combining categories may sometimes be defensible, but only when the combined categories still make sense for the scientific question and the null model.

Pearson’s statistic is a measure of discrepancy, not a signed measure of direction. A large value indicates that the observed counts collectively differ from the null expectations more than would ordinarily be expected under the null model. The chi-square test uses the upper tail because large discrepancies provide evidence against the model. A small value does not prove the model true; it means the data do not show a large discrepancy by this measure.

Common calculation trap: The denominator in each Pearson contribution is the expected count for that category, not the sample size. For the first worked example, dividing each squared deviation by 100 would give $4/100+4/100=0.08$, which is not Pearson’s statistic. The correct calculation is $4/20+4/30=0.3333$, using the corresponding expected counts.

A second frequent error is treating the chi-square reference law as exact or forgetting that fitted parameters affect the degrees of freedom. Always identify what is fixed by the multinomial total, how many parameters were estimated from the tested data, and whether the regularity conditions make the usual degrees-of-freedom rule appropriate. These checks explain why a reference value is being used, rather than merely supplying a number to look up.

Key takeaway: Pearson’s statistic is the squared length of the vector of standardized multinomial deviations. The fixed total leaves $k-1$ asymptotically free directions; regular parameters fitted from the same data generally reduce that number further. The chi-square reference law is useful when the large-sample and model conditions are credible.

Check Your Understanding

For each problem, identify the multinomial setup and distinguish the calculated statistic from the assumptions supporting its chi-square reference law.

A hypothetical survey has four response categories with specified probabilities $(0.10,0.20,0.30,0.40)$. From 200 independent responses, the counts are $(18,42,62,78)$. Find Pearson’s statistic and the usual degrees of freedom. (Answer: $Q=0.4167$; $df=3$.)
A hypothetical device study has three equally likely outcomes under a fully specified model. In 90 independent trials, the counts are $(25,30,35)$. Calculate the statistic and state the reference degrees of freedom. (Answer: $Q=1.6667$; $df=2$.)
A regular four-category model has one parameter estimated from the same data. Its fitted expected counts are $(24,30,36,30)$, and its observed counts are $(26,27,39,28)$. Find Pearson’s statistic and the usual degrees of freedom. (Answer: $Q=0.8500$; $df=2$.)
A five-category null model has probabilities $(0.04,0.16,0.20,0.25,0.35)$. In 24 independent observations, the counts are $(1,4,5,6,8)$. Calculate Pearson’s statistic and explain whether the usual chi-square approximation is well supported. (Answer: $Q\approx0.0357$; $df=4$, but the approximation is not well supported because several expected counts are below 5, including one below 1.)

What You'll Learn