Degrees of Freedom for Fixed Probability Models

Why the Number of Categories Is Not the Degrees of Freedom

A goodness-of-fit question compares observed category counts with counts predicted by a specified probability model. With \(k\) categories, it may seem that there are \(k\) separate ways for the counts to differ from expectation. But the total number of observations is fixed: if some categories have more observations than expected, others must have fewer. That restriction means only \(k-1\) count deviations can vary independently.

This tutorial derives that result for a model whose null probabilities are fixed in advance. The degrees of freedom determine which chi-square reference distribution is associated with the Pearson statistic. They do not count the observations, nor do they count the number of categories without accounting for constraints.

The Fixed-Probability Model

Suppose each of \(n\) independent observations belongs to exactly one of \(k\) mutually exclusive, collectively exhaustive categories. Under the null model, the probability of category \(i\) is a known, fixed value \(p_i\), where each \(p_i>0\) and the probabilities sum to one. Write \(O_i\) for the observed count in category \(i\). The count vector follows a multinomial model with sample size \(n\) and probability vector \((p_1,\ldots,p_k)\).

The expected count in category \(i\) under this model is \(E_i=np_i\). The deviation of the observed count from its expected count is \(O_i-E_i=O_i-np_i\). Pearson's goodness-of-fit statistic sums the squared deviations divided by their expected counts:

The question here is not how to calculate a p-value, but why this statistic is associated with \(k-1\) degrees of freedom when the null probabilities are fixed. To answer it, first examine the count constraint.

The Constraint on the Count Deviations

The observed counts must add to the total sample size, so \(\sum_{i=1}^{k}O_i=n\). The expected counts also add to \(n\), because \(\sum_{i=1}^{k}np_i=n\sum_{i=1}^{k}p_i=n\). Subtracting these totals gives:

Once the deviations for the first \(k-1\) categories are known, the last deviation is determined: \(O_k-np_k=-\sum_{i=1}^{k-1}(O_i-np_i)\). Therefore, the deviations cannot provide \(k\) independent pieces of information. Their possible values lie in a \((k-1)\)-dimensional space.

Why the Pearson Statistic Has a Chi-Square Distribution with \(k-1\) Degrees of Freedom

The constraint gives an intuitive count of the independent deviations. A short asymptotic derivation explains why that count becomes the degrees of freedom of the chi-square reference distribution. Define a standardized deviation for each category by \(Z_i=(O_i-np_i)/\sqrt{np_i}\). Then Pearson's statistic is the sum of the squared standardized deviations, \(X^2=\sum_i Z_i^2\).

The standardized deviations also obey a constraint. Multiplying each \(Z_i\) by \(\sqrt{p_i}\) and summing gives:

In vector terms, the vector of standardized deviations is restricted to the subspace perpendicular to \(v=(\sqrt{p_1},\ldots,\sqrt{p_k})\). Since \(\sum_i p_i=1\), the vector \(v\) has length one. Its perpendicular subspace has dimension \(k-1\).

For completeness, under the fixed multinomial model, the covariance between two counts is \(n(p_i-p_i^2)\) when \(i=j\), and \(-np_ip_j\) when \(i\ne j\). After standardizing by \(\sqrt{np_i}\) and \(\sqrt{np_j}\), these entries form the matrix \(I-vv^\mathsf{T}\): its diagonal entries are \(1-p_i\), and its off-diagonal entries are \(-\sqrt{p_ip_j}\). This matrix projects onto the subspace perpendicular to \(v\) and has rank \(k-1\).

As \(n\) grows, the multinomial central limit theorem gives an approximately multivariate normal distribution for the standardized deviations on that subspace. Choose orthonormal coordinates within it. There are \(k-1\) such coordinates, each approximately standard normal, and the sum of their squares is the Pearson statistic. A chi-square distribution with \(k-1\) degrees of freedom is, by definition, the distribution of a sum of squares of \(k-1\) independent standard normal variables. Thus the fixed-probability Pearson statistic has an approximate chi-square reference distribution with \(k-1\) degrees of freedom when the approximation is appropriate.

This argument explains both parts of the result: the total-count constraint removes one dimension, and the squared standardized deviations give the chi-square form. The chi-square calibration is an approximation, not an exact claim that every finite-sample statistic has that distribution.

Degrees of Freedom in Practice

For a fixed-probability goodness-of-fit model, the calculation is direct: count the categories represented in the model and subtract one. For example, a model with six categories has \(df=6-1=5\), whether its fixed probabilities are equal or unequal. The probability values affect the expected counts and the statistic, but they do not change the number of independent count deviations.

The word fixed matters. In this setting, the probabilities \(p_i\) are specified by the null model rather than estimated from the observed counts. The result derived here applies to that setting. If probabilities are estimated from the same data, the model has additional fitted quantities and the degrees-of-freedom calculation requires separate treatment.

Common Mistakes and Why the Distinction Matters

• Using \(k\) degrees of freedom just because there are \(k\) categories. This ignores the fact that the counts must total \(n\). The deviations sum to zero, leaving only \(k-1\) independent deviations.
• Using \(n-1\) degrees of freedom. That formula belongs to other settings, not this categorical count constraint. Here the degrees of freedom are determined by the number of categories and the one total-count constraint.
• Thinking unequal probabilities change the answer. Unequal fixed probabilities change expected counts, but the deviations still sum to zero. The result remains \(k-1\).
• Assuming a chi-square approximation is exact for every sample size. The derivation uses an asymptotic normal approximation. The statistic's degrees of freedom and the adequacy of its calibration are distinct issues.

Getting the degrees of freedom right matters because the chi-square reference distribution changes with its degrees of freedom. Choosing \(k\) rather than \(k-1\) therefore uses the wrong reference curve, even if the observed counts and the Pearson statistic have been calculated correctly.