Degrees of Freedom When Probabilities Are Estimated

Why Estimation Changes the Reference Distribution

In a goodness-of-fit test, the Pearson statistic compares observed counts with counts predicted by a model. When the model probabilities are fixed in advance, the observed deviations have a known number of independent directions in which they can vary. When probabilities are estimated from the same counts, the fitted model uses some of those directions to match the data. The test therefore has fewer degrees of freedom available to measure remaining lack of fit.

This adjustment is not just a bookkeeping convention. It reflects how the null model was fitted. In particular, the usual degrees-of-freedom result for a fitted Poisson model assumes its rate was estimated from the same grouped categories used in the test. Estimating the rate from more detailed, ungrouped observations can change the calibration.

Definition: For a goodness-of-fit test with \(k\) categories and \(m\) parameters estimated from those same grouped-category counts, the usual asymptotic degrees of freedom are \(k-1-m\), provided the model is identifiable and regular and the parameters are estimated by maximum likelihood from the grouped data. The subtraction of one accounts for the fixed total count; the further subtraction of \(m\) accounts for the fitted parameters.

Fixed and Fitted Probability Models

Let \(O_j\) be the observed count in category \(j\), and let \(E_j\) be its expected count under the null model. The Pearson goodness-of-fit statistic is

X^2=\sum_{j=1}^{k}\frac{(O_j-E_j)^2}{E_j}.

For a fixed probability model, each category probability \(p_j\) is specified before examining the observed counts, and \(E_j=np_j\). Since the counts sum to \(n\), their deviations from expected counts must also sum to zero. Thus, among \(k\) category deviations, only \(k-1\) can vary independently, giving the familiar \(k-1\) degrees of freedom.

If the probabilities depend on \(m\) unknown parameters estimated from the grouped data, fitting constrains the deviations further. Under the usual regularity conditions, each independently estimated parameter uses one additional direction of variation. The resulting reference distribution is approximately chi-square with \(k-1-m\) degrees of freedom. This is an asymptotic result: it describes the reference distribution as the sample size grows, not an exact finite-sample identity.

A Fitted Poisson Model

Suppose independent counts are modeled as Poisson with rate \(\lambda\), so an individual count \(X\) has distribution \(X\sim \operatorname{Poisson}(\lambda)\). For a goodness-of-fit test, the observations may be placed into categories such as 0, 1, and 2 or more. Their probabilities are

q_0(\lambda)=e^{-\lambda},\qquad q_1(\lambda)=\lambda e^{-\lambda},\qquad q_{2+}(\lambda)=1-(1+\lambda)e^{-\lambda}.

These are probabilities for the categories actually tested. If \(\lambda\) is estimated by maximum likelihood from their counts, the fitted expected counts are \(E_j=nq_j(\hat{\lambda})\). There are \(k=3\) categories and one estimated parameter, so the usual reference distribution has \(3-1-1=1\) degree of freedom.

Worked Example: Estimating a Poisson Rate From Grouped Counts

In a hypothetical set of 100 independent observations, suppose the counts in the categories 0, 1, and 2 or more are 37, 37, and 26. Fit a Poisson rate from these grouped data, calculate the Pearson statistic, and find the appropriate degrees of freedom.

Check the model and categories. The example assumes independent observations following one Poisson distribution with a common rate. The three categories are mutually exclusive and exhaustive, and their observed counts total \(37+37+26=100\). The rate will be estimated from these grouped counts, rather than from the underlying ungrouped values.

For grouped-category counts, the likelihood, apart from factors that do not depend on \(\lambda\), is

L(\lambda)\ \propto\ q_0(\lambda)^{37}q_1(\lambda)^{37}q_{2+}(\lambda)^{26}.

The grouped maximum-likelihood estimate maximizes this likelihood. Equivalently, it solves the score equation obtained by differentiating its log:

\frac{d\log L}{d\lambda} =-37+37\left(\frac{1}{\lambda}-1\right) +26\frac{\lambda e^{-\lambda}}{1-(1+\lambda)e^{-\lambda}}=0.

Solving this equation numerically gives \(\hat{\lambda}\approx 0.9909\). The fitted category probabilities are approximately \(0.3712\), \(0.3679\), and \(0.2609\), so the expected counts are

(E_0,E_1,E_{2+}) =100(0.3712,\ 0.3679,\ 0.2609) \approx(37.12,\ 36.79,\ 26.09).

Substituting these expected counts into the Pearson statistic gives

X^2 =\frac{(37-37.12)^2}{37.12} +\frac{(37-36.79)^2}{36.79} +\frac{(26-26.09)^2}{26.09} \approx 0.00196.

There are three categories and one parameter estimated from their grouped counts, so \(df=3-1-1=1\). Using a chi-square reference distribution with 1 degree of freedom, the upper-tail p-value is approximately \(0.965\). The data provide little evidence of lack of fit to this fitted Poisson model. This conclusion is about compatibility with the model, not proof that the model is true.

Why the Parameter-Count Adjustment Works

The adjustment can be understood by considering the vector of differences between observed and expected counts. Because the total observed count is fixed, those differences must sum to zero. This leaves \(k-1\) independent directions before any parameters are fitted. Estimating parameters from the same grouped data then makes the fitted model respond to some of those directions, reducing the residual variation available to the goodness-of-fit statistic.

For a regular model, the grouped maximum-likelihood estimates satisfy score equations: the derivatives of the log-likelihood with respect to the fitted parameters are zero at the estimate. In large samples, these equations make the fitted residuals approximately orthogonal to the directions associated with the parameters. Each independent parameter direction removed leaves one fewer degree of freedom for the residual lack of fit. This gives \(k-1-m\).

The assumptions matter. The parameters must be identifiable from the grouped probabilities, the model must behave regularly near the true parameter values, and the estimate must be based on the same grouped information used to form the Pearson statistic. If these conditions do not hold, simply subtracting the parameter count is not automatically justified.

Worked Example: Comparing With a Rate Specified in Advance

Use the same hypothetical counts, 37, 37, and 26, but now suppose the Poisson rate \(\lambda=1\) was specified before the data were collected. Calculate the fixed-model statistic and compare its degrees of freedom with the fitted-model result.

Check what is fixed. The categories are again 0, 1, and 2 or more, but no parameter is estimated from these counts. The model probabilities are \(e^{-1}\), \(e^{-1}\), and \(1-2e^{-1}\), giving expected counts approximately \(36.79\), \(36.79\), and \(26.42\).

The Pearson statistic is

X^2 =\frac{(37-36.79)^2}{36.79} +\frac{(37-36.79)^2}{36.79} +\frac{(26-26.42)^2}{26.42} \approx 0.00925.

With three categories and no estimated parameters, the degrees of freedom are \(3-1=2\). The upper-tail p-value is approximately \(0.995\). The same observed counts therefore lead to a different reference distribution than in the fitted example: 2 degrees of freedom when the rate is fixed, and 1 when it is estimated from these grouped counts. A p-value depends on both the statistic and its reference distribution.

Why the Estimator Must Match the Grouping

The parameter-count rule is not a license to subtract one degree of freedom whenever any estimate appears in the analysis. For the usual \(k-1-m\) result, the estimate must be fitted using the grouped-category likelihood for the categories in the test. If the rate is estimated from ungrouped counts instead, that estimate can use information not represented by the grouped table. Its effect on the Pearson statistic may not correspond to removing exactly one of the grouped residual directions.

Worked Example: A Raw-Count Estimate Is Not the Grouped Estimate

Suppose the same hypothetical grouped table has 37 zeros, 37 ones, and 26 observations in the 2-or-more category. Further suppose that, in the ungrouped data, all 26 observations in that last category are exactly 2. Compare the raw-count mean with the grouped maximum-likelihood estimate.

The ungrouped observations then consist of 37 zeros, 37 ones, and 26 twos. Their total is \(37(0)+37(1)+26(2)=89\), so the raw-count mean is

\bar{x}=\frac{89}{100}=0.890.

This differs from the grouped-category maximum-likelihood estimate, \(\hat{\lambda}\approx0.9909\), found by maximizing the likelihood for the three categories. The grouped likelihood knows only that each of those 26 observations is at least 2; it does not distinguish a value of 2 from a larger value. The raw mean uses that additional detail.

If the Pearson statistic is calculated from the three grouped categories but \(\lambda\) is estimated from the raw counts, the standard \(k-2\) calibration is not guaranteed. The raw-data estimate's score equation is not generally the same as the grouped score equation, so the fitted estimate need not remove exactly one direction from the grouped statistic. A suitable asymptotic covariance calculation or a parametric bootstrap that reproduces the estimation and grouping procedure may be needed to calibrate the test.

Common Mistake and Practical Use

A common mistake is to count the parameters in the model and subtract that number from \(k-1\) without checking how those parameters were estimated. For example, “one Poisson rate was estimated, so the test has \(k-2\) degrees of freedom” is justified in the standard result when the rate is estimated by grouped maximum likelihood from the same categories. It is not a general rule for every way of estimating the rate.

Before reporting a goodness-of-fit result with estimated probabilities, identify the categories, state which parameters were estimated, and say what data and likelihood were used to estimate them. Then determine whether the usual fitted-model calibration applies. The degrees of freedom affect the p-value, so a mismatch between the fitting method and the reference distribution can lead to an incorrect inferential conclusion.

Key takeaway: For a regular goodness-of-fit model fitted by maximum likelihood to the same grouped counts used in the test, the usual degrees of freedom are \(k-1-m\). For a fitted Poisson model, estimate the rate from those grouped categories before using this standard adjustment.

Check Your Understanding

Answer each question by identifying what is fixed or estimated and how the categories enter the analysis.

A goodness-of-fit test has six categories with fixed probabilities specified in advance. What are the usual degrees of freedom, and why?
A model has five categories and two identifiable parameters estimated by grouped maximum likelihood from the same counts. Under the usual regularity conditions, what are the degrees of freedom?
For Poisson categories 0, 1, and 2 or more, one rate is estimated from the grouped-category likelihood. What are \(k\), \(m\), and the usual degrees of freedom?
Why might estimating a Poisson rate from the raw counts fail to justify the usual \(k-2\) degrees of freedom for a test based on pooled categories?
In the fitted Poisson example, what does the p-value of approximately \(0.965\) describe, and what does it not establish?