Interpreting Goodness-of-Fit P-Values

What a Goodness-of-Fit P-Value Measures

A chi-square goodness-of-fit test asks whether observed category counts are compatible with a specified probability model. After calculating the Pearson chi-square statistic, the next question is how unusual that statistic would be if the model were correct. The p-value answers this question by measuring the probability of getting a statistic at least as large as the one observed, under the null model.

The direction “at least as large” matters. The Pearson statistic is nonnegative, and larger values indicate greater overall disagreement between the observed counts and the counts expected under the null probabilities. Therefore, evidence against the model lies in the upper tail of the chi-square distribution—not in both tails.

Definition: For a one-way Pearson goodness-of-fit test with observed statistic \(X^2_{\text{obs}}\) and \(df\) degrees of freedom, the p-value is the probability, assuming the null model is true, that the test statistic is at least \(X^2_{\text{obs}}\). Using the chi-square calibration, this is \(P(\chi^2_{df} \geq X^2_{\text{obs}})\). When the calibration is approximate, the reported p-value is approximate as well.

The Null Distribution and the Upper Tail

For a fixed null probability model with \(k\) categories, the expected count in category \(i\) is \(E_i=np_i\), where \(n\) is the total count and \(p_i\) is the null probability for that category. The Pearson statistic is \(X^2=\sum_{i=1}^{k}(O_i-E_i)^2/E_i\), where \(O_i\) is the observed count. With fixed probabilities, the usual degrees of freedom are \(df=k-1\).

The chi-square distribution provides the reference distribution for this statistic under the null model. This calibration relies on appropriate conditions: the observations are independent, the categories are mutually exclusive and exhaustive, the null probabilities are specified, and expected counts are large enough for the chi-square approximation to be suitable. If those conditions are not reasonable, a p-value from this approximation may not be trustworthy.

A right-tail probability can be written as one minus the cumulative probability up to the observed statistic:

p\text{-value} =P(\chi^2_{df}\geq X^2_{\text{obs}}) =1-P(\chi^2_{df}<X^2_{\text{obs}})

This expression explains how to obtain a p-value from a chi-square table or software. A table may show that the p-value falls between two cutoffs rather than provide an exact value; software typically reports a numerical approximation. The degrees of freedom must match the test, and the comparison must use the upper-tail area.

Why the upper tail? Each term \((O_i-E_i)^2/E_i\) is nonnegative. If the observed counts lie close to their expected counts, \(X^2\) is relatively small; if they depart more substantially overall, \(X^2\) is larger. Thus, values at least as large as the observed statistic represent discrepancies at least as large by this measure. Small statistics are not evidence of a greater mismatch.

Worked Examples

Worked Example: A Moderate Statistic With a Large P-Value

Suppose, hypothetically, that 100 spins of a four-section spinner are classified by the section on which they land. The null model assigns probability \(0.25\) to each section. The observed counts are 32, 18, 27, and 23. Is there evidence against this equal-probability model at significance level \(\alpha=0.05\)?

Check the model conditions. Assume each spin is conducted independently and each result belongs to exactly one of the four sections. Under the null model, every expected count is \(100(0.25)=25\), which is large enough for the usual chi-square approximation.

Calculate the statistic. The expected counts are all 25, so

X^2 =\frac{(32-25)^2}{25} +\frac{(18-25)^2}{25} +\frac{(27-25)^2}{25} +\frac{(23-25)^2}{25} =\frac{49+49+4+4}{25} =4.24.

There are \(k=4\) fixed-probability categories, so \(df=4-1=3\). The p-value is the upper-tail probability \(P(\chi^2_3\geq4.24)\), approximately \(0.237\).

If the equal-probability model were correct, a statistic of 4.24 or larger would occur in about 23.7% of repeated sets of 100 spins under the stated conditions. Since \(0.237>0.05\), this test does not provide sufficient evidence at the 5% level against equal probabilities. It does not show that the four probabilities are exactly equal.

Worked Example: A Small P-Value and a Test Decision

In a hypothetical test of a two-outcome device, the null model says each outcome has probability \(0.5\). Of 80 independent trials, the observed counts are 49 and 31. Use \(\alpha=0.05\) to assess the null model.

Check the model conditions. Assume the 80 trials are independent and every trial produces exactly one of the two outcomes. Each expected count is \(80(0.5)=40\), so the expected counts are adequate for the chi-square approximation.

Calculate the statistic and degrees of freedom.

X^2 =\frac{(49-40)^2}{40}+\frac{(31-40)^2}{40} =\frac{81}{40}+\frac{81}{40} =4.05, \qquad df=2-1=1.

The upper-tail probability is \(P(\chi^2_1\geq4.05)\), approximately \(0.0442\). If the null model were true, a statistic at least this large would occur in about 4.42% of repeated sets of 80 trials.

Because \(0.0442<0.05\), the result is statistically significant at the 5% level. The data provide evidence against the specified equal-probability model. This conclusion does not establish why the counts differ, nor does the p-value itself identify a cause.

Worked Example: A Very Large P-Value Is Not Proof

Suppose, hypothetically, that 100 independent observations fall into three categories. The specified null probabilities are \(0.50\), \(0.30\), and \(0.20\), and the observed counts are 52, 28, and 20. Interpret the test result at \(\alpha=0.05\).

Check the model conditions. Assume the observations are independent and each is assigned to exactly one category. The expected counts are \(100(0.50)=50\), \(100(0.30)=30\), and \(100(0.20)=20\); all are adequate for the chi-square approximation.

Calculate the statistic.

X^2 =\frac{(52-50)^2}{50} +\frac{(28-30)^2}{30} +\frac{(20-20)^2}{20} =0.0800+0.1333+0 =0.2133.

Here \(df=3-1=2\). The upper-tail probability is \(P(\chi^2_2\geq0.2133)\), approximately \(0.899\). Under the null model, a statistic at least this large is not unusual; the observed statistic is small relative to the reference distribution. Since \(0.899>0.05\), there is not sufficient evidence to reject the specified probability model at the 5% level.

The large p-value does not mean there is an 89.9% probability that the null model is true. It describes how often statistics at least this large would arise if that model were true. A goodness-of-fit test can fail to find evidence against a model without proving that the model is correct.

From a P-Value to a Careful Conclusion

A p-value is interpreted conditionally: it assumes the null model and the test conditions. It is not the probability that the null hypothesis is true, the probability that chance alone caused the observed counts, or the probability of obtaining exactly the observed counts. It is a tail probability for the test statistic under the null reference distribution.

A small p-value means that the observed statistic would be relatively unusual if the specified null model were correct. It is evidence against that model, but it does not measure the size or practical importance of the mismatch. A large p-value means that the statistic is not unusual under the model; it is not affirmative proof that the model is true.

When a significance level is specified, compare the p-value with \(\alpha\). If \(p\leq\alpha\), reject \(H_0\); if \(p>\alpha\), do not reject \(H_0\). The significance level is a decision threshold chosen for the test, while the p-value summarizes the evidence in the observed statistic. State the decision in context and avoid stronger claims than the test supports.

State the null model.
Identify the category probabilities being assessed and confirm that the test conditions make the chi-square calibration appropriate.

Use the observed statistic and its degrees of freedom.
These determine the chi-square reference distribution for the p-value calculation.

Find the upper-tail probability.
Calculate \(P(\chi^2_{df}\geq X^2_{\text{obs}})\), not the lower-tail area.

Conclude in context.
Describe the evidence for or against the specified model, and do not interpret the p-value as the probability that the model is true.

Common Mistakes

Using the lower tail. A learner may calculate the probability of a statistic smaller than the observed value. The goodness-of-fit p-value is the probability of a statistic at least as large, because larger chi-square values indicate greater disagreement with the model.
Calling the p-value the probability that \(H_0\) is true. The p-value assumes \(H_0\) to calculate a probability; it does not assign a probability to \(H_0\).
Claiming a large p-value confirms the model. A large value means the observed statistic is not unusual under the model. It does not prove that the model is correct.
Claiming that a small p-value measures practical importance. It indicates incompatibility with the null model under the test assumptions; by itself, it does not say whether the discrepancy matters in practice.
Reporting only “significant” or “not significant.” Give the p-value and explain what it says about the specified probability model in the application.

Key takeaway: A goodness-of-fit p-value is an upper-tail probability under the null model: the chance of obtaining a chi-square statistic at least as large as the observed one. It measures how unusual the statistic would be under that model, not the probability that the model is true.

Check Your Understanding

Answer each question using the definition of the upper-tail p-value.

In a goodness-of-fit test, the observed statistic is \(X^2_{\text{obs}}=6.2\) with \(df=4\). Write the probability that defines its p-value.
Why does a goodness-of-fit test use the upper tail rather than the lower tail of the chi-square distribution?
A test gives \(p=0.18\) and uses \(\alpha=0.05\). State the decision and a suitable contextual interpretation.
Explain why a p-value of \(0.01\) does not mean there is a 1% probability that the null model is true.
What does a large p-value indicate, and what does it not establish?