Why Measure the Size of a Mismatch?
A goodness-of-fit test addresses whether observed category counts are compatible with a specified probability model. Its p-value helps assess how surprising the observed discrepancy would be if that model were correct. But a p-value alone does not describe the size of the discrepancy. With a sufficiently large sample, even modest differences can produce a small p-value; with a small sample, a meaningful difference may not be detected clearly.
Cohen’s \(w\) provides a standardized summary of the discrepancy between the observed category proportions and the probabilities specified by the null model. It complements the Pearson chi-square statistic and p-value, but it does not show which categories are above or below their expected proportions. For a useful report, consider all three: the effect size, the inferential result, and the observed category pattern.
The formula compares each category’s probability difference with the null probability for that category. Because the differences are squared, departures in either direction add to the overall size: \(\hat{w}\) does not say whether a category occurred more or less often than expected. The category proportions and counts are needed to describe direction.
Why the Sample Formula Works
The relation between Pearson’s statistic and Cohen’s \(w\) follows directly from expressing counts as proportions. Write \(\hat{p}_i=O_i/n\) for the observed proportion in category \(i\). Under the fixed null model, the expected count is \(E_i=np_{0i}\). Substituting these expressions into Pearson’s statistic gives:
\(\chi^2=\sum_{i=1}^{k}(O_i-E_i)^2/E_i\), with \(E_i=np_{0i}\).
Since \(O_i=n\hat{p}_i\) and \(E_i=np_{0i}\), each term becomes \((n\hat{p}_i-np_{0i})^2/(np_{0i})=n(\hat{p}_i-p_{0i})^2/p_{0i}\).
Summing the terms gives \(\chi^2=n\sum_{i=1}^{k}(\hat{p}_i-p_{0i})^2/p_{0i}\).
\(\sqrt{\chi^2/n}=\sqrt{\sum_{i=1}^{k}(\hat{p}_i-p_{0i})^2/p_{0i}}\), which is the sample version of the population discrepancy formula.
This derivation also explains a useful distinction. For observed proportions that stay the same, increasing the sample size increases \(\chi^2\) in proportion to \(n\), while \(\hat{w}\) remains the same. Thus \(\chi^2\) reflects both discrepancy and sample size, whereas \(\hat{w}\) summarizes the discrepancy in proportion terms. The sample value still varies from sample to sample; even when the null model is true, random sampling can produce a nonzero observed \(\hat{w}\).
Worked Calculations
The examples below use a fixed null probability vector. For the chi-square approximation and p-values, assume observations are independent, each observation falls in exactly one of the stated categories, the probabilities in the null model were specified rather than estimated from these data, and expected counts are adequate for the chi-square approximation. Each example checks the expected counts. The degrees of freedom are \(k-1\) because no probability is estimated from the sample.
Worked Example: A Modest Difference in 300 Observations
Suppose a hypothetical process has three categories with null probabilities \(0.50\), \(0.30\), and \(0.20\). In a sample of \(n=300\), the observed counts are \(165\), \(81\), and \(54\). Calculate \(\hat{w}\), and interpret it alongside the p-value and category differences.
Expected counts and conditions. The expected counts are \(300(0.50)=150\), \(300(0.30)=90\), and \(300(0.20)=60\). All are at least 5. Assuming each observation is independent and assigned to one category, the usual chi-square approximation is reasonable.
Pearson’s statistic.
Effect size.
The observed proportions are \(165/300=0.55\), \(81/300=0.27\), and \(54/300=0.18\). Relative to the null probabilities, the first category is higher by \(0.05\), while the second and third are lower by \(0.03\) and \(0.02\). With \(df=3-1=2\), the p-value is \(P(\chi^2_2\geq3.00)=0.2231\). These data show a sample discrepancy of \(\hat{w}=0.100\); the test does not provide strong evidence against the specified distribution at conventional significance levels. The p-value does not establish that the null probabilities are exactly correct.
Worked Example: The Same Proportions in a Larger Sample
Now suppose a second hypothetical sample has \(n=1200\), with observed counts \(660\), \(324\), and \(216\) under the same null probabilities \(0.50\), \(0.30\), and \(0.20\). Compare its effect size and p-value with those in the previous example.
Expected counts and conditions. The expected counts are \(600\), \(360\), and \(240\), all greater than 5. Under the same independence and category-assignment assumptions, the chi-square approximation is appropriate.
Pearson’s statistic.
Effect size.
The observed proportions are again \(0.55\), \(0.27\), and \(0.18\), so their differences from the null probabilities are unchanged. The effect size is also unchanged at \(0.100\). However, with \(df=2\), the p-value is \(P(\chi^2_2\geq12.00)=0.002479\). The larger sample yields stronger evidence against the null model for the same observed proportion pattern. This comparison shows why neither a p-value nor \(\hat{w}\) should be reported as if it answered the other’s question.
Worked Example: A Larger Proportional Discrepancy
In a third hypothetical setting, the null probabilities for three categories are \(0.25\), \(0.50\), and \(0.25\). A sample of \(n=200\) produces counts \(70\), \(90\), and \(40\). Calculate the test statistic and effect size, then describe which categories contribute to the mismatch.
Expected counts and conditions. The expected counts are \(50\), \(100\), and \(50\), each at least 5. Assuming independent observations and one category per observation, the chi-square approximation is reasonable.
Pearson’s statistic.
Effect size.
The observed proportions are \(0.35\), \(0.45\), and \(0.20\). Compared with the null probabilities, the first category is \(0.10\) higher, the second is \(0.05\) lower, and the third is \(0.05\) lower. Their Pearson contributions are \(8.00\), \(1.00\), and \(2.00\), respectively, so the first category accounts for the largest part of the statistic. With \(df=2\), the p-value is \(P(\chi^2_2\geq11.00)=0.004087\). Here the sample shows a larger standardized mismatch than in the first two examples, with the main contribution coming from the first category.
Interpreting and Reporting Cohen’s \(w\)
Cohen’s \(w\) is nonnegative and equals zero when every population category probability matches its null probability. Larger values indicate greater standardized discrepancy from the specified model. It has no units. The formula weights a difference in each category by the corresponding null probability, so the same raw probability difference can contribute differently depending on how probable that category is under the null.
Cohen’s conventional reference values of about \(0.10\), \(0.30\), and \(0.50\) are sometimes described as small, medium, and large. They are rough orientation points, not universal cutoffs. Whether a discrepancy matters depends on the subject, the consequences of being wrong, and which categories differ. In a clinical setting, for example, a small overall effect could still matter if it concentrates in a clinically important category. Do not turn a conventional label into a claim of practical importance without considering context.
A useful report names the model and categories, gives \(\hat{w}\), and also reports the test statistic, degrees of freedom, p-value, and observed-versus-expected pattern. For example, one might report that the sample had \(\hat{w}=0.100\) and \(\chi^2(2)=3.00\), \(p=0.2231\), with observed proportions of \(0.55\), \(0.27\), and \(0.18\) compared with null probabilities of \(0.50\), \(0.30\), and \(0.20\). This states the result without implying that the p-value measures effect size.
Common Mistakes
- Confusing effect size with statistical evidence. A student might say that \(\hat{w}=0.100\) proves the null model is true because the value seems small. The correction is that \(\hat{w}\) summarizes the sample’s discrepancy; it does not prove a model, and it does not give the probability that the null is true.
- Using \(\chi^2\) without dividing by \(n\). The sample calculation is \(\hat{w}=\sqrt{\chi^2/n}\), not \(\sqrt{\chi^2}\). Omitting \(n\) makes the result grow with sample size even if the observed proportions stay fixed.
- Using \(w\) to identify the categories responsible. Because the category differences are squared, \(w\) has no direction information. Report the observed and expected counts or proportions, and use category contributions or residuals to describe where the mismatch occurs.
- Treating conventional benchmarks as decision rules. Labels such as small or large are descriptive guides, not automatic thresholds for clinical importance or a substitute for subject-matter judgment.
Check Your Understanding
Answer each question using the definitions and examples above.
- A fixed-model goodness-of-fit test has \(\chi^2=8.00\) and \(n=200\). Calculate \(\hat{w}\).
- Two samples have the same observed proportions relative to the same fixed null probabilities, but one sample is four times as large. Explain why their \(\hat{w}\) values can be equal while their chi-square statistics differ.
- Why can Cohen’s \(w\) not tell you whether a particular category was observed more often or less often than expected?
- A report gives \(\hat{w}=0.30\) but no observed counts or proportions. What important information about the mismatch is missing?
- Explain why a conventional “small” effect-size label does not, by itself, establish that the discrepancy is unimportant in a clinical context.