Why Small Analysis Errors Matter
A chi-square test of a two-way table can be calculated with a short formula, but a correct calculation does not guarantee a sound conclusion. The table may contain percentages where counts are needed, the wrong conditional distribution may be described, or the study design may not support the inference being made. Each mistake can change what the result means, even when the arithmetic looks plausible.
This tutorial focuses on ways to diagnose those errors. The central object is a table of observed counts: the number of individuals in each combination of categories. A chi-square test compares those counts with counts expected under a null hypothesis. It does not, by itself, establish why an association exists or whether it applies to people outside the study.
In a test of independence, the null hypothesis says that two categorical variables are independent in the population being studied. In a test of homogeneity, it says that the category distribution is the same across the populations or groups being compared. Both use the same Pearson statistic and expected-count calculation, but their study designs and interpretations differ.
Mistake 1: Using Percentages as Though They Were Counts
Percentages describe proportions, but the Pearson statistic uses counts. Percentages alone do not tell us how many observations produced them. For example, a difference between 30% and 45% is based on very different amounts of information if the group sizes are 20 and 60 than if they are 2,000 and 6,000. The raw counts and the group sizes must be retained for the test.
Worked Example: Counts and Percentages Give Different Test Statistics
Suppose, hypothetically, two independently sampled groups are asked whether they support a proposed service. Group A has 100 respondents, and Group B has 300. The observed counts are:
| Group | Support | Do not support | Total |
|---|---|---|---|
| A | 30 | 70 | 100 |
| B | 135 | 165 | 300 |
| Total | 165 | 235 | 400 |
The support rates are 30% in Group A and 45% in Group B. For the test of homogeneity, assume the groups were selected independently, each respondent contributed to exactly one cell, and the expected counts are adequate. The null hypothesis is that the support distribution is the same in the two groups; the alternative is that it differs.
Under the null hypothesis, the expected count in each cell is its row total times its column total, divided by the grand total. Thus the expected counts are 41.25 and 58.75 for Group A, and 123.75 and 176.25 for Group B. All are at least 5. The statistic is
Here \(df=(2-1)(2-1)=1\), giving an upper-tail p-value of approximately 0.00831. At the 0.05 significance level, this is statistically significant evidence that the support distributions differ between the groups.
If we incorrectly enter the percentages as though they were counts, we create a different table: A has 30 and 70, while B has 45 and 55. That artificial table has row totals of 100 in both groups, and its Pearson statistic is 4.800, not 6.963. It produces a different p-value because the substitution has changed the apparent sample sizes and the expected counts. The rates are useful for describing the groups; they are not a substitute for the observed counts in the test.
Mistake 2: Reversing the Conditional Distribution
A row percentage uses the row total as its denominator. It describes the distribution across columns within a row. A column percentage uses the column total as its denominator. It describes the distribution across rows within a column. These answer different questions. Neither is automatically the right description: the denominator must match the question being asked.
Worked Example: Reading Row and Column Percentages Correctly
Imagine a hypothetical comparison of outcomes among patients at three clinics. The counts are:
| Clinic | Improved | Did not improve | Total |
|---|---|---|---|
| A | 36 | 24 | 60 |
| B | 30 | 30 | 60 |
| C | 24 | 36 | 60 |
| Total | 90 | 90 | 180 |
Assume for the homogeneity test that the clinic groups are independent samples, each patient appears once, and expected counts are adequate. The null hypothesis is that the outcome distribution is the same across clinics. Within Clinic A, the improvement percentage is \(36/60=60\%\); within B it is \(30/60=50\%\); within C it is \(24/60=40\%\). These are row percentages and answer, “What fraction improved at each clinic?”
The column percentages answer a different question. Among the 90 patients who improved, 40% were from A, 33.3% from B, and 26.7% from C. Among the 90 who did not improve, 26.7% were from A, 33.3% from B, and 40% from C. These percentages describe clinic membership within each outcome category. They should not be presented as if they were the improvement rates at each clinic.
For the test, the expected count in each clinic-outcome cell is 30: each clinic total is 60, each outcome total is 90, and the grand total is 180. Therefore,
The degrees of freedom are \(df=(3-1)(2-1)=2\), and the upper-tail p-value is approximately 0.09072. At the 0.05 level, the result is not statistically significant. Describing the row percentages correctly does not change that test result; it makes the direction and denominator of the descriptive comparison clear.
Mistake 3: Treating Expected Counts as Observations
An expected count is not another measurement, a guaranteed future count, or a claim that each cell “should” contain exactly that many individuals. It is a reference value calculated from the observed margins under the null hypothesis. The test measures the total discrepancy between the observed and expected counts, with each squared discrepancy scaled by its expected count.
Worked Example: Calculating and Interpreting Expected Counts
Suppose, hypothetically, independent random samples are taken from two clinics, with one outcome recorded for each sampled patient. The counts are:
| Clinic | Outcome A | Outcome B | Total |
|---|---|---|---|
| 1 | 30 | 10 | 40 |
| 2 | 20 | 40 | 60 |
| Total | 50 | 50 | 100 |
For a test of homogeneity, assume the two samples are independent, observations within each sample are independent, and the expected counts are large enough for the chi-square approximation. Under the null hypothesis of equal outcome distributions, the expected count for Clinic 1 and Outcome A is \(E_{11}=(40)(50)/100=20\). The full expected table is 20 and 20 for Clinic 1, and 30 and 30 for Clinic 2. These are null-model reference counts; they are not observed data or required targets.
The statistic is
There are \(df=1\) degree of freedom, and the p-value is approximately 0.0000445. This is strong evidence against equal outcome distributions under the test's assumptions. The large statistic arises because the observed counts are far from their null-model expected counts relative to the expected-count scale. It does not mean that an expected count is a prediction that must be met exactly.
Mistake 4: Checking the Wrong Conditions—or Skipping Them
A chi-square calculation does not repair a poor design. Before interpreting a p-value, check that the data consist of counts for mutually exclusive cells; that each individual contributes appropriately to the table; that observations are independent under the study design; and that the expected counts meet the approximation condition taught for the analysis. In particular, check the expected counts, not merely the observed counts. A large observed cell does not guarantee that every expected count is large enough.
Random sampling and random assignment serve different purposes. A probability sample can support generalization to the population from which it was sampled, subject to the design and response limitations. Random assignment in an experiment can support a causal interpretation of treatment differences. One does not substitute for the other. If the study is observational, a significant association alone does not show that one variable caused the other.
Worked Example: A Small P-Value Does Not Fix a Convenience Sample
Imagine a hypothetical online poll promoted through student clubs. Among the respondents, 45 club members support a proposed campus service and 15 do not; among 60 nonmembers, 30 support it and 30 do not. Suppose each person responded once and the observations are treated as independent for the calculation. The table is:
| Group | Support | Do not support | Total |
|---|---|---|---|
| Club members | 45 | 15 | 60 |
| Nonmembers | 30 | 30 | 60 |
| Total | 75 | 45 | 120 |
For the independence test, the null hypothesis is that club membership and support are independent among the population under consideration. The expected counts are \(60(75)/120=37.5\) for support and \(60(45)/120=22.5\) for non-support in each row. They all exceed 5. The Pearson calculation is
With \(df=1\), the nominal upper-tail p-value is approximately 0.004678. The poll shows a clear difference in support among its respondents. However, recruitment through student clubs is not a probability sample of all students. The p-value does not make the respondents representative, so broad generalization to the student population is not justified by this design. Nor were students randomly assigned to be club members; the association does not establish that club membership caused support. The calculation and the limits of the design must be reported together.
A Practical Error-Checking Sequence
Before writing a conclusion, use a short sequence to catch the most common problems:
Confirm that entries are observed counts, categories are mutually exclusive, and row, column, and grand totals agree.
Decide whether the question is about independence between variables or equal distributions across groups; state the null hypothesis in context.
Calculate expected counts from the margins, then check the count condition and whether the observations and sampling or assignment process support the intended inference.
Label percentages as row or column percentages, and explain what group each percentage is calculated within.
Use the statistic, degrees of freedom, and p-value consistently. State evidence of association or distributional difference, then respect the limits on generalization and causation.
Common Mistakes to Avoid
- “The percentages are the data for the test.” Percentages are useful summaries, but use the observed counts to compute expected counts and the Pearson statistic.
- “Forty percent of improved patients were from Clinic A, so 40% of Clinic A patients improved.” Those statements use different denominators. The first is a column percentage; the second is a row percentage.
- “The expected count is what should actually have happened.” It is the null-model reference count calculated from the margins, not a second observation or a guaranteed outcome.
- “All observed counts are large, so the approximation condition is met.” Check every expected count against the applicable condition; observed counts and expected counts are not interchangeable.
- “A significant result proves a cause or represents everyone.” Statistical significance does not create random assignment or random sampling. Interpret the result according to the study design.
Check Your Understanding
For each question, explain the reasoning rather than giving only a label or number.
- Why can two tables with the same group percentages but different sample sizes produce different chi-square statistics?
- A report says that 35% of all respondents who chose “Yes” came from Group A. What denominator defines this percentage, and is it a row or column percentage?
- In a cell, the observed count is 12 and the expected count is 8. Explain what each number represents under a chi-square test.
- Why should the expected counts, rather than just the observed counts, be checked when assessing the chi-square approximation?
- A convenience poll produces a small p-value for an association between membership and preference. What can the result say about the respondents, and what does it not establish about the larger population or causation?