From a P-Value to a Decision
In “Running the Chi-Square Test on a Calculator,” you learned how to obtain the chi-square statistic, degrees of freedom, p-value, and expected counts. The next step is to compare the p-value with the significance level, \(\alpha\), chosen for the test. That comparison determines whether the sample provides enough evidence to reject the null hypothesis.
For a chi-square test, the p-value is an upper-tail probability: assuming the null hypothesis is true, it is the probability of getting a chi-square statistic at least as large as the one observed. A small p-value means the observed differences between the observed and expected counts would be unusual if the null model were true. The decision rule is the same whether the test is for independence or homogeneity.
The significance level is the cutoff for deciding how much evidence against \(H_0\) is required before rejecting it. Common choices include \(\alpha=0.05\) and \(\alpha=0.01\). It should be set before examining the results, rather than selected afterward to produce a preferred decision.
The decision rule has two possible outcomes. Reject \(H_0\) means that the data provide convincing evidence for the alternative hypothesis at the chosen significance level. Fail to reject \(H_0\) means that the data do not provide convincing evidence for the alternative at that level. It does not establish that the null hypothesis is true, or that the variables are definitely independent or the distributions definitely equal.
Making the Comparison Carefully
A chi-square test’s p-value already accounts for the observed statistic and its degrees of freedom. Compare that p-value directly with \(\alpha\); do not compare the chi-square statistic itself with \(\alpha\). The statistic and \(\alpha\) are on different scales.
Also keep the stated direction of the comparison in mind: reject when the p-value is at most \(\alpha\), including equality. If the p-value is greater than \(\alpha\), fail to reject. When the calculator displays more digits than you plan to report, use its unrounded value to make the decision, especially when the result is close to \(\alpha\).
Use the stated \(\alpha\) for the test. Do not substitute a different level after seeing the p-value.
Confirm the test’s p-value and check that the reported degrees of freedom match the table.
Reject \(H_0\) if \(p\le\alpha\); otherwise, fail to reject \(H_0\).
Describe the evidence for the alternative in the context of the variables or groups being studied. Do not claim that failing to reject proves \(H_0\).
This comparison does not replace the checks covered in earlier tutorials, including “Random Condition for Chi-Square Procedures,” “Independence and 10 Percent Condition for Chi-Square,” and “Checking the Expected Count Condition for Chi-Square.” A small p-value does not make an inappropriate design or unmet condition acceptable.
Worked Example: Rejecting at the Five-Percent Level
Worked Example: Rejecting at the Five-Percent Level
An invented survey takes a random sample of 100 residents from a community of 2,000 residents. Each resident is classified by whether they usually commute by bicycle or another method, and whether they use a neighborhood travel-planning app. The observed counts are:
| Usual commute | Uses app | Does not use app | Total |
|---|---|---|---|
| Bicycle | 36 | 14 | 50 |
| Other method | 24 | 26 | 50 |
| Total | 60 | 40 | 100 |
State. The population of interest is the community’s residents. The null hypothesis is that usual commute method and app use are independent in this population; the alternative is that they are associated.
Plan. This is a chi-square test of independence because one sample is classified by two categorical variables. The sample is random, and each resident is counted in only one cell. The 10% condition holds because \(100\le0.10(2000)=200\). Under independence, the expected counts are 30 and 20 in each row: for example, \(50(60)/100=30\) for bicycle commuters who use the app, and \(50(40)/100=20\) for bicycle commuters who do not. All expected counts are at least 5.
Do. The chi-square statistic is the sum of the cell contributions:
The degrees of freedom are \((2-1)(2-1)=1\). The calculator gives an upper-tail p-value of approximately \(0.0143\), rounded to four decimal places. As a check, the chi-square distribution with 1 degree of freedom gives an upper-tail probability of about \(0.0143\) for \(X^2=6.00\).
Decide and conclude. At \(\alpha=0.05\), \(0.0143<0.05\), so reject \(H_0\). The random sample provides convincing evidence that usual commute method and travel-planning app use are associated among residents of this community.
Worked Example: Failing to Reject at the Ten-Percent Level
Worked Example: Failing to Reject at the Ten-Percent Level
In an invented survey, a random sample of 80 subscribers from a service with 2,000 subscribers is classified by device platform and preferred notification format. The observed table is:
| Device platform | Text | In-app notice | Total | |
|---|---|---|---|---|
| Platform A | 12 | 14 | 14 | 40 |
| Platform B | 18 | 16 | 6 | 40 |
| Total | 30 | 30 | 20 | 80 |
State. The question is whether device platform and preferred notification format are independent among the service’s subscribers. The null hypothesis states that they are independent; the alternative states that they are associated.
Plan. Use a chi-square test of independence. The random sample meets the random condition, each subscriber is counted once, and the 10% condition holds because \(80\le0.10(2000)=200\). The expected count in each row is \(40(30)/80=15\) for email, \(40(30)/80=15\) for text, and \(40(20)/80=10\) for in-app notices. Every expected count is at least 5.
Do. The six cell contributions are:
Their sum is \(0.6+0.0667+1.6+0.6+0.0667+1.6\approx4.5333\). The degrees of freedom are \((2-1)(3-1)=2\). The calculator reports \(p\approx0.1037\), rounded to four decimal places.
Decide and conclude. At \(\alpha=0.10\), \(0.1037>0.10\), so fail to reject \(H_0\). The sample does not provide convincing evidence of an association between device platform and preferred notification format among subscribers. This decision does not prove that the variables are independent; the sample may simply not provide sufficiently strong evidence against independence.
Worked Example: The Same P-Value, a Different Alpha
Worked Example: The Same P-Value, a Different Alpha
An invented random sample of 100 residents from a city of 5,000 is classified by whether they participate in a neighborhood tool library and whether they prefer weekday or weekend events. The observed counts are:
| Tool-library participation | Weekday preference | Weekend preference | Total |
|---|---|---|---|
| Participates | 30 | 20 | 50 |
| Does not participate | 20 | 30 | 50 |
| Total | 50 | 50 | 100 |
The null hypothesis is that participation and event-day preference are independent among city residents; the alternative is that they are associated. This is a chi-square test of independence. The sample is random, each resident contributes to one cell, and the 10% condition holds because \(100\le0.10(5000)=500\). All four expected counts equal \(50(50)/100=25\), so the expected-count condition is met.
The chi-square statistic is:
The degrees of freedom are \((2-1)(2-1)=1\), and the upper-tail p-value is approximately \(0.0455\), rounded to four decimal places.
At \(\alpha=0.05\), \(0.0455\le0.05\), so reject \(H_0\). At \(\alpha=0.01\), \(0.0455>0.01\), so fail to reject \(H_0\). The p-value has not changed; the decision changes because the significance level changes. For a given analysis, use the alpha specified or chosen before examining the data.
Common Mistakes and AP Exam Tip
- Comparing \(X^2\) with \(\alpha\): Compare the p-value with \(\alpha\). The chi-square statistic is used to determine the p-value, but it is not the decision threshold.
- Reversing the inequality: A smaller p-value gives stronger evidence against \(H_0\). Reject when \(p\le\alpha\), not when \(p\ge\alpha\).
- Saying “accept \(H_0\)” or “prove independence”: When \(p>\alpha\), say “fail to reject \(H_0\).” Explain that the data do not provide convincing evidence for the alternative; do not claim that the null has been proven.
- Choosing alpha after seeing the output: That changes the decision rule after the evidence is known. Use the significance level specified in the question or set in advance.
- Rounding before deciding: If the p-value is close to \(\alpha\), use the calculator’s unrounded value for the comparison. Report a suitably rounded value afterward.
- Making a claim the test does not support: A significant chi-square test indicates evidence of an association or a difference in distributions. It does not, by itself, identify which cells account for the result or establish causation.
For full credit, state the chosen \(\alpha\), report the p-value, explicitly compare the two, and say either “reject \(H_0\)” or “fail to reject \(H_0\).” Then describe the evidence in context using the alternative hypothesis. A decision is about evidence at a stated threshold, not certainty about whether the null hypothesis is true.
Check Your Understanding
For each question, use the stated significance level and remember that chi-square p-values are upper-tail probabilities.
- A test has \(p=0.032\) and \(\alpha=0.05\). What is the decision?
- A test has \(p=0.084\) and \(\alpha=0.05\). What is the decision, and what should you avoid claiming?
- A test has \(p=0.010\) and \(\alpha=0.01\). Should you reject or fail to reject \(H_0\)?
- Why is it important to choose \(\alpha\) before examining the p-value?
- A chi-square test rejects the null hypothesis of independence. Does that alone show which cells differ most or prove a causal relationship?