Why Small Calculation Choices Matter
A classical test statistic is not just a formula to evaluate. Its standard error, tail, and reference distribution are determined by the null hypothesis and the test design. A calculation can look plausible and still answer the wrong question—for example, by using a confidence-interval standard error in a test or by counting an estimated parameter incorrectly in the degrees of freedom.
This tutorial focuses on checks that help catch such errors before interpreting a p-value. The procedures assume the sampling conditions relevant to each test are met: for example, independent observations for the usual one-sample proportion test, or adequate expected counts for a chi-square approximation. If those conditions are not credible, a correctly calculated statistic does not by itself make the test appropriate.
Standard Errors Must Match the Question
A standard error measures the sampling variability of a statistic. In a hypothesis test, the null hypothesis often supplies a value to use when calculating that variability. In a confidence interval, the variability is usually estimated from the sample. These are different tasks, so their standard errors need not be the same.
For a one-sample test of \(H_0:p=p_0\), with \(n\) independent Bernoulli observations, the null-based standard error of the sample proportion is \(\sqrt{p_0(1-p_0)/n}\). By contrast, the usual estimated standard error used for a confidence interval for \(p\) is \(\sqrt{\hat p(1-\hat p)/n}\), where \(\hat p=x/n\). The test statistic compares the observed \(\hat p\) to the null value \(p_0\); the interval estimates uncertainty around \(\hat p\).
Worked Example: Test a Proportion Without Substituting the Sample Rate
A hypothetical quality audit samples \(n=100\) independent units and finds \(x=62\) with a specified feature. Test \(H_0:p=0.50\) against \(H_A:p>0.50\) using a normal-approximation z test. Then calculate the usual 95% confidence interval for \(p\). Assume the audit sample is representative and the normal approximation is acceptable for both calculations.
Method and test calculation. The sample proportion is \(\hat p=62/100=0.62\). Under the null hypothesis, the standard error is \(\sqrt{0.50(1-0.50)/100}=\sqrt{0.0025}=0.05\). Thus \(z=(0.62-0.50)/0.05=2.40\).
The alternative is “greater than,” so this is an upper-tail test. The p-value is \(P(Z\geq2.40)\approx0.0082\) for a standard normal random variable \(Z\). At the 5% significance level, reject \(H_0\).
Interval calculation. For the interval, use the sample-based standard error: \(\sqrt{0.62(0.38)/100}=\sqrt{0.002356}\approx0.04854\). The 95% Wald interval is \(0.62\pm1.96(0.04854)=0.62\pm0.09514\), or approximately \((0.5249,0.7151)\). The interval and test both indicate values above 0.50 in this example, but they use different standard errors for different purposes.
Check. The null value 0.50 lies below the interval, consistent with rejecting the two-sided null at 5%; the one-sided test also rejects. The p-value is rounded to four decimals and interval endpoints to four decimals. Common mistake: using \(\hat p=0.62\) in the test standard error. That would use \(\sqrt{0.62(0.38)/100}\) and produce a different statistic; it is not the standard error specified by this null-based z test.
The same distinction appears in two-proportion procedures. For testing \(H_0:p_1=p_2\), the null says the populations share a proportion. The usual pooled estimate is therefore used to estimate that common proportion. For a confidence interval estimating \(p_1-p_2\), the two sample proportions are estimated separately, so the usual interval standard error is unpooled.
Worked Example: Pool for the Equality Test, Not for the Interval
In a hypothetical study, 44 of 80 users of one interface complete a task successfully, compared with 40 of 100 users of another interface. Test \(H_0:p_1=p_2\) against \(H_A:p_1>p_2\) with a two-proportion z test. Assume independent random samples and an adequate normal approximation.
The sample proportions are \(\hat p_1=44/80=0.55\) and \(\hat p_2=40/100=0.40\). Under \(H_0\), the pooled estimate is \(\hat p=(44+40)/(80+100)=84/180=0.4667\).
The pooled standard error is \(\sqrt{0.4667(1-0.4667)(1/80+1/100)}\). Since \(0.4667(0.5333)\approx0.2489\) and \(1/80+1/100=0.0225\), this is \(\sqrt{0.0056}\approx0.07483\).
The test statistic is \(z=(0.55-0.40)/0.07483\approx2.005\). Because the alternative is \(p_1>p_2\), use the upper tail: the p-value is approximately \(0.0225\). At the 5% level, reject equality in favor of a larger success proportion for interface 1.
For comparison, the unpooled standard error used for a usual interval is \(\sqrt{0.55(0.45)/80+0.40(0.60)/100} =\sqrt{0.00309375+0.0024} =\sqrt{0.00549375}\approx0.07412\). It is not the standard error for this pooled equality test. Check: the observed difference is positive, and the one-sided alternative and upper-tail p-value have matching directions. Common mistake: choosing a pooled or unpooled formula by habit rather than by the question. The null of equality justifies pooling for this test; estimating a difference for an interval does not.
Tail Selection and Reference Distributions
The alternative hypothesis determines the tail for a signed statistic such as a z or t statistic: \(H_A:\theta>\theta_0\) gives an upper-tail test, \(H_A:\theta<\theta_0\) gives a lower-tail test, and \(H_A:\theta\ne\theta_0\) gives a two-sided test. A two-sided p-value is not obtained by doubling every p-value indiscriminately; it is based on outcomes at least as extreme in either direction under the relevant null distribution.
For Pearson chi-square goodness-of-fit and independence tests, the statistic is a sum of nonnegative squared discrepancies. Large values indicate disagreement with the null model, so the p-value is always an upper-tail probability. The fact that a chi-square statistic cannot be negative does not make every observed value evidence against the null: a small statistic is generally not unusual under the null.
Worked Example: Goodness of Fit With Known Probabilities
A hypothetical system records one of four event types. Under a specified model, their probabilities are \(0.10, 0.20, 0.30,\) and \(0.40\), respectively. In 100 independent observations, the counts are \(5,25,30,\) and \(40\). Test the specified model using a Pearson chi-square goodness-of-fit test. Assume the probabilities are fully specified in advance and the chi-square approximation is appropriate.
The expected counts are \(100(0.10)=10\), \(100(0.20)=20\), \(100(0.30)=30\), and \(100(0.40)=40\). Each expected count is at least 5.
The statistic is \(\chi^2=(5-10)^2/10+(25-20)^2/20+(30-30)^2/30+(40-40)^2/40\) \(=25/10+25/20+0+0=2.5+1.25=3.75\).
There are \(k=4\) categories and no parameters estimated from the data, so the degrees of freedom are \(k-1=3\). The p-value is \(P(\chi^2_3\geq3.75)\approx0.2893\). At the 5% level, do not reject the specified model. Check: the test uses an upper-tail probability and all expected counts meet the stated minimum check. Common mistake: using a lower tail because the first observed count is below its expectation, or treating the four separate cell differences as four independent tests. The test uses the single summed statistic.
Degrees of Freedom and Expected Counts
Degrees of freedom depend on which quantities were fixed and which were estimated. For a goodness-of-fit test with \(k\) categories and fully specified probabilities, the Pearson statistic has \(k-1\) degrees of freedom: the \(k\) observed counts sum to the fixed total, imposing one constraint. If \(m\) independent model parameters are estimated from the data in the standard fitted-model setting, the usual degrees of freedom become \(k-1-m\). Do not subtract a parameter that was fixed in advance, and do not forget one that was estimated from the same data.
For an \(r\)-by-\(c\) independence table, the usual degrees of freedom are \((r-1)(c-1)\), reflecting the constraints from the row and column margins. A one-sample chi-square test for a normal population variance uses \(n-1\) degrees of freedom under the normal-population assumption. These formulas belong to different test designs; selecting the right reference distribution includes selecting its degrees of freedom.
Expected counts are calculated under the null model, not copied from observed counts. In a contingency table, the expected count in row \(i\), column \(j\), under independence is \(E_{ij}=(\text{row }i\text{ total})(\text{column }j\text{ total})/(\text{grand total})\). For goodness of fit, it is \(E_i=np_i\), where \(p_i\) is the null probability for category \(i\). Rules of thumb for adequate expected counts help assess the chi-square approximation; they do not change the expected counts or the degrees-of-freedom formula. When counts are too sparse, consider whether categories can be combined meaningfully or whether another method is needed.
Worked Example: Independence Counts Come From Margins
A hypothetical survey records whether 100 respondents prefer one of three service options, grouped by membership status. The observed counts are: members, \(15,10,15\); nonmembers, \(15,25,20\). Test independence using Pearson’s chi-square test. Assume respondents are independent and the approximation conditions are acceptable.
The row totals are 40 and 60; the column totals are 30, 35, and 35; the grand total is 100. Under independence, the expected counts in the first row are \(40(30)/100=12\), \(40(35)/100=14\), and \(40(35)/100=14\). The second-row expected counts are \(60(30)/100=18\), \(60(35)/100=21\), and \(60(35)/100=21\). All are at least 12.
The six contributions are \(9/12=0.7500\), \(16/14\approx1.1429\), \(1/14\approx0.0714\), \(9/18=0.5000\), \(16/21\approx0.7619\), and \(1/21\approx0.0476\). Their sum is \(\chi^2\approx3.2738\). With \(r=2\) rows and \(c=3\) columns, the degrees of freedom are \((2-1)(3-1)=2\). The upper-tail p-value is approximately \(0.1945\), so do not reject independence at 5%.
Check. The expected counts reproduce the observed row and column totals when summed, as they should under this calculation. Common mistake: using the observed cell count as its own expected count. Expected counts must come from the null hypothesis—in this case, independence and the table margins.
Continuity Corrections in Two-by-Two Tables
A two-by-two table contains discrete counts, while the chi-square reference distribution is continuous. Yates’ continuity correction adjusts each absolute observed-minus-expected difference downward by \(0.5\), but not below zero. The corrected statistic is \(\sum (|O-E|-0.5)^2/E\) for the four cells when \(|O-E|\geq0.5\). Use it only when the procedure calls for the correction; do not apply it to every chi-square test or subtract \(0.5\) from a signed difference before taking its absolute value.
Worked Example: Apply Yates’ Correction Cell by Cell
A hypothetical two-group study produces the table: group A has 20 successes and 10 failures; group B has 8 successes and 22 failures. Test equality of success proportions using a two-sided two-by-two chi-square test with Yates’ correction. Assume independent groups and adequate conditions for the corrected approximation.
The row totals are 30 and 30, the column totals are 28 and 32, and the grand total is 60. The expected counts are \(30(28)/60=14\), \(30(32)/60=16\), \(30(28)/60=14\), and \(30(32)/60=16\). The absolute discrepancies are \(6,6,6,6\).
After correction, each discrepancy is \(6-0.5=5.5\). Thus \(\chi^2_Y=5.5^2/14+5.5^2/16+5.5^2/14+5.5^2/16\) \(=30.25(1/14+1/16+1/14+1/16)\) \(=30.25(15/56)\approx8.1027\).
The degrees of freedom are \((2-1)(2-1)=1\). The upper-tail p-value is approximately \(0.0044\), so reject equality at the 5% level. Check. Without correction, the statistic would be \(36(15/56)\approx9.6429\), which is larger; the correction reduced the statistic as intended. Common mistake: subtracting 0.5 from \(O-E\) with its sign intact. The correction uses the absolute discrepancy, then subtracts 0.5.
A Practical Error-Checking Sequence
A brief audit can prevent several mistakes at once. Use the following sequence before reporting a test result:
Confirm whether the claim is directional or two-sided, and whether the test is about a mean, proportion, variance, or counts.
Check whether a null value or pooled estimate is required. Keep interval-estimation formulas distinct from test formulas.
Calculate expected counts from the null model or margins, then assess whether the approximation is reasonable.
Use the test design, count constraints, and number of parameters estimated from the data.
Use the correct tail; for Pearson chi-square tests, large statistics give upper-tail evidence against the null.
The most effective exam habit is to write a one-line justification beside each choice: “pooled because the null equates the proportions,” “upper tail because \(H_A\) is greater,” or “\(k-1-m\) because \(m\) parameters were fitted.” That makes a wrong formula easier to spot than a final number alone and is aligned with the SOA VEE Mathematical Statistics syllabus.
Check Your Understanding
For each problem, identify the relevant diagnostic before calculating.
- A hypothetical audit finds 44 qualifying cases in a sample of 64. Test \(H_0:p=0.60\) against \(H_A:p>0.60\) using the null-based one-proportion z statistic. (Answer: \(\hat p=0.6875\), standard error \(=\sqrt{0.60(0.40)/64}\approx0.0612\), \(z\approx1.429\), upper-tail \(p\approx0.0765\).)
- Two independent hypothetical samples have 36 successes in 60 trials and 24 successes in 60 trials. For \(H_0:p_1=p_2\) against a two-sided alternative, give the pooled estimate, pooled standard error, z statistic, and approximate p-value. (Answer: pooled estimate \(=60/120=0.50\), standard error \(\sqrt{0.25(1/60+1/60)}\approx0.0913\), \(z\approx2.191\), two-sided \(p\approx0.0285\).)
- A goodness-of-fit test has four categories, specified probabilities \(0.20,0.30,0.10,0.40\), and observed counts \(18,32,8,42\) from 100 observations. Give the expected counts, Pearson statistic, degrees of freedom, and approximate upper-tail p-value. (Answer: expected counts \(20,30,10,40\); \(\chi^2=0.2+0.1333+0.4+0.1=0.8333\); \(df=3\); \(p\approx0.841\).)
- An independence test uses a table with three rows and two columns. One model parameter was estimated from the cell counts before calculating the expected counts. What is the standard independence-test degrees-of-freedom formula, and should that fitted parameter automatically be subtracted from it? (Answer: \(df=(3-1)(2-1)=2\); the ordinary independence formula is used for that test structure, and a fitted parameter is not automatically subtracted as in a goodness-of-fit test.)
- A two-by-two table has row totals 25 and 25, column totals 26 and 24, and observed rows \((17,8)\) and \((9,16)\). Find the expected counts and the Yates-corrected statistic. (Answer: expected counts \(13,12,13,12\); each absolute discrepancy is 4, so \(\chi^2_Y=3.5^2(2/13+2/12)\approx3.9263\), with \(df=1\).)