Why Sample Size Changes Goodness-of-Fit Results
A goodness-of-fit test asks whether observed data are compatible with a specified distributional model. Compatibility is not the same as truth: a model may be a useful approximation even when the data come from a slightly different distribution. Whether a test detects that difference depends both on its form and on the sample size.
For a test with significance level \(\alpha\), the power against a particular alternative distribution is the probability of rejecting the null model when that alternative is the data-generating distribution. Equivalently, power is \(1-\beta\), where \(\beta\) is the probability of a Type II error for that alternative. Power is not a single property of a test: it depends on the alternative, sample size, significance level, and (for grouped tests) the category design.
The three tests considered here summarize discrepancy differently. The Kolmogorov–Smirnov (KS) statistic uses the largest vertical difference between the empirical and specified cumulative distribution functions. The Anderson–Darling (AD) statistic accumulates squared differences with extra weight near the tails. Pearson’s chi-square statistic compares observed and expected counts across categories. These summaries can respond differently to the same departure; sample size generally makes a persistent departure easier to detect.
How Sample Size Enters the Three Tests
For a fully specified continuous null distribution \(F_0\), let \(F_n\) be the empirical distribution function. The one-sample KS statistic is \(D_n=\sup_x|F_n(x)-F_0(x)|\). Under the null, its critical value is approximately \(c_\alpha/\sqrt n\), where \(c_\alpha\) depends on the significance level. At the 5% level, \(c_\alpha\) is approximately 1.36 in the usual large-sample approximation.
Under an alternative \(G\), define the population KS discrepancy \(D=\sup_x|G(x)-F_0(x)|\). When \(D>0\), the empirical discrepancy tends to this positive value as the sample grows, while the null critical threshold shrinks. The convergence claim requires more than convergence at one selected value of \(x\).
The AD statistic also accumulates discrepancies, but its weighting makes differences in regions where \(F_0(x)\) is close to 0 or 1 more influential. For a continuous, fully specified null, a common form is \(n\int (F_n-F_0)^2/[F_0(1-F_0)]\,dF_0\). Under an alternative, the corresponding population discrepancy is the weighted integral with \(G\) in place of \(F_n\). If that integral is finite and positive, the systematic part of the statistic grows on the scale of \(n\), rather than remaining at its null scale. Exact power still depends on the statistic’s distribution and its calibrated critical value.
For Pearson’s chi-square test, sample size appears through expected cell counts. If there are \(k\) fixed categories with null probabilities \(p_{0i}>0\), the expected count in category \(i\) is \(np_{0i}\). Under a fixed alternative with category probabilities \(p_{1i}\), departures in counts grow proportionally to \(n\), while the standard deviation of a count grows proportionally to \(\sqrt n\). This is why a fixed difference in category probabilities becomes more detectable as \(n\) increases. The category probabilities must represent the full alternative: a departure that does not change the category probabilities is invisible to this grouped statistic.
Worked Power Calculations
Worked Example: Approximate Power for a Binary Chi-Square Test
A hypothetical audit classifies each claim as either “routine” or “requires review.” Under the null model, the probabilities are \((0.50,0.50)\). Consider the alternative \((0.60,0.40)\), with independent claims and fixed category probabilities. Estimate the large-sample power of Pearson’s chi-square test at the 5% level for \(n=100\) and \(n=400\). Assume the null probabilities are fully specified.
Method. For fixed category probabilities, the Pearson statistic has an approximate noncentral chi-square distribution under the alternative. Its degrees of freedom are \(k-1=1\), and its noncentrality parameter is \(\lambda=n\sum_i(p_{1i}-p_{0i})^2/p_{0i}\). The 5% critical value for one degree of freedom is 3.841.
First calculate the discrepancy per observation: \((0.60-0.50)^2/0.50+(0.40-0.50)^2/0.50=0.01/0.50+0.01/0.50=0.02+0.02=0.04\). Thus \(\lambda=0.04n\).
For one degree of freedom, the noncentral chi-square variable can be written as \((Z+\sqrt{\lambda})^2\), where \(Z\) is standard normal. The approximate rejection probability is therefore \(P(Z>1.96-\sqrt{\lambda})+P(Z<-1.96-\sqrt{\lambda})\).
At \(n=100\), \(\lambda=4\) and \(\sqrt{\lambda}=2\). The power is \(P(Z>-0.04)+P(Z<-3.96)=\Phi(0.04)+\Phi(-3.96)\), approximately \(0.5160+0.0000=0.5160\).
At \(n=400\), \(\lambda=16\) and \(\sqrt{\lambda}=4\). The power is \(P(Z> -2.04)+P(Z<-5.96)=\Phi(2.04)+\Phi(-5.96)\), approximately \(0.9793+0.0000=0.9793\). The larger sample makes the same 10-percentage-point category departure much easier to detect.
Check. Both noncentrality parameters equal \(0.04n\), so increasing \(n\) from 100 to 400 multiplies \(\lambda\) by four. The calculated power increases, as expected. This is an asymptotic approximation, not an exact finite-sample power calculation. The common mistake is to report \(\lambda\) itself as power; \(\lambda\) is a parameter used to obtain the approximate rejection probability.
Worked Example: KS Separation and a Sample-Size Threshold
Suppose a hypothetical continuous measurement has null distribution \(F_0(x)=x\) on \(0\leq x\leq1\), but its actual distribution is \(G(x)=x^2\) on that interval. Find the population KS discrepancy and compare it with the approximate 5% null critical values at \(n=25\) and \(n=100\).
Method. The difference is \(|G(x)-F_0(x)|=|x^2-x|=x(1-x)\) on the interval. Its derivative is \(1-2x\), so the maximum occurs at \(x=1/2\); the endpoints have difference 0. Thus \(D=(1/2)(1-1/2)=1/4=0.25\).
At \(n=25\), the approximate critical value is \(1.36/\sqrt{25}=1.36/5=0.272\). The population discrepancy 0.25 is below this threshold. At \(n=100\), the threshold is \(1.36/10=0.136\), below the population discrepancy.
The threshold crosses the population discrepancy when \(1.36/\sqrt n<0.25\). Rearranging gives \(\sqrt n>1.36/0.25=5.44\), or \(n>29.5936\); thus the first integer sample size satisfying this comparison is 30. This is only a separation-scale comparison. It does not say that a sample of size 30 has any particular power: the empirical distribution fluctuates around \(G\), and power is a probability over those samples.
Check. The null threshold at \(n=100\) is exactly half the threshold at \(n=25\), because the sample size quadruples and the threshold scales as \(1/\sqrt n\). The common trap is to call the threshold crossing a guaranteed rejection or a specified power. Neither follows from the comparison.
Worked Example: AD Tail Weighting and Departure Scale
For a hypothetical model, let the null distribution be uniform on \([0,1]\), \(F_0(x)=x\), and let the alternative be \(G(x)=2x-x^2\). Calculate the population AD discrepancy \(A=\int_0^1 [G(x)-F_0(x)]^2/[F_0(x)(1-F_0(x))]\,dF_0(x)\), then compare its systematic scale at \(n=12\) and \(n=48\).
Method. Since \(dF_0(x)=dx\), \(G(x)-F_0(x)=2x-x^2-x=x(1-x)\). For \(0<x<1\), the integrand is \([x(1-x)]^2/[x(1-x)]=x(1-x)\). Therefore \(A=\int_0^1(x-x^2)\,dx=[x^2/2-x^3/3]_0^1=1/2-1/3=1/6\).
The population discrepancy multiplied by sample size is \(nA\). At \(n=12\), this scale is \(12/6=2\); at \(n=48\), it is \(48/6=8\). The latter is four times as large because the sample size is four times as large. This calculation illustrates the growth of the systematic discrepancy; it does not equal exact power or determine rejection without the statistic’s sampling variation and a calibrated critical value.
The denominator \(F_0(1-F_0)\) gives tail-region differences more weight than an unweighted squared-distance measure would. In this example, algebra cancels the endpoint factors, leaving the finite integral \(1/6\). In other alternatives, the weighted integral can be especially influenced by tail behavior, and it may fail to be finite; in that case this simple linear signal approximation requires additional care.
Check. The discrepancy is positive, and \(nA\) increases with \(n\), consistent with increasing evidence against the null for a fixed alternative. The common mistake is to interpret \(nA=8\) as an 8% or 80% rejection probability. It is a discrepancy scale, not a probability.
What Determines Power Besides Sample Size?
Increasing sample size does not make every departure equally visible. For the KS test, the population discrepancy is the single largest vertical gap; a broad departure and a narrow departure can have the same maximum gap even though their overall shapes differ. For AD, departures in the tails receive enhanced weight, but the result also depends on how much of the distribution is affected. For chi-square, the departure is reduced to changes in category probabilities; category boundaries can either reveal or conceal a difference.
The chi-square noncentrality calculation shows this dependence directly. Under the fixed-category alternative, its noncentrality parameter is \(\lambda=n\sum_i(p_{1i}-p_{0i})^2/p_{0i}\). The sum measures the departure relative to the null probabilities, while \(n\) scales its strength. If \(p_{1i}=p_{0i}\) for every category, then \(\lambda=0\): that chi-square grouping has no power to detect an alternative that changes the distribution within categories but leaves all category probabilities unchanged.
The familiar statement that “large samples detect small departures” should be interpreted carefully. For a fixed alternative with a nonzero discrepancy, these tests can become increasingly likely to reject as \(n\) grows. A departure that shrinks as the sample grows is a different problem: its detectability depends on the rate at which the discrepancy shrinks relative to sampling variability. Also, a statistically detectable departure need not be important for a practical actuarial decision.
State how the actual distribution differs from the null, including whether the difference is in the center, tails, or category probabilities.
Use a population CDF distance for KS, a tail-weighted integrated discrepancy for AD, or category-probability differences for Pearson chi-square.
A discrepancy or noncentrality calculation helps compare sample sizes, but actual power is a rejection probability and requires the alternative sampling distribution or a justified approximation.
A further condition matters in practice: the null critical value must be appropriate for the way the model was specified and tested. The simple KS threshold and the calculations here assume a fully specified continuous null; estimating distribution parameters from the same data can change calibration. Chi-square approximations also depend on adequate expected counts and appropriate degrees of freedom. If these conditions do not hold, a numerical “power” based on the simple formulas may be misleading.
Check Your Understanding
For each problem, distinguish an exact probability claim from a discrepancy or asymptotic approximation.
- A binary Pearson goodness-of-fit test has null probabilities \((0.50,0.50)\) and alternative probabilities \((0.55,0.45)\). Find the noncentrality parameter for \(n=200\). (Answer: \(\lambda=2\).)
- For a KS test on \([0,1]\), \(F_0(x)=x\) and \(G(x)=x^3\). Find \(D=\sup_{0\leq x\leq1}|G(x)-F_0(x)|\). (Answer: \(D=2/(3\sqrt3)\approx0.3849\), attained at \(x=1/\sqrt3\).)
- With uniform null \(F_0(x)=x\) and alternative \(G(x)=2x-x^2\), find the AD population discrepancy \(A\) and its scale \(nA\) at \(n=48\). (Answer: \(A=1/6\) and \(nA=8\); this is not exact power.)
- A three-category chi-square test has null probabilities \((0.20,0.30,0.50)\), alternative probabilities \((0.25,0.25,0.50)\), and \(n=200\). Calculate \(\lambda\). (Answer: \(\lambda=200[(0.05)^2/0.20+(0.05)^2/0.30]=25/6\approx4.1667\).)
- For a binary test with null probabilities \((0.50,0.50)\) and alternative probabilities \((0.55,0.45)\), what sample size gives noncentrality \(\lambda=9\)? State what this target does and does not tell you. (Answer: \(\lambda=0.01n\), so \(n=900\); this fixes a noncentrality target, not a power level unless the critical value and alternative distribution are also used.)