Chi-Square Calibration and Large-Sample P-Values

From a Test Statistic to a P-Value

A Wald, score, or likelihood-ratio test may produce a statistic whose null distribution is not known exactly in a finite sample. Under suitable regularity conditions, however, its null distribution approaches a chi-square distribution as the sample size grows. Chi-square calibration uses that limiting distribution to turn an observed statistic into an approximate p-value.

The key is the upper tail: large values of these test statistics indicate disagreement with the null hypothesis. The p-value is therefore the reference distribution's probability of producing a value at least as large as the observed one. It is not the probability that the null hypothesis is true, nor is it generally an exact finite-sample probability.

Definition: Suppose a test statistic $T_n$ has, under the null hypothesis and stated regularity conditions, the limiting distribution $T_n \xrightarrow{d} \chi^2_q$. If the observed statistic is $t_{\mathrm{obs}}$, its chi-square-calibrated p-value is $$ p_{\mathrm{asy}} = P(\chi^2_q \geq t_{\mathrm{obs}}) = 1-F_q(t_{\mathrm{obs}}), $$ where $F_q$ is the cumulative distribution function of a chi-square random variable with $q$ degrees of freedom. The subscript “asy” emphasizes that this is an asymptotic calibration.

A chi-square random variable with $q$ degrees of freedom, for positive integer $q$, has density $$ f_q(x)=\frac{x^{q/2-1}e^{-x/2}}{2^{q/2}\Gamma(q/2)}, \qquad x>0. $$ Thus, the calibrated p-value can always be expressed as the upper-tail integral of this density. In practice, a statistical table or software evaluates the integral. The degrees of freedom must be chosen before computing the tail: using the wrong $q$ can substantially change the reported p-value.

Choosing Degrees of Freedom

For a regular test of $q$ independent restrictions, the standard large-sample reference law is $\chi^2_q$. In a nested-model likelihood-ratio test, $q$ is the dimension of the unrestricted model minus the dimension of the restricted model. For smooth restrictions written as $r(\theta)=0$, the relevant count is the local codimension of the null constraint set. Under a regularity condition such as a constant-rank condition near the null parameter, this codimension is the rank of the Jacobian of $r$ at that parameter.

The rank qualification matters. A list can contain repeated or locally dependent equations, so counting written equations can overstate the number of independent restrictions. Conversely, the rank rule is not a universal rule for arbitrary equations. It applies when the restrictions define a regular constraint set locally. If they are singular, one must examine the null set and establish the limiting distribution separately; the Jacobian rank alone does not determine the degrees of freedom or reference law.

Regular degrees-of-freedom rule: If the null restriction set is a smooth, locally regular constraint set of codimension $q$, and the test's standard chi-square limit theorem applies, use $q$ degrees of freedom. For a restriction map $r:\mathbb{R}^k\to\mathbb{R}^m$, a sufficient local condition is that its Jacobian has constant rank $q$ near the null parameter; the local codimension is then $q$. If regularity fails, do not infer the reference law from the Jacobian rank alone.

For example, if $\theta=(\theta_1,\theta_2,\theta_3,\theta_4)$ and the null hypothesis sets $\theta_2,\theta_3,\theta_4$ to specified values, the three coordinate restrictions are independent and have rank three. Under the standard regularity conditions, a Wald, score, or likelihood-ratio statistic for this null is calibrated against $\chi^2_3$, not against a chi-square law whose degrees of freedom equal the total number of parameters.

Useful Tail Calculations

Two special cases make the upper-tail calculation especially transparent. For one degree of freedom, if $Z\sim N(0,1)$, then $Z^2\sim\chi^2_1$. Consequently, a chi-square p-value can be written as a two-sided standard normal tail. For two degrees of freedom, the density simplifies to an exponential density, giving a particularly direct formula.

Result: If $T\sim\chi^2_1$ and $t\geq 0$, then $$ P(T\geq t)=2\{1-\Phi(\sqrt{t})\}, $$ where $\Phi$ is the standard normal cumulative distribution function. If $T\sim\chi^2_2$, then $$ P(T\geq t)=e^{-t/2}, \qquad t\geq 0. $$

For the first identity, write $T=Z^2$. Then $Z^2\geq t$ exactly when $Z\leq-\sqrt{t}$ or $Z\geq\sqrt{t}$. Symmetry of the standard normal distribution gives the stated two-tail probability. For the second identity, substitute $q=2$ in the chi-square density: $f_2(x)=\tfrac12 e^{-x/2}$ for $x>0$. Integrating its density from $t$ to infinity gives $$ P(T\geq t)=\int_t^\infty \frac12 e^{-x/2}\,dx=e^{-t/2}. $$

Worked Example: A One-Restriction Wald Test

Suppose $X_1,\ldots,X_{100}$ are independent Bernoulli observations with success probability $p$. Test $H_0:p=0.5$ using the Wald statistic, and suppose 60 successes are observed. Compute the asymptotic p-value.

The maximum-likelihood estimate is $\hat p=60/100=0.6$. Its estimated variance is $\hat p(1-\hat p)/n$, so $$ \widehat{\operatorname{Var}}(\hat p)=\frac{0.6(0.4)}{100}=0.0024. $$ The Wald statistic is $$ T=\frac{(\hat p-0.5)^2}{\widehat{\operatorname{Var}}(\hat p)} =\frac{(0.1)^2}{0.0024} =\frac{0.01}{0.0024} =4.1667. $$

There is one scalar restriction, $p=0.5$, and its derivative with respect to $p$ is 1, so it has rank one. Under the Wald test's large-sample conditions, the reference law is $\chi^2_1$. Therefore $$ p_{\mathrm{asy}}=P(\chi^2_1\geq 4.1667) =2\{1-\Phi(\sqrt{4.1667})\} \approx 2\{1-\Phi(2.0412)\} \approx 0.0412. $$

This is the asymptotic Wald p-value, not an exact binomial-test p-value. In particular, the calculation does not assert that the finite-sample distribution of this Wald statistic is exactly $\chi^2_1$.

Worked Example: Two Restrictions and the Exponential Tail

A regular likelihood-ratio test compares a model with two unrestricted parameters to a nested model that imposes two independent restrictions. The observed likelihood-ratio statistic is 6. Find the asymptotic p-value.

The dimension difference is two, so the standard reference law is $\chi^2_2$. Using the two-degree-of-freedom tail formula, $$ p_{\mathrm{asy}}=P(\chi^2_2\geq 6)=e^{-6/2}=e^{-3}\approx 0.0497871. $$

The result is just below 0.05, so a test using the asymptotic 5% cutoff would reject. That conclusion is based on the approximate reference law. It does not guarantee that the test's actual finite-sample rejection probability under the null is exactly 0.05.

Why the Calibration Is Approximate

The reference distribution describes a limit as $n$ grows, not necessarily the exact distribution at the sample size in hand. If $T_n\xrightarrow{d}\chi^2_q$ under the null, the distribution of $T_n$ is close to that of $\chi^2_q$ in the large-sample sense. The chi-square tail at the observed statistic is then a useful approximation. How accurate it is at a particular sample size depends on the model, the statistic, and how well the assumptions supporting the limit apply.

The limiting result also gives a precise reason that the p-value scale is appropriate. Let $F_q$ be the continuous, strictly increasing chi-square cumulative distribution function on its support. Define $p_n=1-F_q(T_n)$. Under the null, convergence $T_n\xrightarrow{d}\chi^2_q$ implies, for $0<u<1$, $$ P(p_n\leq u) =P\bigl(T_n\geq F_q^{-1}(1-u)\bigr) \longrightarrow P\bigl(\chi^2_q\geq F_q^{-1}(1-u)\bigr) =u. $$ The convergence follows from convergence in distribution at the threshold $F_q^{-1}(1-u)$, which is a continuity point of the limiting distribution. The last equality follows from the definition of the quantile and continuity of $F_q$. Thus $p_n$ converges in distribution to a Uniform$(0,1)$ random variable under the null. This is asymptotic validity, not exact uniformity at each finite $n$.

In particular, for a fixed significance level $\alpha$ with $0<\alpha<1$, the asymptotic test rejects when $p_n\leq\alpha$, equivalently when $T_n\geq F_q^{-1}(1-\alpha)$. The same convergence argument gives $$ P_{H_0}\bigl(T_n\geq F_q^{-1}(1-\alpha)\bigr)\longrightarrow \alpha. $$ So the limiting null rejection probability is $\alpha$. A finite-sample rejection probability can be either above or below that target.

Worked Example: Three Independent Restrictions

Suppose a regular test concerns a four-parameter model with null hypothesis $H_0:\theta_2=\theta_3=\theta_4=0$. The restriction map is $r(\theta)=(\theta_2,\theta_3,\theta_4)$. Its Jacobian has three linearly independent rows, so the null set has codimension three. If the observed likelihood-ratio statistic is 9, calculate its asymptotic p-value.

The reference distribution is $\chi^2_3$. Its upper-tail probability has the expression $$ P(\chi^2_3\geq x) =\operatorname{erfc}\!\left(\sqrt{\frac{x}{2}}\right) +\sqrt{\frac{2x}{\pi}}e^{-x/2}. $$ At $x=9$, this gives $$ p_{\mathrm{asy}} =\operatorname{erfc}(\sqrt{4.5}) +\sqrt{\frac{18}{\pi}}e^{-4.5} \approx 0.00270+2.3937(0.011109) \approx 0.0293. $$

The reported value is an upper-tail probability under the $\chi^2_3$ reference law. It says that a value at least as large as 9 is relatively unusual under that law; it does not give the probability that the three null restrictions are true.

Common Mistakes and Practical Checks

Using the wrong tail. These statistics are calibrated using large values as evidence against the null, so use $P(\chi^2_q\geq t_{\mathrm{obs}})$, not the lower-tail probability.
Counting parameters instead of restrictions. The degrees of freedom are tied to the number of independent regular restrictions (or, for nested models, the dimension difference), not automatically to the unrestricted parameter count.
Counting equations without checking rank. Dependent restrictions do not contribute separate degrees of freedom. The rank rule requires a regular constraint set; a singular restriction cannot be assigned degrees of freedom by rank alone.
Calling the approximation exact. A limiting chi-square law supports a large-sample p-value. It does not make a finite-sample statistic exactly chi-square distributed.
Interpreting a p-value as a null probability. The p-value is calculated under the null reference law. It is not $P(H_0\mid\text{data})$.

As a warning about the rank condition, the single equation $r(\theta)=\theta^2=0$ has null set $\{0\}$, but its derivative at zero is 0. The Jacobian rank therefore cannot be used as a general-purpose degrees-of-freedom count for such a singular restriction. The appropriate null distribution must be established from the specific problem rather than assumed to be the usual rank-based chi-square law.

Key takeaway: A large-sample chi-square p-value is the upper-tail probability under the test statistic's limiting null law. Choose its degrees of freedom from the number of independent regular restrictions, and interpret the resulting probability as an approximation whose accuracy depends on the asymptotic conditions and sample size.

Check Your Understanding

For each question, identify the reference law or calculation needed and explain the reasoning.

A regular test imposes four independent restrictions and gives an observed statistic of 10. Which chi-square distribution defines its asymptotic p-value, and which tail is used?
For an observed statistic of 4 under a $\chi^2_1$ reference law, express the p-value using the standard normal cumulative distribution function.
A likelihood-ratio comparison has unrestricted and restricted model dimensions of 8 and 5. Under regular nested-model conditions, what are the degrees of freedom?
Why does convergence of a test statistic to a chi-square distribution justify approximate, rather than necessarily exact, finite-sample p-values?
For the restriction $r(\theta)=\theta^2=0$ at $\theta=0$, why does the derivative rank not by itself determine the standard chi-square degrees of freedom?