Why Degrees of Freedom Come From Restrictions
A likelihood ratio test compares a restricted model with a more flexible model that contains it. The chi-square degrees of freedom are not determined by the sample size, the number of observations, or simply the number of equations written in the null hypothesis. They are determined by how many independent parameter restrictions the null imposes on the alternative model.
This distinction matters in applications such as testing whether several normal populations share a mean or whether selected categories in a multinomial model have equal probabilities. A parameter that is estimated under both hypotheses is a nuisance parameter, but it does not add a restriction merely because it appears in both models. This tutorial is aligned with the SOA VEE Mathematical Statistics syllabus.
For a likelihood ratio test, the statistic is commonly written
where \(\ell\) is the log-likelihood, \(\widehat{\theta}\) maximizes it over the larger model, and \(\widehat{\theta}_0\) maximizes it subject to the null restrictions. The degrees-of-freedom count concerns the difference in the models' dimensions; it does not require computing either maximum likelihood estimate.
Counting Independent Restrictions
Suppose the larger model has parameter vector \(\theta\) with \(p\) free components. Write the null hypothesis as a set of equations \(g(\theta)=0\). If these restrictions are independent in a neighborhood of the true null parameter, the null model has dimension \(p-r\), where \(r\) is the rank of the restriction equations. Thus the degrees of freedom are \(r\).
A useful way to check the count is to parameterize the null model directly. Count how many free quantities remain after applying the null and all model constraints, such as probabilities summing to one. Subtract that count from the number of free quantities in the larger model. This method avoids treating a redundant equation as a new restriction.
Use only free parameters. For example, \(k\) multinomial probabilities have \(k-1\) free parameters because they sum to one.
Parameterize the distributions allowed by the null and count the free parameters that remain.
The difference is the number of independent restrictions, and hence the usual chi-square degrees of freedom when the regularity conditions hold.
Why the Rank Gives the Count
The dimension rule follows from how independent equations reduce the set of possible parameter values. Near an interior parameter value, a smooth restriction \(g(\theta)=0\) has a first-order change described by its derivative matrix, the Jacobian. If that matrix has rank \(r\), then \(r\) locally independent directions are removed from the \(p\)-dimensional parameter space. The null set therefore has local dimension \(p-r\).
For example, if three written equations have gradients that are linearly dependent, they do not remove three independent directions. The rank, not the number of displayed equations, is the relevant count. Equivalently, a direct parameterization of the null set reveals how many free coordinates remain. These are two ways to establish the same dimension difference.
The chi-square calibration is an asymptotic result for regular nested models; it is not a claim that the statistic has an exact chi-square distribution for every finite sample. The usual conditions include an identifiable model, an interior null parameter, and sufficient smoothness. Boundary constraints or other failures of regularity can lead to a different limiting distribution. Here the focus is how to count the degrees of freedom when the regular nested-model setting applies.
Worked Normal-Model Comparisons
Worked Example: Testing Whether Three Normal Means Are Equal
Three independent groups each contribute two measurements. Assume observations in group \(j\) are independent and normally distributed with mean \(\mu_j\) and a common, unknown variance \(\sigma^2>0\). The observed pairs are \((1,3)\), \((4,6)\), and \((7,9)\). Test the nested-model dimension for \(H_0:\mu_1=\mu_2=\mu_3\), and calculate the likelihood ratio statistic.
Method: compare the free parameters and, as a check, compute the likelihood ratio from the residual sums of squares. In the larger model, the three means and the shared variance are free, so its dimension is \(3+1=4\). Under the null, there is one common mean and the same unknown variance, giving dimension \(1+1=2\). The variance is a nuisance parameter in both models; it is counted in both dimensions and cancels in their difference. Thus the null imposes \(4-2=2\) independent restrictions.
The group means are \((1+3)/2=2\), \((4+6)/2=5\), and \((7+9)/2=8\). The unrestricted residual sum of squares is
Under the null, the common maximum likelihood estimate of the mean is the overall mean, \((1+3+4+6+7+9)/6=30/6=5\). Therefore,
For independent normal observations with a common unknown variance, maximizing over the variance gives \(D=N\log(\operatorname{RSS}_0/\operatorname{RSS}_1)\). Here \(N=6\), so
The statistic would be compared with a chi-square distribution with \(2\) degrees of freedom under the regular large-sample approximation; the dimension count is two whether or not this small sample makes that approximation accurate. The check is consistent: the null leaves one mean and one variance, rather than three means and one variance. Common mistake avoided: counting only the two mean restrictions in the alternative and forgetting that the shared unknown variance is present in both models. Adding it to both dimensions leaves the difference unchanged.
Worked Example: A Multinomial Test of Two Equal Cell Probabilities
A hypothetical audit classifies \(60\) independent records into four mutually exclusive categories. Assume a multinomial model with probabilities \(p_1,p_2,p_3,p_4>0\), summing to one. The counts are \((24,12,12,12)\). Under \(H_0:p_1=p_2\), find the degrees of freedom and the likelihood ratio statistic.
Method: parameterize the null and subtract dimensions. The unrestricted four-category multinomial has \(4-1=3\) free probabilities. Under the null, write \(p_1=p_2=q\). The sum constraint gives \(2q+p_3+p_4=1\), so two free quantities remain; for instance, \(q\) and \(p_3\) determine \(p_4\). The dimension difference is \(3-2=1\), so there is one independent restriction.
In the unrestricted model, the maximum likelihood estimates are the observed proportions: \((24/60,12/60,12/60,12/60)=(0.4,0.2,0.2,0.2)\). Under the null, the pooled count for categories 1 and 2 is \(24+12=36\), so their fitted probabilities are \(36/(2\cdot60)=0.3\) each. The other fitted probabilities remain \(12/60=0.2\) each. The null fitted expected counts are therefore \((18,18,12,12)\).
The multinomial likelihood ratio statistic is twice the sum of each observed count times the logarithm of its unrestricted fitted probability divided by its null fitted probability. The equal multinomial coefficient cancels:
The reference degrees of freedom are \(1\), the dimension difference, not four categories and not the number of observed counts. The calculation uses positive fitted probabilities, as assumed. Common mistake avoided: treating the four probabilities as four free parameters, despite their sum-to-one constraint.
Worked Example: Three Written Equations but Only Two Restrictions
A five-category multinomial model has positive probabilities \(p_1,\ldots,p_5\) summing to one. The null hypothesis is written as \(p_1=p_2\), \(p_2=p_3\), and \(p_1=p_3\). Determine the degrees of freedom for a regular likelihood ratio comparison with the unrestricted model.
Method: use both a direct parameterization and a rank check. The unrestricted dimension is \(5-1=4\). Under the null, set \(p_1=p_2=p_3=q\). The remaining probabilities satisfy \(3q+p_4+p_5=1\). There are three probabilities \(q,p_4,p_5\) subject to one sum constraint, so the null dimension is \(2\). The difference is \(4-2=2\).
The rank check gives the same result. The three equations can be represented by \(g_1=p_1-p_2\), \(g_2=p_2-p_3\), and \(g_3=p_1-p_3\). Since \(g_3=g_1+g_2\), the third equation adds no new restriction. Thus the rank is \(2\), not \(3\), and the usual reference degrees of freedom are \(2\). Common mistake avoided: counting the number of equations as though every equation were independent.
Common Counting Traps
- Counting equations instead of independent restrictions. Check for equations implied by the others, or use the rank of the constraint system.
- Forgetting built-in constraints. A \(k\)-category probability vector has \(k-1\), not \(k\), free parameters because its components sum to one.
- Counting a shared nuisance parameter as a restriction. A variance estimated in both normal models contributes to both dimensions; it does not change their difference.
- Using the number of observations. Sample size affects the data and the approximation, but the dimension difference determines the degrees-of-freedom count.
- Assuming the reference distribution is always chi-square. The usual result requires regularity; boundary parameters and other nonregular cases need separate treatment.
Check Your Understanding
For each regular nested-model comparison, determine the degrees of freedom from the model dimensions. Answers are provided for checking.
- Four independent normal groups have separate means and one common unknown variance in the larger model. The null says all four means are equal, with the same variance structure. What are the two model dimensions and the degrees of freedom? (Answer: larger dimension \(5\), null dimension \(2\), degrees of freedom \(3\).)
- A six-category multinomial model is compared with a null model that fixes all six probabilities at specified positive values summing to one. How many free parameters are in each model, and what are the degrees of freedom? (Answer: dimensions \(5\) and \(0\), degrees of freedom \(5\).)
- A five-category multinomial model has null restrictions \(p_1=p_2\) and \(p_4=p_5\). All probabilities are positive and sum to one. What is the null dimension and the degrees of freedom? (Answer: null dimension \(2\), degrees of freedom \(2\).)
- A four-category multinomial model has null restrictions \(p_1=p_2\), \(p_2=p_3\), and \(p_1=p_3\). What is the degrees-of-freedom count, and why is it not three? (Answer: \(2\); the third equality follows from the first two.)
- In a normal model, the larger model has three separate means and a common unknown variance. The null equates the first two means but leaves the third unrestricted, with the same variance. Give the model dimensions and degrees of freedom. (Answer: larger dimension \(4\), null dimension \(3\), degrees of freedom \(1\).)