Degrees of Freedom From Parameter Dimensions

Why Parameter Dimensions Determine the Comparison

A likelihood-ratio comparison between nested generalized linear models asks whether the larger model’s additional directions in the parameter space improve the fit enough to matter. The degrees of freedom for that comparison count those independent directions, not simply the number of coefficient labels written in a formula. If two coefficients are aliased, changing them in certain combinations leaves every fitted linear predictor unchanged; those changes cannot contribute separate dimensions to the comparison.

For GLMs with a linear predictor, the design matrix makes the count explicit. The rank of the design matrix is the dimension of the set of distinct linear predictors the model can produce. For nested models, the difference between the larger and smaller model’s ranks is the number of additional estimable dimensions. This is the parameter-dimension count used for the likelihood-ratio comparison. The conditions under which the likelihood-ratio statistic has a particular asymptotic reference distribution are a separate question.

Definition: For a GLM with fixed design matrix $X$, the estimable parameter dimension of the linear predictor is $\operatorname{rank}(X)$. It is the dimension of the column space of $X$, or equivalently the number of linearly independent directions in the fitted linear predictors $X\beta$. A coefficient vector $\beta$ need not be uniquely determined for this dimension to be well defined.

Aliased Coefficients and Model Dimension

Write the linear predictor for observation $i$ as $\eta_i=x_i^\mathsf{T}\beta$, so the full vector of predictors is $\eta=X\beta$. If there is a nonzero vector $v$ such that $Xv=0$, then

X(\beta+v)=X\beta+Xv=X\beta.

Thus, $\beta$ and $\beta+v$ produce exactly the same linear predictors. In a GLM with the same link and fixed offsets, they also produce the same fitted means and likelihood. Such a coefficient direction is not separately identified by the design. This is called aliasing. Aliasing does not mean that the model has no identifiable information; it means some coefficient combinations cannot be distinguished, and the model dimension is the rank rather than the number of coefficient entries.

For example, a coefficient is not automatically estimable just because it appears in the model formula. An estimable linear combination $c^\mathsf{T}\beta$ is one whose value is the same for every coefficient vector that produces the same $X\beta$. Equivalently, $c$ must belong to the row space of $X$. Individual coefficients can fail this condition while certain combinations of them remain estimable.

Result: The dimension of the identifiable linear-predictor space is $\operatorname{rank}(X)$, even when the coefficient vector is not unique. To see this, the linear map $\beta\mapsto X\beta$ has image equal to the column space of $X$. By the rank-nullity theorem, its image has dimension equal to the rank of $X$; directions in its null space change coefficients without changing the predictors. $\square$

The Degrees of Freedom for Nested Models

Let $X_0$ and $X_1$ be the design matrices for a reduced and a full GLM, respectively, with the same observations, response distribution, link, and fixed offsets. Suppose the models are nested in their linear predictors: every predictor vector available under the reduced model is also available under the full model. In matrix terms, this means $\mathcal{C}(X_0)\subseteq\mathcal{C}(X_1)$, where $\mathcal{C}(X)$ denotes the column space of $X$.

Theorem: For nested GLMs of this form, the difference in estimable parameter dimensions is $$ \operatorname{df}_{\text{comparison}} =\dim\mathcal{C}(X_1)-\dim\mathcal{C}(X_0) =\operatorname{rank}(X_1)-\operatorname{rank}(X_0). $$ This is the number of new independent linear-predictor directions in the full model, and hence the parameter-dimension degrees of freedom for the likelihood-ratio comparison.

Proof. Because the models are nested, $\mathcal{C}(X_0)$ is a subspace of $\mathcal{C}(X_1)$. Extend a basis of $\mathcal{C}(X_0)$ to a basis of $\mathcal{C}(X_1)$. The number of added basis vectors is the dimension of the larger space minus the dimension of the smaller one. The numbers of basis vectors are, respectively, $\operatorname{rank}(X_1)$ and $\operatorname{rank}(X_0)$. Therefore the increase in model dimension is their difference. Since each added basis vector is an independent predictor direction unavailable in the reduced model, this difference is the comparison’s parameter-dimension count. $\square$

The same count can be seen from restrictions. If the full model has $r_1$ estimable dimensions and the reduced model has $r_0$, expressing the reduced model inside the full one imposes $r_1-r_0$ independent restrictions on the full model’s identifiable predictor space. Counting written restrictions without checking their independence can give the wrong answer.

For the likelihood-ratio statistic, the models are fitted by maximizing the same likelihood, and the difference is $2(\hat\ell_1-\hat\ell_0)$. The rank difference tells you the parameter-dimension count associated with this comparison. It does not, by itself, establish that the statistic follows an exact or approximate chi-square distribution. That distributional calibration requires additional conditions, which are considered separately.

Worked Example: A Three-Level Factor Adds Two Dimensions

Suppose a logistic regression has an intercept and a factor with three observed levels, A, B, and C. The reduced model has an intercept only. The full model includes the factor. Use A as the reference level, with two indicator columns $I_B$ and $I_C$. Assume all three levels are represented in the data.

The reduced design matrix has one column, the intercept, so $\operatorname{rank}(X_0)=1$. The full design matrix has columns $1,I_B,I_C$. These columns are linearly independent: in a row from level A the two indicator entries are both zero; in a row from B only $I_B$ is one; in a row from C only $I_C$ is one. A linear combination of the three columns that is zero on all three types of rows must therefore have all three coefficients zero. Hence $\operatorname{rank}(X_1)=3$.

The comparison has

\operatorname{df}_{\text{comparison}}=3-1=2.

The factor has three levels but contributes two dimensions beyond the intercept. Counting the three factor levels as three added parameters would be incorrect: the intercept and the factor contrasts together span three dimensions in the full model, of which one was already present in the reduced model.

Worked Rank Calculations

Worked Example: An Added Coefficient Is Aliased

Consider four observations with a logistic regression design whose columns are an intercept, a treatment indicator $T$, and a site indicator $S$. Suppose treatment and site are perfectly confounded in the observed data: $T=S$ for every observation. The reduced model has an intercept and $T$; the full model adds $S$.

For example, the design rows could be $(1,0,0),(1,0,0),(1,1,1),(1,1,1)$. In the full model, the $S$ column equals the $T$ column. The full predictor is

\eta=\beta_0\mathbf{1}+\beta_T T+\beta_S S =\beta_0\mathbf{1}+(\beta_T+\beta_S)T.

Only the sum $\beta_T+\beta_S$ is identified; the separate coefficients cannot be distinguished from these observations. The reduced design has two independent columns, the intercept and $T$, so its rank is 2. The full design still has rank 2 because its added column duplicates an existing one. Therefore,

\operatorname{df}_{\text{comparison}}=2-2=0.

The full model produces no new predictor direction and no likelihood improvement unavailable to the reduced model. There is no separate site effect to test in this design. Additional observations in which treatment and site vary independently would be needed to distinguish those effects.

Worked Example: Three Added Columns Contribute Two Dimensions

Consider four observations. The reduced design has an intercept and a binary predictor $x$. The full design adds three columns $z_1,z_2,z_3$, with $z_3=z_1+z_2$. Let the rows of the full design matrix, in column order $1,x,z_1,z_2,z_3$, be

X_1= \begin{pmatrix} 1&0&0&0&0\\ 1&0&1&0&1\\ 1&1&0&0&0\\ 1&1&1&1&2 \end{pmatrix}.

The reduced design consists of the first two columns. They are independent: the intercept is one in every row, while $x$ changes between the first two and last two rows. Thus $\operatorname{rank}(X_0)=2$.

In the full design, the last column is the sum of the third and fourth columns, so it cannot increase the rank. The first four columns are independent. To check this, suppose $a\mathbf{1}+bx+cz_1+dz_2=0$. The first row gives $a=0$. The second then gives $c=0$, the third gives $b=0$, and the fourth gives $d=0$. Therefore $\operatorname{rank}(X_1)=4$, and

\operatorname{df}_{\text{comparison}}=4-2=2.

There are three newly written coefficient columns, but only two independent new directions. The rank calculation, rather than a count of coefficient names, gives the correct parameter-dimension difference.

Practical Interpretation and Common Pitfalls

The rank calculation is also useful for checking model summaries. If $n$ observations are used and the design has rank $r$, the mean-model dimension is $r$, not necessarily the number of displayed coefficients. Residual degrees of freedom in a model summary and degrees of freedom for comparing two nested models answer different questions: the former compares the fitted model dimension with the number of observations, while the latter compares the dimensions of two model spaces.

A fixed offset contributes no estimated parameter dimension. Likewise, a nuisance parameter estimated under both models does not change the difference in dimensions; a nuisance parameter whose structure differs between the models must be included in the parameter-dimension comparison if the likelihood comparison is otherwise valid.

Mistake: count coefficient labels. A formula may display several coefficients that are linearly dependent in the observed design. Compute the design ranks instead.
Mistake: count the factor’s levels as added parameters. The relevant count is the number of new independent contrasts relative to the reduced model, not the number of level labels.
Mistake: treat an aliased effect as estimable because it appears in the formula. If its column is a combination of existing columns, it adds no independent predictor direction.
Mistake: assume a positive rank difference guarantees a valid chi-square calibration. The rank difference gives the dimension count; distributional validity also depends on regularity conditions and the comparison setup.

Key takeaway: For nested GLMs with the same observations and likelihood structure, calculate comparison degrees of freedom as $\operatorname{rank}(X_1)-\operatorname{rank}(X_0)$. This automatically accounts for aliased coefficients and counts only independent estimable directions.

Check Your Understanding

Use model-space dimensions, not just the number of coefficient names, in your answers.

A reduced design has rank 3 and a nested full design has rank 6. What is the parameter-dimension degrees of freedom for their comparison?
In a full model, two added columns are identical to each other but both independent of the reduced design. How many dimensions do they add?
A factor has four observed levels and is compared with an intercept-only model. Under ordinary indicator coding with all levels represented, how many dimensions does the factor add?
Why does an added coefficient whose design column equals an existing column contribute zero comparison degrees of freedom?
Does a positive rank difference alone prove that the likelihood-ratio statistic has a chi-square distribution? Explain.