Comparing Nested Loglinear Models

Why Compare Nested Loglinear Models?

A loglinear model summarizes expected cell counts through selected main effects and interactions. Comparing a simpler model with a more complex one asks whether the added terms improve the fit enough to justify their additional parameters. In a contingency table, that is often a substantive question: does the table show association, or does it satisfy a particular form of independence?

The comparison is based on the models’ maximized likelihoods. The larger model can fit at least as well as the smaller one, because it allows every fit available under the smaller model as a special case. The likelihood-ratio statistic measures the improvement. Its degrees of freedom count the additional independent parameters, not the number of cells by which the models differ.

Definition: A model \(M_0\) is nested in a model \(M_1\) if every distribution allowed by \(M_0\) is also allowed by \(M_1\). Write \(d_0\) and \(d_1\) for their parameter dimensions. For regular nested models with \(d_1>d_0\), the likelihood-ratio statistic is \(2\{\ell(\hat\theta_1)-\ell(\hat\theta_0)\}\), where each log-likelihood is maximized within its model.

The Difference in Deviance Is the Likelihood-Ratio Statistic

For independent Poisson cell counts \(Y_r\), let \(y_r\) be the observed count and let \(\hat\mu_{r,M}\) be the fitted mean under model \(M\). The deviance of \(M\), measured against the saturated model, is

D(M)=2\sum_r\left\{y_r\log\left(\frac{y_r}{\hat\mu_{r,M}}\right)-(y_r-\hat\mu_{r,M})\right\}.

A term with \(y_r=0\) is interpreted using \(0\log(0/\hat\mu)=0\). The saturated model fits each cell mean separately, so its fitted mean is \(y_r\) when \(y_r>0\). The deviance records the loss in maximized log-likelihood relative to that saturated fit.

Result: For nested models \(M_0\subset M_1\), fitted by maximum likelihood to the same data, \(D(M_0)-D(M_1)=2\{\ell(\hat\theta_1)-\ell(\hat\theta_0)\}\). Thus the difference in deviances is exactly the likelihood-ratio statistic.

To verify this identity, write the Poisson log-likelihood, apart from no omitted terms, as

\ell_M=\sum_r\left\{y_r\log\hat\mu_{r,M}-\hat\mu_{r,M}-\log(y_r!)\right\}.

The saturated log-likelihood \(\ell_{\mathrm{sat}}\) is the same reference for both models. From the deviance definition, \(D(M)=2(\ell_{\mathrm{sat}}-\ell_M)\). Subtracting the two deviances therefore gives

D(M_0)-D(M_1) =2(\ell_{\mathrm{sat}}-\ell_{M_0})-2(\ell_{\mathrm{sat}}-\ell_{M_1}) =2(\ell_{M_1}-\ell_{M_0}).

This argument also explains why the statistic is nonnegative: maximizing over the larger model cannot produce a lower maximized likelihood. A negative difference beyond numerical rounding signals a fitting or model-specification problem. \(\square\)

Degrees of Freedom and the Reference Distribution

The degrees of freedom for a regular nested comparison are \(q=d_1-d_0\), the number of additional independent parameters. For standard identifiable hierarchical loglinear models, parameter dimensions can be counted by the independent main-effect and interaction contrasts. Constraints used to identify the parameters affect their labels, but not the dimension.

For a two-way table with \(I\) levels of \(A\) and \(J\) levels of \(B\), the independence model has an intercept and main effects, with dimension \(1+(I-1)+(J-1)\). The saturated model has one free mean per cell and dimension \(IJ\). Their dimension difference is

IJ-\{1+(I-1)+(J-1)\}=(I-1)(J-1).

This is also the number of independent \(A\)-by-\(B\) interaction contrasts added when moving from independence to the saturated model.

For a three-way table, the model \([AB][AC][BC]\) includes all main effects and all two-factor interactions, but no three-factor interaction. Adding the three-factor interaction gives the saturated model. The number of added contrasts is \((I-1)(J-1)(K-1)\). This follows by counting the full model’s interaction contrasts and subtracting the lower-order ones: the saturated model has \(IJK-1\) dimensions, while \([AB][AC][BC]\) has \(1+(I-1)+(J-1)+(K-1)+(I-1)(J-1)+(I-1)(K-1)+(J-1)(K-1)\). Their difference is \((I-1)(J-1)(K-1)\).

Wilks’ theorem for nested models: Suppose the models are identifiable, have fixed finite dimensions, and are regular, with the null parameter in the interior of the larger model’s parameter space. If the smaller model is true and the codimension of that model in the larger one is \(q\), then, under the usual increasing-information asymptotics, the likelihood-ratio statistic converges in distribution to \(\chi^2_q\). For loglinear tables, these conditions are appropriate when the cell probabilities or mean proportions are positive and the table dimensions remain fixed as information increases.

The resulting \(p\)-value is the upper-tail probability \(P(\chi^2_q\geq D(M_0)-D(M_1))\). The chi-square reference is an asymptotic result, not an exact finite-sample distribution. The model comparison and its degrees of freedom can still be defined when the approximation needs caution.

Worked Comparisons

Worked Example: Testing Independence in a Two-Way Table

Suppose counts for two binary variables \(A\) and \(B\) are

\begin{array}{c|rr|r} & B_1 & B_2 & \text{Total}\\ \hline A_1&30&10&40\\ A_2&10&30&40\\ \hline \text{Total}&40&40&80 \end{array}

Compare the independence model \([A][B]\) with the saturated model \([AB]\). Under independence, the fitted count in cell \((i,j)\) is the row total times the column total divided by the grand total. Each fitted count is therefore \(40(40)/80=20\). The saturated fit reproduces the four observed counts.

The deviance of the independence fit is

D([A][B])=2\left\{2(30\log(30/20))+2(10\log(10/20))\right\} =4\{30\log(1.5)+10\log(0.5)\}\approx20.93.

The saturated deviance is zero, so the difference is \(20.93\). The independence model has dimension \(1+1+1=3\), while the saturated model has dimension \(4\); hence \(q=1\), also given by \((2-1)(2-1)=1\). The comparison is a one-degree-of-freedom test of the \(A\)-by-\(B\) interaction, equivalently a test of independence. The statistic is large relative to \(\chi^2_1\) (its upper-tail probability is less than \(0.001\)), providing strong evidence against independence.

Worked Example: A Two-by-Three Independence Test

Consider this table of counts for binary \(A\) and three-level \(B\):

\begin{array}{c|rrr|r} & B_1&B_2&B_3&\text{Total}\\ \hline A_1&10&20&30&60\\ A_2&20&20&20&60\\ \hline \text{Total}&30&40&50&120 \end{array}

Under independence, the fitted counts in each row are half the column totals: \((15,20,25)\). For the first row, the deviance contributions before multiplying by \(2\) are \(10\log(10/15)\), \(20\log(20/20)\), and \(30\log(30/25)\). For the second row they are \(20\log(20/15)\), \(20\log(20/20)\), and \(20\log(20/25)\). Thus

D([A][B])=2\{10\log(2/3)+30\log(1.2)+20\log(4/3)+20\log(0.8)\}\approx5.412.

The saturated model has six cell parameters; the independence model has \(1+(2-1)+(3-1)=4\). The test therefore has \(q=2\) degrees of freedom, matching \((2-1)(3-1)=2\). Its approximate chi-square upper-tail probability is \(e^{-5.412/2}\approx0.067\). This comparison tests whether the two variables are independent; it does not test a single one-degree-of-freedom contrast.

Worked Example: Testing for a Three-Factor Interaction

Let \(A,B,C\) each have two levels, coded \(0\) and \(1\). Suppose the observed count is \(30\) in the four cells where \(a+b+c\) is even and \(10\) in the other four cells. Every two-way marginal table has four counts of \(40\): for any fixed pair of levels, summing over the remaining factor gives \(30+10=40\).

Consequently, the fitted counts under the model \([AB][AC][BC]\) are \(20\) in all eight cells. These fitted counts reproduce every observed two-way marginal, which is sufficient for this model’s maximum-likelihood fit. Compare it with the saturated model. The deviance difference is

2\{4(30\log(30/20))+4(10\log(10/20))\} =8\{30\log(1.5)+10\log(0.5)\}\approx41.86.

The no-three-factor-interaction model has seven parameters and the saturated model has eight, so \(q=1\). The resulting statistic is compared with \(\chi^2_1\). This test asks whether the three-way interaction is needed after allowing all two-way interactions; it is not a test that \(A,B,C\) are mutually independent. The strong discrepancy here is evidence against the no-three-factor-interaction model.

Interpreting Added Interactions

An interaction test gets its meaning from the particular terms already present in the smaller model. Adding an \(AB\) interaction to \([A][B]\) tests independence of \(A\) and \(B\). By contrast, adding a three-factor interaction to \([AB][AC][BC]\) tests whether the pairwise associations can be represented without a three-way term. The latter model already allows pairwise association, so its comparison with the saturated model cannot be read as a test of overall independence.

More generally, name both models before interpreting a deviance difference. State which terms are added, how many independent parameters they contribute, and what null hypothesis the smaller model imposes. This prevents a numerical test result from being detached from the scientific question.

Common Mistakes

Using the larger model’s deviance alone. A deviance against saturation is not the nested-model statistic unless the larger model is saturated. Subtract the larger model’s deviance from the smaller model’s deviance.
Counting added terms instead of added parameters. One named interaction may represent several independent contrasts. Use the difference in model dimensions.
Calling every interaction test a test of independence. Independence is the null only when the smaller model excludes the relevant association terms in the appropriate way.
Comparing nonnested models with a deviance difference test. The likelihood-ratio chi-square result described here requires nesting; for nonnested models, this difference does not have the stated interpretation.

The comparison is useful because it turns a model-building question into a targeted inferential one: does the additional structure improve fit beyond what would be expected from the extra parameters? Its answer is meaningful only when the models are genuinely nested, the degrees of freedom are counted correctly, and the tested interaction is interpreted in the context of the smaller model.

Check Your Understanding

For each question, identify the relevant model comparison or calculation.

For an \(I\)-by-\(J\) table, what is the degrees-of-freedom difference between independence and saturation, and why?
Why is the deviance difference \(D(M_0)-D(M_1)\), rather than the larger model’s deviance alone, the likelihood-ratio statistic?
In a two-way table, what hypothesis is tested by comparing \([A][B]\) with \([AB]\)?
What does comparing \([AB][AC][BC]\) with \([ABC]\) test, and why is it not a test of mutual independence?
If two models have deviances \(18.6\) and \(11.2\), with the first model nested in the second and three extra parameters in the second, what is the test statistic and its reference distribution?