Saturated Loglinear Models

What Saturation Means

A loglinear model describes the expected count in each cell of a categorical table. A model is saturated when its terms are sufficiently numerous to represent every possible vector of cell means. It therefore places no restrictions on the relative sizes of the cell means: the table dimensions determine how many means must be represented, and the fitted mean in each cell matches its observed count.

Suppose a table has \(m\) possible cells, with observed counts \(Y_1,\ldots,Y_m\). Under the independent-Poisson formulation, the counts are independent and \(Y_r\sim\operatorname{Poisson}(\mu_r)\), where \(\mu_r>0\) is the expected count in cell \(r\). A loglinear model specifies \(\log(\mu_r)\) as a linear predictor. For the ordinary case considered here, every cell is possible; a sampling-zero cell is allowed, even when its observed count is zero.

Definition: A loglinear model for \(m\) possible cells is saturated if its design matrix has rank \(m\), so its linear predictors can take any value in \(\mathbb{R}^m\). Equivalently, it imposes no constraints on the vector of log means. Its nominal residual degrees of freedom are \(m-m=0\).

“Nominal” matters when counts are zero. A zero observed count has maximum-likelihood fitted mean zero, which is on the boundary of the nonnegative mean space. Because a finite log mean always produces a strictly positive mean, that estimate requires a log mean tending to \(-\infty\), not a finite coefficient. The saturated model still has zero nominal residual degrees of freedom; boundary estimation does not change the number of cells or the design-matrix rank.

The Cell-Indicator Construction

The most direct saturated model assigns one parameter to every cell. Let \(I(r=s)\) equal 1 when an observation belongs to cell \(s\), and 0 otherwise. Then

\log(\mu_r)=\sum_{s=1}^{m}\theta_s I(r=s)=\theta_r,\qquad r=1,\ldots,m.

The design matrix of these cell indicators is the \(m\times m\) identity matrix. Thus it has rank \(m\), and each cell’s log mean can be selected independently. This representation is compact and unambiguous, although factor-effect notation is often more informative about which combinations of factors are included.

For factors with \(I,J,\ldots\) levels, a full factorial loglinear model contains a main effect for each factor, every two-factor interaction, every higher-order interaction, and the highest-order interaction across all factors. With reference-level indicator coding, the model has one intercept and \(I-1,J-1,\ldots\) indicator columns and their products. The resulting number of coefficients is

$$ 1+(I-1)+(J-1)+(I-1)(J-1)=IJ $$

for a two-way table, and \(\prod_{q=1}^{K} I_q\) for a table of \(K\) factors with \(I_q\) levels in factor \(q\). The product terms are what allow the model to represent cell-specific departures from lower-order terms. Omitting a term can impose restrictions; including the complete set yields a saturated model.

Worked Example: A Saturated Two-Way Table

Consider a \(2\times2\) table with observed counts \(y_{11}=12,\ y_{12}=8,\ y_{21}=6,\ y_{22}=14\). Code the second level of each factor with an indicator taking value 1, and use the model

\log(\mu_{ij})=\beta_0+\beta_A x_i+\beta_B z_j+\beta_{AB}x_i z_j.

The four cell means are unrestricted because there are four coefficients for four cells. Set each fitted mean equal to its observed count. In the reference cell \((1,1)\), \(x_1=z_1=0\), so \(\beta_0=\log(12)\). In cell \((2,1)\), \(x_2=1,z_1=0\), giving \(\beta_A=\log(6)-\log(12)=\log(1/2)\). In cell \((1,2)\), \(\beta_B=\log(8)-\log(12)=\log(2/3)\). Finally, the \((2,2)\) cell gives

\beta_{AB}=\log(14)-\beta_0-\beta_A-\beta_B =\log\!\left(\frac{14\cdot12}{6\cdot8}\right) =\log(3.5).

Substitution verifies the fitted means: \(\exp(\beta_0)=12\), \(\exp(\beta_0+\beta_A)=6\), \(\exp(\beta_0+\beta_B)=8\), and \(\exp(\beta_0+\beta_A+\beta_B+\beta_{AB})=14\). There are \(2\cdot2=4\) cells and four independent coefficients, so the residual degrees of freedom are \(4-4=0\).

Why the Fitted Counts Equal the Observed Counts

This property follows directly from the likelihood; it is not merely a consequence of having a flexible-looking formula. For independent Poisson cells, the log-likelihood, apart from no omitted terms, is

\ell(\boldsymbol{\mu})=\sum_{r=1}^{m} \left\{y_r\log(\mu_r)-\mu_r-\log(y_r!)\right\}.

In a saturated model, each \(\mu_r\) can be chosen independently of the other cell means. Thus maximizing the full log-likelihood reduces to maximizing one Poisson contribution at a time.

Theorem: For independent Poisson counts in \(m\) possible cells, the saturated model has maximum-likelihood fitted means \(\hat{\mu}_r=y_r\) for all cells, when the mean space includes its boundary at zero. If every observed count is positive, the maximum is attained at finite log means. If any observed count is zero, the maximum is attained at \(\mu_r=0\) for that cell, corresponding to a log mean of \(-\infty\) in the closure of the ordinary loglinear parameterization.

Proof. For a cell with \(y_r>0\), differentiate its log-likelihood contribution with respect to \(\mu_r>0\):

\frac{\partial \ell_r}{\partial\mu_r}=\frac{y_r}{\mu_r}-1,\qquad \frac{\partial^2 \ell_r}{\partial\mu_r^2}=-\frac{y_r}{\mu_r^2}<0.

The first derivative is zero only at \(\mu_r=y_r\), and the strictly negative second derivative shows this is the unique maximum. If \(y_r=0\), then \(\ell_r=-\mu_r\), since \(\log(0!)=0\). This decreases as \(\mu_r\) increases, so its maximum over \(\mu_r\geq0\) is at \(\mu_r=0\). The likelihood separates across cells, and saturation permits each mean to be selected independently. Combining the cellwise maxima proves the result. \(\square\)

When all counts are positive, the fitted value \(\hat{\mu}_r=y_r\) is achieved by the finite cell parameter \(\hat{\theta}_r=\log(y_r)\) in the cell-indicator model. When a count is zero, \(\log(y_r)\) is undefined; the mean estimate remains meaningful, but it is a boundary estimate rather than a finite log-mean estimate.

Worked Example: A Zero Count in a Saturated Table

Suppose a \(2\times3\) table has counts

\begin{array}{c|ccc} & B_1 & B_2 & B_3\\ \hline A_1 & 5 & 0 & 9\\ A_2 & 2 & 4 & 0 \end{array}

There are six possible cells, so a cell-indicator saturated model has six mean parameters. Maximizing the Poisson likelihood cell by cell gives fitted means

\begin{array}{c|ccc} & B_1 & B_2 & B_3\\ \hline A_1 & 5 & 0 & 9\\ A_2 & 2 & 4 & 0 \end{array}.

The four positive-count cells have finite fitted log means \(\log(5),\log(9),\log(2),\log(4)\), respectively. For each zero-count cell, the likelihood is maximized at mean zero; no finite value of its log mean gives exactly zero. The nominal residual degrees of freedom are still \(6-6=0\). Treating a sampling zero as if it removed a cell would incorrectly change the table’s model dimension.

Dimensions and Residual Degrees of Freedom

For a \(K\)-factor table with \(I_q\) levels for factor \(q\), the number of possible cells is \(m=\prod_{q=1}^{K}I_q\). A cell-indicator saturated model has \(m\) columns and rank \(m\). In full factorial notation, the same count results by choosing, for each factor, either its intercept contribution or one of its \(I_q-1\) reference-level indicators. The total number of coefficients is therefore

\prod_{q=1}^{K}\{1+(I_q-1)\} =\prod_{q=1}^{K} I_q=m.

This is a dimension count, not a claim that every chosen parameterization is automatically identifiable. For example, writing an intercept plus all level effects without constraints can introduce redundant coefficients. Reference coding, or suitable sum-to-zero constraints, removes that redundancy. The resulting identifiable saturated model has \(m\) parameters.

Result: In a table with \(m\) possible cells and a loglinear design matrix of rank \(p\), the nominal residual degrees of freedom are \(m-p\). In particular, for a saturated model \(p=m\), so the residual degrees of freedom are zero.

Justification. There is one observed count for each of the \(m\) cells. A rank-\(p\) design matrix has \(p\) linearly independent estimable directions in its vector of linear predictors. The residual degrees of freedom are the dimension left after accounting for those \(p\) directions, namely \(m-p\). For cell indicators the design matrix is the identity, hence \(p=m\) and \(m-p=0\). This also shows why the relevant count is the design-matrix rank rather than a possibly redundant number of symbols in an unconstrained formula. \(\square\)

Worked Example: Counting Terms in a Three-Way Table

Consider a \(2\times2\times2\) table with all eight observed counts positive:

\begin{array}{c|cc|cc} & \multicolumn{2}{c|}{A_1} & \multicolumn{2}{c}{A_2}\\ & C_1 & C_2 & C_1 & C_2\\ \hline B_1 & 2 & 3 & 6 & 7\\ B_2 & 4 & 5 & 8 & 9 \end{array}

Code level 2 of each factor by \(x_A,x_B,x_C=1\), and level 1 by 0. The complete three-way model includes one intercept, three main effects, three two-factor products, and one three-factor product:

\log(\mu)=\beta_0+\beta_Ax_A+\beta_Bx_B+\beta_Cx_C+ \beta_{AB}x_Ax_B+\beta_{AC}x_Ax_C+\beta_{BC}x_Bx_C+ \beta_{ABC}x_Ax_Bx_C.

There are \(1+3+3+1=8\) coefficients for eight cells. To verify that the model reproduces all counts, order the counts by \((x_A,x_B,x_C)\): \(2,4,3,5,6,8,7,9\) at \(000,010,001,011,100,110,101,111\). The coefficients obtained successively from these cells are

\begin{aligned} \beta_0&=\log 2, & \beta_A&=\log 3, & \beta_B&=\log 2, & \beta_C&=\log(3/2),\\ \beta_{AB}&=\log(2/3), & \beta_{AC}&=\log(7/9), & \beta_{BC}&=\log(5/6), & \beta_{ABC}&=\log(81/70). \end{aligned}

For instance, at \(110\) the linear predictor is \(\log 2+\log3+\log2+\log(2/3)=\log8\); at \(101\) it is \(\log2+\log3+\log(3/2)+\log(7/9)=\log7\). At \(111\), adding all eight coefficients gives \(\log9\). The first four cells are reproduced by the intercept and the corresponding single-factor terms, and the calculations for \(110\) and \(101\) verify the next two cells. The \(011\) cell is \(\exp(\beta_0+\beta_B+\beta_C+\beta_{BC})=5\). Thus all eight fitted means equal their observed counts. The residual degrees of freedom are \(8-8=0\).

Common Mistakes and Why Saturation Matters

Assuming every coefficient must be finite. With a zero observed count, the saturated maximum is a zero mean. The associated log mean tends to \(-\infty\); it is not a finite logarithm of zero.
Counting written terms without checking rank. Redundant level indicators can make a formula appear to have more coefficients than estimable parameters. Use an identifiable coding or count the design-matrix rank.
Removing a zero-count cell from the table. A sampling zero is still one of the possible cells. It contributes to the table dimension, even though its fitted mean is zero at the likelihood maximum.
Interpreting zero residual degrees of freedom as a useful model check. Saturation guarantees a perfect match to the observed counts, so it leaves no residual degrees of freedom for assessing lack of fit. Its main role is as a fully flexible reference model.

The central calculation is straightforward once the distinction between cells and parameters is kept clear: count the possible cells, determine the rank of the model matrix, and subtract. Saturation means the rank equals the number of cells; the Poisson likelihood then fits each cell mean to its observed count, with boundary care for zeros.

Check Your Understanding

Work through these questions using the definitions and derivations above.

A \(3\times4\) table has all cells possible. How many cell means does a saturated model represent, and what are its nominal residual degrees of freedom?
Why does a zero observed count have a maximum-likelihood fitted mean of zero, but no finite log mean?
For a \(2\times3\times4\) table, count the coefficients in a full factorial reference-coded model and determine the residual degrees of freedom.
A model formula lists ten coefficients for a table with eight cells, but two coefficients are redundant. What rank should be used when calculating residual degrees of freedom?
Explain why a saturated model’s exact reproduction of the observed counts does not, by itself, provide a lack-of-fit assessment.

What You'll Learn