Comparing Non-Nested Factor Models With Information Criteria

Why Compare Models With Information Criteria?

Two factor models can offer different descriptions of the same observed variables without one being a special case of the other. For example, alternative loading patterns may assign variables to different groups of factors. A likelihood-ratio test is designed for nested models; it does not generally provide a valid comparison for such non-nested alternatives. Information criteria instead combine each fitted model’s likelihood with a penalty for its complexity. They give a common numerical basis for comparing models fitted to the same observations.

The comparison is not simply “choose the model with the larger likelihood.” A more flexible model can usually fit at least as well as a less flexible one, but its additional parameters can also make its apparent fit less useful as a guide to new data. AIC and BIC penalize this flexibility differently, so it is possible for them to prefer different models.

Definition: For a model with maximized log-likelihood $\ell(\hat{\theta})$ and $k$ free, identifiable parameters, the Akaike information criterion and Bayesian information criterion are $$ \operatorname{AIC}=-2\ell(\hat{\theta})+2k, \qquad \operatorname{BIC}=-2\ell(\hat{\theta})+k\log n, $$ where $n$ is the number of independent observational units. When comparing models fitted to the same data, the smaller criterion is preferred by that criterion.

The factor $-2$ puts the likelihood term on the deviance scale. Because a larger maximized likelihood means a larger log-likelihood, it makes $-2\ell(\hat{\theta})$ smaller. The added term is the complexity penalty: AIC charges $2$ per parameter, whereas BIC charges $\log n$ per parameter. Neither criterion is a probability that a model is true.

Using the Criteria for Factor Models

Suppose each of $n$ independent observations is a $p$-dimensional vector $X_i$, and a candidate factor model specifies a distribution for those vectors. For example, a centered Gaussian factor model may specify $\operatorname{Cov}(X_i)=\Lambda\Lambda^\mathsf{T}+\Psi$, where $\Psi$ is diagonal. Fit each candidate by maximizing its likelihood for the same observed vectors, then calculate the criterion using that model’s maximized log-likelihood and parameter count.

The count $k$ is the dimension of the model’s locally identifiable parameterization, not necessarily the number of entries in the matrix as it is first written. Rotationally equivalent loading matrices can represent the same covariance model. Counting every loading as independent when some changes only rotate the factors overcounts the effective dimension. Conversely, if the mean is estimated rather than fixed, its free parameters must be included. State what is held fixed and how the factor model is identified before giving a count.

For a centered Gaussian model with known zero mean and $p=5$, consider the generic orthogonal factor covariance model $\Lambda\Lambda^\mathsf{T}+\Psi$, with positive diagonal $\Psi$. With one factor, there are five loading and five uniqueness parameters; the sign ambiguity is discrete and does not reduce the local dimension, so $k=10$. With two factors, there are ten loading and five uniqueness entries, but a continuous rotation of the two factor axes has one degree of freedom and leaves the covariance unchanged. Under generic local identification, the effective dimension is therefore $k=10+5-1=14$. If the five observed means were also estimated, five parameters would be added to each count.

Worked Example: One-Factor and Two-Factor Gaussian Models

For $n=120$ centered observations on five variables, suppose the fitted one-factor model has $\ell_1=-430$ and $k_1=10$, while the fitted two-factor model has $\ell_2=-421$ and $k_2=14$. Use $\log(120)\approx4.7875$.

One factor. Its AIC is $-2(-430)+2(10)=860+20=880$. Its BIC is $860+10(4.7875)=907.875$.

Two factors. Its AIC is $-2(-421)+2(14)=842+28=870$. Its BIC is $842+14(4.7875)=842+67.025=909.025$.

AIC prefers the two-factor model because $870<880$; BIC prefers the one-factor model because $907.875<909.025$. The two-factor fit improves the maximized log-likelihood by $9$, but it also adds four parameters. The criteria weigh that improvement differently.

How the Penalties Change a Comparison

For two candidate models $A$ and $B$, define $\Delta\ell=\ell_B(\hat{\theta}_B)-\ell_A(\hat{\theta}_A)$ and $\Delta k=k_B-k_A$. Subtracting the criteria gives exact comparison rules:

\operatorname{AIC}_B-\operatorname{AIC}_A=-2\Delta\ell+2\Delta k, \qquad \operatorname{BIC}_B-\operatorname{BIC}_A=-2\Delta\ell+(\log n)\Delta k.

If $B$ has more parameters, AIC prefers it precisely when $\Delta\ell>\Delta k$; BIC prefers it precisely when $\Delta\ell>\tfrac12(\log n)\Delta k$. These are thresholds for the increase in maximized log-likelihood, not likelihood-ratio tests. When $n>e^2$, $\log n>2$, so BIC’s per-parameter penalty exceeds AIC’s. At very large sample sizes, even a modest improvement in fit can justify extra parameters under AIC, while BIC demands a larger improvement.

Worked Example: A Fit Improvement That Clears Only the AIC Threshold

Suppose two non-nested factor models are fitted to the same $n=80$ observations. Model $A$ has $\ell_A=-250$, $k_A=8$; model $B$ has $\ell_B=-246$, $k_B=11$. Thus $\Delta\ell=4$, $\Delta k=3$, and $\log(80)\approx4.3820$.

The criteria themselves are $\operatorname{AIC}_A=500+16=516$ and $\operatorname{AIC}_B=492+22=514$. AIC prefers $B$. For BIC, $\operatorname{BIC}_A=500+8(4.3820)=535.056$ and $\operatorname{BIC}_B=492+11(4.3820)=540.202$, so BIC prefers $A$.

The threshold calculation gives the same answer. AIC requires $\Delta\ell>\Delta k=3$, which holds because $4>3$. BIC requires $\Delta\ell>\tfrac12(4.3820)(3)=6.573$, which fails because $4<6.573$.

Why the Criteria Have Different Penalties

AIC’s penalty is motivated by estimating predictive performance. Under regularity conditions, fitting parameters to a sample makes the fitted model look better on that sample than it will on fresh observations. The following derivation gives the leading size of that optimism. Assume independent observations from a correctly specified, regular $k$-parameter model, with the true parameter $\theta_0$ in the interior, a nonsingular Fisher information matrix $I$, and the usual differentiability and moment conditions for maximum-likelihood asymptotics. Let $\hat{\theta}$ be the maximum-likelihood estimator and compare a training sample of size $n$ with an independent test sample of the same size.

Result: Under those conditions, the expected difference between the fitted log-likelihood on the training sample and its log-likelihood on the independent test sample is $k+o(1)$. Thus the leading optimism on the $-2\ell$ scale is $2k$, the AIC penalty.

To see this, write $S_n$ for the training score at $\theta_0$ and $I$ for the per-observation Fisher information. The score has mean zero and covariance $nI$. A second-order likelihood expansion around $\theta_0$, together with $\hat{\theta}-\theta_0=I^{-1}S_n/n+o_p(n^{-1/2})$, gives the training-sample log-likelihood gain $\ell_{\mathrm{train}}(\hat{\theta})-\ell_{\mathrm{train}}(\theta_0)=\tfrac12 S_n^\mathsf{T}I^{-1}S_n/n+o_p(1)$. Taking expectations, the leading quadratic term has expectation $\tfrac12\operatorname{tr}(I^{-1}nI)/n=k/2$.

For the independent test sample, its score at $\theta_0$ has conditional mean zero given the training sample. Its first-order contribution to the expected test log-likelihood difference therefore vanishes. The second-order contribution is $-\tfrac n2(\hat{\theta}-\theta_0)^\mathsf{T}I(\hat{\theta}-\theta_0)$; its expectation is $-k/2+o(1)$, using the same score covariance calculation. The expected training-minus-test log-likelihood gap is consequently $k/2-(-k/2)+o(1)=k+o(1)$. Multiplying by two gives the AIC penalty on the deviance scale. This is an asymptotic argument, not a guarantee that AIC always selects the best predictive model in a particular data set. $\square$

BIC has a different derivation. It approximates a model’s marginal likelihood, which averages the likelihood over parameter values using a prior density. For a regular, identifiable $k$-parameter model with an interior maximum-likelihood estimate and a prior density $\pi$ continuous and positive there, a Laplace approximation yields the following result.

Result: For fixed $k$, under the regularity conditions above, $\log \int L(\theta)\pi(\theta)\,d\theta=\ell(\hat{\theta})-\tfrac{k}{2}\log n+O_p(1)$. Therefore minus twice the log marginal likelihood equals $\operatorname{BIC}+O_p(1)$.

Near the maximum, the log-likelihood has the quadratic approximation $\ell(\theta)=\ell(\hat{\theta})-\tfrac n2(\theta-\hat{\theta})^\mathsf{T}I(\hat{\theta})(\theta-\hat{\theta})+o_p(1)$ on the $n^{-1/2}$ scale, where $I(\hat{\theta})$ is the per-observation information matrix. The integral’s leading contribution comes from a neighborhood of width $n^{-1/2}$ in each of the $k$ parameter directions. Rescaling $u=\sqrt n(\theta-\hat{\theta})$ contributes a volume factor $n^{-k/2}$. The remaining Gaussian integral, the prior density near the estimate, and the determinant of the information matrix contribute terms whose logarithms are $O_p(1)$. Hence the log integral is $\ell(\hat{\theta})-\tfrac{k}{2}\log n+O_p(1)$; multiplying by $-2$ gives BIC up to $O_p(1)$. $\square$

Thus AIC’s constant penalty targets leading predictive optimism, while BIC’s penalty grows with sample size because it approximates evidence averaged over parameter values. Under suitable regularity conditions, BIC also selects the true model consistently when the true model belongs to a finite candidate set and receives positive prior probability. These theoretical interpretations have assumptions; in particular, they do not make the criteria interchangeable.

Worked Comparison With a Parameter-Counting Check

Worked Example: Counting Parameters Before Comparing Fits

Consider two fitted covariance models for the same $n=150$ centered observations on $p=4$ variables. The mean is fixed at zero. Model $A$ is a generic one-factor model with diagonal uniquenesses, so its four loadings and four uniquenesses give $k_A=8$. Model $B$ is an unrestricted positive-definite covariance model, with $4(5)/2=10$ distinct covariance entries, so $k_B=10$. Assume both fits are interior and the factor model is locally identified. Suppose $\ell_A=-310$ and $\ell_B=-306$.

With $\log(150)\approx5.0106$, the AIC values are $\operatorname{AIC}_A=620+16=636$ and $\operatorname{AIC}_B=612+20=632$. AIC prefers the unrestricted covariance model.

The BIC values are $\operatorname{BIC}_A=620+8(5.0106)=660.085$ and $\operatorname{BIC}_B=612+10(5.0106)=662.106$. BIC prefers the one-factor model. The unrestricted fit gains $4$ log-likelihood units but uses two more parameters. That gain exceeds the AIC threshold $2$, but not the BIC threshold $\tfrac12(5.0106)(2)=5.0106$.

Common Pitfalls and Practical Interpretation

An information-criterion comparison is meaningful only when both likelihoods use the same observations and the same likelihood convention. For example, do not compare a log-likelihood reported per observation with a total log-likelihood, or fit one model after dropping cases that were used for the other. Use $n$ for the number of independent observational units, not the number of measured variables or the total number of matrix entries.

Choosing the largest likelihood alone. This ignores the fact that extra flexibility tends to improve in-sample fit. Compare the penalized criteria instead.
Using a nested-model likelihood-ratio reference distribution for non-nested models. That reference distribution does not generally apply. AIC and BIC are alternative comparison criteria, not a replacement chi-square test.
Counting every written loading as an independent parameter. In factor models, rotational redundancy can make that count too large. Count identifiable dimensions, and state any fixed means, variances, or constraints that affect the count.
Treating a small criterion difference as decisive evidence. The smaller value identifies the preference under that criterion, but does not establish that the model is true or that the difference is practically important.

Factor models can have singularities, boundary parameters, or unresolved identification, situations in which the regular likelihood approximations motivating standard AIC and BIC may not apply as stated. The formulas remain commonly used, but their interpretation requires caution when regularity fails. When competing models represent the same covariance structure under a rotation, a different orientation alone is not evidence of a better-fitting model: the likelihood and effective model dimension should reflect the represented distribution, not an arbitrary choice of factor coordinates.

Check Your Understanding

Work through these questions using the definitions and comparison rules above.

Two models have $\ell_A=-180,\ k_A=6$ and $\ell_B=-175,\ k_B=9$, fitted to the same $n=100$ observations. Calculate both criteria and state which model each prefers.
For a model with two additional parameters, what increase in maximized log-likelihood is required for it to improve AIC? What is the corresponding BIC threshold at sample size $n$?
Why can counting all entries of a loading matrix overstate the number of free parameters in a factor model?
Explain in one sentence why AIC’s penalty is constant per parameter while BIC’s penalty increases with $n$.
Name two conditions that make a direct AIC or BIC comparison inappropriate or in need of caution.