AIC and BIC for Choosing the Number of Factors

Why Use Information Criteria to Choose a Factor Count?

A model with more factors can usually fit the observed covariance structure at least as well as a more restricted model. Better fit alone therefore tends to favor larger factor counts. AIC and BIC address this by adding a penalty for model complexity to a measure of lack of fit. Their balance is not identical: AIC uses a fixed penalty per parameter, whereas BIC’s penalty grows with the sample size.

For this comparison, each candidate factor count must be fitted by maximum likelihood to the same observations under the same likelihood convention. The criteria are not likelihood-ratio tests: they produce scores for comparing candidate models, not a test statistic with a null distribution.

Definition: For a fitted model with maximized log-likelihood $\ell(\hat{\theta})$ and $q$ free parameters, define $$ \mathrm{AIC}=-2\ell(\hat{\theta})+2q, \qquad \mathrm{BIC}=-2\ell(\hat{\theta})+q\log n, $$ where $n$ is the number of independent observations. Among the candidates being compared, the model with the smaller criterion value is preferred by that criterion. The quantity $-2\ell(\hat{\theta})$ is the likelihood-based lack-of-fit term, and the remaining term penalizes model dimension.

The log-likelihood must be evaluated consistently for every candidate. In particular, use the same observations and include the same likelihood constants across models. Omitting a constant that is common to all candidates does not change their ranking, but inconsistently changing the likelihood calculation can.

Counting Parameters in a Common-Factor Model

Consider $p$-dimensional observations with a Gaussian common-factor covariance model $$ X_i\sim N_p(0,\Sigma_k), \qquad \Sigma_k=\Lambda\Lambda^\mathsf{T}+\Psi, \qquad \Psi=\operatorname{diag}(\psi_1,\ldots,\psi_p), $$ where $\Lambda$ is a $p\times k$ loading matrix and all uniquenesses $\psi_j$ are positive. The mean is taken as known to be zero here. If the mean is estimated instead, there are $p$ additional mean parameters.

Counting all $pk$ entries of $\Lambda$ as distinct free parameters would overcount. For every orthogonal $k\times k$ matrix $R$, the loading matrix $\Lambda R$ gives the same covariance, because $(\Lambda R)(\Lambda R)^\mathsf{T}=\Lambda\Lambda^\mathsf{T}$. Thus rotations of the factors do not create different covariance models.

Result: At a generic, locally identified point of the $k$-factor covariance model, the number of free covariance parameters is $$ q_k=pk-\frac{k(k-1)}{2}+p. $$ The terms count loading parameters, subtract the continuous rotational redundancy, and add the $p$ uniquenesses. This is a generic dimension count: special or nonidentified points can require separate treatment.

To justify the count, the orthogonal group has $k(k-1)/2$ continuous degrees of freedom. One way to see this is to examine an orthogonal matrix near the identity: to first order it has the form $I+A$, and the condition $(I+A)^\mathsf{T}(I+A)=I$ implies $A^\mathsf{T}=-A$. A skew-symmetric $k\times k$ matrix has one freely chosen entry for each pair above the diagonal, giving $k(k-1)/2$ dimensions. At a generic loading matrix of full column rank, these rotations trace a redundancy of that dimension in the $pk$ loading coordinates. Adding the $p$ freely varying uniquenesses gives the stated formula. Discrete sign changes and permutations do not affect a dimension count.

This count applies when the model is considered in locally identified coordinates, with no additional constraints that change its dimension. If an implementation estimates the mean, add $p$ to $q_k$. If other parameters or constraints are used, count the parameters of the fitted model actually used for the likelihood. A common addition such as the same $p$ mean parameters for every candidate changes the absolute AIC and BIC values but not their rankings across factor counts.

Worked Example: Counting Parameters and Computing Scores

Suppose $p=6$ centered variables are fitted with two factors, the mean is fixed at zero, and the maximized log-likelihood gives $-2\ell(\hat{\theta})=480$. The sample size is $n=120$. Calculate the generic parameter count, AIC, and BIC.

For $k=2$, the loading matrix contributes $pk=6(2)=12$ coordinates. The rotational redundancy is $2(1)/2=1$, and the diagonal uniqueness matrix contributes 6 parameters. Therefore, $$ q_2=12-1+6=17. $$

The AIC is $$ 480+2(17)=480+34=514. $$ Since $\log(120)\approx4.78749$, the BIC is $$ 480+17\log(120)\approx480+17(4.78749)=480+81.3873=561.3873. $$

These are scores for this fitted model, not a decision by themselves. To select a factor count, compare them with the corresponding scores from the other candidate models fitted to the same observations.

What Improvement Is Needed to Add Factors?

The criteria’s penalties give a direct way to understand a comparison. Let candidate $j$ have deviance $D_j=-2\ell(\hat{\theta}_j)$ and parameter count $q_j$, and let candidate $k$ be another model. Smaller scores are preferred. Subtracting the score for $k$ from the score for $j$ gives the exact comparison.

Result: Candidate $j$ has a lower AIC than candidate $k$ exactly when $$ D_k-D_j>2(q_j-q_k), $$ and it has a lower BIC exactly when $$ D_k-D_j>(q_j-q_k)\log n. $$ If $j$ is the model with more factors and $q_j>q_k$, then $D_k-D_j$ is the reduction in deviance obtained by adding those factors.

For AIC, subtracting the two scores yields $$ \mathrm{AIC}_j-\mathrm{AIC}_k=(D_j-D_k)+2(q_j-q_k). $$ This difference is negative precisely when $D_k-D_j>2(q_j-q_k)$. Replacing the penalty $2q$ by $q\log n$ gives $$ \mathrm{BIC}_j-\mathrm{BIC}_k=(D_j-D_k)+(q_j-q_k)\log n, $$ which proves the BIC condition. Equality means a tie. In particular, if $n>e^2$, then $\log n>2$, so BIC requires a greater deviance reduction per added parameter than AIC. This comparison assumes the same $n$ and likelihood convention for both candidates.

Worked Example: AIC and BIC Disagree on an Added Factor

Compare one- and two-factor models for $p=5$ variables and $n=100$ observations. Assume the mean is fixed at zero. The one-factor model has deviance $D_1=900$, and the two-factor model has deviance $D_2=891$. Find the parameter counts and determine what each criterion prefers.

The counts are $$ q_1=5(1)-0+5=10, \qquad q_2=5(2)-\frac{2(1)}{2}+5=14. $$ Adding a factor reduces the deviance by $D_1-D_2=900-891=9$ and adds $14-10=4$ parameters.

The AIC penalty for those parameters is $2(4)=8$. The deviance reduction of 9 exceeds 8, so AIC favors the two-factor model. Direct calculation confirms this: $$ \mathrm{AIC}_1=900+2(10)=920, \qquad \mathrm{AIC}_2=891+2(14)=919. $$

For BIC, the penalty increase is $4\log(100)\approx4(4.60517)=18.4207$, which is larger than the deviance reduction of 9. Thus BIC favors one factor. Its scores are $$ \mathrm{BIC}_1=900+10\log(100)\approx946.0517, \qquad \mathrm{BIC}_2=891+14\log(100)\approx955.4724. $$ The disagreement follows from the penalties, not from different fitted likelihoods.

Comparing Several Fitted Factor Counts

The next example computes both criteria across a set of candidates. The deviance values below are assumed to come from maximum-likelihood fits to the same data. They decrease as factors are added, illustrating that the largest likelihood need not produce the preferred criterion value.

Worked Example: Choosing Among One, Two, and Three Factors

Suppose $p=8$, $n=200$, and the mean is fixed at zero. The fitted deviances are $D_1=1200$, $D_2=1176$, and $D_3=1162$. Calculate the generic parameter counts and both criteria.

Using $q_k=pk-k(k-1)/2+p$, $$ q_1=8+8=16,\qquad q_2=16-1+8=23,\qquad q_3=24-3+8=29. $$ The AIC values are $$ \mathrm{AIC}_1=1200+2(16)=1232, $$ $$ \mathrm{AIC}_2=1176+2(23)=1222, \qquad \mathrm{AIC}_3=1162+2(29)=1220. $$ Therefore AIC selects three factors, the candidate with the smallest AIC.

For BIC, $\log(200)\approx5.29832$. Hence $$ \mathrm{BIC}_1=1200+16(5.29832)\approx1284.77, $$ $$ \mathrm{BIC}_2=1176+23(5.29832)\approx1297.86, \qquad \mathrm{BIC}_3=1162+29(5.29832)\approx1315.65. $$ BIC selects one factor. Its larger complexity penalty outweighs the observed deviance reductions in this example.

The three-factor model has the largest maximized likelihood, but that fact alone does not settle the choice. AIC and BIC answer a comparison question using different penalties, so their selections can differ even when all fits and parameter counts have been computed correctly.

Interpretation, Assumptions, and a Common Pitfall

AIC is commonly motivated as an estimate of relative expected predictive information loss, with the $2q$ term correcting for the optimism of evaluating a fitted model on the data used to estimate it. BIC is commonly motivated through an approximation to the log marginal likelihood. In a regular, locally identifiable model with an interior parameter, nonsingular information, and a prior density that is positive and continuous near the fitted parameter, a Laplace approximation gives the reason for its $q\log n$ penalty.

In outline, write the model’s marginal likelihood as the integral of $\exp\{\ell(\theta)\}$ times a smooth prior density. Near an interior maximum, the log-likelihood is locally quadratic, with curvature of order $n$. Rescaling the $q$ local coordinates by $\sqrt{n}$ makes the integral’s local volume proportional to $n^{-q/2}$. Thus its leading log form is $\ell(\hat{\theta})-(q/2)\log n+O_p(1)$; multiplying by $-2$ gives $-2\ell(\hat{\theta})+q\log n+O_p(1)$. This regular approximation is a motivation for conventional BIC, not a guarantee that every factor-analysis fit satisfies its assumptions.

Factor models have rotational redundancy unless parameters are expressed in locally identified coordinates, and special or boundary estimates can violate ordinary regularity conditions. The generic parameter count is therefore the appropriate standard count for regular candidate fits, but a nonidentified or boundary fit calls for care in interpreting the usual asymptotic rationale. AIC and BIC are also relative criteria: their values do not establish that any candidate fits well in an absolute sense.

Common mistake: “The model with the most factors has the highest likelihood, so it must be the best choice.” This ignores the fact that a more flexible model can improve likelihood while using additional parameters. The correction is to compute the criterion for every candidate from its maximized likelihood and the parameter count, then compare the resulting values. Another frequent error is to count all loading entries without subtracting rotational redundancy, or to count a mean in one candidate but not the others.

Key takeaway: AIC and BIC combine maximum-likelihood fit with a complexity penalty. Use the generic factor-model parameter count only under its stated identification conditions, keep the likelihood convention consistent, and report which criterion selects which factor count rather than treating the choice as an automatic test result.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

For a centered Gaussian common-factor model with $p=7$ variables and $k=2$ factors, what is the generic covariance parameter count?
For two candidate models with parameter-count difference $q_j-q_k=5$, what deviance reduction does AIC require to prefer $j$ over $k$?
Why does estimating the same $p$-dimensional mean in every candidate change criterion values but not their ranking?
In the three-factor-count example, why does BIC select a different model from AIC?
Give one reason the usual generic parameter count or regular BIC approximation may need caution in a particular factor-analysis fit.