Likelihood-Ratio Assessment of Nested Loading Restrictions

Why Compare Nested Loading Restrictions?

A factor model often has loadings that are difficult to interpret or costly to estimate freely. A restriction, such as setting selected loadings to zero, expresses a simpler structure: perhaps an indicator is not associated with a factor, or a group of indicators does not load on a particular factor. The likelihood-ratio test asks whether the fit gained by relaxing such restrictions is large relative to sampling variation.

The comparison must be between nested models: every covariance matrix permitted by the restricted model must also be permitted by the less restricted model. The test compares their maximized likelihoods, not the loading estimates in isolation. In particular, a loading restriction should be interpreted through the distributions it permits, with identification and regularity checked before using a reference distribution.

Definition: Let \(H_0\) be a restricted model and \(H_1\) a less restricted model. They are nested when the set of distributions allowed by \(H_0\) is contained in the set allowed by \(H_1\). Write \(\ell_0\) and \(\ell_1\) for their respective maximized log-likelihoods. The likelihood-ratio statistic is \(T=-2\log\{\sup_{H_0}L/\sup_{H_1}L\}=2(\ell_1-\ell_0)\).

Because maximization over the larger model cannot yield a smaller likelihood, \(\ell_1\geq\ell_0\), so \(T\geq0\). A large value indicates that restricting the loadings causes a substantial loss of fit. Whether that loss is statistically unusual depends on the reference distribution conditions below.

Gaussian Factor-Model Likelihood

For clarity, consider independent observations \(X_1,\ldots,X_n\) from a \(p\)-variate normal distribution with mean \(\mu\) and covariance matrix \(\Sigma\). In a factor model, \(\Sigma=\Lambda\Phi\Lambda'+\Psi\), where \(\Lambda\) is the loading matrix, \(\Phi\) is the factor-covariance matrix, and \(\Psi\) is the uniqueness covariance matrix, usually diagonal. Loading restrictions specify a smaller set of allowable \(\Lambda\) values and therefore a smaller set of allowable covariance matrices.

Assume the same mean model is used in both fits and that \(\mu\) is unrestricted. Its maximum-likelihood estimate is \(\bar X\). Define the maximum-likelihood sample covariance \(S=n^{-1}\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)'\). For a candidate positive-definite covariance matrix \(\Sigma\), the log-likelihood after maximizing over \(\mu\), apart from a constant common to both models, is

\ell(\Sigma)=-\frac{n}{2}\left\{\log|\Sigma|+\operatorname{tr}(S\Sigma^{-1})\right\}.

This follows by summing the multivariate normal log densities: the quadratic terms sum to \(n\operatorname{tr}(S\Sigma^{-1})\), and the determinant term contributes \(n\log|\Sigma|\). Let \(\hat\Sigma_0\) and \(\hat\Sigma_1\) denote the covariance matrices at the restricted and less restricted maximum-likelihood fits. Subtracting their log-likelihoods gives the general statistic.

Derived likelihood-ratio formula: When the two models use the same Gaussian mean model and \(S\) is defined with divisor \(n\),

T=n\left[ \log\frac{|\hat\Sigma_0|}{|\hat\Sigma_1|} +\operatorname{tr}\left\{S(\hat\Sigma_0^{-1}-\hat\Sigma_1^{-1})\right\} \right].

The trace terms generally do not cancel: neither fitted covariance matrix need equal \(S\). The formula does simplify when the less restricted model is saturated, so that \(\hat\Sigma_1=S\), provided \(S\) is positive definite. Then \(\operatorname{tr}(S\hat\Sigma_1^{-1})=p\). In many nested comparisons, however, it is safest to use the maximized log-likelihoods directly.

Reference Distribution and Degrees of Freedom

The usual reference distribution is a consequence of Wilks' theorem. Its degrees of freedom are the difference in the dimensions of the two identifiable model spaces, not automatically the number of loading entries that look fixed in a parameter table.

Theorem (Wilks' likelihood-ratio theorem): Suppose independent observations follow a regular parametric model, the null model is a smooth submodel of the alternative of codimension \(q\), and the true distribution lies in the null model. Assume the true parameter is an interior point, the model is locally identifiable, the log-likelihood is sufficiently smooth, and the Fisher information is finite and nonsingular in identifiable coordinates. Under these conditions, as \(n\to\infty\), \(T\xrightarrow{d}\chi^2_q\).

When the null is written as \(g(\theta)=0\), the codimension is \(q\) if the derivative of \(g\) has rank \(q\) at the true parameter. Thus, two written restrictions count as two degrees of freedom only if they are locally independent in the identifiable model. If the null is false, the statistic does not generally have this central chi-square limit.

Factor models require particular care because rotations can produce different loading matrices for the same covariance matrix. A loading-based test needs a well-defined, locally identifiable formulation of the hypotheses; otherwise counting restrictions on \(\Lambda\) can overstate the degrees of freedom. Boundary parameters, singular information, and unidentified factor directions also invalidate the routine Wilks approximation. A zero loading is not automatically a boundary restriction: a real-valued loading may vary on either side of zero. But the overall model must still meet the regularity conditions.

Worked Example: Testing a Zero Loading in a Bivariate Model

Suppose \(n=100\) independent observations follow a bivariate normal one-factor model with factor variance fixed at one and diagonal uniqueness covariance. The fitted sample covariance is \(S=\begin{pmatrix}4&1.2\\1.2&9\end{pmatrix}\). Compare \(H_0:\lambda_2=0\) with the less restricted one-factor model.

Restricted fit. Under \(H_0\), the covariance is diagonal. The maximum-likelihood diagonal entries are the sample variances, so \(\hat\Sigma_0=\begin{pmatrix}4&0\\0&9\end{pmatrix}\). This covariance is allowed by the restriction: for example, take \(\lambda_2=0\), \(\lambda_1=1\), and uniqueness variances \(3\) and \(9\).

Less restricted fit. The sample covariance itself is attainable in the one-factor model. For example, choose \(\lambda_1=\sqrt{0.8}\), \(\lambda_2=\sqrt{1.8}\), and uniqueness variances \(3.2\) and \(7.2\). Then \(\lambda_1\lambda_2=1.2\), and adding the uniqueness variances to the loading squares gives diagonal entries \(4\) and \(9\). Thus \(\hat\Sigma_1=S\).

The determinants are \(|\hat\Sigma_0|=4(9)=36\) and \(|S|=4(9)-(1.2)^2=34.56\). Since \(\hat\Sigma_1=S\), both trace terms in the likelihood difference equal \(2\), and therefore \(T=100\log(36/34.56)=100\log(1.0416667)\approx4.0822\). At the covariance-model level, this comparison tests zero covariance against a free bivariate covariance, a one-dimensional restriction. Under the regularity conditions, the reference distribution is \(\chi^2_1\); the approximate upper-tail probability is \(0.043\). This is evidence against the restricted covariance structure at the five-percent level.

Two Further Comparisons

Worked Example: Restricting the Third Indicator's Loading

Consider \(n=120\) observations in a three-indicator, one-factor Gaussian model. Compare the model with all loadings free to the restriction \(\lambda_3=0\). Suppose the observed maximum-likelihood covariance is \(S=\begin{pmatrix}2&1&1\\1&3&1\\1&1&4\end{pmatrix}\).

The less restricted model attains \(S\): take all three loadings equal to \(1\), with uniqueness variances \(1,2,3\). Under the restriction, indicator 3 is uncorrelated with indicators 1 and 2. The restricted maximum-likelihood covariance is therefore \(\hat\Sigma_0=\begin{pmatrix}2&1&0\\1&3&0\\0&0&4\end{pmatrix}\); the first two indicators' block is attainable with loadings \(1,1\) and uniqueness variances \(1,2\).

The determinant of the restricted fit is \((2\cdot3-1)\cdot4=20\). The determinant of \(S\) is \(2(12-1)-(4-1)+(1-3)=17\). Both trace terms equal \(3\), because the alternative fit is \(S\) and the restricted fit reproduces the sample covariance within its two-by-two block and the third variance. Hence \(T=120\log(20/17)\approx120(0.162519)=19.5023\).

At the covariance level, the restriction sets two distinct covariances, \(\Sigma_{13}\) and \(\Sigma_{23}\), to zero, so the codimension is \(q=2\). The \(\chi^2_2\) upper-tail probability is \(\exp(-19.5023/2)\approx0.000058\). This example illustrates why the degrees of freedom follow the independent restrictions on the model distributions.

Worked Example: Using Maximized Log-Likelihoods

A fitted, identified factor model is compared with a nested model that fixes three locally independent loadings to zero. Assume the true parameter is an interior point of the null and the regularity conditions for Wilks' theorem hold. With \(n=80\), suppose the maximized log-likelihoods are \(\ell_0=-127.9\) and \(\ell_1=-124.6\).

The alternative is less restricted and has the larger log-likelihood, as required. The statistic is \(T=2\{(-124.6)-(-127.9)\}=2(3.3)=6.6\). The codimension is \(q=3\), because the three restrictions are assumed independent in identifiable coordinates. The \(\chi^2_3\) upper-tail probability is approximately \(0.086\); the result is not significant at the five-percent level. This calculation uses the reported maximized log-likelihoods, not a comparison of the raw loading estimates or their standard errors.

Proof Outline and Practical Checks

Fit both models by maximum likelihood.
The restricted model must be nested in the less restricted model, and both fits must use the same observations and compatible mean assumptions.

Calculate the likelihood-ratio statistic.
Use \(T=2(\ell_1-\ell_0)\), or the covariance-matrix formula when the fitted matrices are available. Check that \(T\geq0\).

Find the effective restriction count.
Use the codimension of the null in the identifiable model space, not a raw count that ignores redundant restrictions or rotational equivalence.

Check the approximation before reporting it.
Use a \(\chi^2_q\) reference distribution only when the null is true for the reference calculation and the regularity conditions are credible.

A common mistake is to count every fixed loading as one degree of freedom and immediately compare \(T\) with a chi-square distribution. This reasoning fails if some restrictions are redundant, if rotation leaves the loading hypotheses unidentified, or if the null lies on a boundary. The correction is to define the hypotheses as sets of distributions, establish the effective codimension under an identified parameterization, and check the theorem's conditions. If those conditions fail, the ordinary chi-square calibration should not be presented as justified.

The likelihood-ratio assessment matters because it turns a structural question about loadings into a comparison of how well two nested probability models explain the observed data. A small likelihood loss supports the simpler restriction; a large loss indicates that the added loading freedom materially improves fit. This is evidence about the specified models and assumptions, not proof that a particular rotated loading pattern is uniquely correct.

Key takeaway: For nested models, calculate \(T=2(\ell_1-\ell_0)\). Its usual \(\chi^2_q\) reference requires a regular, identifiable null model, with \(q\) equal to the null's effective codimension. Loading counts alone do not establish either condition.

Check Your Understanding

Use these questions to check both the calculation and the conditions behind the test.

Why must the log-likelihood under the less restricted model be at least as large as under the restricted model?
For Gaussian covariance fits, which terms appear in the likelihood-ratio formula when neither fitted covariance matrix equals \(S\)?
What does \(q\) count in Wilks' theorem, and why might it differ from the number of loading entries fixed to zero?
In the three-indicator example, why is the covariance-level codimension two when \(\lambda_3=0\)?
Name two conditions under which the ordinary chi-square reference distribution may not be justified.