Why Compare Nested ARMA Models?
Information criteria such as AIC and BIC compare models by balancing fit against complexity. A likelihood-ratio test asks a different, more focused question: does imposing a specified set of restrictions cause a sufficiently large loss of fit to count as evidence against those restrictions? This is useful when a simpler ARMA specification is obtained from a larger one by setting particular coefficients to zero.
The test is not simply a comparison of two reported likelihoods. Its usual chi-square calibration relies on regularity conditions, including identifiable parameters and restrictions that do not put the null model on the boundary of the parameter space. In ARMA applications, the likelihoods must also be computed on a comparable basis.
Nested Models and the Likelihood-Ratio Statistic
Write a Gaussian ARMA model as \(\phi(B)(Y_t-\mu)=\theta(B)\epsilon_t\), where \(B\) is the lag operator, \(\phi(B)=1-\phi_1B-\cdots-\phi_pB^p\), and \(\theta(B)=1+\theta_1B+\cdots+\theta_qB^q\). The innovations are independent \(N(0,\sigma^2)\), and the model is restricted to an identifiable, stationary and invertible parameter region when those properties are required.
A restricted model is nested in an unrestricted model if every distribution allowed by the restricted model is also allowed by the unrestricted model. For example, an AR(1) is nested in an ARMA(1,1) by setting the MA coefficient \(\theta_1\) to zero. Let \(\ell_U\) be the maximized log likelihood under the unrestricted model and \(\ell_R\) the maximized log likelihood under the restricted model, computed for the same observations and likelihood formulation.
The hypotheses are \(H_0\): all \(q\) restrictions hold, and \(H_a\): at least one does not. A large value of \(LR\) indicates that the unrestricted model fits substantially better than the restricted model. The p-value is the upper-tail probability \(P(\chi^2_q\ge LR)\), calculated under the null reference distribution. At significance level \(\alpha\), reject \(H_0\) if this p-value is less than \(\alpha\).
The degrees of freedom count restrictions, not the total number of parameters in either model. If two coefficients are set to zero, \(q=2\), provided those are two independent restrictions and the regularity conditions hold.
Worked Example: Testing Whether an MA Term Is Needed
Suppose 180 hypothetical observations are fitted with an ARMA(1,1) and, under the restriction \(\theta_1=0\), an AR(1). Both are fitted using the same exact Gaussian likelihood and the same observations. Assume the estimates are interior to the stationary and invertible region, the models are locally identifiable, and the innovation variance is positive. The maximized log likelihoods are \(\ell_U=-142.310\) for the ARMA(1,1) and \(\ell_R=-144.820\) for the AR(1).
There is one restriction, so \(q=1\). The statistic is
Under \(H_0:\theta_1=0\), the reference distribution is approximately \(\chi^2_1\). Its upper-tail probability at 5.020 is approximately 0.0251. At \(\alpha=0.05\), reject the restriction that the MA coefficient is zero. In this hypothetical analysis, the data provide evidence that the ARMA(1,1) fits better than the nested AR(1), under the stated likelihood and regularity assumptions. This does not establish that the extra term represents a causal mechanism.
Why the Chi-Square Reference Distribution Applies
The asymptotic result is a consequence of how the log likelihood behaves near its maximum under regularity. Informally, near the true parameter, the log likelihood is approximately a quadratic function. The unrestricted fit can move in directions unavailable under the restrictions; the likelihood-ratio statistic measures the improvement from those additional directions.
For a short justification, let \(\eta\) denote the \(q\) parameters constrained by \(H_0\), and let the remaining parameters be nuisance parameters. Under the null, the constrained estimate sets \(\eta\) to its null value. With a correctly specified likelihood, an interior true parameter, and standard differentiability and information conditions, the log likelihood has a local quadratic approximation around the true parameter. After accounting for nuisance parameters, the unrestricted estimate's \(q\) additional directions have an asymptotically standard-normal score after suitable scaling. The quadratic improvement in log likelihood is therefore asymptotically the sum of squares of \(q\) independent standard normal variables:
where \(Z_1,\ldots,Z_q\) are independent \(N(0,1)\) variables. This is the source of the chi-square calibration. It is an asymptotic result, not a claim that the statistic has an exact chi-square distribution in a finite sample.
Worked Example: Testing Two ARMA Restrictions
Consider a hypothetical ARMA(2,1) fitted to 240 observations, compared with a nested AR(1). In the common specification, the restricted model sets \(\phi_2=0\) and \(\theta_1=0\), leaving the AR(1) coefficient, mean, and innovation variance unrestricted. Suppose both fits use the same exact Gaussian likelihood, and the relevant estimates are interior and identifiable. Let \(\ell_U=-318.70\) and \(\ell_R=-321.00\).
The null imposes two independent restrictions, so \(q=2\). The likelihood-ratio statistic is
For two degrees of freedom, the chi-square upper-tail probability is approximately 0.1003. At \(\alpha=0.05\), do not reject the joint restrictions. The result does not show that both coefficients are zero or prove the AR(1) is the true model; it says that this test does not find a statistically significant likelihood loss from imposing the two restrictions at the 5% level.
Regularity Conditions and Warning Signs
The usual chi-square approximation is most defensible when the null parameter is in the interior of the larger model's admissible region, the parameters are identifiable, the information matrix is nonsingular, and the likelihood is sufficiently smooth. The observations and likelihood construction must also be comparable across the two fits.
Several ARMA features can make these requirements questionable:
- A boundary restriction. If the null places a parameter at a stationarity or invertibility boundary, the ordinary interior-parameter derivation may fail. The null distribution need not be \(\chi^2_q\).
- Nonidentification or near-cancellation. If different parameter values represent the same process, or an AR and MA factor nearly cancel, the likelihood may not have the regular local shape assumed by the standard result.
- A poorly determined nuisance parameter. If a parameter is unidentified under the null, counting restrictions alone does not establish a chi-square reference distribution.
- Different likelihood treatments. Exact Gaussian likelihoods and conditional likelihoods handle initial observations differently. Comparing maxima from different treatments, different effective samples, or inconsistent constants can make the likelihood difference uninterpretable.
- Small samples or weak identification. Even when the formal conditions hold, the asymptotic approximation can be inaccurate in a short series or when the likelihood is unusually flat.
These are reasons to investigate the model fit and the likelihood construction, not reasons to assume that a different calibration automatically solves the problem. In a doubtful case, report the concern and seek a justified reference distribution or a suitable simulation-based calibration for the specific model.
Worked Example: Recognizing a Boundary Problem
Suppose a hypothetical AR(1) coefficient is constrained to the stationary region \(|\phi_1|<1\), and a proposed null fixes \(\phi_1=1\). Assume the likelihood is otherwise computed consistently for the restricted and unrestricted fits. The unrestricted model is stationary, but the null value lies on the boundary of the stationary region rather than in its interior.
The arithmetic \(2(\ell_U-\ell_R)\) can still be computed, but the ordinary argument for a \(\chi^2_1\) reference distribution does not apply: it requires an interior null parameter. Therefore, looking up a \(\chi^2_1\) p-value and treating it as a standard likelihood-ratio p-value is not justified here. One must use a result or calibration appropriate to this boundary problem. This example illustrates a limitation of the usual test; it does not provide that alternative calibration.
Likelihood-Ratio and Wald Tests
The earlier material on Wald inference for ARMA parameters gives a test based on the estimated restriction and its standard error. For a single restriction \(r(\beta)=0\), a Wald statistic measures the squared, standard-error-scaled distance between the unrestricted estimate and the null value. For several restrictions, it uses the estimated covariance matrix of the restriction estimates. Under regularity, the Wald statistic is also asymptotically \(\chi^2_q\).
The tests use different information. The likelihood-ratio test compares the maxima under the restricted and unrestricted models. The Wald test uses the unrestricted estimate and its estimated covariance; it does not require fitting the restricted model. In finite samples their statistics and p-values can differ, even when they test the same null. Under the regularity conditions and large samples, they have the same asymptotic null calibration.
Worked Example: Comparing Likelihood-Ratio and Wald Results
Return to the hypothetical ARMA(1,1) comparison, with one restriction \(\theta_1=0\). Suppose the unrestricted fit gives \(\hat{\theta}_1=0.180\) with estimated standard error \(0.080\), and the likelihood-ratio statistic from the two fits is 5.020. Assume the estimate is interior, the coefficient is identified, and the standard error is valid for this fit.
The Wald statistic is the squared estimate divided by its estimated variance:
Using \(\chi^2_1\), the Wald p-value is approximately 0.0244; the likelihood-ratio p-value was approximately 0.0251. Both tests reject at the 5% level, and their small numerical difference is expected because they use different finite-sample calculations. The standard error is in the units of \(\theta_1\); the test statistics themselves are dimensionless.
Common Pitfalls and Practical Interpretation
A frequent mistake is to calculate \(2(\ell_U-\ell_R)\) and immediately compare it with a chi-square distribution, without checking that the models are nested, the log likelihoods are comparable, and the null is regular. The correction is to verify those conditions before interpreting a p-value. If the models are not nested, this likelihood-ratio test is not the appropriate comparison.
Another mistake is to treat a non-significant result as proof that the restricted model is correct. A failure to reject means only that the observed likelihood improvement is not sufficiently large under the chosen test and significance level. It does not assign a probability to the null hypothesis. Likewise, a statistically significant result is evidence against the restrictions, not proof that the unrestricted model is substantively adequate.
In a report, state the two fitted specifications, the restrictions, how many were imposed, the likelihood formulation, the statistic and reference distribution, and the p-value. Note any questionable regularity conditions. This makes clear both what the test assessed and what limits its conclusion.
Check Your Understanding
Use the definitions and conditions in this tutorial to answer the following questions.
- An unrestricted model has maximized log likelihood \(-210.4\), and its nested restricted model has maximized log likelihood \(-212.1\). Calculate the likelihood-ratio statistic.
- If the null imposes three independent, regular restrictions, what is the asymptotic reference distribution for the likelihood-ratio statistic?
- Why must the restricted and unrestricted likelihoods be computed on a comparable basis?
- What feature of a null parameter at the boundary makes ordinary chi-square calibration questionable?
- In one sentence, explain why likelihood-ratio and Wald test results can differ in a finite sample.