Moment Estimation for ARMA Models

Why Estimate ARMA Parameters From Moments?

An ARMA model connects a series’ autocovariances to a small set of autoregressive and moving-average parameters. Moment estimation uses that connection directly: calculate sample autocovariances, then choose parameter values whose theoretical autocovariances match them. It is often a transparent way to see what the dependence pattern can—and cannot—tell us about model parameters.

The equations differ by model. For a pure autoregression, the Yule–Walker equations are linear in the autoregressive coefficients. For a moving-average or mixed model, the corresponding equations are generally nonlinear, and a given set of moments may not identify one admissible parameter vector. In all cases, estimates are based on sample moments and therefore vary from sample to sample.

Model and Sample Moments

Write the centered ARMA(\(p,q\)) model as

Here \(\mu=E(X_t)\), and the innovations \(\varepsilon_t\) have mean zero, variance \(\sigma^2\), and zero covariance at distinct times. Assume the process is covariance-stationary and the ARMA representation is causal; for invertible MA parameterization, impose the usual invertibility root condition. These conditions ensure the theoretical autocovariances are well defined. Gaussian innovations are not required for the moment equations below.

Let \(\gamma(k)=\operatorname{Cov}(X_t,X_{t-k})\). For a sample \(X_1,\ldots,X_n\), define the sample mean \(\bar{x}=n^{-1}\sum_{t=1}^n X_t\), and use the convention

The denominator is \(n\) at every lag; other conventions exist, so stating the convention matters when reproducing a calculation. Moment estimation substitutes \(\hat{\gamma}(k)\) for \(\gamma(k)\) in the model’s theoretical equations. It does not make the sample moments equal to their population counterparts.

Yule–Walker Estimation for an AR Model

For a zero-mean AR(\(p\)) process, \(X_t=\sum_{j=1}^p\phi_jX_{t-j}+\varepsilon_t\), covariance with \(X_{t-k}\) gives, for \(k\geq1\),

Use \(\gamma(-k)=\gamma(k)\) to write the first \(p\) equations as \(\boldsymbol{\Gamma}_p\boldsymbol{\phi}=\boldsymbol{\gamma}_p\), where \(\boldsymbol{\Gamma}_p\) has entries \(\gamma(|i-j|)\), \(\boldsymbol{\phi}=(\phi_1,\ldots,\phi_p)^\mathsf{T}\), and \(\boldsymbol{\gamma}_p=(\gamma(1),\ldots,\gamma(p))^\mathsf{T}\). Replacing theoretical autocovariances by sample autocovariances yields the Yule–Walker estimator, provided the sample matrix is invertible:

The innovation variance follows from the lag-zero equation, \(\gamma(0)=\sum_{j=1}^p\phi_j\gamma(j)+\sigma^2\). Its moment estimate is therefore \(\hat{\sigma}^2=\hat{\gamma}(0)-\sum_{j=1}^p\hat{\phi}_j\hat{\gamma}(j)\). These equations are specific to a pure AR(\(p\)); applying them as if they were the exact equations for an ARMA model generally does not estimate its parameters correctly.

Moving-Average Moments and Identification

For an MA(1), \(X_t-\mu=\varepsilon_t+\theta\varepsilon_{t-1}\). Directly taking covariances, using the zero covariance of distinct innovations, gives

The ratio \(r=\gamma(1)/\gamma(0)\) removes the scale parameter and satisfies \(r=\theta/(1+\theta^2)\). This equation can have two solutions: if \(\theta\neq0\), then \(\theta\) and \(1/\theta\) give the same ratio. The standard invertible MA(1) representation imposes \(|\theta|<1\), selecting at most one of these roots. A real solution also requires \(|r|\leq 1/2\), since \(1+\theta^2\geq2|\theta|\).

With sample moments, set \(\hat{r}=\hat{\gamma}(1)/\hat{\gamma}(0)\) and solve the resulting quadratic. If an invertible real root exists, estimate the innovation variance by \(\hat{\sigma}^2=\hat{\gamma}(0)/(1+\hat{\theta}^2)\). If \(|\hat{r}|>1/2\), the sample moments do not satisfy the exact MA(1) moment restriction, so there is no real exact solution. This can happen from sampling variation even when an MA(1) is a reasonable population model; it is not by itself a proof that no such model could have generated the data.

Moment Equations for a Mixed ARMA(1,1)

For an ARMA(1,1), let \(Y_t=X_t-\mu\), so \(Y_t=\phi Y_{t-1}+\varepsilon_t+\theta\varepsilon_{t-1}\). Under causality, \(|\phi|<1\). Its theoretical moments are

Thus, when \(\gamma(1)\neq0\), \(\gamma(2)/\gamma(1)=\phi\). One possible moment procedure first estimates \(\phi\) by \(\hat{\gamma}(2)/\hat{\gamma}(1)\), then solves the \(\gamma(1)/\gamma(0)\) equation for \(\theta\). This ratio approach is sensitive when \(\hat{\gamma}(1)\) is close to zero. More generally, one can choose parameters to minimize a weighted sum of squared discrepancies between several sample autocovariances and their theoretical values. Such a fitting rule must be stated: matching more sample moments than there are parameters will not usually produce exact equality.

Identification and Sampling Variability

A parameter is identified by a set of moments when those moments determine it uniquely within the stated parameter space. In the MA(1) example, the moment ratio alone cannot distinguish \(\theta\) from \(1/\theta\); invertibility supplies a convention that selects one representation. In mixed models, identification can also become weak or fail. For example, if an AR and MA factor cancel, different parameter descriptions can represent the same process. Ratios used in estimation may be unstable when their denominators are near zero.

Even when the population equations uniquely determine the parameters locally, estimates use random sample autocovariances. Under suitable weak-dependence conditions, sample autocovariances are consistent, and their sampling errors shrink as the sample grows. A moment estimator inherits consistency when the population moment equations have a unique, sufficiently regular solution and the sample equations are solved appropriately. Near multiple roots, boundaries, or weakly identified configurations, small changes in sample moments can cause large changes in estimated parameters. Therefore, an apparently precise numerical solution to the sample equations does not by itself establish reliable estimation.

Common Mistakes

• Treating AR equations as valid for every ARMA model. The pure AR Yule–Walker equations do not generally hold in that form when moving-average terms are present. Use the theoretical autocovariance equations for the specified ARMA model.
• Ignoring invertibility when solving an MA moment equation. Reciprocal roots can yield the same moments. State the invertibility restriction and select the admissible root.
• Forcing an exact MA(1) solution when \(|\hat{r}|>1/2\). The sample ratio then violates the population equation’s feasible range. Recognize that the exact sample equations have no real solution, or specify an approximate fitting procedure.
• Interpreting estimates as exact population quantities. Sample autocovariances fluctuate. Report moment estimates as estimates, and consider whether the moments identify the parameters stably.