Standard Errors and Wald Inference for ARMA Parameters

Why ARMA Estimates Need Standard Errors

Maximum likelihood produces estimates of an ARMA model’s parameters, but an estimate alone does not show how precisely the sample has determined them. Standard errors describe the estimated sampling variability; they support confidence intervals and tests of parameter restrictions. The calculations rely on a large-sample approximation, so the model and the conditions behind that approximation matter.

Write the parameter vector as \(\eta\), the log-likelihood based on \(n\) observations as \(\ell_n(\eta)\), and its maximizer as \(\hat{\eta}\). The vector may include autoregressive coefficients, moving-average coefficients, a mean or intercept, and the innovation variance. The likelihood may be exact Gaussian likelihood or a conditional likelihood, as appropriate to the fitted model. In either case, inference is based on the likelihood actually optimized.

The Hessian measures local curvature: a sharply peaked likelihood indicates relatively precise estimates, while a flatter likelihood indicates greater uncertainty. Because the likelihood sums information over observations, its curvature typically increases with sample size, and the resulting covariance typically decreases at rate \(1/n\). Numerical optimization software often returns the Hessian, its inverse, or a covariance estimate derived from it.

Conditions for Wald Inference

Ordinary likelihood-based Wald inference uses an asymptotic normal approximation. In an ARMA setting, its standard justification requires a correctly specified and identifiable model, suitable differentiability and moment conditions, a stable stationary and invertible parameter value in the interior of the admissible parameter region, and a nonsingular information matrix. The fitted solution must also be a genuine local maximum rather than a saddle point or a boundary solution. Under these conditions, the estimate is approximately multivariate normal around the true parameter, with covariance estimated by the inverse information matrix.

The notation \(\dot{\sim}\) indicates an approximate large-sample distribution, not an exact finite-sample result. In particular, the standard errors do not account for uncertainty from choosing the ARMA orders or selecting a model after examining the data. They describe uncertainty conditional on the fitted model and its assumptions.

Sandwich Covariance for Score-Based Inference

The inverse Hessian is model-based: it relies on the likelihood curvature being an appropriate description of sampling variability. A sandwich covariance can instead combine curvature with the variability of the score. This can provide a more robust estimate when the likelihood’s variance assumptions are imperfect, provided the estimator still targets a well-defined parameter and the score covariance is estimated appropriately.

Let \(s_t(\eta)\) denote the contribution of observation \(t\) to the score, so the full score is \(S_n(\eta)=\sum_{t=1}^n s_t(\eta)\). Define the average negative Hessian \(A_n=-n^{-1}\partial^2\ell_n(\hat{\eta})/\partial\eta\partial\eta'\). Let \(B_n\) estimate the long-run covariance of the score contributions. The sandwich covariance estimate is

The middle matrix \(B_n\) must reflect dependence across score contributions when such dependence is present. For example, a long-run covariance estimate accounts for autocovariances of the score sequence; simply summing individual score outer products is not generally adequate for dependent scores. Under a correctly specified likelihood with suitable conditions, score contributions may have a martingale-difference property, in which case their cross-time covariances vanish and a simpler score outer-product estimate can be appropriate. The method used should match the dependence and likelihood assumptions.

This distinction is important in time series. “Robust” does not mean assumption-free: the sandwich estimate still needs a consistent estimate of the score’s long-run covariance and regularity conditions for the estimator. If the model is correctly specified, the model-based and sandwich estimates should be close in large samples under the information identity. Large discrepancies can signal misspecification or an unstable estimate, but they do not identify the cause by themselves.

Wald Tests and Confidence Intervals

For a scalar parameter \(\eta_j\), a two-sided Wald test of \(H_0:\eta_j=\eta_{j,0}\) uses

Equivalently, its square is approximately \(\chi^2_1\). For \(q\) linear restrictions \(R\eta=r\), where \(R\) is a known \(q\)-row matrix, use

These formulae apply whether the covariance matrix is model-based or a valid sandwich estimate. The chosen covariance estimate must be used consistently in both the test and the corresponding interval. For a scalar coefficient, the usual approximate \(100(1-\alpha)\%\) interval is \(\hat{\eta}_j\pm z_{1-\alpha/2}SE(\hat{\eta}_j)\), where \(z_{1-\alpha/2}\) is the standard normal quantile.

Functions of ARMA Parameters: The Delta Method

Inference often concerns a function of several estimated coefficients rather than a single coefficient. If \(g(\eta)\) is differentiable near the true parameter and the estimate is asymptotically normal, its approximate variance is obtained by evaluating the gradient at \(\hat{\eta}\):

The square root is the delta-method standard error. A normal confidence interval then uses \(g(\hat{\eta})\pm z_{1-\alpha/2}SE\{g(\hat{\eta})\}\). This approximation is most useful when the function is smooth near the estimate and the estimate is not close to a point where the function becomes unstable.

For example, in the ARMA(1,1) equation \(Y_t=c+\phi Y_{t-1}+\varepsilon_t+\psi\varepsilon_{t-1}\), with a stationary autoregressive coefficient, the long-run mean is \(\mu=c/(1-\phi)\). Its estimate depends on both \(c\) and \(\phi\), so its uncertainty must include their covariance.

Derivation: Why Inverse Curvature Gives a Covariance

The key approximation follows by expanding the score around the true parameter \(\eta_0\). At an interior maximum the score is zero, so a first-order Taylor expansion gives \(0\approx S_n(\eta_0)+\partial S_n(\eta_0)/\partial\eta'(\hat{\eta}-\eta_0)\). Rearranging yields

where \(A\) is the probability limit of the average negative Hessian. If the normalized score has limiting covariance \(B\), the covariance of this approximation is \(A^{-1}BA^{-1}/n\) on the scale of \(\hat{\eta}\). Under a correctly specified regular likelihood, the information identity gives \(A=B\), reducing the expression to \(A^{-1}/n\), the large-sample counterpart of the inverse observed information. This derivation explains both the Hessian formula and the sandwich formula, and why the sandwich middle matrix must represent score variability.

Common Pitfalls and Practical Limits

• Ignoring covariance in a function or restriction. Variances of coefficient combinations include cross-covariance terms. Use the full estimated covariance matrix, not only its diagonal.
• Calling a Wald interval exact. It is an asymptotic approximation. Small samples, weak identification, or highly skewed likelihoods can make its coverage inaccurate.
• Using ordinary inference at a boundary. Near-unit roots or parameters close to the stationarity or invertibility boundary can produce nonstandard behavior and unreliable normal approximations. A positive-definite Hessian alone does not resolve this concern.
• Treating a sandwich estimate as automatically robust. Its score covariance must account for relevant time dependence. A misspecified or inconsistent estimate of that covariance does not become valid merely by using the sandwich formula.
• Interpreting a nonsignificant test as proof of the null. Failure to reject a restriction means the data did not provide sufficiently strong evidence against it under the test; it does not establish that the restriction is true.
• Forgetting model-selection uncertainty. Standard errors computed after choosing the ARMA order generally condition on that chosen order and do not include the uncertainty from the selection process.