Exact Gaussian Likelihood via the Innovations Algorithm

Why the Initial Observation Matters

For a stationary ARMA model, the observed series is not generated by starting the process at the first recorded value. Its first observation is part of a process that has been running over time, and therefore has a stationary marginal distribution. An exact likelihood for a finite sample must include that distribution as well as the dependence among later observations.

The innovations algorithm handles this by building the joint distribution one observation at a time. At each step it computes the one-step prediction error and its variance using the model’s autocovariances. For Gaussian data, these quantities determine the likelihood exactly, without discarding the initial observation.

Model and Exact Gaussian Likelihood

Let \(Y_1,\ldots,Y_n\) be consecutive observations from a stationary, causal Gaussian ARMA process with unconditional mean \(\mu\). Write \(X_t=Y_t-\mu\), and use the model

where \(B\) is the lag operator, \(\phi(B)=1-\phi_1B-\cdots-\phi_pB^p\), and \(\theta(B)=1+\theta_1B+\cdots+\theta_qB^q\). The Gaussian innovations \(\varepsilon_t\) are independent over time. The AR parameters satisfy the stationarity condition, so the process has a stationary distribution; the usual invertibility restriction gives a standard identifiable ARMA representation. Denote the autocovariance at lag \(h\) by \(\gamma(h)=\operatorname{Cov}(X_t,X_{t-h})\), with \(\gamma(-h)=\gamma(h)\).

For a candidate parameter vector, form the centered observations \(x_t=y_t-\mu\). Define the one-step innovation \(e_t\) as the observation minus its linear prediction from the preceding observations, and let \(F_t\) be its variance. For a Gaussian process, the linear predictor is also the conditional mean. The likelihood is

The likelihood depends on the ARMA parameters through the mean, the autocovariances, and hence the recursively computed \(e_t\) and \(F_t\). The first innovation is simply \(e_1=x_1\), and its variance is \(F_1=\gamma(0)\), not a variance obtained by assuming that the process began at time 1.

The Innovations Recursion

The recursion uses only the autocovariances implied by the candidate model. For each observation \(t\), let \(c_{t,j}\) be the coefficient on the earlier innovation \(e_j\) in the linear prediction of \(X_t\). Also define \(K_{t,j}=\operatorname{Cov}(X_t,e_j)\). Initialize \(F_1=\gamma(0)\) and \(e_1=x_1\). For \(t=2,\ldots,n\), calculate in order:

The sum in the formula for \(K_{t,j}\) is empty when \(j=1\), so \(K_{t,1}=\gamma(t-1)\). Each \(K_{t,j}\) is computed using values with smaller second index, and each \(c_{t,j}\) uses an already computed innovation variance \(F_j\). The recursion therefore specifies a sequential calculation.

Why the Recursion Gives the Exact Likelihood

The argument is a sequence of projections. Suppose \(e_1,\ldots,e_{t-1}\) have been constructed as mutually uncorrelated prediction errors. The best linear prediction of \(X_t\) from them is \(\sum_{j<t}c_{t,j}e_j\). Since the innovations are uncorrelated, the coefficient on \(e_j\) is its covariance with \(X_t\) divided by its variance: \(c_{t,j}=\operatorname{Cov}(X_t,e_j)/F_j\). The covariance recursion follows directly from \(e_j=X_j-\sum_{k<j}c_{j,k}e_k\):

Subtracting the projection from \(X_t\) gives \(e_t\). Its covariance with any earlier innovation \(e_j\) is zero, because the coefficient \(c_{t,j}\) removes exactly the component along \(e_j\). Orthogonality and the variance formula for a projection then give

These projection statements establish the recursion for second moments. Gaussianity supplies the further fact needed for a likelihood: a Gaussian prediction error uncorrelated with the preceding observations is independent of them. Thus, conditional on \(X_1,\ldots,X_{t-1}\), \(X_t\) is normal with mean \(\sum_{j<t}c_{t,j}e_j\) and variance \(F_t\). Multiplying these conditional densities, starting with \(X_1\sim N(0,\gamma(0))\), gives the stated joint density. This is why the resulting likelihood is exact for the finite sample, rather than an approximation based on fixed initial values.

Using and Interpreting the Result

For each admissible candidate parameter vector, the model supplies \(\mu\) and \(\gamma(h)\). The recursion then supplies a set of one-step errors and variances, and the log-likelihood formula gives a numerical score for that candidate. Likelihood maximization selects parameter values with comparatively high density for the observed series; it does not imply that the fitted model is true or that a coefficient is causal.

This calculation also clarifies the meaning of “exact.” It refers to evaluating the finite-sample Gaussian likelihood using the stationary model distribution, including all initial-observation effects. It does not mean that the parameters are known without uncertainty, nor that the likelihood calculation is free of numerical issues. In a nondegenerate stationary model, each \(F_t\) is positive. If a computed variance is nonpositive, check the parameter constraints, autocovariances, and recursion rather than taking its logarithm.

Common Mistakes

• Using the innovation variance for the first observation. The first observation has variance \(\gamma(0)\), which generally differs from \(\sigma_\varepsilon^2\). Use the stationary autocovariance implied by the model.
• Ignoring the conditioning structure. The \(e_t\) values are prediction errors, not generally the raw centered observations. Compute each using the preceding innovations and their coefficients.
• Treating uncorrelatedness as independence without an assumption. The recursion constructs uncorrelated innovations from covariance information. Their independence, needed for the Gaussian density factorization, follows here from joint Gaussianity.
• Forgetting to center by the candidate mean. Use \(x_t=y_t-\mu\) throughout. A different candidate mean changes the errors and therefore the likelihood.
• Assuming the recursion eliminates the need for model assumptions. The autocovariances must correspond to a valid stationary model, and a Gaussian likelihood requires Gaussianity.