Conditional Likelihood and Initial-Value Approximations

Why Condition on Initial Values?

An ARMA model links each observation to past observations and past innovations. At the beginning of a finite sample, some of those past values are unavailable. Exact Gaussian likelihood accounts for their distribution; conditional likelihood instead fixes or approximates them so the likelihood can be computed using a residual recursion. This can be substantially quicker, but it changes what probability model is being evaluated, especially for short samples.

Use the centered ARMA model

\phi(B)(y_t-\mu)=\theta(B)\varepsilon_t,\qquad \phi(B)=1-\phi_1B-\cdots-\phi_pB^p,\quad \theta(B)=1+\theta_1B+\cdots+\theta_qB^q,

where \(B y_t=y_{t-1}\), \(\mu\) is the process mean, and the innovations are independent \(N(0,\sigma^2)\). Assume the model is causal and invertible when discussing its stationary Gaussian distribution. Write \(x_t=y_t-\mu\). The signs in the residual recursion below follow this lag-polynomial convention.

Definition: A conditional Gaussian likelihood evaluates the density of the observed portion of a series after treating specified presample values as fixed, rather than integrating over their joint distribution with the observations. For an ARMA model, these values may include presample observations and presample innovations. Their treatment is called the initialization.

Residual Recursion and Conditional Likelihood

For given parameter values, define the innovation residual \(e_t\) recursively by rearranging the ARMA equation:

e_t=x_t-\sum_{i=1}^{p}\phi_i x_{t-i} -\sum_{j=1}^{q}\theta_j e_{t-j}.

If the necessary past \(x\)'s and \(e\)'s are available, this gives the model's one-step innovation for each \(t\). In a finite sample, the recursion cannot begin without an initialization. One common conditional construction fixes the presample observations used by the autoregressive part and sets unavailable presample residuals to zero. Another uses approximate presample residuals, for example estimates obtained by a backcasting procedure. These choices produce different residual sequences and therefore different likelihood values.

Suppose the recursion is evaluated for \(t=r+1,\ldots,n\), where the first \(r\) observations are treated as conditioning values. Conditional on those values and the chosen initial residuals, the likelihood contribution of each remaining observation is Gaussian with mean given by the ARMA recursion and variance \(\sigma^2\). Thus the conditional log likelihood is

\ell_c(\mu,\phi_1,\ldots,\phi_p,\theta_1,\ldots,\theta_q,\sigma^2) =-\frac{n-r}{2}\log(2\pi\sigma^2) -\frac{1}{2\sigma^2}\sum_{t=r+1}^{n}e_t^2.

For a pure autoregression, conditioning on the first \(p\) observed values gives a natural choice \(r=p\). For a model with moving-average terms, the presample innovations are not observed and must also be fixed or approximated. Some implementations instead evaluate a conditional density from the first observation onward after imposing an initialization for every unavailable value. The precise convention matters: it determines which observations contribute and how initial values enter the recursion.

When \(\sigma^2\) is estimated by maximizing this conditional likelihood for fixed dynamic parameters, its maximizing value is the conditional residual sum of squares divided by the number of included observations:

\widehat{\sigma}^2_c=\frac{1}{n-r}\sum_{t=r+1}^{n}e_t^2.

This is a likelihood maximizer, not an unbiased degrees-of-freedom correction. The formula follows by differentiating the log likelihood with respect to \(\sigma^2\) and setting the derivative to zero.

Worked Example: Conditional Likelihood for an AR(1)

Suppose hypothetical observations are \(y_1=2\) and \(y_2=1.5\), and consider the zero-mean AR(1) model \(y_t=\phi y_{t-1}+\varepsilon_t\), with \(\phi=0.5\) and \(\sigma^2=1\). Condition on \(y_1=2\); the only likelihood contribution is from \(y_2\).

The conditional residual is

e_2=y_2-\phi y_1=1.5-(0.5)(2)=0.5.

Therefore,

\ell_c=-\frac{1}{2}\log(2\pi)-\frac{1}{2}(0.5)^2 =-0.918939-0.125=-1.04394.

This value is the log density of \(y_2\) conditional on \(y_1\), under the specified parameters. It is not the log density of the two observations jointly because the probability distribution of \(y_1\) has been left out.

Initial-Value Approximations in an MA Model

For an MA model there may be no observed lagged \(x\)'s to condition on, but the residual recursion still needs presample innovations. Fixing those innovations at zero is simple and deterministic. It is an approximation because, under a stationary model, presample innovations are random rather than known to be zero. A different initialization, such as backcasting, attempts to approximate their values from the available series.

Worked Example: Zero Initialization for an MA(1)

Consider the hypothetical zero-mean model \(x_t=\varepsilon_t+0.5\varepsilon_{t-1}\), with \(\sigma^2=1\), and observations \(x_1=1\), \(x_2=1.5\), and \(x_3=0.25\). Use the conditional convention \(e_0=0\) and include all three observations.

The residual recursion is \(e_t=x_t-0.5e_{t-1}\). It gives

e_1=1,\qquad e_2=1.5-(0.5)(1)=1,\qquad e_3=0.25-(0.5)(1)=-0.25.

The residual sum of squares is \(1^2+1^2+(-0.25)^2=2.0625\). Consequently,

\ell_c=-\frac{3}{2}\log(2\pi)-\frac{1}{2}(2.0625) \approx-2.75682-1.03125=-3.78807.

This calculation is conditional on the chosen value \(e_0=0\). It does not assert that the actual presample innovation was zero.

Worked Example: Sensitivity to the Initial Innovation

Keep the MA(1) model and observations from the previous calculation, but initialize with \(e_0=0.4\) instead of zero. This value is hypothetical and is used only to show how initialization changes the conditional calculation.

The recursion now gives

e_1=1-(0.5)(0.4)=0.8,\qquad e_2=1.5-(0.5)(0.8)=1.1,\qquad e_3=0.25-(0.5)(1.1)=-0.3.

The new residual sum of squares is \(0.8^2+1.1^2+(-0.3)^2=0.64+1.21+0.09=1.94\). With three Gaussian contributions and \(\sigma^2=1\),

\ell_c=-\frac{3}{2}\log(2\pi)-\frac{1}{2}(1.94) \approx-2.75682-0.97000=-3.72682.

Changing the fixed initial innovation changes the log likelihood by about \(0.0613\) for these data and this parameter setting. In a fitted model, initialization can also change which dynamic parameters maximize the likelihood.

Conditional and Exact Likelihoods

For a stationary Gaussian ARMA process, exact likelihood uses the joint distribution of the observed vector \(Y=(y_1,\ldots,y_n)^\mathsf{T}\). If its mean vector is \(m\) and covariance matrix is \(\Sigma\), its log likelihood is

\ell_{\mathrm{exact}} =-\frac{n}{2}\log(2\pi)-\frac{1}{2}\log|\Sigma| -\frac{1}{2}(Y-m)^\mathsf{T}\Sigma^{-1}(Y-m).

This expression accounts for the distribution of the initial observations and their dependence on later observations. The innovations algorithm computes an equivalent Gaussian likelihood by sequentially finding conditional means and variances, rather than directly inverting \(\Sigma\). Conditional likelihood takes a different route: it fixes or approximates the presample values and uses a simpler residual recursion. Its likelihood is generally not equal to the exact likelihood.

Worked Example: Exact Versus Conditional Likelihood for an AR(1)

Use again the hypothetical observations \(y_1=2\), \(y_2=1.5\), with \(\phi=0.5\), \(\mu=0\), and \(\sigma^2=1\). Under the stationary AR(1) model, the variance of \(y_1\) is \(\sigma^2/(1-\phi^2)=4/3\). Given \(y_1\), the conditional mean of \(y_2\) is \(\phi y_1=1\), and its conditional variance is \(1\).

The exact log likelihood factors into the marginal density of \(y_1\) and the conditional density of \(y_2\) given \(y_1\):

\ell_{\mathrm{exact}} =-\frac{1}{2}\log\left(2\pi\frac{4}{3}\right) -\frac{1}{2}\frac{2^2}{4/3} -\frac{1}{2}\log(2\pi) -\frac{1}{2}(1.5-0.5(2))^2.

The four terms are approximately \(-1.06278\), \(-1.50000\), \(-0.918939\), and \(-0.12500\), giving \(\ell_{\mathrm{exact}}\approx -3.607\). The conditional log likelihood from the first worked example is \(-1.04394\). The difference is the contribution from the marginal distribution of \(y_1\); the two likelihood values refer to different probability calculations, so their raw magnitudes should not be compared as if they were computed from the same set of density contributions.

Computational Trade-Off and Finite-Sample Behavior

For fixed ARMA orders, conditional residuals can usually be computed with a number of operations proportional to \(n(p+q)\), which is linear in sample size when \(p\) and \(q\) are fixed. This makes conditional likelihood attractive for large samples and for repeated evaluations during parameter estimation. Exact likelihood also has efficient implementations: the innovations algorithm avoids direct matrix inversion, and state-space methods can exploit the model's fixed-dimensional structure. The practical cost depends on the algorithm and model representation; conditional likelihood is not the only computationally feasible option.

The finite-sample difference is most visible when the sample is short or the process is highly persistent. Initial values then make up a larger share of the information, and plausible initialization choices can produce noticeably different residuals. Exact likelihood explicitly models the initial observations under stationarity, while a conditional approximation discards their density or substitutes fixed values for unobserved presample quantities.

Under standard regularity conditions—such as a stationary, causal and invertible model of fixed order, identifiable parameters, and an initialization whose effect is controlled—the contribution of a fixed number of initial values becomes small relative to the full sample as \(n\) grows. Conditional and exact likelihood procedures can then have the same large-sample behavior. This does not make their finite-sample likelihoods identical, nor does it guarantee a negligible initialization effect in a short or highly persistent series.

Key takeaway: Conditional likelihood replaces integration over the presample distribution with a stated initialization. Its residual recursion is convenient, but the initialization and the set of included observations are part of the likelihood definition and should be reported.

Common Mistake

A common error is to set presample innovations to zero and then describe the resulting likelihood as the exact Gaussian likelihood. Zero initialization is a computational convention, not a claim about the stationary distribution of those innovations. Exact likelihood accounts for the joint distribution of the observed data, including the initial observations; conditional likelihood fixes or approximates presample quantities. Another mistake is to compare their raw log-likelihood values without checking whether they include the same observations and density terms. A numerical difference can arise simply because one likelihood conditions on the first observation while the other includes its marginal density.

Specify the model and convention.
State the ARMA sign convention, which observations are conditioning values, and how unobserved presample innovations are initialized.

Run the residual recursion.
Compute \(e_t\) from the observed data, dynamic parameters, and fixed initial values.

Evaluate the included density contributions.
Use the Gaussian conditional log likelihood for the stated set of observations, and do not label it exact unless the initial distribution has been accounted for.

Check Your Understanding

Try these questions before moving on.

For a conditional ARMA likelihood, what quantities may need to be fixed or approximated before the residual recursion can begin?
In the MA(1) model \(x_t=\varepsilon_t+\theta\varepsilon_{t-1}\), write the residual recursion and explain what setting \(e_0=0\) means.
Why does the conditional likelihood in the AR(1) example omit a density contribution for \(y_1\), while the exact likelihood includes one?
For fixed ARMA orders, why is the conditional residual recursion typically linear in the sample size?
Name one reason conditional and exact likelihoods can differ noticeably in a finite sample.