Yule-Walker Equations For Higher-Order AR Models

Why Yule-Walker Equations Matter

An autoregressive model describes a time-series value through earlier values of the series and a random innovation. The Yule-Walker equations connect the model’s autoregressive coefficients to the series’ autocovariances. They let us solve for coefficients from theoretical autocovariances, or estimate coefficients by substituting sample autocovariances. The equations are especially useful for seeing how the same autocovariance sequence enters several equations with shifted lags.

This tutorial focuses on constructing and solving that system for AR(p) models. The model must be weakly stationary for the population autocovariances used here to depend only on the lag. The treatment is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: A weakly stationary AR(p) process with mean $\mu$ satisfies $$ X_t-\mu=\phi_1(X_{t-1}-\mu)+\cdots+\phi_p(X_{t-p}-\mu)+\varepsilon_t, $$ where the innovations $\varepsilon_t$ have mean 0 and variance $\sigma_\varepsilon^2$, and are uncorrelated with the process values at earlier times. Weak stationarity means that $E[X_t]=\mu$ is constant, the variance is finite and constant, and the autocovariance depends only on the lag. Write $\gamma_k=\operatorname{Cov}(X_t,X_{t-k})$; stationarity gives $\gamma_{-k}=\gamma_k$.

Constructing the Yule-Walker System

To obtain an equation for lag $k$, multiply the centered AR(p) equation by $X_{t-k}-\mu$ and take expectations. For $k\geq 1$, the innovation term has zero covariance with $X_{t-k}-\mu$, because that value is from an earlier time. Each autoregressive term contributes its autocovariance at the difference between its time index and $t-k$. Consequently,

\gamma_k=\sum_{j=1}^{p}\phi_j\gamma_{k-j},\qquad k\geq 1,

with negative subscripts replaced using $\gamma_{-h}=\gamma_h$. In particular, the first $p$ equations form a linear system. For $k=1,\ldots,p$, the coefficient of $\phi_j$ is $\gamma_{k-j}$. This creates a symmetric Toeplitz matrix: entries are constant along each diagonal.

\begin{pmatrix} \gamma_0 & \gamma_1 & \cdots & \gamma_{p-1}\\ \gamma_1 & \gamma_0 & \cdots & \gamma_{p-2}\\ \vdots & \vdots & \ddots & \vdots\\ \gamma_{p-1} & \gamma_{p-2} & \cdots & \gamma_0 \end{pmatrix} \begin{pmatrix} \phi_1\\ \phi_2\\ \vdots\\ \phi_p \end{pmatrix} = \begin{pmatrix} \gamma_1\\ \gamma_2\\ \vdots\\ \gamma_p \end{pmatrix}.

If the autocovariance matrix is nonsingular, the system has a unique solution. The Yule-Walker equations at positive lags determine the autoregressive coefficients; a separate equation at lag zero gives the innovation variance:

\gamma_0=\sum_{j=1}^{p}\phi_j\gamma_j+\sigma_\varepsilon^2, \qquad \sigma_\varepsilon^2=\gamma_0-\sum_{j=1}^{p}\phi_j\gamma_j.

This variance formula follows by multiplying the centered AR equation by $X_t-\mu$ and taking expectations. The left side is $\gamma_0$. The $j$th autoregressive term contributes $\phi_j\gamma_j$, while the innovation contributes $\operatorname{Cov}(\varepsilon_t,X_t-\mu)=\sigma_\varepsilon^2$. Here $X_t-\mu$ includes the current innovation and a part determined by earlier values, which are uncorrelated with that innovation. The formula does not apply if the assumed innovation is correlated with the past or if the process is not stationary.

Worked AR(2) System

For AR(2), the first two positive-lag equations have a particularly useful form. The first uses $\gamma_{-1}=\gamma_1$; the second uses $\gamma_0$:

\gamma_1=\phi_1\gamma_0+\phi_2\gamma_1,\qquad \gamma_2=\phi_1\gamma_1+\phi_2\gamma_0.

Worked Example: Solve and Check an AR(2) Model

A hypothetical stationary series has autocovariances $\gamma_0=12$, $\gamma_1=8$, and $\gamma_2=7$. Use the Yule-Walker equations to find $\phi_1$ and $\phi_2$, calculate the innovation variance, and check AR(2) stationarity.

Method. Use the first two positive-lag equations and then check the resulting coefficients against the AR(2) stationarity conditions. The equations are

\begin{pmatrix}12&8\\8&12\end{pmatrix} \begin{pmatrix}\phi_1\\\phi_2\end{pmatrix} = \begin{pmatrix}8\\7\end{pmatrix}.

The determinant is $12(12)-8(8)=144-64=80$, so the system has a unique solution. Solving,

\phi_1=\frac{8(12)-8(7)}{80}=\frac{40}{80}=0.5,\qquad \phi_2=\frac{12(7)-8(8)}{80}=\frac{20}{80}=0.25.

The innovation variance is

\sigma_\varepsilon^2 =12-(0.5)(8)-(0.25)(7) =12-4-1.75 =6.25.

For an AR(2), stationarity requires $|\phi_2|<1$, $\phi_1+\phi_2<1$, and $\phi_2-\phi_1<1$. Here, $|0.25|<1$, $0.5+0.25=0.75<1$, and $0.25-0.5=-0.25<1$, so the model is stationary. As a check on the lag equations, $0.5(12)+0.25(8)=8=\gamma_1$ and $0.5(8)+0.25(12)=7=\gamma_2$. The calculated innovation variance is positive as required.

Common mistake avoided: the lag-two equation uses $\gamma_1$ with $\phi_1$ and $\gamma_0$ with $\phi_2$; it does not repeat the lag-one row.

A Higher-Order System

The same rule constructs an AR(p) system for any order. In an AR(3), for example, the three equations are

\begin{aligned} \gamma_1&=\phi_1\gamma_0+\phi_2\gamma_1+\phi_3\gamma_2,\\ \gamma_2&=\phi_1\gamma_1+\phi_2\gamma_0+\phi_3\gamma_1,\\ \gamma_3&=\phi_1\gamma_2+\phi_2\gamma_1+\phi_3\gamma_0. \end{aligned}

Each row corresponds to a lag $k$, and each coefficient of $\phi_j$ is $\gamma_{k-j}$. A proposed covariance sequence should be compatible with a valid stationary process. When equations are formed from sample autocovariances instead, the solution is an estimate; sampling variability means it need not reproduce every property of the underlying population model.

Worked Example: Solve a Candidate AR(3) System

A modeler fits a candidate AR(3) to a hypothetical centered series and obtains autocovariances $\gamma_0=10$, $\gamma_1=3$, $\gamma_2=0.9$, and $\gamma_3=0.27$. Solve the Yule-Walker system, find the innovation variance, and interpret the result.

Method. Set up all three equations. The system is

\begin{pmatrix} 10&3&0.9\\ 3&10&3\\ 0.9&3&10 \end{pmatrix} \begin{pmatrix}\phi_1\\\phi_2\\\phi_3\end{pmatrix} = \begin{pmatrix}3\\0.9\\0.27\end{pmatrix}.

The coefficient matrix has determinant $828.1$, which is nonzero, so the solution is unique. Check the candidate $(\phi_1,\phi_2,\phi_3)=(0.3,0,0)$ row by row: $10(0.3)+3(0)+0.9(0)=3$; $3(0.3)+10(0)+3(0)=0.9$; and $0.9(0.3)+3(0)+10(0)=0.27$. Thus it solves the system.

The innovation variance is $10-(0.3)(3)-(0)(0.9)-(0)(0.27)=10-0.9=9.1$. The candidate third-order model has zero coefficients for its second and third lags, so it reduces to an AR(1) with coefficient $0.3$. Its nonzero autoregressive root is $1/0.3=10/3$, outside the unit circle, so the reduced model is stationary.

Common mistake avoided: a fitted AR(3) form does not imply that all three estimated coefficients must be nonzero. The system can reveal that the higher-order terms contribute nothing.

Mean, Intercept, and Stationarity

The centered model is often convenient for Yule-Walker calculations, but data may be written with an intercept:

X_t=c+\phi_1X_{t-1}+\cdots+\phi_pX_{t-p}+\varepsilon_t.

Taking expectations under stationarity gives $\mu=c+(\phi_1+\cdots+\phi_p)\mu$, and therefore $c=\mu(1-\sum_{j=1}^{p}\phi_j)$. This relationship matters because the intercept is not generally the process mean. Yule-Walker equations for the autoregressive coefficients use centered autocovariances, not raw second moments that include the squared mean.

Worked Example: Convert a Centered AR(2) to Intercept Form

A hypothetical stationary process has mean $\mu=20$, with centered-model coefficients $\phi_1=0.5$ and $\phi_2=0.25$. Find the intercept $c$, state the intercept-form model, and verify its unconditional mean.

The coefficient sum is $0.5+0.25=0.75$. Therefore $c=20(1-0.75)=5$. The intercept form is $X_t=5+0.5X_{t-1}+0.25X_{t-2}+\varepsilon_t$.

Taking expectations gives $E[X_t]=5+0.5(20)+0.25(20)=5+10+5=20$, as specified. The coefficient stationarity checks are $|0.25|<1$, $0.5+0.25=0.75<1$, and $0.25-0.5=-0.25<1$. The intercept changes the mean level; it does not change these AR(2) stationarity checks.

Common mistake avoided: setting $c=\mu$ would give $c=20$, which does not satisfy the model’s mean equation.

Common Indexing Trap

The lag index is the most common source of errors in constructing this system. For a general lag $k$, the $j$th term is $\phi_j\gamma_{k-j}$. In an AR(2), the correct lag-two equation is $\gamma_2=\phi_1\gamma_1+\phi_2\gamma_0$. A tempting but incorrect equation is $\gamma_2=\phi_1\gamma_0+\phi_2\gamma_1$, which simply repeats the lag-one pattern. In the worked AR(2) example, substituting $(0.5,0.25)$ into this incorrect expression gives $0.5(12)+0.25(8)=8$, not the specified $\gamma_2=7$. The correct equation gives $0.5(8)+0.25(12)=7$.

Also distinguish population equations from estimation. With theoretical autocovariances, the equations describe the stationary process. With sample autocovariances, solving the same system produces Yule-Walker estimates, not exact population coefficients. Check that the system is nonsingular and assess whether the resulting model is stationary before interpreting it as a stationary AR fit.

Key takeaway: For each positive lag $k$, write $\gamma_k=\sum_{j=1}^{p}\phi_j\gamma_{k-j}$, using symmetry for negative lags. Solve the resulting Toeplitz system, calculate $\sigma_\varepsilon^2=\gamma_0-\sum_{j=1}^{p}\phi_j\gamma_j$, and separately check stationarity.

Check Your Understanding

For each problem, construct the indicated Yule-Walker calculation and check the requested property.

A hypothetical AR(2) series has $\gamma_0=5$, $\gamma_1=2$, and $\gamma_2=1$. Find both coefficients, check AR(2) stationarity, and calculate the innovation variance. (Answer: $\phi_1=8/21,\ \phi_2=1/21$; stationary; $\sigma_\varepsilon^2=88/21$.)
For an AR(2) candidate, $\gamma_0=9$, $\gamma_1=3$, and $\gamma_2=0$. Solve for the coefficients and innovation variance, then check stationarity. (Answer: $\phi_1=3/8,\ \phi_2=-1/8$; stationary; $\sigma_\varepsilon^2=63/8$.)
A candidate AR(3) has $\gamma_0=4$, $\gamma_1=1$, $\gamma_2=0.25$, and $\gamma_3=0.0625$. Solve the system and calculate the innovation variance. (Answer: $\phi_1=0.25,\ \phi_2=0,\ \phi_3=0$; $\sigma_\varepsilon^2=3.75$.)
A stationary AR(2) has mean $\mu=15$ and coefficients $\phi_1=0.4,\ \phi_2=0.2$. Find its intercept in the uncentered model and verify the mean equation. (Answer: $c=6$; $6+0.4(15)+0.2(15)=15$.)
An AR(2) candidate has $\gamma_0=1,\ \gamma_1=0.9,\ \gamma_2=0.95$. Solve for its coefficients, check stationarity, and calculate the innovation variance. (Answer: $\phi_1=9/38,\ \phi_2=14/19$; stationary; $\sigma_\varepsilon^2=33/380$.)

What You'll Learn