Deriving the Stationary AR(1) Covariance

From a Recursion to a Covariance Structure

In a GLMM, a time-indexed latent effect can represent dependence among observations made at successive occasions. An AR(1) specification describes that effect recursively: the current value carries forward part of the preceding value, while a new innovation adds variation. The recursion is useful for defining the process, but a covariance model is often the form needed to describe dependence across all observation times.

This tutorial derives that covariance model and makes explicit the role of initialization. The derivation concerns the latent process itself. It does not, by itself, give the covariance of the GLMM response, whose distribution also depends on the link and conditional response model.

Definition: A stationary AR(1) process $b_t$, indexed by equally spaced integer times $t$, satisfies $$ b_t-\mu=\phi(b_{t-1}-\mu)+\epsilon_t, $$ where $\mu$ is its constant mean, $\phi$ is the autoregressive coefficient, and the innovations $\epsilon_t$ have mean $0$, variance $\sigma_\epsilon^2$, and are independent across time and independent of the process history before time $t$. For the causal stationary process, $|\phi|<1$. Its variance is denoted $\sigma_b^2$, and its lag-$k$ covariance is $\gamma_k=\operatorname{Cov}(b_t,b_{t-k})$.

The innovation assumptions matter: they ensure that a new innovation has zero covariance with earlier process values. A finite innovation variance is needed for the covariance calculations below. For strict stationarity, a standard sufficient condition is that the innovations are identically distributed as well as independent, with finite variance. In the Gaussian case, the resulting process is Gaussian; Gaussian innovations are not required for the covariance formulas.

The process must also be initialized consistently with stationarity. One way is to define it over all integer times through its convergent infinite-past representation. For a process begun at a finite time, the initial value must have mean $\mu$, variance $\sigma_b^2$, and be independent of future innovations. Merely using the recursion does not guarantee stationarity.

Deriving the Stationary Variance and Covariance

Write $x_t=b_t-\mu$, so that $x_t=\phi x_{t-1}+\epsilon_t$ has mean zero under the stated assumptions. If the process is stationary with finite variance, then $\operatorname{Var}(x_t)=\operatorname{Var}(x_{t-1})=\sigma_b^2$. Since $\epsilon_t$ is independent of the past, it is uncorrelated with $x_{t-1}$. Taking variances gives

\sigma_b^2 =\operatorname{Var}(\phi x_{t-1}+\epsilon_t) =\phi^2\sigma_b^2+\sigma_\epsilon^2.

Rearranging yields the stationary variance. For lag $k\geq 1$, multiply the recursion for $x_t$ by $x_{t-k}$ and take expectations. The innovation $\epsilon_t$ is uncorrelated with $x_{t-k}$, because $x_{t-k}$ depends only on the process history up to time $t-k$, before the new innovation at time $t$. Thus the covariance at lag $k$ is $\phi$ times that at lag $k-1$.

\sigma_b^2=\frac{\sigma_\epsilon^2}{1-\phi^2}, \qquad \gamma_k=\phi\gamma_{k-1} \quad\Longrightarrow\quad \gamma_k=\phi^k\sigma_b^2 =\frac{\sigma_\epsilon^2\phi^k}{1-\phi^2}, \quad k\geq 0.

Covariance is symmetric in its two time arguments, so the result for any integer lag is $\gamma_k=\sigma_b^2\phi^{|k|}$. Dividing by the variance gives the lag-$k$ correlation, $\rho_k=\phi^{|k|}$. A negative $\phi$ therefore produces correlations that alternate in sign, while their magnitude declines geometrically when $|\phi|<1$.

Why $|\phi|<1$ is the stationarity condition: Rearranging the variance equation requires $1-\phi^2>0$ for a positive finite variance when $\sigma_\epsilon^2>0$. More directly, repeated substitution gives $$ x_t=\phi^m x_{t-m}+\sum_{j=0}^{m-1}\phi^j\epsilon_{t-j}. $$ When $|\phi|<1$, the contribution from the distant starting value decays, and the innovation weights have finite squared sum $1+\phi^2+\phi^4+\cdots=1/(1-\phi^2)$. If $|\phi|\geq 1$, the causal recursion with positive innovation variance does not have a finite-variance stationary solution: the accumulated innovation variance does not converge. This condition concerns the usual causal AR(1) process driven by present and past innovations.

The same argument also gives an infinite-past representation when $|\phi|<1$:

x_t=\sum_{j=0}^{\infty}\phi^j\epsilon_{t-j}.

The variance of this sum is $\sigma_\epsilon^2\sum_{j=0}^{\infty}\phi^{2j}$, because distinct innovations are uncorrelated. The geometric-series sum gives $\sigma_\epsilon^2/(1-\phi^2)$, agreeing with the variance-equation derivation. This representation explains why the stationary covariance depends only on the time lag, not on the absolute observation times.

Worked Example: Positive Dependence and a Four-Time Covariance Matrix

Suppose a hypothetical latent process has $\phi=0.60$ and innovation variance $\sigma_\epsilon^2=1.44$ in squared latent-effect units. Assume equally spaced times, independent mean-zero innovations with constant variance, and stationary initialization. These conditions allow the stationary formulas to apply.

The stationary variance and standard deviation are $$ \sigma_b^2=\frac{1.44}{1-0.60^2} =\frac{1.44}{0.64}=2.25, \qquad \sigma_b=\sqrt{2.25}=1.50. $$

The lag-one covariance is $\gamma_1=0.60(2.25)=1.35$, and the lag-three covariance is $\gamma_3=0.60^3(2.25)=0.486$. The corresponding lag-three correlation is $0.60^3=0.216$. These covariances are in squared latent-effect units; the correlations have no units.

For four successive times, the covariance matrix is $$ \begin{pmatrix} 2.25&1.35&0.810&0.486\\ 1.35&2.25&1.35&0.810\\ 0.810&1.35&2.25&1.35\\ 0.486&0.810&1.35&2.25 \end{pmatrix}. $$ Each entry uses the variance $2.25$ multiplied by $0.60^{|i-j|}$. The constant diagonal and dependence on lag alone are consequences of stationarity.

Worked Example: Negative Autoregression

Consider a hypothetical stationary process with $\phi=-0.50$ and $\sigma_\epsilon^2=0.75$, with the same innovation and initialization assumptions as above. The condition $|\phi|=0.50<1$ holds.

Its stationary variance is $$ \sigma_b^2=\frac{0.75}{1-(-0.50)^2} =\frac{0.75}{0.75}=1. $$

The first three lag covariances are $\gamma_1=(-0.50)(1)=-0.50$, $\gamma_2=(-0.50)^2(1)=0.250$, and $\gamma_3=(-0.50)^3(1)=-0.125$. Because the variance is one, these values are also the lag correlations. Adjacent latent effects tend to move in opposite directions, while effects two steps apart tend to move in the same direction. The alternating signs are not negative variances; the variance remains the positive lag-zero covariance $\gamma_0=1$.

Initialization and the Approach to Stationarity

The stationary covariance formula describes a process whose distribution is stationary, not every sequence generated by the AR(1) recursion. Suppose instead that $x_0=b_0-\mu$ has mean zero and variance $V_0$, is independent of future innovations, and the recursion begins at time 1. Repeated substitution gives $$ x_t=\phi^t x_0+\sum_{j=0}^{t-1}\phi^j\epsilon_{t-j}. $$

The initial term and all future innovations are uncorrelated. Consequently, the variance at time $t$ is $$ \operatorname{Var}(b_t)=\phi^{2t}V_0+\sigma_\epsilon^2\sum_{j=0}^{t-1}\phi^{2j} =\sigma_b^2+\phi^{2t}(V_0-\sigma_b^2), \quad |\phi|<1. $$

Unless $V_0=\sigma_b^2$, this variance depends on $t$. It approaches the stationary variance as $\phi^{2t}$ tends to zero, but early observations need not have stationary covariance. For $t\geq s$, the same assumptions give $\operatorname{Cov}(b_t,b_s)=\phi^{t-s}\operatorname{Var}(b_s)$, which depends on the earlier time $s$ when its variance has not reached stationarity.

Worked Example: A Fixed Initial Value Produces Transient Variance

For a hypothetical process, let $\phi=0.80$, $\sigma_\epsilon^2=0.36$, and set $b_0=\mu$ exactly. Thus $V_0=0$; the process does not start with its stationary variance. The stationary variance implied by the recursion is $$ \sigma_b^2=\frac{0.36}{1-0.80^2} =\frac{0.36}{0.36}=1. $$

At time 1, the variance is $0.36$. At time 2, it is $0.80^2(0.36)+0.36=0.5904$. At time 3, it is $0.80^2(0.5904)+0.36=0.737856$. Equivalently, the formula gives $\operatorname{Var}(b_3)=1+0.80^6(0-1)=0.737856$.

The variances increase toward 1, but none of these three early-time variances equals the stationary variance. Applying the stationary covariance matrix from time 0 would therefore be inappropriate for this initialization. The calculation illustrates a transient effect, not a failure of the stationary variance formula.

Using the Result and Recognizing Its Limits

For equally spaced observations at times $1,\ldots,T$, stationary AR(1) effects have covariance entries $\operatorname{Cov}(b_i,b_j)=\sigma_b^2\phi^{|i-j|}$. In a GLMM, this supplies a structured covariance for the latent effects. It is parsimonious: a variance and an autoregressive coefficient determine all entries, rather than estimating a separate covariance for each pair of times.

The structure imposes substantive restrictions. The marginal latent variance must be constant across time, and two pairs of times separated by the same number of equally spaced steps must have the same covariance. Dependence must decline geometrically in magnitude, with the sign pattern governed by $\phi$. These restrictions may be unsuitable when the process has a trend, changing variability, irregular time gaps, or a dependence pattern that does not decay geometrically.

Structure	Implied temporal dependence	Important distinction
Independent latent effects	Zero covariance at nonzero lags	The AR(1) model reduces to this case when $\phi=0$.
Stationary AR(1)	$\sigma_b^2\phi^{\|k\|}$ at lag $k$	Requires equally spaced steps and $\|\phi\|<1$ for the causal stationary process.
Random-walk boundary, $\phi=1$	Does not have the finite stationary variance derived here when $\sigma_\epsilon^2>0$	It is not a stationary AR(1) covariance model.

In practice, examine whether the chosen time scale is equally spaced and whether the fitted process plausibly has a constant mean and variance. Check whether estimates lie near the boundary $|\phi|=1$, and whether model-based simulations reproduce the temporal patterns relevant to the analysis. For GLMMs, observed response correlations are not direct measurements of latent correlations: conditional response variation and the link function also matter. A discrepancy in response data should therefore not be interpreted as a direct estimate of $\phi$.

A common mistake is to write down $\gamma_k=\sigma_b^2\phi^{|k|}$ simply because the recursion contains $\phi$, without checking stationarity or initialization. The recursion alone does not imply constant variance. First verify the innovation assumptions and the condition $|\phi|<1$, then verify that the initial distribution is stationary if using stationary covariance entries from the first observation.

Key takeaway: Under independent mean-zero innovations with finite variance and stationary initialization, the causal AR(1) process has $\sigma_b^2=\sigma_\epsilon^2/(1-\phi^2)$ and $\gamma_k=\sigma_b^2\phi^{|k|}$. The absolute-value condition and initialization are part of the result, not optional details.

Check Your Understanding

Work each question from the stated model. Solutions are included after the questions.

A stationary AR(1) process has $\phi=0.40$ and innovation variance $0.84$. Find its stationary variance and lag-two covariance.
A process has $\phi=-0.70$ and innovation variance $0.51$. Is the causal stationary condition met? Find its stationary variance and lag-one correlation.
For $\phi=0.90$ and positive innovation variance, explain why the causal stationary variance formula cannot be used.
A process has $\phi=0.50$, innovation variance $0.75$, and $b_0=\mu$ fixed. Find the stationary variance and the variance at time 2. Is the time-2 marginal variance stationary?

Solutions. (1) Since $1-\phi^2=1-0.16=0.84$, $\sigma_b^2=0.84/0.84=1$. Then $\gamma_2=1(0.40)^2=0.160$. (2) The condition holds because $|-0.70|=0.70<1$. The variance is $0.51/(1-0.49)=0.51/0.51=1$, and the lag-one correlation is $-0.70$. (3) The condition does hold: $0.90<1$. In fact, the stationary formula can be used, giving $\sigma_b^2=\sigma_\epsilon^2/(1-0.90^2)=\sigma_\epsilon^2/0.19$. A coefficient near one implies a large stationary variance relative to the innovation variance; it does not invalidate the formula. (4) The stationary variance is $0.75/(1-0.25)=1$. Since $V_0=0$, the time-2 variance is $1+0.50^4(0-1)=0.9375$. It is not the stationary variance, because the process was initialized at a fixed value rather than with variance 1.

Deriving the Stationary AR(1) Covariance

What You'll Learn