Autocovariance Restrictions and Positive Semidefinite Matrices

Why Autocovariances Cannot Be Chosen Arbitrarily

For a weakly stationary time series, the covariance between two observations depends on their separation in time, not on the date at which they occur. But this invariance does not mean that every list of numbers indexed by time lag can be an autocovariance sequence. Symmetry, variance bounds, and positive semidefiniteness impose restrictions that any such sequence must satisfy.

These restrictions matter because an impossible covariance sequence cannot describe any weakly stationary process, even if each individual number seems plausible. The key test is that every finite linear combination of observations must have nonnegative variance.

Definition: Let $\{X_t\}$ be a weakly stationary process with finite second moments and constant mean $\mu$. Its autocovariance at integer lag $h$ is $$ \gamma(h)=\operatorname{Cov}(X_t,X_{t-h}). $$ Weak stationarity makes this quantity independent of $t$. In particular, $\gamma(0)=\operatorname{Var}(X_t)$, so $\gamma(0)\geq 0$. The collection of values $\{\gamma(h):h\in\mathbb{Z}\}$ is the autocovariance sequence.

Here $h$ can be positive, zero, or negative. A negative lag reverses which observation appears first in the covariance. This immediately implies a symmetry restriction.

Symmetry and the Variance Bound

Covariance is symmetric in its two arguments. Therefore, for any integer $h$,

\gamma(-h) =\operatorname{Cov}(X_t,X_{t+h}) =\operatorname{Cov}(X_{t+h},X_t) =\gamma(h).

Thus the autocovariance sequence is an even function of the lag: the covariance at lag $h$ equals the covariance at lag $-h$. This is a restriction on the full sequence, not merely an approximation that might hold in a large sample.

There is also a bound on the size of an autocovariance. The two random variables $X_t-\mu$ and $X_{t-h}-\mu$ have the same variance $\gamma(0)$. The variance of either their sum or their difference must be nonnegative:

\begin{aligned} 0 &\leq \operatorname{Var}\bigl((X_t-\mu)+(X_{t-h}-\mu)\bigr) =2\gamma(0)+2\gamma(h),\\ 0 &\leq \operatorname{Var}\bigl((X_t-\mu)-(X_{t-h}-\mu)\bigr) =2\gamma(0)-2\gamma(h). \end{aligned}

Together these inequalities give $-\gamma(0)\leq\gamma(h)\leq\gamma(0)$. The argument also covers the case $\gamma(0)=0$: then every autocovariance must be zero.

Takeaway: Every weakly stationary autocovariance sequence satisfies $$ \gamma(-h)=\gamma(h) \qquad\text{and}\qquad |\gamma(h)|\leq\gamma(0)\quad\text{for every integer }h. $$ These are necessary restrictions, but they do not by themselves guarantee that a proposed sequence is valid.

Worked Example: Applying the Variance Bound

Suppose a hypothetical stationary process has variance $\gamma(0)=4$ units squared. Consider two proposed values for its autocovariance at lag 2: $-3$ units squared and $5$ units squared.

The bound requires $$ |\gamma(2)|\leq\gamma(0)=4. $$ For the first proposal, $|-3|=3\leq4$, so $-3$ is consistent with this bound. A negative covariance is not automatically impossible. For the second proposal, $|5|=5>4$, so $5$ is impossible as an autocovariance at lag 2 for a process with variance 4 units squared.

Passing the bound does not establish that the first proposal belongs to a valid full autocovariance sequence. Other restrictions must still be checked.

Covariance Matrices Must Be Positive Semidefinite

A matrix $C$ is positive semidefinite if $a^\top C a\geq0$ for every real vector $a$ of matching dimension. The requirement is stronger than having nonnegative entries on the diagonal: every weighted combination represented by the matrix must also have nonnegative variance.

Take any finite collection of time points $t_1,\ldots,t_m$, and let $a_1,\ldots,a_m$ be real numbers. The variance of the centered linear combination of observations is

\begin{aligned} \operatorname{Var}\left(\sum_{j=1}^m a_j(X_{t_j}-\mu)\right) &=\sum_{i=1}^m\sum_{j=1}^m a_i a_j \operatorname{Cov}(X_{t_i},X_{t_j})\\ &=\sum_{i=1}^m\sum_{j=1}^m a_i a_j\gamma(t_i-t_j) \geq 0. \end{aligned}

The equality uses the lag convention $\gamma(h)=\operatorname{Cov}(X_t,X_{t-h})$: setting $t=t_i$ and $h=t_i-t_j$ gives $\operatorname{Cov}(X_{t_i},X_{t_j})=\gamma(t_i-t_j)$. The inequality follows because a variance cannot be negative.

For consecutive observations $X_1,\ldots,X_m$, their covariance matrix is Toeplitz: each diagonal is constant, because the lag between two observations is constant along that diagonal. Symmetry of the matrix follows from $\gamma(-h)=\gamma(h)$.

\Gamma_m= \begin{pmatrix} \gamma(0)&\gamma(1)&\cdots&\gamma(m-1)\\ \gamma(1)&\gamma(0)&\cdots&\gamma(m-2)\\ \vdots&\vdots&\ddots&\vdots\\ \gamma(m-1)&\gamma(m-2)&\cdots&\gamma(0) \end{pmatrix}.

The variance calculation shows that $\Gamma_m$ must be positive semidefinite for every positive integer $m$. More generally, the same requirement applies to covariance matrices from any finite selection of time points, including nonconsecutive ones. A negative value of $a^\top\Gamma_m a$ for even one vector $a$ proves that the proposed sequence is impossible.

Worked Example: Rejecting a Sequence With a Negative Variance

Consider the proposed sequence $\gamma(0)=1$, $\gamma(1)=\gamma(-1)=0.8$, and $\gamma(h)=0$ for $|h|\geq2$. It satisfies symmetry and the individual bound $|\gamma(h)|\leq\gamma(0)$. Check whether it can be an autocovariance sequence by considering three consecutive observations.

The proposed covariance matrix and a candidate weight vector are $$ \Gamma_3= \begin{pmatrix} 1&0.8&0\\ 0.8&1&0.8\\ 0&0.8&1 \end{pmatrix}, \qquad a= \begin{pmatrix} 1\\-\sqrt{2}\\1 \end{pmatrix}. $$

If this matrix were a valid covariance matrix, the variance of the corresponding weighted sum would be $a^\top\Gamma_3a\geq0$. Instead, direct multiplication gives $$ a^\top\Gamma_3a =1+2+1+2(0.8)(-\sqrt{2}-\sqrt{2}) =4-3.2\sqrt{2} \approx -0.5255. $$

A variance cannot equal a negative number. Therefore this proposed sequence is impossible, despite passing both the symmetry and the individual variance-bound checks. The matrix test detects a restriction that checking each lag separately misses.

Valid Sequences and the Limits of Small Checks

A useful way to demonstrate that a proposed sequence is valid is to construct a process with exactly that covariance. For example, let $A$ and $B$ be independent random variables with mean zero and variance one, and define, for integer $t$,

X_t=A\cos(\omega t)+B\sin(\omega t).

The mean is zero, and independence gives $$ \operatorname{Cov}(X_t,X_{t-h}) =\cos(\omega t)\cos(\omega(t-h)) +\sin(\omega t)\sin(\omega(t-h)) =\cos(\omega h). $$ Thus $\gamma(h)=\cos(\omega h)$ is a valid autocovariance sequence for this process. The construction also explains why some autocovariances may be negative: for example, at $\omega=\pi/3$, $\gamma(3)=\cos(\pi)=-1$, while $\gamma(0)=1$. The negative value is consistent with the variance bound and with the existence of the process.

Worked Example: Verifying a Sequence by Construction

Take $\omega=\pi/3$ in the process just defined. Its proposed autocovariances include $\gamma(0)=1$, $\gamma(1)=1/2$, $\gamma(2)=-1/2$, and $\gamma(3)=-1$, with the values at negative lags equal to those at the corresponding positive lags.

These numbers can be checked directly from $\gamma(h)=\cos(\pi h/3)$. More importantly, the random-variable construction verifies all lags at once. For any finite real coefficients $a_1,\ldots,a_m$ at times $t_1,\ldots,t_m$, the weighted sum is $$ \sum_{j=1}^m a_jX_{t_j} =A\sum_{j=1}^m a_j\cos(\omega t_j) +B\sum_{j=1}^m a_j\sin(\omega t_j). $$

Since $A$ and $B$ are independent, mean zero, and each have variance one, the variance of this sum is $$ \left(\sum_{j=1}^m a_j\cos(\omega t_j)\right)^2 + \left(\sum_{j=1}^m a_j\sin(\omega t_j)\right)^2 \geq0. $$

This confirms positive semidefiniteness for every finite collection of times. In this example, validity is established by an explicit process, not just by checking a few entries of a covariance matrix.

A finite matrix check can reject a sequence, but passing one small check does not establish that all larger covariance matrices are positive semidefinite. The distinction is important when a proposed sequence specifies infinitely many lags.

Worked Example: A Small Matrix Passes but a Larger One Fails

Consider the proposed sequence $\gamma(0)=1$, $\gamma(1)=\gamma(-1)=0.6$, and $\gamma(h)=0$ for $|h|\geq2$. For three consecutive observations, the matrix is $$ \Gamma_3= \begin{pmatrix} 1&0.6&0\\ 0.6&1&0.6\\ 0&0.6&1 \end{pmatrix}. $$

Its eigenvalues are $1$, $1+\sqrt{2}(0.6)$, and $1-\sqrt{2}(0.6)$. The smallest is approximately $0.1515$, so this particular matrix is positive definite. The sequence also satisfies the symmetry and variance-bound restrictions.

Now consider five consecutive observations and the vector $a=(1,-\sqrt{3},2,-\sqrt{3},1)^\top$. The corresponding covariance matrix has ones on the diagonal, $0.6$ on the immediately adjacent diagonals, and zeros elsewhere. Its quadratic form is $$ a^\top\Gamma_5a =12+1.2(-6\sqrt{3}) =12-7.2\sqrt{3} \approx -0.4708. $$

The negative result rules out the proposed sequence. The three-observation matrix passed, but the five-observation matrix did not. A sequence must satisfy the positive-semidefinite requirement for every finite collection, not just for one convenient matrix size.

Common Pitfall and Practical Check

A common mistake is to treat symmetry and the bound $|\gamma(h)|\leq\gamma(0)$ as a complete validity test. They are necessary, but the example with lag-one covariance $0.8$ shows that a sequence can satisfy both and still imply a negative variance. Another mistake is to check a single small Toeplitz matrix and conclude that an infinite sequence is valid. A successful finite check rules out a violation at that size only.

When assessing a proposed autocovariance sequence, first check symmetry and the variance bound. Then form covariance matrices for informative finite collections of times. If a matrix fails, look for a vector $a$ with $a^\top\Gamma a<0$; that vector identifies a linear combination that would have impossible negative variance. To establish validity rather than merely fail to find a problem, use a construction or verify the positive-semidefinite condition for every finite collection.

Key takeaway: A weakly stationary process has a symmetric autocovariance sequence bounded in magnitude by its variance. In addition, every finite covariance matrix generated by that sequence must be positive semidefinite, because every finite linear combination of observations has nonnegative variance.

Check Your Understanding

Use the restrictions and variance arguments from this tutorial to answer the following questions.

If $\gamma(0)=9$ units squared, what range of values is permitted for $\gamma(4)$ by the variance bound? Does the bound alone establish that every value in this range is valid?
Explain why $\gamma(-2)=\gamma(2)$ for a weakly stationary process.
What does a negative value of $a^\top\Gamma_m a$ imply about a proposed autocovariance sequence?
Why does passing the positive-semidefinite check for a three-observation covariance matrix not establish that an entire sequence is valid?
For the constructed process $X_t=A\cos(\omega t)+B\sin(\omega t)$, state its autocovariance at lag $h$.