Positive Definiteness of Covariance Functions

Why Covariance Functions Must Be Positive Definite

A covariance function cannot assign values to pairs of times independently. All its values at any finite collection of times must fit together as the covariances of random variables. The key restriction is that every linear combination of those variables has nonnegative variance. Written in terms of the covariance function, this becomes a nonnegative quadratic form—a condition called positive definiteness. It is a necessary consistency check for any proposed covariance function, whether or not one is building a Gaussian model.

Throughout, let $(X_t)_{t\in T}$ be a real-valued process on a probability space $(\Omega,\mathcal{F},\mathbb{P})$, with state space $S=\mathbb{R}$ and time index set $T$. Assume each $X_t$ has finite second moment. Write $m_t=\mathbb{E}[X_t]$ and $K(s,t)=\operatorname{Cov}(X_s,X_t)$. For selected times $t_1,\ldots,t_r$, the covariance matrix is the $r$ by $r$ matrix $C$ with entries $C_{ij}=K(t_i,t_j)$. Times need not be distinct.

Definition: A real-valued kernel $K$ on $T\times T$ is positive definite in the covariance sense if, for every integer $r\ge1$, every choice of times $t_1,\ldots,t_r\in T$, and every choice of complex numbers $c_1,\ldots,c_r$, the sum $\sum_{i=1}^r\sum_{j=1}^r c_i\overline{c_j}K(t_i,t_j)$ is real and nonnegative. Equivalently, every finite matrix $(K(t_i,t_j))$ is positive semidefinite. “Positive semidefinite” permits a zero quadratic form for a nonzero vector; it does not mean strictly positive definite.

Complex coefficients are convenient notation, even though the process itself is real-valued. If all coefficients are real, the condition says directly that every real linear combination of the selected variables has nonnegative variance. Allowing complex coefficients gives the same condition: writing $c_i=a_i+\mathrm{i}b_i$, the quadratic form splits into the corresponding nonnegative forms for the real coefficient vectors $a$ and $b$, because the covariance matrix is real and symmetric.

Theorem (Covariance quadratic-form condition): For any finite selection $t_1,\ldots,t_r$ from $T$, the covariance matrix $C=(K(t_i,t_j))$ is positive semidefinite. In particular, for all complex $c_1,\ldots,c_r$, $$ \sum_{i=1}^r\sum_{j=1}^r c_i\overline{c_j}K(t_i,t_j)\ge 0. $$

Proof. Set $Y_i=X_{t_i}-m_{t_i}$, so that $\mathbb{E}[Y_i]=0$ and $\mathbb{E}[Y_iY_j]=K(t_i,t_j)$. For any complex coefficients $c_1,\ldots,c_r$, define $Z=\sum_{i=1}^r c_iY_i$. Finite second moments ensure that $Z$ has finite second moment. Expanding the squared modulus gives $$ \mathbb{E}[|Z|^2] =\mathbb{E}\left[\left(\sum_{i=1}^r c_iY_i\right) \left(\sum_{j=1}^r\overline{c_j}Y_j\right)\right] =\sum_{i=1}^r\sum_{j=1}^r c_i\overline{c_j}K(t_i,t_j). $$ Since $|Z|^2\ge0$ for every outcome, its expectation is real and nonnegative. This proves the asserted condition. If the selection is empty, the associated sum is zero, so the condition is vacuously satisfied in that case as well. $\square$

The proof uses only the definition of covariance and nonnegativity of squared magnitude. It does not depend on Gaussianity, stationarity, or a Markov property. Consequently, positive definiteness is necessary for the covariance kernel of any finite-second-moment process, not only for covariance functions that depend on time differences.

Stationary Covariance Sequences

For a weakly stationary process with integer time index, the mean is constant and the covariance depends only on the lag. In this setting let $(X_n)_{n\in\mathbb{Z}}$ be a real-valued process on $(\Omega,\mathcal{F},\mathbb{P})$, with state space $S=\mathbb{R}$, constant mean, and finite second moments. Its autocovariance sequence is $\gamma(h)=\operatorname{Cov}(X_{n+h},X_n)$; weak stationarity makes this value independent of $n$. Thus $\operatorname{Cov}(X_{n_i},X_{n_j})=\gamma(n_i-n_j)$.

Corollary: The autocovariance sequence of any weakly stationary, finite-second-moment process is positive definite: for every $r\ge1$, integers $n_1,\ldots,n_r$, and complex coefficients $c_1,\ldots,c_r$, $$ \sum_{i=1}^r\sum_{j=1}^r c_i\overline{c_j}\gamma(n_i-n_j)\ge0. $$

Proof. Fix $r$, the integer times $n_1,\ldots,n_r$, and the coefficients $c_1,\ldots,c_r$. Apply the covariance quadratic-form condition to the selected random variables $X_{n_1},\ldots,X_{n_r}$. Their covariance entries are $\operatorname{Cov}(X_{n_i},X_{n_j})=\gamma(n_i-n_j)$ by weak stationarity. Substitution into that condition gives exactly the stated inequality. Since the selection and coefficients were arbitrary, the inequality holds for every finite choice. $\square$

This corollary explains the practical significance of the condition: if a proposed stationary covariance sequence produces a negative quadratic form for even one finite selection of times and coefficients, it cannot be the autocovariance sequence of a weakly stationary process. The condition is a necessary consistency requirement. Establishing sufficiency or constructing a process from a proposed sequence requires additional results and is not part of this argument.

Worked Quadratic Forms

Worked Example: A Singular Covariance Matrix from Dependent Variables

Let $U$ and $V$ be independent, mean-zero random variables with variance $1$, and define $X_1=U$, $X_2=U+V$, and $X_3=V$. These variables live on the probability space carrying $(U,V)$, have state space $\mathbb{R}$, and are indexed here by the finite set of labels $\{1,2,3\}$. Find their covariance matrix and evaluate its quadratic form at $c=(1,2,-1)$.

Independence and zero means give $\operatorname{Cov}(U,V)=0$. The diagonal entries are $\operatorname{Var}(X_1)=1$, $\operatorname{Var}(X_2)=\operatorname{Var}(U)+\operatorname{Var}(V)=2$, and $\operatorname{Var}(X_3)=1$. The off-diagonal entries are $\operatorname{Cov}(X_1,X_2)=1$, $\operatorname{Cov}(X_1,X_3)=0$, and $\operatorname{Cov}(X_2,X_3)=1$. Therefore $$ C=\begin{bmatrix}1&1&0\\1&2&1\\0&1&1\end{bmatrix}. $$ Multiplying by the chosen vector gives $$ Cc=\begin{bmatrix}1+2\\1+4-1\\2-1\end{bmatrix} =\begin{bmatrix}3\\4\\1\end{bmatrix}, \qquad c^\mathsf{T}Cc=1(3)+2(4)+(-1)(1)=10. $$ This agrees with the variance of the corresponding linear combination: $$ X_1+2X_2-X_3=U+2(U+V)-V=3U+V, \qquad \operatorname{Var}(3U+V)=9(1)+1(1)=10. $$ The calculation verifies the quadratic-form identity directly. The matrix is not strictly positive definite: for $c=(1,-1,1)$, the combination $X_1-X_2+X_3=U-(U+V)+V=0$, so its variance and quadratic form are both $0$.

Worked Example: A Moving-Average Covariance Matrix

Let $(\varepsilon_n)_{n\in\mathbb{Z}}$ be independent mean-zero random variables with variance $1$, on a probability space supporting that sequence. Define the real-valued, discrete-time process $X_n=\varepsilon_n+\varepsilon_{n-1}$, with state space $\mathbb{R}$. Its mean is zero. Compute the covariance matrix at times $0,1,2$ and the quadratic form for $c=(1,-1,1)$.

Each $X_n$ contains two independent terms of variance $1$, so $\gamma(0)=\operatorname{Var}(X_n)=2$. Adjacent variables share one innovation, so $\gamma(1)=\gamma(-1)=1$. Variables separated by at least two time units share no innovations, so $\gamma(2)=\gamma(-2)=0$. The resulting covariance matrix, with every entry given by $\operatorname{Cov}(X_i,X_j)$, is $$ C=\begin{bmatrix}2&1&0\\1&2&1\\0&1&2\end{bmatrix}. $$ Then $$ Cc=\begin{bmatrix}2-1\\1-2+1\\-1+2\end{bmatrix} =\begin{bmatrix}1\\0\\1\end{bmatrix}, \qquad c^\mathsf{T}Cc=1(1)+(-1)(0)+1(1)=2. $$ As an independent check, $X_0-X_1+X_2=\varepsilon_{-1}+\varepsilon_2$: the $\varepsilon_0$ and $\varepsilon_1$ terms cancel. The two remaining independent terms have variance $1$ each, so the variance is $2$, matching the quadratic form.

Worked Example: A Positive-Semidefinite Sequence with a Zero Direction

Let $A$ and $B$ be independent mean-zero random variables with variance $1$, on a probability space supporting them. Define a real-valued process on integer times $n\in\mathbb{Z}$, with state space $\mathbb{R}$, by $X_n=A\cos(n\pi/3)+B\sin(n\pi/3)$. Compute its covariance matrix at times $0,1,2$, then check the quadratic form at $c=(1,-1,1)$.

The mean is zero. Independence and the unit variances give $$ \operatorname{Cov}(X_n,X_m) =\cos(n\pi/3)\cos(m\pi/3)+\sin(n\pi/3)\sin(m\pi/3) =\cos((n-m)\pi/3). $$ Using $\cos(0)=1$, $\cos(\pi/3)=1/2$, and $\cos(2\pi/3)=-1/2$, the matrix at $0,1,2$ is $$ C=\begin{bmatrix}1&\tfrac12&-\tfrac12\\ \tfrac12&1&\tfrac12\\ -\tfrac12&\tfrac12&1\end{bmatrix}. $$ Multiplication gives $$ Cc=\begin{bmatrix}1-\tfrac12-\tfrac12\\ \tfrac12-1+\tfrac12\\ -\tfrac12-\tfrac12+1\end{bmatrix} =\begin{bmatrix}0\\0\\0\end{bmatrix}, \qquad c^\mathsf{T}Cc=0. $$ The zero also follows from the process itself: $X_0-X_1+X_2=0\cdot A+0\cdot B=0$, since the coefficients of $A$ are $1-\tfrac12-\tfrac12=0$ and those of $B$ are $0-\tfrac{\sqrt{3}}2+\tfrac{\sqrt{3}}2=0$. Thus nonzero coefficient vectors can have zero quadratic form when the selected variables satisfy a deterministic linear relation.

A Consequence: Autocovariance Is Bounded by the Variance

The two-time case already places a useful restriction on any stationary covariance sequence. In particular, no lag covariance can have magnitude larger than the variance at lag zero. This consequence is also a quick way to detect some invalid proposals, though it does not replace the full family of finite quadratic-form conditions.

Proposition: For a weakly stationary real-valued process with finite second moments, $\gamma(0)\ge0$ and $|\gamma(h)|\le\gamma(0)$ for every integer $h$.

Proof. The value $\gamma(0)=\operatorname{Var}(X_n)$ is nonnegative by the definition of variance. For a fixed integer $h$, the covariance matrix at times $0$ and $h$ is $$ \begin{bmatrix}\gamma(0)&\gamma(h)\\\gamma(h)&\gamma(0)\end{bmatrix}. $$ For any real $x$, its quadratic form at $(1,x)$ must be nonnegative by the covariance quadratic-form condition. Writing $v=\gamma(0)$ and $g=\gamma(h)$, this says $v+2xg+x^2v\ge0$ for every real $x$. If $v>0$, this quadratic polynomial has its minimum at $x=-g/v$. Substitution gives $v-g^2/v\ge0$, hence $g^2\le v^2$, and therefore $|g|\le v$. If $v=0$, the inequality reduces to $2xg\ge0$ for every real $x$. Taking $x=1$ gives $g\ge0$, and taking $x=-1$ gives $g\le0$, so $g=0$. Thus the bound holds in both cases. $\square$

A Common Mistake and the Correct Test

Positive variances on the diagonal do not guarantee a valid covariance matrix. For example, consider the proposed two-time matrix $$ C=\begin{bmatrix}1&2\\2&1\end{bmatrix}. $$ Both diagonal entries are positive, but for $c=(1,-1)$ the quadratic form is $1-2-2+1=-2$. A covariance matrix cannot have this value: it would assert that the variance of the corresponding centered linear combination is negative. The correction is to test the whole quadratic form, including both cross terms. Here the proposed off-diagonal covariance $2$ is already incompatible with two variances equal to $1$, as the two-time bound also shows.

For a proposed covariance function, the essential check is therefore not merely that each value at zero is nonnegative, or that each pair of times appears plausible. Every finite selection must produce a positive-semidefinite covariance matrix. Zero quadratic forms are allowed and often meaningful: they indicate a linear combination with zero variance, which is constant almost surely after centering. Strict positivity is an extra property and cannot be assumed for general processes.

Check Your Understanding

For each question, use the quadratic-form identity and state which finite set of times or variables you are considering.

Why does $\operatorname{Var}(\sum_i a_iX_{t_i})\ge0$ imply a restriction on every finite covariance matrix?
For the matrix $\begin{bmatrix}2&1\\1&2\end{bmatrix}$, compute the quadratic form at $c=(1,-1)$. Is the value consistent with positive semidefiniteness?
In the cosine-process example, explain why a nonzero coefficient vector can have zero quadratic form.
If $\gamma(0)=0$ for a weakly stationary process, what does the two-time quadratic-form argument imply about $\gamma(h)$ for each $h$?
Why do positive diagonal entries alone fail to establish that a proposed covariance matrix is valid?