Gaussian Processes and Finite-Dimensional Distributions

From a Collection of Variables to a Gaussian Process

A Gaussian process is a way to describe a possibly large collection of random variables while requiring a precise form of joint normality. The requirement concerns every finite selection of values—not just the distribution at each individual index. This distinction matters whenever values at different times or inputs are analyzed together.

Let $(\Omega,\mathcal{F},\mathbb{P})$ be a probability space, let $T$ be an index set, and let the state space be $\mathbb{R}$. A real-valued stochastic process indexed by $T$ is a collection $(X_t)_{t\in T}$ of random variables on this same probability space, each taking values in $\mathbb{R}$. The set $T$ need not be a time interval: it might be a finite collection of labels, a subset of $\mathbb{R}$, or a more general input space. The index $t$ identifies which random variable is being considered; it is not the value of that random variable.

Definition: The process $(X_t)_{t\in T}$ is a Gaussian process if, for every integer $n\ge1$ and every choice $t_1,\ldots,t_n\in T$, the vector $(X_{t_1},\ldots,X_{t_n})^\mathsf{T}$ has a multivariate normal distribution. The selected indices may be taken distinct; allowing repetitions gives the same definition and includes the resulting possibly degenerate normal vectors.

Here, a multivariate normal distribution is allowed to be singular: its covariance matrix need only be positive semidefinite, not invertible. Thus a Gaussian vector can be concentrated on a line or a lower-dimensional plane. Requiring only that each $X_t$ separately be normally distributed is weaker and does not suffice.

For a Gaussian process, its mean function and covariance function are, respectively,

m(t)=\mathbb{E}[X_t],\qquad K(s,t)=\operatorname{Cov}(X_s,X_t) =\mathbb{E}\big[(X_s-m(s))(X_t-m(t))\big], \quad s,t\in T.

These quantities exist and are finite because every individual $X_t$ is normally distributed. For selected indices $t_1,\ldots,t_n$, the mean vector has entries $m(t_r)$, while the covariance matrix has $(r,q)$ entry $K(t_r,t_q)$. The finite-dimensional distribution at those indices is the joint probability distribution of $(X_{t_1},\ldots,X_{t_n})^\mathsf{T}$. The index set specifies where the process is indexed; the mean and covariance describe first- and second-order features of its values.

Two Basic Consequences of the Definition

Theorem: If $(X_t)_{t\in T}$ is a Gaussian process, then every finite linear combination $\sum_{r=1}^n a_rX_{t_r}$, with real coefficients $a_r$, is normally distributed. Its mean and variance are $$ \mathbb{E}\!\left[\sum_{r=1}^n a_rX_{t_r}\right] =\sum_{r=1}^n a_rm(t_r),\qquad \operatorname{Var}\!\left(\sum_{r=1}^n a_rX_{t_r}\right) =\sum_{r=1}^n\sum_{q=1}^n a_ra_qK(t_r,t_q). $$ A zero variance is permitted and means the linear combination is constant almost surely.

Proof. By the definition of a Gaussian process, $Y=(X_{t_1},\ldots,X_{t_n})^\mathsf{T}$ is a multivariate normal vector. Write $Y=b+AZ$, where $b$ is a fixed vector, $A$ is a fixed matrix, and the coordinates of $Z$ are independent standard normal random variables; this representation also permits singular normal vectors. With $a=(a_1,\ldots,a_n)^\mathsf{T}$, the linear combination is $a^\mathsf{T}Y=a^\mathsf{T}b+(A^\mathsf{T}a)^\mathsf{T}Z$. A linear combination of independent standard normal variables is normal: its characteristic function is the product of the characteristic functions of the terms, namely $\prod_j\exp(-u^2c_j^2/2)=\exp(-u^2\sum_jc_j^2/2)$ for coefficients $c_j$. If every coefficient is zero, the combination is constant instead. Linearity of expectation gives the stated mean. Expanding the variance of the sum gives $\sum_{r,q}a_ra_q\operatorname{Cov}(X_{t_r},X_{t_q})$, which is the stated formula. $\square$

Proposition: Selecting finitely many coordinates of a Gaussian vector, or changing their order, gives another multivariate normal vector. Consequently, finite-dimensional distributions of a Gaussian process are consistent with taking marginals and reordering indices.

Proof. Let $Y$ be a multivariate normal vector and write $Y=b+AZ$ as above. Selecting coordinates or reordering them is multiplication by a fixed matrix $C$. The resulting vector is $CY=Cb+CAZ$, which is again an affine transformation of a vector of independent standard normal variables, and hence is multivariate normal, possibly singular. Applying this to a finite vector of process values proves the assertion about its subvectors and their orderings. $\square$

These facts help check a proposed process: finite linear combinations must be normal, and a finite joint law must retain the appropriate normal form when coordinates are discarded. They do not impose continuity of sample paths or stationarity. Those are additional properties that a process may or may not have.

Worked Examples

Worked Example: Brownian Motion at Three Times

Let $(B_t)_{t\ge0}$ be standard Brownian motion on a probability space $(\Omega,\mathcal{F},\mathbb{P})$. Its index set is $T=[0,\infty)$, and its state space is $\mathbb{R}$. Verify that $(B_1,B_2,B_3)^\mathsf{T}$ is multivariate normal and calculate its mean vector and covariance matrix.

The increments $\Delta_1=B_1$, $\Delta_2=B_2-B_1$, and $\Delta_3=B_3-B_2$ are independent $N(0,1)$ variables. The Brownian values are their cumulative sums:

\begin{pmatrix}B_1\\B_2\\B_3\end{pmatrix} = \begin{pmatrix} 1&0&0\\ 1&1&0\\ 1&1&1 \end{pmatrix} \begin{pmatrix}\Delta_1\\\Delta_2\\\Delta_3\end{pmatrix} = \begin{pmatrix} \Delta_1\\ \Delta_1+\Delta_2\\ \Delta_1+\Delta_2+\Delta_3 \end{pmatrix}.

The increment vector is multivariate normal, so its displayed linear transformation is multivariate normal. Each increment has mean zero, hence the mean vector of the Brownian values is $(0,0,0)^\mathsf{T}$. To calculate the covariance matrix, take dot products of the rows of the transformation matrix, because the increments have covariance matrix equal to the $3\times3$ identity:

\operatorname{Cov}\begin{pmatrix}B_1\\B_2\\B_3\end{pmatrix} = \begin{pmatrix} 1&1&1\\ 1&2&2\\ 1&2&3 \end{pmatrix}.

For example, the covariance of $B_1$ and $B_3$ is the dot product $(1,0,0)\cdot(1,1,1)=1$; the variance of $B_3$ is $(1,1,1)\cdot(1,1,1)=3$. The other entries follow from the same row products: the first row has dot products $1,1,1$, the second has $1,2,2$, and the third has $1,2,3$. This also verifies the familiar formula $K(s,t)=\min(s,t)$ at the selected times $1,2,3$.

Worked Example: A Degenerate Gaussian Process on Three Indices

Let $Z,W$ be independent standard normal variables on a probability space, and set $T=\{-1,0,2\}$. Define $X_{-1}=Z$, $X_0=Z+W$, and $X_2=2Z-W$. Find the mean vector and covariance matrix in the displayed index order, and verify joint normality.

The vector is the linear transformation

\begin{pmatrix}X_{-1}\\X_0\\X_2\end{pmatrix} = \begin{pmatrix} 1&0\\ 1&1\\ 2&-1 \end{pmatrix} \begin{pmatrix}Z\\W\end{pmatrix}.

Since $(Z,W)^\mathsf{T}$ is multivariate normal, the displayed vector is multivariate normal. Each coordinate has mean zero, so the mean vector is $(0,0,0)^\mathsf{T}$. For centered linear combinations of independent standard normals, covariance is the dot product of their coefficient rows. The row products give $\operatorname{Var}(X_{-1})=1$, $\operatorname{Cov}(X_{-1},X_0)=1$, $\operatorname{Cov}(X_{-1},X_2)=2$, $\operatorname{Var}(X_0)=2$, $\operatorname{Cov}(X_0,X_2)=2-1=1$, and $\operatorname{Var}(X_2)=4+1=5$. Therefore

\operatorname{Cov}\begin{pmatrix}X_{-1}\\X_0\\X_2\end{pmatrix} = \begin{pmatrix} 1&1&2\\ 1&2&1\\ 2&1&5 \end{pmatrix}.

As a check, $X_2=3X_{-1}-X_0$, since $3Z-(Z+W)=2Z-W$. The covariance matrix must therefore be singular. The computed matrix has that same relation: its third column equals three times its first column minus its second column, namely $(2,1,5)^\mathsf{T}=3(1,1,2)^\mathsf{T}-(1,2,1)^\mathsf{T}$. Thus singularity is consistent with the exact dependence among the values and does not prevent this from being a Gaussian process.

Worked Example: Normal Marginals That Do Not Form a Gaussian Process

Let $U$ be standard normal and let $\varepsilon$, independent of $U$, take the values $1$ and $-1$ with probability $\tfrac12$ each. On $T=\{a,b\}$, set $X_a=U$ and $X_b=\varepsilon U$. Both values are normal marginally. Determine their means and covariance, then check whether this two-index process is Gaussian.

Clearly $\mathbb{E}[X_a]=0$ and $\operatorname{Var}(X_a)=1$. Conditional on $\varepsilon=1$, $X_b=U$; conditional on $\varepsilon=-1$, $X_b=-U$, which has the same standard normal distribution by symmetry. Thus $X_b$ is also standard normal, with mean zero and variance one. Independence of $\varepsilon$ and $U$ gives

\operatorname{Cov}(X_a,X_b) =\mathbb{E}[\varepsilon U^2] =\mathbb{E}[\varepsilon]\mathbb{E}[U^2] =0\cdot1=0.

The covariance matrix is therefore the identity matrix. That calculation alone does not prove joint normality. In fact $X_b^2=X_a^2$ with probability one, so the joint vector lies on the two lines $y=x$ and $y=-x$. If the vector were a bivariate normal vector with the computed identity covariance matrix, its joint density would be the product of two standard normal densities; those lines have probability zero under that density. This contradiction shows that the vector is not jointly Gaussian. Hence the process is not Gaussian, despite having normal marginals.

What the Definition Does—and Does Not—Require

The index set, mean function, and covariance function answer different questions. The index set $T$ states which variables belong to the process. The mean function assigns a number to each index. The covariance function assigns a number to each pair of indices and records their second-order dependence. For a chosen finite list of indices, these functions provide the entries of the corresponding mean vector and covariance matrix.

A Gaussian process need not be stationary: its mean or covariance structure may depend on the particular indices, not just on their separation. It need not have continuous sample paths either. The definition requires multivariate normality for every finite selection; it does not impose a path-regularity condition. Conversely, checking only a few selected finite vectors is not enough to establish that a process indexed by a larger set is Gaussian. The defining condition quantifies over every finite selection from that set.

Common pitfall. A frequent error is to calculate normal marginal distributions and then declare the process Gaussian. In the preceding example, both marginal variances are $1$, and the covariance is $0$, but the pair satisfies $X_b^2=X_a^2$ almost surely. A bivariate normal vector with identity covariance would instead have a density and assign probability zero to those two lines. The correction is to test joint normality of every finite vector, not merely normality of its coordinates or a plausible-looking covariance matrix.

When reading or constructing a Gaussian process, state its index set explicitly, identify the common probability space on which its variables are defined, and check the joint law requirement. Then compute $m(t)$ and $K(s,t)$ with the indices kept visible. This prevents confusion between the process as a family of random variables and any one finite-dimensional vector extracted from it.

Check Your Understanding

For each question, distinguish the joint-distribution requirement from statements about individual coordinates.

What must be true of $(X_{t_1},\ldots,X_{t_n})^\mathsf{T}$ for every finite selection of indices in a Gaussian process?
For the three-index process built from $Z$ and $W$, verify directly that $X_2=3X_{-1}-X_0$.
In the Brownian-motion example, why is the vector of values obtained by cumulative sums of the increments?
Can a Gaussian process have a singular covariance matrix for a finite vector of values? Give the dependence relation from the worked example that demonstrates this.
Why do normal marginal distributions and zero covariance fail to establish joint Gaussianity in the two-index counterexample?

Gaussian Processes and Finite-Dimensional Distributions

What You'll Learn