Means and Covariances of Gaussian Vectors

Why Means and Covariances Are Encoded in the Characteristic Function

For a Gaussian vector, the characteristic function gives more than a compact description of its distribution. Its linear term records the component means, while its quadratic term records the variances and covariances. Recovering these quantities carefully requires distinguishing derivatives of the characteristic function from derivatives of its logarithm: they do not encode exactly the same moments when the mean is nonzero.

Let \(X=(X_1,\ldots,X_p)^{\mathsf T}\) be a Gaussian random vector with characteristic function \(\phi_X(t)=E[\exp(i t^{\mathsf T}X)]\), where \(t\in\mathbb R^p\). In the Gaussian form, this function is \[ \phi_X(t)=\exp\left(i t^{\mathsf T}\mu-\frac12 t^{\mathsf T}\Sigma t\right), \] where \(\mu\in\mathbb R^p\) and \(\Sigma\) is a symmetric matrix. The goal is to show that these parameters are the actual mean vector and covariance matrix of \(X\).

Definition: For a random vector \(X=(X_1,\ldots,X_p)^{\mathsf T}\) with finite second moments, its mean vector has entries \(E[X_j]\), and its covariance matrix has entries \(\operatorname{Cov}(X_j,X_k)=E[(X_j-E[X_j])(X_k-E[X_k])]\). The diagonal entries are component variances; the off-diagonal entries are pairwise covariances.

Deriving the Moment Formulas

Differentiating an expectation under its integral or expectation sign requires moment conditions. Here those conditions follow from the Gaussian characteristic function itself. First restrict the characteristic function to coordinate \(j\), by setting \(t=s e_j\), where \(e_j\) is the \(j\)th coordinate vector. Then \(X_j\) has characteristic function \(\exp(i\mu_j s-\Sigma_{jj}s^2/2)\).

For any real random variable \(Y\) with characteristic function \(\phi_Y\), \(E[1-\cos(sY)]=1-\operatorname{Re}\phi_Y(s)\). Also, for each fixed value of \(Y\), \((1-\cos(sY))/s^2\) tends to \(Y^2/2\) as \(s\) tends to zero. Fatou's lemma and the expansion of the Gaussian characteristic function therefore give

\frac12 E[X_j^2] \leq \liminf_{s\to 0}\frac{1-\operatorname{Re}\phi_{X_j}(s)}{s^2} =\frac{\mu_j^2+\Sigma_{jj}}{2}<\infty.

Theorem: If \(X\) has characteristic function \(\phi_X(t)=\exp(i t^{\mathsf T}\mu-\tfrac12 t^{\mathsf T}\Sigma t)\), then \(E[X]=\mu\) and \(\operatorname{Cov}(X)=\Sigma\). More explicitly, \[ \left.\frac{\partial \phi_X}{\partial t_j}\right|_{t=0}=iE[X_j]=i\mu_j,\qquad \left.\frac{\partial^2\phi_X}{\partial t_j\partial t_k}\right|_{t=0} =-E[X_jX_k]=-\Sigma_{jk}-\mu_j\mu_k. \]

Proof. The moment bounds above allow differentiation under the expectation: \[ \frac{\partial\phi_X(t)}{\partial t_j} =E[iX_j\exp(i t^{\mathsf T}X)],\qquad \frac{\partial^2\phi_X(t)}{\partial t_j\partial t_k} =E[-X_jX_k\exp(i t^{\mathsf T}X)]. \] At \(t=0\), these derivatives are \(iE[X_j]\) and \(-E[X_jX_k]\), respectively. On the other hand, differentiating the stated Gaussian form gives \[ \frac{\partial\phi_X(t)}{\partial t_j} =(i\mu_j-(\Sigma t)_j)\phi_X(t) \] and \[ \frac{\partial^2\phi_X(t)}{\partial t_j\partial t_k} =\left[-\Sigma_{jk} +(i\mu_j-(\Sigma t)_j)(i\mu_k-(\Sigma t)_k)\right]\phi_X(t). \] Since \(\phi_X(0)=1\), evaluating at zero yields \(i\mu_j\) and \(-\Sigma_{jk}-\mu_j\mu_k\). Equating the two expressions for each derivative gives \(E[X_j]=\mu_j\) and \(E[X_jX_k]=\Sigma_{jk}+\mu_j\mu_k\). Therefore \[ \operatorname{Cov}(X_j,X_k) =E[X_jX_k]-E[X_j]E[X_k] =\Sigma_{jk}. \] This holds for every pair of indices, proving both vector identities. \(\square\)

The calculation also gives a useful second derivative result for the logarithm of the characteristic function. Because \(\phi_X(0)=1\) and \(\phi_X\) is continuous, \(\phi_X\) is nonzero in some neighborhood of zero, so a continuous local logarithm can be used there.

Result: At the origin, the Hessian matrix of \(\log\phi_X(t)\) is the negative covariance matrix: \[ \nabla^2\log\phi_X(0)=-\operatorname{Cov}(X). \]

Derivation. For indices \(j,k\), the second derivative of a logarithm is \[ \frac{\partial^2\log\phi_X}{\partial t_j\partial t_k} =\frac{\partial^2\phi_X/\partial t_j\partial t_k}{\phi_X} -\frac{(\partial\phi_X/\partial t_j)(\partial\phi_X/\partial t_k)}{\phi_X^2}. \] At zero, \(\phi_X(0)=1\), the first derivatives are \(i\mu_j\) and \(i\mu_k\), and the mixed second derivative is \(-\Sigma_{jk}-\mu_j\mu_k\). Hence the displayed expression is \[ -\Sigma_{jk}-\mu_j\mu_k-(i\mu_j)(i\mu_k)=-\Sigma_{jk}. \] The \(\mu_j\mu_k\) terms cancel, proving the result entry by entry. \(\square\)

Worked Calculations

Worked Example: Reading a Bivariate Gaussian Characteristic Function

Suppose \(X=(X_1,X_2)^{\mathsf T}\) has characteristic function \[ \phi_X(t_1,t_2) =\exp\left\{i(2t_1-t_2)-\frac12(4t_1^2+6t_1t_2+9t_2^2)\right\}. \] Identify both component means, both variances, and their covariance.

The linear term is \(i(2t_1-t_2)=i t^{\mathsf T}\mu\), so \[ \mu=\begin{pmatrix}2\\-1\end{pmatrix}. \] The quadratic expression must equal \(t^{\mathsf T}\Sigma t\). Since \(t^{\mathsf T}\Sigma t=\Sigma_{11}t_1^2+2\Sigma_{12}t_1t_2+\Sigma_{22}t_2^2\), matching coefficients gives \[ \Sigma=\begin{pmatrix}4&3\\3&9\end{pmatrix}. \] In particular, \(2\Sigma_{12}=6\), so the covariance is \(3\), not \(6\). The component variances are \(\operatorname{Var}(X_1)=4\) and \(\operatorname{Var}(X_2)=9\); their means are \(2\) and \(-1\). As a check, the correlation is \(3/(\sqrt{4}\sqrt{9})=3/6=0.5\).

Worked Example: Interpreting Covariances in Three Dimensions

Consider the characteristic function \[ \phi_X(t)=\exp\left\{i(t_1-2t_2)-\frac12\left[(t_1+t_2)^2+(t_2+t_3)^2\right]\right\}. \] Find the mean vector and covariance matrix, and explain what the off-diagonal entries say.

The linear term gives \[ E[X]=\mu=\begin{pmatrix}1\\-2\\0\end{pmatrix}. \] To read the quadratic term, expand it: \[ (t_1+t_2)^2+(t_2+t_3)^2 =t_1^2+2t_1t_2+2t_2^2+2t_2t_3+t_3^2. \] Matching this with \(t^{\mathsf T}\Sigma t\), where each off-diagonal entry appears twice, gives \[ \Sigma=\begin{pmatrix}1&1&0\\1&2&1\\0&1&1\end{pmatrix}. \] Thus the variances of \(X_1,X_2,X_3\) are \(1,2,1\), respectively. The covariance of \(X_1\) and \(X_2\) is \(1\); the covariance of \(X_2\) and \(X_3\) is also \(1\). The covariance of \(X_1\) and \(X_3\) is zero. These entries describe pairwise linear association in the sense of covariance; a zero covariance does not, by itself, mean that arbitrary random variables are independent.

Worked Example: Recovering a Raw Second Moment

Suppose a bivariate Gaussian vector has characteristic function \[ \phi_X(t_1,t_2) =\exp\left\{i(-t_1+3t_2)-\frac12(4t_1^2-2t_1t_2+t_2^2)\right\}. \] Find \(E[X_1X_2]\) and verify it using the covariance definition.

The mean vector is \(\mu=(-1,3)^{\mathsf T}\). Comparing the quadratic term with \(t^{\mathsf T}\Sigma t\) gives \[ \Sigma=\begin{pmatrix}4&-1\\-1&1\end{pmatrix}, \] because the coefficient of \(t_1t_2\) is \(2\Sigma_{12}=-2\). The raw cross-moment formula from the theorem is \[ E[X_1X_2]=\Sigma_{12}+\mu_1\mu_2=-1+(-1)(3)=-4. \] To check, expand the definition: \[ \operatorname{Cov}(X_1,X_2) =E[(X_1+1)(X_2-3)] =E[X_1X_2]-3E[X_1]+E[X_2]-3. \] Substituting \(E[X_1]=-1\) and \(E[X_2]=3\) makes the right side \(E[X_1X_2]+3+3-3=E[X_1X_2]+3\). Since the covariance is \(-1\), it follows that \(E[X_1X_2]=-4\), as before.

How to Read the Quadratic Term

The covariance matrix is symmetric. Its \(j\)th diagonal entry is the variance of \(X_j\), and its \((j,k)\) entry is the covariance of \(X_j\) and \(X_k\). When expanding \(t^{\mathsf T}\Sigma t\), this symmetry is why each off-diagonal covariance appears with a factor of two: \[ t^{\mathsf T}\Sigma t =\sum_{j=1}^p\Sigma_{jj}t_j^2 +2\sum_{1\leq j<k\leq p}\Sigma_{jk}t_jt_k. \] A reliable coefficient-reading procedure is to identify the coefficient of \(t_j^2\) directly as \(\Sigma_{jj}\), and divide the coefficient of \(t_jt_k\) by two to obtain \(\Sigma_{jk}\).

There are two equivalent ways to read the covariance information. In the Gaussian exponent, the matrix multiplying the quadratic form is \(\Sigma\). In the second derivatives of the characteristic function itself, the result is instead \(-E[X_jX_k]=-\Sigma_{jk}-\mu_j\mu_k\). In the second derivatives of its logarithm, the result is \(-\Sigma_{jk}\). The logarithm removes the contribution of the nonzero mean.

Common Mistake

Wrong reasoning: “The second derivative of the characteristic function at zero is minus the covariance.” This overlooks that differentiating the characteristic function gives \(-E[X_jX_k]\), a raw second moment with a negative sign.
Correction: Subtract the product of the means to obtain covariance: \(\operatorname{Cov}(X_j,X_k)=E[X_jX_k]-\mu_j\mu_k\). Equivalently, the Hessian of the local logarithm of the characteristic function at zero is \(-\Sigma\).
Another coefficient trap: The coefficient of \(t_jt_k\) in \(t^{\mathsf T}\Sigma t\) is \(2\Sigma_{jk}\), not \(\Sigma_{jk}\).

These distinctions matter when translating a characteristic function into parameters for a statistical model or checking a derivation. The linear term identifies location, and the quadratic term identifies covariance; keeping track of which function is being differentiated prevents mixing centered and uncentered moments.

Check Your Understanding

Use the derivative formulas and coefficient-matching rule to answer the following.

If the linear term in a Gaussian characteristic function is \(i(4t_1-t_2)\), what are the component means?
In \(t^{\mathsf T}\Sigma t\), what is the coefficient of \(t_1t_2\) in terms of \(\Sigma_{12}\)?
If \(E[X_1]=2\), \(E[X_2]=-1\), and \(\operatorname{Cov}(X_1,X_2)=3\), what is \(E[X_1X_2]\)?
What does the \((j,k)\) entry of the Hessian of \(\log\phi_X(t)\) at zero equal?
Why does a second derivative of \(\phi_X\), rather than of its logarithm, include a product of component means?