Affine Equivariance of Gaussian Location and Scatter Estimates

Why Coordinate Changes Matter

A Gaussian model can describe the same observations in different coordinates: lengths may be converted from one unit to another, or measurements may be rotated or combined into new coordinates. A sound estimate should respond to such a change in a predictable way. In particular, changing the coordinates should not alter the underlying fitted model; it should change the reported location and scatter to match the new coordinates.

This property is called affine equivariance. Here we derive it for the unrestricted Gaussian maximum-likelihood estimates. The key assumptions are that the observations are transformed by an invertible affine map and that the Gaussian covariance MLE exists.

Setup and Definitions

Let \(X_1,\ldots,X_n\) be independent observations in \(\mathbb{R}^p\), each with distribution \(N(\mu,\Sigma)\), where \(\mu\in\mathbb{R}^p\) and \(\Sigma\) is positive definite. Write \(x_1,\ldots,x_n\) for the observed vectors and define their sample mean by

\bar{x}=\frac{1}{n}\sum_{i=1}^n x_i.

When the unrestricted Gaussian covariance MLE exists, the maximum-likelihood estimates are

\hat{\mu}=\bar{x}, \qquad \hat{\Sigma}=\frac{1}{n}\sum_{i=1}^n (x_i-\bar{x})(x_i-\bar{x})^T.

The divisor is \(n\), not \(n-1\), because this is the maximum-likelihood estimate. It is useful to call \(\hat{\Sigma}\) the MLE scatter as well as the MLE covariance: it describes dispersion in the coordinates of the observations.

Definition: An estimate of location and scatter is affine equivariant if, after replacing every observation \(x_i\) by \(z_i=Ax_i+b\), its location estimate becomes \(A\hat{\mu}+b\) and its scatter estimate becomes \(A\hat{\Sigma}A^T\). Here \(A\) is an invertible \(p\times p\) matrix and \(b\in\mathbb{R}^p\).

For the unrestricted Gaussian MLE, existence requires the centered observations to span \(\mathbb{R}^p\), or equivalently that \(\hat{\Sigma}\) be positive definite. In particular, this requires \(n\geq p+1\). If the scatter is singular, the unrestricted Gaussian likelihood has no finite maximum over positive-definite covariance matrices.

How the Gaussian Likelihood Changes

Consider the transformed observations \(z_i=Ax_i+b\), with \(A\) invertible. If the original Gaussian parameters are \((\mu,\Sigma)\), the corresponding parameters in the transformed coordinates are

\mu_z=A\mu+b, \qquad \Sigma_z=A\Sigma A^T.

The parameter map is one-to-one: given \((\mu_z,\Sigma_z)\), recover \(\mu=A^{-1}(\mu_z-b)\) and \(\Sigma=A^{-1}\Sigma_z(A^{-1})^T\). The following likelihood identity makes the effect of the coordinate change precise.

Result: For a fixed invertible \(A\) and fixed \(b\), the Gaussian log-likelihood of the transformed observations at \((\mu_z,\Sigma_z)=(A\mu+b,A\Sigma A^T)\) is the original log-likelihood minus \(n\log|\det A|\).

Proof. The multivariate Gaussian log-likelihood for observations \(x_1,\ldots,x_n\) is

\ell_X(\mu,\Sigma) =-\frac{np}{2}\log(2\pi)-\frac{n}{2}\log|\Sigma| -\frac{1}{2}\sum_{i=1}^n(x_i-\mu)^T\Sigma^{-1}(x_i-\mu).

Since \(\Sigma_z=A\Sigma A^T\), its determinant satisfies \(|\Sigma_z|=|\det A|^2|\Sigma|\). Also, \(\Sigma_z^{-1}=(A^{-1})^T\Sigma^{-1}A^{-1}\), and \(z_i-\mu_z=A(x_i-\mu)\). Thus each transformed quadratic term is

(z_i-\mu_z)^T\Sigma_z^{-1}(z_i-\mu_z) =(x_i-\mu)^T\Sigma^{-1}(x_i-\mu).

The difference is constant in the parameters. Because the parameter map is a bijection, maximizing one likelihood is equivalent to maximizing the other and mapping the maximizer through that bijection. This is the likelihood explanation for the transformation of the MLEs; the sample formulas also verify it directly.

Transformation of the Gaussian MLEs

Theorem: Suppose the unrestricted Gaussian MLE exists for \(x_1,\ldots,x_n\), and let \(z_i=Ax_i+b\) for invertible \(A\). Then the transformed data have unrestricted Gaussian MLEs \(\hat{\mu}_z=A\hat{\mu}+b\) and \(\hat{\Sigma}_z=A\hat{\Sigma}A^T\).

Proof. The sample mean of the transformed observations is

\bar{z} =\frac{1}{n}\sum_{i=1}^n(Ax_i+b) =A\left(\frac{1}{n}\sum_{i=1}^n x_i\right)+b =A\bar{x}+b.

For each observation, \(z_i-\bar{z}=A(x_i-\bar{x})\). Therefore the transformed MLE scatter is

\begin{aligned} \hat{\Sigma}_z &=\frac{1}{n}\sum_{i=1}^n(z_i-\bar{z})(z_i-\bar{z})^T\\ &=\frac{1}{n}\sum_{i=1}^n A(x_i-\bar{x})(x_i-\bar{x})^TA^T\\ &=A\hat{\Sigma}A^T. \end{aligned}

Finally, \(\hat{\Sigma}\) is positive definite by the existence assumption, so \(A\hat{\Sigma}A^T\) is positive definite for invertible \(A\). The transformed-data MLE therefore exists, and the displayed sample formulas give its location and scatter estimates. \(\square\)

Worked Calculations

Worked Example: Changing the Measurement Units

Four two-dimensional observations are \(x_1=(0,0)^T\), \(x_2=(2,0)^T\), \(x_3=(0,2)^T\), and \(x_4=(2,2)^T\). Rescale the first coordinate by \(10\), the second by \(0.1\), and translate by \(b=(5,-3)^T\). Find the transformed MLEs.

The original mean is \(\bar{x}=(1,1)^T\). The four centered observations are \((-1,-1)^T,(1,-1)^T,(-1,1)^T,(1,1)^T\). Their outer products sum to \(\left(\begin{smallmatrix}4&0\\0&4\end{smallmatrix}\right)\), so, using divisor \(n=4\), \(\hat{\Sigma}=I_2\).

The transformation matrix is \(A=\left(\begin{smallmatrix}10&0\\0&0.1\end{smallmatrix}\right)\). Thus

\hat{\mu}_z=A\bar{x}+b =\begin{pmatrix}10&0\\0&0.1\end{pmatrix} \begin{pmatrix}1\\1\end{pmatrix} +\begin{pmatrix}5\\-3\end{pmatrix} =\begin{pmatrix}15\\-2.9\end{pmatrix},

and

\hat{\Sigma}_z=A I_2 A^T =\begin{pmatrix}100&0\\0&0.01\end{pmatrix}.

The translation changes location but not scatter. The two diagonal entries of scatter change by the squares of the corresponding scale factors, as covariance units require.

Worked Example: A Shear Creates Covariance

Now take observations \((1,1)^T,(3,1)^T,(1,3)^T,(3,3)^T\). Their mean is \((2,2)^T\); the centered vectors are the same four sign combinations as in the previous calculation, so \(\hat{\Sigma}=I_2\). Apply \(A=\left(\begin{smallmatrix}1&1\\0&2\end{smallmatrix}\right)\) and \(b=(-1,0)^T\).

The matrix is invertible because \(\det A=2\). The transformed mean is

\hat{\mu}_z=A\begin{pmatrix}2\\2\end{pmatrix} +\begin{pmatrix}-1\\0\end{pmatrix} =\begin{pmatrix}3\\4\end{pmatrix}.

The transformed scatter is

\hat{\Sigma}_z=A I_2 A^T =\begin{pmatrix}1&1\\0&2\end{pmatrix} \begin{pmatrix}1&0\\1&2\end{pmatrix} =\begin{pmatrix}2&2\\2&4\end{pmatrix}.

Although the original coordinates had zero estimated covariance, the transformed coordinates have estimated covariance \(2\). The shear combines the original coordinates, so they are no longer uncorrelated. The off-diagonal entries are a consequence of the coordinate transformation, not a failure of equivariance.

Worked Example: Checking the Likelihood Jacobian

Use the four observations \((0,0)^T,(2,0)^T,(0,2)^T,(2,2)^T\), for which \(\hat{\mu}=(1,1)^T\) and \(\hat{\Sigma}=I_2\). Transform them using \(A=\left(\begin{smallmatrix}2&0\\0&3\end{smallmatrix}\right)\) and \(b=(1,-1)^T\). Compare the maximized log-likelihoods.

Here \(|\det A|=6\). The transformed MLEs are \(\hat{\mu}_z=(3,2)^T\) and \(\hat{\Sigma}_z=\left(\begin{smallmatrix}4&0\\0&9\end{smallmatrix}\right)\), with \(|\hat{\Sigma}_z|=36\). At the MLE, the sum of the four quadratic terms is \(np=8\): for the original data each squared distance is \(2\), and the same quadratic values hold after transforming both data and covariance.

For \(n=4,p=2\), the original maximized log-likelihood is \(\ell_X(\hat{\mu},\hat{\Sigma})=-4\log(2\pi)-2\log|I_2|-4=-4\log(2\pi)-4\). The transformed maximum is

\ell_Z(\hat{\mu}_z,\hat{\Sigma}_z) =-4\log(2\pi)-2\log(36)-4 =\ell_X(\hat{\mu},\hat{\Sigma})-4\log(6).

This agrees with the likelihood transformation, which subtracts \(n\log|\det A|=4\log 6\). The fitted parameters transform with the data, while the numerical density changes because the coordinate system has a different volume scale.

Interpretation and a Common Pitfall

Affine equivariance is a consistency property: estimates expressed in new coordinates agree with estimates transformed from the original coordinates. It is useful when variables have different units or when a multivariate analysis uses a new linear coordinate system. For any invertible transformation, the location estimate follows the same affine map as the data, and the scatter estimate follows the associated congruence transformation.

A common mistake is to say that the covariance estimate is invariant and therefore should remain numerically unchanged. That is wrong: invariance would mean \(\hat{\Sigma}_z=\hat{\Sigma}\), which is generally false. Under rescaling by \(A\), the correct rule is \(\hat{\Sigma}_z=A\hat{\Sigma}A^T\). A second error is to use \(A\hat{\Sigma}A^{-1}\); covariance matrices transform with a transpose, not an inverse. For example, if \(A\) is diagonal with entries \(c_1,c_2\), the variance in coordinate \(j\) is multiplied by \(c_j^2\), as the formula \(A\hat{\Sigma}A^T\) gives.

Invertibility matters. If \(A\) is singular, the transformed observations lie in a lower-dimensional subspace and \(A\Sigma A^T\) is singular even when \(\Sigma\) is positive definite. The one-to-one parameter correspondence and the unrestricted positive-definite Gaussian covariance model used in the proof no longer apply.

Key takeaway: Under \(z_i=Ax_i+b\) with invertible \(A\), the Gaussian MLEs satisfy \(\hat{\mu}_z=A\hat{\mu}+b\) and \(\hat{\Sigma}_z=A\hat{\Sigma}A^T\), provided the unrestricted covariance MLE exists. Location transforms affinely; scatter transforms by congruence.

Check Your Understanding

Use the transformation rules and likelihood identity to answer the following.

For \(z_i=3x_i\) in one dimension, by what factor does the Gaussian MLE variance change?
Why does translating every observation by \(b\) leave the MLE scatter unchanged?
If \(A\) is invertible and \(\hat{\Sigma}\) is positive definite, explain why \(A\hat{\Sigma}A^T\) is positive definite.
For an invertible affine transformation, what constant is added to the log-likelihood of the transformed data relative to the original log-likelihood?
Why is using \(A\hat{\Sigma}A^{-1}\) generally an incorrect covariance transformation?