Invertible Linear Transformations of Random Vectors

Why Invertible Linear Maps Are Useful

A vector of measurements is often re-expressed through linear combinations: two measurements may be replaced by their sum and difference, or several coordinates may be changed by a rotation, rescaling, or shear. When the transformation is invertible, no information about the vector is lost. Its distribution can be calculated by mapping each transformed point back to the original coordinates and accounting for the change in volume.

This tutorial treats the affine transformation $Y=AX+b$, where $X$ and $Y$ are $p$-dimensional random vectors, $A$ is a fixed nonsingular $p\times p$ matrix, and $b$ is a fixed vector. The key ingredients are the inverse map $x=A^{-1}(y-b)$ and the constant volume factor $|\det A|$.

Density and Support Under an Invertible Affine Map

Definition: Let $X$ be a random vector in $\mathbb{R}^p$ with density $f_X$ with respect to $p$-dimensional Lebesgue measure. For a nonsingular matrix $A$ and fixed $b\in\mathbb{R}^p$, define $Y=AX+b$. The inverse transformation is $x=A^{-1}(y-b)$, and the density of $Y$ is $$ f_Y(y)=\frac{1}{|\det A|}f_X\!\left(A^{-1}(y-b)\right). $$ This equality is understood almost everywhere; a density’s values on a set of Lebesgue measure zero may be chosen arbitrarily.

The determinant factor is constant because the transformation is linear apart from the translation $b$. A small region in $x$-space has its volume multiplied by $|\det A|$ when mapped into $y$-space. Consequently, the density per unit volume is divided by that factor.

The formula follows directly from the multivariate change-of-variables theorem for a differentiable bijection. For any measurable set $C\subseteq\mathbb{R}^p$,

\begin{aligned} P(Y\in C) &=P(AX+b\in C)\\ &=\int_{\{x:Ax+b\in C\}} f_X(x)\,dx\\ &=\int_C f_X\!\left(A^{-1}(y-b)\right) \left|\det(A^{-1})\right|\,dy. \end{aligned}

Since $\left|\det(A^{-1})\right|=1/|\det A|$, the integrand is the stated density. Nonsingularity matters: it ensures that the inverse exists and that the transformed vector remains $p$-dimensional with a density whenever $X$ has one.

The support also transforms simply, provided “support” means topological support: the set of points every open neighborhood of which has positive probability. If $S$ is the topological support of $X$, then the topological support of $Y$ is $AS+b=\{Ax+b:x\in S\}$. This statement concerns the support as a closed set; it is not a claim that the density must be positive at every point of that set.

Theorem: If $X$ has topological support $S$, and $Y=AX+b$ with $A$ nonsingular, then $Y$ has topological support $AS+b$.

Proof. The map $T(x)=Ax+b$ is a homeomorphism: both $T$ and its inverse $T^{-1}(y)=A^{-1}(y-b)$ are continuous. Fix $x\in S$, and let $V$ be any open neighborhood of $T(x)$. The inverse image $T^{-1}(V)$ is an open neighborhood of $x$, so $P(X\in T^{-1}(V))>0$. Therefore $P(Y\in V)>0$, showing that $T(x)$ is in the support of $Y$.

Conversely, if $y\notin T(S)$, then $T^{-1}(y)\notin S$. By the definition of support, there is an open neighborhood $U$ of $T^{-1}(y)$ with $P(X\in U)=0$. The image $T(U)$ is an open neighborhood of $y$, and $P(Y\in T(U))=0$. Thus $y$ is not in the support of $Y$. These two inclusions prove the result. $\square$

In calculations, it is often convenient to describe the transformed support by applying the inverse map: $y$ belongs to $AS+b$ exactly when $A^{-1}(y-b)$ belongs to $S$. Boundary inequalities must reflect the support being transformed. For example, the topological support of a uniform distribution on an open rectangle’s interior includes the rectangle’s boundary, even though the density is positive only in the interior.

Means and Covariance Matrices

Theorem: Suppose $E[\|X\|^2]<\infty$, and write $\mu_X=E[X]$ and $\Sigma_X=\operatorname{Cov}(X)$. For $Y=AX+b$, $$ E[Y]=A\mu_X+b,\qquad \operatorname{Cov}(Y)=A\Sigma_XA^{T}. $$

Proof. The finite second-moment condition ensures that the means and covariance matrices in these expressions exist. By linearity of expectation,

E[Y]=E[AX+b]=AE[X]+b=A\mu_X+b.

Center $Y$ by its mean. Then $Y-E[Y]=A(X-\mu_X)$. Using the definition $\operatorname{Cov}(Y)=E[(Y-E[Y])(Y-E[Y])^T]$ gives

\begin{aligned} \operatorname{Cov}(Y) &=E\!\left[A(X-\mu_X)(X-\mu_X)^TA^T\right]\\ &=A\,E\!\left[(X-\mu_X)(X-\mu_X)^T\right]A^T\\ &=A\Sigma_XA^T. \end{aligned}

The fixed translation changes the mean but not the covariance. The matrix $A$ mixes and rescales coordinates, which is why it acts on the left of the covariance matrix and $A^T$ acts on the right. $\square$

Worked Example: Sum and Difference of Two Uniform Variables

Let $X=(X_1,X_2)^T$, where $X_1$ and $X_2$ are independent Uniform$(0,1)$ variables. Define $Y=AX$, with $A=\begin{pmatrix}1&1\\1&-1\end{pmatrix}$. Find the density, topological support, mean, and covariance of $Y$.

The determinant is $\det A=-2$, so $A$ is nonsingular and $|\det A|=2$. Writing $Y=(U,V)^T$, the transformation is $u=x_1+x_2$, $v=x_1-x_2$. Solving these equations gives

A^{-1}=\frac12 \begin{pmatrix}1&1\\1&-1\end{pmatrix}, \qquad x_1=\frac{u+v}{2},\quad x_2=\frac{u-v}{2}.

The original density is 1 on the unit square and 0 outside it. Thus the transformed density is $1/2$ when both inverse coordinates are in $[0,1]$, and 0 otherwise. The topological support is specified by

0\le \frac{u+v}{2}\le 1, \qquad 0\le \frac{u-v}{2}\le 1,

or equivalently $0\le u+v\le2$ and $0\le u-v\le2$. The inequalities are non-strict because they describe the image of the closed topological support $[0,1]^2$. The density equals $1/2$ in the interior of this parallelogram and can be assigned any values on its boundary without changing the distribution.

As a check, the parallelogram has area $|\det A|\cdot1=2$; density $1/2$ over that area integrates to 1. Also, $E[X]=(1/2,1/2)^T$, so $E[Y]=A E[X]=(1,0)^T$. Independence gives $\Sigma_X=(1/12)I$, and therefore

\operatorname{Cov}(Y) =\frac{1}{12}AA^T =\frac{1}{12} \begin{pmatrix}2&0\\0&2\end{pmatrix} =\begin{pmatrix}1/6&0\\0&1/6\end{pmatrix}.

The sum and difference are uncorrelated here, as the zero off-diagonal entries show.

Worked Example: Transforming Independent Exponentials

Let $X_1,X_2$ be independent exponential random variables with rate 1, so their joint density is $e^{-x_1-x_2}$ for $x_1>0,\ x_2>0$, and zero otherwise. Set $Y=(U,V)^T$, where $U=2X_1+X_2$ and $V=X_1+X_2$.

Here $A=\begin{pmatrix}2&1\\1&1\end{pmatrix}$, with determinant $2(1)-1(1)=1$. Its inverse is

A^{-1}= \begin{pmatrix}1&-1\\-1&2\end{pmatrix}, \qquad x_1=u-v,\quad x_2=2v-u.

The inverse coordinates are both positive precisely when $u>v$ and $u<2v$. These conditions imply $v>0$, and the image of the open positive quadrant is $v<u<2v$. The topological support, however, is the closure:

v\ge0,\qquad v\le u\le2v.

Because $|\det A|=1$, the density in the interior is

f_Y(u,v) =e^{-(u-v)-(2v-u)} =e^{-v},\qquad v<u<2v,

and it is zero outside that region, up to arbitrary choices on the boundary. The exponent simplifies to $-v$ because $(u-v)+(2v-u)=v$. In particular, the density is not constant on its support.

For a normalization check, the change of variables has determinant magnitude 1, and the integral over the density-positive region is

\int_0^\infty\int_v^{2v}e^{-v}\,du\,dv =\int_0^\infty v e^{-v}\,dv =1.

The original mean is $(1,1)^T$, and its covariance matrix is $I$. Hence $E[Y]=(3,2)^T$, and

\operatorname{Cov}(Y)=AA^T =\begin{pmatrix}5&3\\3&2\end{pmatrix}.

Worked Example: A Shifted Linear Transformation of a Rectangle

Suppose $X_1\sim\operatorname{Uniform}(-1,1)$ and $X_2\sim\operatorname{Uniform}(0,2)$, independently. Define

Y=AX+b,\qquad A=\begin{pmatrix}1&2\\3&1\end{pmatrix}, \qquad b=\begin{pmatrix}1\\-1\end{pmatrix}.

The determinant is $1-6=-5$, so $|\det A|=5$, and direct inversion gives

A^{-1}=-\frac15 \begin{pmatrix}1&-2\\-3&1\end{pmatrix} =\begin{pmatrix}-1/5&2/5\\3/5&-1/5\end{pmatrix}.

For $y=(y_1,y_2)^T$, first subtract $b$, then apply the inverse:

x_1=\frac{-(y_1-1)+2(y_2+1)}5,\qquad x_2=\frac{3(y_1-1)-(y_2+1)}5.

The original topological support is $[-1,1]\times[0,2]$. Thus the transformed topological support is exactly the set of $y$ satisfying $-1\le x_1\le1$ and $0\le x_2\le2$ for the inverse coordinates above. The original density is $1/4$ on the rectangle, whose area is 4. The transformed density is therefore $1/(4\cdot5)=1/20$ in the image’s interior, and zero outside, with boundary values immaterial.

The image has area $5\cdot4=20$, so its constant density integrates to $20/20=1$. The original mean is $(0,1)^T$, giving $E[Y]=A(0,1)^T+b=(3,0)^T$. Both original variances are $1/3$, and independence gives $\Sigma_X=(1/3)I$. Therefore

\operatorname{Cov}(Y) =\frac13AA^T =\frac13 \begin{pmatrix}5&5\\5&10\end{pmatrix} =\begin{pmatrix}5/3&5/3\\5/3&10/3\end{pmatrix}.

The off-diagonal covariance is positive because the two transformed coordinates both increase with $X_1$ and $X_2$.

Common Pitfall: Strict Inequalities and Support

A frequent error is to use strict inequalities when describing the topological support of a transformed continuous vector. For example, since a Uniform$(0,1)^2$ density is positive on the open square, one might conclude that its topological support is only $(0,1)^2$, then map only that open square. This confuses a region where a particular density is positive with the support of the probability distribution.

The correction is to identify the original topological support first. For the uniform vector it is $[0,1]^2$, since every neighborhood of every boundary point intersects the square in a region of positive probability. Its image under an invertible affine map is the corresponding closed parallelogram. Strict inequalities describe the image of the open square, or the interior where the transformed density is positive; non-strict inequalities describe the topological support.

A second useful check is normalization: the determinant in the density formula is in the denominator, and the transformed support’s volume is multiplied by $|\det A|$. Reversing the determinant factor would make the density integrate to $|\det A|^2$ rather than 1 in a constant-density example.

Key takeaway: For $Y=AX+b$ with nonsingular $A$, invert the map, evaluate the original density at $A^{-1}(y-b)$, and divide by $|\det A|$. Map the topological support as a set, retaining its boundary, and use $A\Sigma_XA^T$ for covariance when second moments exist.

Check Your Understanding

For each question, distinguish the density-positive region from topological support when boundaries matter.

If $A$ is a nonsingular $p\times p$ matrix, what determinant factor appears in the density of $AX+b$, and why?
A random vector has topological support $[0,1]^2$. What set should be transformed to find the topological support after an invertible affine map?
For the exponential example, verify from the inverse coordinates that the density-positive region satisfies $v<u<2v$.
If $E[\|X\|^2]<\infty$, write the transformed mean and covariance for $Y=AX+b$. Which expression depends on $b$?
Why can a density be zero on part of a topological support without contradicting the definition of support?

Invertible Linear Transformations of Random Vectors

What You'll Learn