Determinant and Orientation of Orthogonal Matrices

Orthogonality Restricts the Determinant

An orthogonal matrix preserves inner products and lengths, but it may or may not preserve orientation. The determinant distinguishes these possibilities. In this tutorial, matrices and vectors are real, and an orthogonal matrix means a square matrix \(\mathbf{Q}\in\mathbb{R}^{n\times n}\) satisfying \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}_n\). The central fact is that its determinant cannot have an arbitrary magnitude or sign: it must be exactly \(1\) or \(-1\).

We use the standard determinant identities \(\det(\mathbf{A}\mathbf{B})=\det(\mathbf{A})\det(\mathbf{B})\), \(\det(\mathbf{A}^{\top})=\det(\mathbf{A})\), and \(\det(\mathbf{I}_n)=1\). We also use the determinant criterion that a square matrix is invertible exactly when its determinant is nonzero. These identities apply to square matrices of the same size where a product is taken.

Proof. Since \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}_n\), take determinants of both sides and use the determinant identities stated above:

Thus \((\det(\mathbf{Q})-1)(\det(\mathbf{Q})+1)=0\). Because the determinant is real, this implies \(\det(\mathbf{Q})=1\) or \(\det(\mathbf{Q})=-1\). Either value is nonzero, so the determinant criterion gives that \(\mathbf{Q}\) is invertible. \(\square\)

The argument gives a square of the determinant, not the determinant itself. That distinction is essential: both signs satisfy the equation \(\det(\mathbf{Q})^2=1\). The orthogonality equation rules out every other determinant, but it does not select between \(1\) and \(-1\).

Orientation of an Ordered Basis

Orientation is a property of an ordered basis. To define it in \(\mathbb{R}^n\), write an ordered basis as \(\mathcal{B}=(\mathbf{b}_1,\dots,\mathbf{b}_n)\) and form its basis matrix \(\mathbf{B}=[\mathbf{b}_1\ \cdots\ \mathbf{b}_n]\). Since the vectors are a basis, \(\det(\mathbf{B})\ne0\). We call \(\mathcal{B}\) positively oriented when \(\det(\mathbf{B})>0\), and negatively oriented when \(\det(\mathbf{B})<0\). The standard ordered basis \((\mathbf{e}_1,\dots,\mathbf{e}_n)\) is positively oriented because its basis matrix is \(\mathbf{I}_n\), whose determinant is \(1\).

Swapping two vectors in an ordered basis reverses its orientation: swapping two columns changes the sign of the determinant. This illustrates why orientation belongs to the ordering, not merely to the set of basis vectors. Applying a linear transformation to each vector in a basis also produces an ordered list, and its orientation can be compared with the original one.

Proof. Let \(\mathbf{B}\) be the basis matrix of \(\mathcal{B}\). The matrix whose columns are the image vectors \(\mathbf{A}\mathbf{b}_1,\dots,\mathbf{A}\mathbf{b}_n\) is \(\mathbf{A}\mathbf{B}\). Because \(\mathbf{A}\) and \(\mathbf{B}\) are invertible, their product is invertible, so its columns form a basis. Its determinant is

Since \(\det(\mathbf{B})\ne0\), multiplying it by a positive number preserves its sign, while multiplying it by a negative number reverses its sign. Therefore the image basis has the same orientation as \(\mathcal{B}\) when \(\det(\mathbf{A})>0\), and the opposite orientation when \(\det(\mathbf{A})<0\). \(\square\)

This proof uses invertibility for an important reason: the image of a basis need not be a basis under a noninvertible transformation. For example, the zero matrix sends every basis vector to \(\mathbf{0}\). In the theorem, invertibility guarantees that \(\mathbf{A}\mathbf{B}\) is a basis matrix and that the image list has a well-defined orientation.

For an orthogonal matrix, the determinant theorem sharpens this distinction. The positive case has determinant exactly \(1\); the negative case has determinant exactly \(-1\). Thus an orthogonal matrix preserves orientation precisely when its determinant is \(1\), and reverses orientation precisely when its determinant is \(-1\).

Worked Examples: Both Determinant Signs Occur

Orientation for a Basis That Is Not Standard

The determinant sign does not depend on starting with the standard basis. For any ordered basis, the orientation theorem applies to its basis matrix and the images of its vectors. The following calculation shows this explicitly for an orientation-preserving or reversing test on a nonstandard basis.

Common Pitfall: Orthogonal Does Not Mean Determinant \(1\)

A frequent error is to start from \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}_n\), take determinants, obtain \(\det(\mathbf{Q})^2=1\), and conclude \(\det(\mathbf{Q})=1\). That conclusion drops one of the two possible real solutions. The concrete calculation with \(\mathbf{R}=\begin{bmatrix}1&0\\0&-1\end{bmatrix}\) disproves it: \(\mathbf{R}^{\top}\mathbf{R}=\mathbf{I}_2\), yet \(\det(\mathbf{R})=-1\).

The correction is to factor the equation rather than take only its positive square root:

Another useful distinction is that orientation is not a claim that each individual vector points in a particular direction. It is a property of the ordered basis as a whole. An orthogonal transformation preserves lengths and angles, while the determinant sign records whether the handedness of an ordered basis is maintained or reversed.

The sign also behaves consistently under composition. For two orthogonal matrices \(\mathbf{Q}_1,\mathbf{Q}_2\), the product is orthogonal, and \(\det(\mathbf{Q}_1\mathbf{Q}_2)=\det(\mathbf{Q}_1)\det(\mathbf{Q}_2)\). If both reverse orientation, their product has determinant \((-1)(-1)=1\) and preserves orientation. If exactly one reverses orientation, the product has determinant \(-1\) and reverses orientation. This follows directly from the determinant product identity and the two possible signs.