Maximum Stretching by a Matrix
A matrix can stretch different directions by different amounts. Its singular values record these amounts along orthogonal directions, but one quantity is especially important: the largest stretch the matrix can produce from any unit input. The operator norm measures precisely this maximum. We will prove that it is the largest singular value, show which inputs attain it, and use SVDs to calculate it in square and rectangular examples.
Let \(\mathbf{A}\in\mathbb{F}^{m\times n}\), where \(m,n\geq 1\) and \(\mathbb{F}\) is \(\mathbb{R}\) or \(\mathbb{C}\). For a vector \(\mathbf{x}\), the Euclidean norm is \(\|\mathbf{x}\|=(\sum_j |x_j|^2)^{1/2}\); over \(\mathbb{R}\), the absolute values may be replaced by squares. A unit vector is a vector with norm \(1\). The operator norm below is induced by these Euclidean norms on the input and output spaces.
The maximum in this definition is attained: the unit sphere in a finite-dimensional real or complex vector space is compact, and the function \(\mathbf{x}\mapsto\|\mathbf{A}\mathbf{x}\|\) is continuous. The SVD gives more than existence, however; it identifies the maximum explicitly. We use the full SVD \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{*}\), where \(\mathbf{U}\) and \(\mathbf{V}\) are orthogonal in the real case and unitary in the complex case. The matrix \(\boldsymbol{\Sigma}\) is \(m\times n\), with nonnegative diagonal entries \(\sigma_1\geq\cdots\geq\sigma_p\geq0\), where \(p=\min(m,n)\), and zeros elsewhere. These \(\sigma_i\) are the singular values. The columns of \(\mathbf{V}\) give input directions; multiplication by \(\mathbf{V}^{*}\) changes to their coordinates without changing vector length.
The Largest Singular Value Gives the Norm
Proof. Write \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{*}\), and let \(\mathbf{x}\) be any unit vector in \(\mathbb{F}^n\). Set \(\mathbf{y}=\mathbf{V}^{*}\mathbf{x}\). Since \(\mathbf{V}\) is orthogonal or unitary, \(\|\mathbf{y}\|=\|\mathbf{x}\|=1\). Also, \(\mathbf{U}\) preserves norms, so \(\|\mathbf{A}\mathbf{x}\|=\|\boldsymbol{\Sigma}\mathbf{y}\|\). For \(p=\min(m,n)\), the nonzero-shaped diagonal of \(\boldsymbol{\Sigma}\) gives \(\|\boldsymbol{\Sigma}\mathbf{y}\|^2=\sum_{i=1}^{p}\sigma_i^2|y_i|^2\). Because \(0\leq\sigma_i\leq\sigma_1\), \[ \|\mathbf{A}\mathbf{x}\|^2 \leq \sigma_1^2\sum_{i=1}^{p}|y_i|^2 \leq \sigma_1^2\sum_{j=1}^{n}|y_j|^2 =\sigma_1^2. \] Thus every unit input produces an output of norm at most \(\sigma_1\). Let \(\mathbf{v}_1\) be the first column of \(\mathbf{V}\), so \(\mathbf{V}^{*}\mathbf{v}_1=\mathbf{e}_1\) and \(\|\mathbf{v}_1\|=1\). Then \(\mathbf{A}\mathbf{v}_1=\sigma_1\mathbf{u}_1\), where \(\mathbf{u}_1\) is the first column of \(\mathbf{U}\), and \(\|\mathbf{A}\mathbf{v}_1\|=\sigma_1\). This also holds when \(\sigma_1=0\): both sides are zero. Hence the maximum equals \(\sigma_1\). \(\square\)
The inequality in the proof also gives a useful scaled form. For \(\mathbf{x}\ne\mathbf{0}\), apply the unit-vector bound to \(\mathbf{x}/\|\mathbf{x}\|\) and multiply by \(\|\mathbf{x}\|\); for \(\mathbf{x}=\mathbf{0}\), both sides vanish. Therefore \(\|\mathbf{A}\mathbf{x}\|\leq\sigma_1\|\mathbf{x}\|\) for every input. The right singular vector \(\mathbf{v}_1\) shows that no smaller constant can replace \(\sigma_1\).
Worked Example: A Diagonal Matrix
Find the operator norm of \(\mathbf{A}=\begin{bmatrix}3&0\\0&1\end{bmatrix}\) and identify a unit vector attaining it.
This is already an SVD: take \(\mathbf{U}=\mathbf{I}_2\), \(\mathbf{V}=\mathbf{I}_2\), and \(\boldsymbol{\Sigma}=\begin{bmatrix}3&0\\0&1\end{bmatrix}\). The factors multiply back to \(\mathbf{A}\), and the singular values are \(3\) and \(1\). The theorem gives \(\|\mathbf{A}\|_2=3\). Directly, for a unit vector \(\mathbf{x}=(x_1,x_2)^{\top}\), \(x_1^2+x_2^2=1\), and \[ \|\mathbf{A}\mathbf{x}\|^2 =\|(3x_1,x_2)^{\top}\|^2 =9x_1^2+x_2^2 \leq 9(x_1^2+x_2^2)=9. \] For \(\mathbf{x}=\mathbf{e}_1=(1,0)^{\top}\), \(\mathbf{A}\mathbf{e}_1=(3,0)^{\top}\), whose norm is \(3\). This verifies both the upper bound and its attainment.
Which Unit Vectors Attain the Maximum?
If the largest singular value is repeated, there need not be just one maximizing direction. To describe all such inputs, let \(\tau_j=\sigma_j\) for \(1\leq j\leq p\), and set \(\tau_j=0\) for \(p<j\leq n\). The extra zeros account for right-coordinate directions that are not among the \(p\) displayed singular values when \(n>m\). The value \(\tau_j\) is the stretch associated with the \(j\)-th right singular vector.
Proof. Since \(\|\mathbf{y}\|=1\), \[ \sigma_1^2-\|\mathbf{A}\mathbf{x}\|^2 =\sum_{j=1}^{n}(\sigma_1^2-\tau_j^2)|y_j|^2. \] Every summand is nonnegative, since \(0\leq\tau_j\leq\sigma_1\). The output norm equals \(\sigma_1\) exactly when the left side is zero, which happens exactly when every summand is zero. For an index with \(\tau_j<\sigma_1\), its coefficient \(\sigma_1^2-\tau_j^2\) is positive, so that summand is zero exactly when \(y_j=0\). For indices with \(\tau_j=\sigma_1\), the coefficient is zero and \(y_j\) is unrestricted. This proves both directions. If \(\mathbf{A}=\mathbf{O}\), then all \(\tau_j=0=\sigma_1\), so there are no indices with \(\tau_j<\sigma_1\); every unit vector satisfies the condition. \(\square\)
Worked Example: A Rectangular Matrix with a Repeated Largest Singular Value
Find the operator norm of \(\mathbf{A}=\begin{bmatrix}1&1&0\\1&-1&0\end{bmatrix}\) and determine all unit inputs that attain it.
Use \(\mathbf{U}=\mathbf{I}_2\), \(\boldsymbol{\Sigma}=\begin{bmatrix}\sqrt{2}&0&0\\0&\sqrt{2}&0\end{bmatrix}\), and \(\mathbf{V}=\begin{bmatrix}\frac{1}{\sqrt{2}}&\frac{1}{\sqrt{2}}&0\\\frac{1}{\sqrt{2}}&-\frac{1}{\sqrt{2}}&0\\0&0&1\end{bmatrix}\). The columns of \(\mathbf{V}\) have unit length and pairwise dot product zero, so \(\mathbf{V}\) is orthogonal. Multiplying the SVD factors gives \[ \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top} =\begin{bmatrix}\sqrt{2}&0&0\\0&\sqrt{2}&0\end{bmatrix} \begin{bmatrix}\frac{1}{\sqrt{2}}&\frac{1}{\sqrt{2}}&0\\\frac{1}{\sqrt{2}}&-\frac{1}{\sqrt{2}}&0\\0&0&1\end{bmatrix} =\begin{bmatrix}1&1&0\\1&-1&0\end{bmatrix} =\mathbf{A}. \] Both singular values are \(\sqrt{2}\), so the operator norm is \(\sqrt{2}\). The third right singular direction has stretch zero, so the equality criterion says a maximizing unit input has third coordinate zero. Indeed, for \(\mathbf{x}=(x_1,x_2,x_3)^{\top}\), \[ \|\mathbf{A}\mathbf{x}\|^2 =(x_1+x_2)^2+(x_1-x_2)^2 =2x_1^2+2x_2^2. \] If \(\|\mathbf{x}\|=1\), this equals \(2\) exactly when \(x_3=0\). For example, \(\mathbf{e}_1\) maps to \((1,1)^{\top}\), of norm \(\sqrt{2}\). Thus the full unit circle in the \(x_1x_2\)-plane consists of maximizing inputs.
Worked Example: A Matrix That Swaps the Stretching Directions
Find the operator norm of \(\mathbf{A}=\begin{bmatrix}0&2\\1&0\end{bmatrix}\), and identify a maximizing unit input.
First compute \[ \mathbf{A}^{\top}\mathbf{A} =\begin{bmatrix}0&1\\2&0\end{bmatrix} \begin{bmatrix}0&2\\1&0\end{bmatrix} =\begin{bmatrix}1&0\\0&4\end{bmatrix}. \] The corresponding right singular directions are \(\mathbf{e}_1\) and \(\mathbf{e}_2\), with singular values \(1\) and \(2\), respectively. An SVD is \[ \mathbf{U}=\mathbf{I}_2,\qquad \boldsymbol{\Sigma}=\begin{bmatrix}2&0\\0&1\end{bmatrix},\qquad \mathbf{V}=\begin{bmatrix}0&1\\1&0\end{bmatrix}. \] Here \(\mathbf{V}\) is orthogonal, and multiplying verifies the factorization: \[ \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top} =\begin{bmatrix}2&0\\0&1\end{bmatrix} \begin{bmatrix}0&1\\1&0\end{bmatrix} =\begin{bmatrix}0&2\\1&0\end{bmatrix} =\mathbf{A}. \] Therefore \(\|\mathbf{A}\|_2=2\). The maximizing input is \(\mathbf{v}_1=\mathbf{e}_2\): \(\mathbf{A}\mathbf{e}_2=(2,0)^{\top}\) has norm \(2\). For a direct check, if \(\mathbf{x}=(x_1,x_2)^{\top}\) is unit, then \(\|\mathbf{A}\mathbf{x}\|^2=x_1^2+4x_2^2\leq4(x_1^2+x_2^2)=4\), with equality at \(\mathbf{e}_2\).
A Common Mistake: Combining Singular Values Instead of Maximizing
For \(\mathbf{A}=\begin{bmatrix}3&0\\0&1\end{bmatrix}\), a tempting but incorrect calculation is to combine the singular values as \(\sqrt{3^2+1^2}=\sqrt{10}\) and call that the operator norm. The definition asks for the largest output from one unit input, not a combination of the stretches in two different directions. The actual calculation for a unit input gives \(9x_1^2+x_2^2\leq9\), and \(\mathbf{e}_1\) attains the bound. Thus the operator norm is \(3\), the largest singular value.
Another useful check is to distinguish an input vector from its image. The largest singular value is attained along a right singular vector; its image points along the corresponding left singular vector and has length \(\sigma_1\). In the last example, \(\mathbf{e}_2\) is the input direction and \(\mathbf{e}_1\) is the output direction. Keeping those roles separate prevents reversing which vector is being maximized over.
The operator norm summarizes the strongest possible action of a matrix on an input of fixed length. From an SVD, no optimization over all unit vectors is needed: read off \(\sigma_1\). When the largest singular value is repeated, the equality criterion also tells exactly which combinations of right singular vectors attain that stretch; when it is unique, every maximizing unit input is a scalar multiple of its right singular vector, with scalar of absolute value \(1\).
Check Your Understanding
Use the SVD or the definition to justify each answer.
- If the singular values of a matrix are \(5,2,0\), what is its operator norm?
- For a unit vector \(\mathbf{x}\), explain why \(\|\mathbf{A}\mathbf{x}\|\leq\sigma_1\).
- If the two largest singular values are equal, how does that affect the maximizing unit inputs?
- For \(\mathbf{A}=\begin{bmatrix}0&0\\0&4\end{bmatrix}\), give a unit input that attains the operator norm and verify its output norm.
- Why is every unit vector a maximizing input when \(\mathbf{A}=\mathbf{O}\)?
