Extremes of the Rayleigh Quotient
For a real symmetric matrix, the Rayleigh quotient connects a quadratic expression to the matrix’s eigenvalues. It assigns a scalar to each nonzero vector, but scaling a vector does not change that scalar. The quotient can therefore be studied on the unit sphere, where its largest and smallest values are attained. The key step is to show that a vector where either extreme occurs must be an eigenvector.
The quotient is a continuous function of \(\mathbf{x}\) wherever \(\mathbf{x}\ne\mathbf{0}\). The unit sphere in \(\mathbb{R}^n\) is compact, so the quotient attains a maximum and a minimum there. Equivalently, by the existence result for extremal Rayleigh quotients, there are unit vectors where these two values occur. Because the quotient is unchanged by nonzero scaling, these are also its maximum and minimum over all nonzero vectors.
We use the standard definition that \(\lambda\) is an eigenvalue of \(\mathbf{A}\) if there is a nonzero vector \(\mathbf{v}\) such that \(\mathbf{A}\mathbf{v}=\lambda\mathbf{v}\). When discussing a real matrix, eigenvalues may initially be taken over \(\mathbb{C}\). For a real symmetric matrix, they are real. Indeed, if \(\mathbf{A}\mathbf{v}=\lambda\mathbf{v}\) for nonzero \(\mathbf{v}\in\mathbb{C}^n\), then \(\mathbf{v}^{*}\mathbf{A}\mathbf{v}\) is real because \(\mathbf{A}^{*}=\mathbf{A}\). Since \(\mathbf{v}^{*}\mathbf{A}\mathbf{v}=\lambda\mathbf{v}^{*}\mathbf{v}\) and \(\mathbf{v}^{*}\mathbf{v}>0\), \(\lambda\) is real. Writing \(\mathbf{v}=\mathbf{u}+i\mathbf{w}\) then gives \(\mathbf{A}\mathbf{u}=\lambda\mathbf{u}\) and \(\mathbf{A}\mathbf{w}=\lambda\mathbf{w}\); at least one of \(\mathbf{u},\mathbf{w}\) is nonzero. Thus each eigenvalue has a nonzero real eigenvector.
Why a Stationary Point Gives an Eigenvector
For a unit vector \(\mathbf{x}\), the tangent directions to the sphere at \(\mathbf{x}\) are precisely the vectors \(\mathbf{y}\) satisfying \(\mathbf{x}^{\top}\mathbf{y}=0\). At a local maximum or minimum on the sphere, the derivative of the quotient in each such direction must vanish. Symmetry lets us turn this collection of scalar derivative conditions into the vector equation \(\mathbf{A}\mathbf{x}=r\mathbf{x}\).
Proof. Fix any \(\mathbf{y}\in\mathbb{R}^n\) with \(\mathbf{x}^{\top}\mathbf{y}=0\). The curve \(\boldsymbol{\gamma}(t)=\dfrac{\mathbf{x}+t\mathbf{y}}{\sqrt{1+t^2\mathbf{y}^{\top}\mathbf{y}}}\) lies on the unit sphere: the squared length of its numerator is \(1+t^2\mathbf{y}^{\top}\mathbf{y}\). Because \(\mathbf{A}\) is symmetric, \((\mathbf{x}+t\mathbf{y})^{\top}\mathbf{A}(\mathbf{x}+t\mathbf{y})\) equals \(\mathbf{x}^{\top}\mathbf{A}\mathbf{x}+2t\mathbf{y}^{\top}\mathbf{A}\mathbf{x}+t^2\mathbf{y}^{\top}\mathbf{A}\mathbf{y}\). Consequently, \(R_{\mathbf{A}}(\boldsymbol{\gamma}(t))\) is this expression divided by \(1+t^2\mathbf{y}^{\top}\mathbf{y}\), so its derivative at \(t=0\) is \(2\mathbf{y}^{\top}\mathbf{A}\mathbf{x}\). Stationarity makes this derivative zero for every \(\mathbf{y}\perp\mathbf{x}\). Set \(r=\mathbf{x}^{\top}\mathbf{A}\mathbf{x}\) and \(\mathbf{w}=\mathbf{A}\mathbf{x}-r\mathbf{x}\). Then \(\mathbf{x}^{\top}\mathbf{w}=r-r=0\). Taking \(\mathbf{y}=\mathbf{w}\) in the derivative condition gives \(0=\mathbf{w}^{\top}\mathbf{A}\mathbf{x}=\mathbf{w}^{\top}(\mathbf{w}+r\mathbf{x})=\mathbf{w}^{\top}\mathbf{w}\). Thus \(\mathbf{w}=\mathbf{0}\), so \(\mathbf{A}\mathbf{x}=r\mathbf{x}\), and \(r=R_{\mathbf{A}}(\mathbf{x})\) because \(\mathbf{x}\) is a unit vector. \(\square\)
The symmetry assumption is used when expanding the quadratic expression: the two cross terms combine to \(2t\mathbf{y}^{\top}\mathbf{A}\mathbf{x}\). Without symmetry, the derivative instead involves the symmetric part of the matrix, and the conclusion need not hold.
The Extreme Quotients Are the Extreme Eigenvalues
Proof. The maximum and minimum are attained at unit vectors because the sphere is compact and the quotient is continuous there. At either extremum, the derivative along every tangent direction is zero. By the proposition, each extremizing vector is an eigenvector, and its eigenvalue is the quotient value at that vector. Thus \(M\) and \(m\) are eigenvalues. Now take any eigenvalue \(\lambda\) and a nonzero real eigenvector \(\mathbf{v}\), whose existence for a real symmetric matrix was established above. By the definition of eigenvector, \(\mathbf{v}^{\top}\mathbf{A}\mathbf{v}=\mathbf{v}^{\top}(\lambda\mathbf{v})=\lambda\mathbf{v}^{\top}\mathbf{v}\), so \(R_{\mathbf{A}}(\mathbf{v})=\lambda\). By the definitions of minimum and maximum, \(m\le R_{\mathbf{A}}(\mathbf{v})\le M\), giving \(m\le\lambda\le M\). Since every eigenvalue lies in this interval and both endpoints are eigenvalues, \(M\) is the largest and \(m\) the smallest eigenvalue. \(\square\)
This argument does not require the eigenvalues to be distinct. If an extreme eigenvalue is repeated, there may be more than one independent eigenvector attaining the same quotient. The theorem identifies the extreme values, not a unique extremizing vector.
Worked Examples
Worked Example: Finding the Full Range in Two Dimensions
Let \(\mathbf{A}=\begin{bmatrix}4&2\\2&1\end{bmatrix}\). Find the range of its Rayleigh quotient and verify eigenvectors for the endpoints.
For a unit vector \(\mathbf{x}=(x,y)^{\top}\), where \(x^2+y^2=1\), direct multiplication gives \(\mathbf{x}^{\top}\mathbf{A}\mathbf{x}=4x^2+4xy+y^2=(2x+y)^2\). By the Cauchy–Schwarz inequality, \((2x+y)^2\le(2^2+1^2)(x^2+y^2)=5\). The quotient is a square, so it is also at least \(0\). These bounds are attained: \((2,1)^{\top}/\sqrt5\) gives \(2x+y=\sqrt5\), while \((-1,2)^{\top}/\sqrt5\) gives \(2x+y=0\).
To verify that every value between the bounds occurs, write \((x,y)=(\cos t,\sin t)\). Choose \(\phi\) with \(\cos\phi=2/\sqrt5\) and \(\sin\phi=1/\sqrt5\). Then \(2\cos t+\sin t=\sqrt5\cos(t-\phi)\). As \(t\) moves continuously from \(\phi\) to \(\phi+\pi/2\), the quotient changes continuously from \(5\) to \(0\). The intermediate value theorem therefore guarantees that it takes every value in \([0,5]\). The range is exactly \([0,5]\).
Check the endpoints: Multiplying gives \(\mathbf{A}\begin{bmatrix}2\\1\end{bmatrix}=\begin{bmatrix}10\\5\end{bmatrix}=5\begin{bmatrix}2\\1\end{bmatrix}\) and \(\mathbf{A}\begin{bmatrix}-1\\2\end{bmatrix}=\begin{bmatrix}0\\0\end{bmatrix}=0\begin{bmatrix}-1\\2\end{bmatrix}\). The endpoint vectors are eigenvectors with eigenvalues \(5\) and \(0\), respectively, as the theorem predicts.
Worked Example: Bounding Eigenvalues of a Diagonal Matrix
For \(\mathbf{A}=\begin{bmatrix}-2&0&0\\0&1&0\\0&0&4\end{bmatrix}\), find the extreme Rayleigh quotients and check the eigenvalues.
For \(\mathbf{x}=(x,y,z)^{\top}\ne\mathbf{0}\), \(R_{\mathbf{A}}(\mathbf{x})=\dfrac{-2x^2+y^2+4z^2}{x^2+y^2+z^2}\). The numerator is a weighted sum of the three squares, with weights \(-2,1,4\). Subtracting \(-2\) times the denominator from the numerator gives \(3y^2+6z^2\ge0\), so \(R_{\mathbf{A}}(\mathbf{x})\ge-2\). Subtracting the numerator from \(4\) times the denominator gives \(6x^2+3y^2\ge0\), so \(R_{\mathbf{A}}(\mathbf{x})\le4\). The lower bound is attained at \((1,0,0)^{\top}\); the upper bound is attained at \((0,0,1)^{\top}\). Thus the minimum is \(-2\) and the maximum is \(4\).
Check: Multiplication gives \(\mathbf{A}(1,0,0)^{\top}=(-2,0,0)^{\top}=-2(1,0,0)^{\top}\) and \(\mathbf{A}(0,0,1)^{\top}=(0,0,4)^{\top}=4(0,0,1)^{\top}\). The middle standard basis vector satisfies \(\mathbf{A}(0,1,0)^{\top}=(0,1,0)^{\top}\), giving eigenvalue \(1\). All three eigenvalues lie between the extreme values.
Worked Example: A Symmetric Matrix with Two Distinct Extremes
For \(\mathbf{A}=\begin{bmatrix}3&1\\1&3\end{bmatrix}\), find the minimum and maximum Rayleigh quotients and verify the corresponding eigenvectors.
For \(\mathbf{x}=(x,y)^{\top}\ne\mathbf{0}\), \(R_{\mathbf{A}}(\mathbf{x})=\dfrac{3x^2+2xy+3y^2}{x^2+y^2}\). Subtracting \(2\) from the quotient gives \(\dfrac{x^2+2xy+y^2}{x^2+y^2}=\dfrac{(x+y)^2}{x^2+y^2}\ge0\). Subtracting the quotient from \(4\) gives \(\dfrac{x^2-2xy+y^2}{x^2+y^2}=\dfrac{(x-y)^2}{x^2+y^2}\ge0\). Therefore every quotient lies in \([2,4]\). Equality at \(2\) occurs for \((1,-1)^{\top}\), and equality at \(4\) occurs for \((1,1)^{\top}\).
Check: \(\mathbf{A}\begin{bmatrix}1\\-1\end{bmatrix}=\begin{bmatrix}2\\-2\end{bmatrix}=2\begin{bmatrix}1\\-1\end{bmatrix}\) and \(\mathbf{A}\begin{bmatrix}1\\1\end{bmatrix}=\begin{bmatrix}4\\4\end{bmatrix}=4\begin{bmatrix}1\\1\end{bmatrix}\). These are the eigenvectors for the minimum and maximum quotient values.
Common Pitfall: Symmetry Is Essential
A common mistake is to apply the theorem to a matrix that is not symmetric. For example, take \(\mathbf{B}=\begin{bmatrix}0&1\\0&0\end{bmatrix}\). For \(\mathbf{x}=(x,y)^{\top}\ne\mathbf{0}\), its quotient is \(\dfrac{\mathbf{x}^{\top}\mathbf{B}\mathbf{x}}{\mathbf{x}^{\top}\mathbf{x}}=\dfrac{xy}{x^2+y^2}\). Since \((x-y)^2\ge0\) gives \(2xy\le x^2+y^2\), this quotient is at most \(1/2\). Since \((x+y)^2\ge0\) gives \(-2xy\le x^2+y^2\), it is at least \(-1/2\). These bounds are attained at \((1,1)^{\top}\) and \((1,-1)^{\top}\), respectively.
The matrix \(\mathbf{B}\) has only the eigenvalue \(0\): if \(\mathbf{B}\mathbf{v}=\lambda\mathbf{v}\) for nonzero \(\mathbf{v}\), then \(\mathbf{B}^2=\mathbf{O}\) implies \(\lambda^2\mathbf{v}=\mathbf{0}\), so \(\lambda=0\). The quotient’s extrema, \(-1/2\) and \(1/2\), are not eigenvalues. The error is dropping the symmetry hypothesis: this example shows why the theorem is specifically about real symmetric matrices.
How to Use the Result
The theorem is useful when a quadratic expression needs to be bounded. Instead of testing every nonzero vector individually, find or identify the smallest and largest eigenvalues: every Rayleigh quotient lies between them, and eigenvectors for those eigenvalues attain the bounds. Conversely, if an extremizing direction is known, the stationary-point argument proves it is an eigenvector. The result also distinguishes a bound from an attained value: compactness guarantees that the minimum and maximum are achieved, not merely approached.
Check Your Understanding
Use the definitions and arguments above to answer these questions.
- Why can the Rayleigh quotient be maximized by restricting attention to unit vectors?
- In the stationary-point proof, why is it enough to show that \(\mathbf{A}\mathbf{x}-r\mathbf{x}\) is orthogonal to every vector perpendicular to \(\mathbf{x}\)?
- For a real symmetric matrix, explain why an eigenvalue has a nonzero real eigenvector even if the eigenvalue is first obtained using a complex eigenvector.
- For \(\mathbf{A}=\begin{bmatrix}2&0\\0&-3\end{bmatrix}\), find the minimum and maximum Rayleigh quotients and give vectors attaining them.
- For the nonsymmetric matrix \(\mathbf{B}\) in the pitfall, why does the interval of Rayleigh quotient values not identify its eigenvalues?
