The Rayleigh Quotient And Its Bounds

From a Quadratic Form to a Quotient

A real symmetric matrix assigns a real number \(\mathbf{x}^{\top}\mathbf{A}\mathbf{x}\) to each vector \(\mathbf{x}\). This quadratic form depends on the size of the vector as well as its direction: replacing \(\mathbf{x}\) by \(c\mathbf{x}\) multiplies the value by \(c^2\). Dividing by \(\mathbf{x}^{\top}\mathbf{x}\), the squared length of \(\mathbf{x}\), removes this dependence on scale. The resulting quantity is the Rayleigh quotient. Its central bound is particularly useful because it connects the values of a quadratic form to the eigenvalues of its matrix.

Throughout this tutorial, \(\mathbf{A}\) is a real symmetric \(n\times n\) matrix with \(n\ge 1\), and \(\mathbf{x}\) is a nonzero vector in \(\mathbb{R}^n\). Because \(\mathbf{x}\ne\mathbf{0}\), at least one component of \(\mathbf{x}\) is nonzero, so \(\mathbf{x}^{\top}\mathbf{x}\), the sum of the squares of its components, is positive.

The quotient is unchanged by nonzero scaling. In fact, if \(c\in\mathbb{R}\) and \(c\ne0\), then

Thus the quotient depends on the direction of a nonzero vector, not on its length. In particular, it is enough to calculate it on unit vectors, for which \(\mathbf{x}^{\top}\mathbf{x}=1\). This observation does not say which directions give the smallest or largest values; the bound below is a statement about every nonzero vector.

Why the Extreme Eigenvalues Bound Every Value

The Spectral Theorem for real symmetric matrices states that a real symmetric matrix has an orthonormal basis of real eigenvectors. Equivalently, there is an orthogonal matrix \(\mathbf{Q}\) and a diagonal matrix \(\mathbf{D}\) such that \(\mathbf{A}=\mathbf{Q}\mathbf{D}\mathbf{Q}^{\top}\). Here, orthogonal means \(\mathbf{Q}^{\top}\mathbf{Q}=\mathbf{I}_n\), and the diagonal entries of \(\mathbf{D}\) are the real eigenvalues of \(\mathbf{A}\). We use this earlier result to express the quotient as a weighted average of those eigenvalues.

Proof. By the Spectral Theorem, choose an orthogonal matrix \(\mathbf{Q}\) whose columns are an orthonormal eigenbasis of \(\mathbf{A}\), and write \(\mathbf{A}=\mathbf{Q}\mathbf{D}\mathbf{Q}^{\top}\), where \(\mathbf{D}=\operatorname{diag}(\lambda_1,\dots,\lambda_n)\). Every \(\lambda_i\) lies between \(\lambda_{\min}\) and \(\lambda_{\max}\). Set \(\mathbf{y}=\mathbf{Q}^{\top}\mathbf{x}\). Since \(\mathbf{Q}\) is invertible and \(\mathbf{x}\ne\mathbf{0}\), \(\mathbf{y}\ne\mathbf{0}\). Also, \(\mathbf{x}=\mathbf{Q}\mathbf{y}\), so orthogonality gives \(\mathbf{x}^{\top}\mathbf{x}=\mathbf{y}^{\top}\mathbf{y}=\sum_{i=1}^n y_i^2\), and \(\mathbf{x}^{\top}\mathbf{A}\mathbf{x}=\mathbf{y}^{\top}\mathbf{D}\mathbf{y}=\sum_{i=1}^n\lambda_i y_i^2\). Consequently, \(R_{\mathbf{A}}(\mathbf{x})=\dfrac{\sum_{i=1}^n\lambda_i y_i^2}{\sum_{i=1}^n y_i^2}\). For each \(i\), \(\lambda_{\min}\le\lambda_i\le\lambda_{\max}\). Multiplying these inequalities by \(y_i^2\ge0\) and summing gives \(\lambda_{\min}\sum_{i=1}^n y_i^2\le\sum_{i=1}^n\lambda_i y_i^2\le\lambda_{\max}\sum_{i=1}^n y_i^2\). The sum \(\sum_{i=1}^n y_i^2\) is positive because \(\mathbf{y}\ne\mathbf{0}\). Dividing by it proves both claimed inequalities. \(\square\)

This proof reveals the structure of the bound. The numbers \(y_i^2/\sum_j y_j^2\) are nonnegative and add to \(1\). The quotient is therefore an average of the eigenvalues, with weights determined by the squared components of \(\mathbf{x}\) in an orthonormal eigenbasis. No weight is negative, so the average cannot fall below every eigenvalue or rise above every eigenvalue.

Proof. Use the orthonormal eigenbasis and coordinates \(y_i\) from the preceding proof. For the lower bound, subtract \(\lambda_{\min}\) from the quotient: \(R_{\mathbf{A}}(\mathbf{x})-\lambda_{\min}=\dfrac{\sum_{i=1}^n(\lambda_i-\lambda_{\min})y_i^2}{\sum_{i=1}^n y_i^2}\). The denominator is positive, and every term in the numerator is nonnegative. The quotient equals \(\lambda_{\min}\) if and only if the numerator is zero. A sum of nonnegative terms is zero exactly when each term is zero. Thus \(y_i=0\) whenever \(\lambda_i>\lambda_{\min}\); the only possibly nonzero coordinates are those belonging to eigenvectors with eigenvalue \(\lambda_{\min}\). This is exactly the condition that \(\mathbf{x}\) belongs to the eigenspace for \(\lambda_{\min}\). Conversely, if \(\mathbf{x}\) belongs to that eigenspace, its coordinates in the chosen eigenbasis are zero outside the \(\lambda_{\min}\) eigenspace, so the quotient is \(\lambda_{\min}\). For the upper bound, \(\lambda_{\max}-R_{\mathbf{A}}(\mathbf{x})=\dfrac{\sum_{i=1}^n(\lambda_{\max}-\lambda_i)y_i^2}{\sum_{i=1}^n y_i^2}\). The same reasoning shows that this is zero exactly when \(y_i=0\) for every \(\lambda_i<\lambda_{\max}\), which is equivalent to \(\mathbf{x}\) belonging to the \(\lambda_{\max}\) eigenspace. \(\square\)

Computing the Quotient and Reading the Weights

Common Mistake: Omitting the Denominator

The Rayleigh quotient is not the quadratic form by itself. For \(\mathbf{A}=\begin{bmatrix}2&0\\0&5\end{bmatrix}\) and \(\mathbf{x}=\begin{bmatrix}2\\1\end{bmatrix}\), the smallest and largest eigenvalues are \(2\) and \(5\). A wrong calculation treats \(\mathbf{x}^{\top}\mathbf{A}\mathbf{x}=2\cdot4+5\cdot1=13\) as the quotient and concludes that the value exceeds \(5\). The correction is to include the denominator: \(\mathbf{x}^{\top}\mathbf{x}=4+1=5\), so \(R_{\mathbf{A}}(\mathbf{x})=13/5\), which is between \(2\) and \(5\). The bound applies to the ratio, not to the unnormalized numerator.

A second useful check is to confirm that the matrix is real and symmetric before applying this theorem. Its proof uses an orthonormal basis of real eigenvectors, supplied by the Spectral Theorem for real symmetric matrices. The bound as stated does not automatically apply to a matrix that fails these hypotheses.

What the Bound Tells Us

The quotient turns eigenvalue information into a bound on a quadratic expression evaluated at any nonzero vector. It also gives a way to compare a direction with the extreme eigendirections: a vector in the smallest or largest eigenspace gives the corresponding endpoint, while a vector mixing different eigenvalues gives an average weighted by its squared eigenbasis coordinates. Repeated extreme eigenvalues cause no difficulty; equality holds for every nonzero vector in the corresponding eigenspace, not just for one selected eigenvector.

When using the quotient, check three things: the matrix is symmetric, the input vector is nonzero, and the denominator is \(\mathbf{x}^{\top}\mathbf{x}\). Then identify the smallest and largest eigenvalues and compare the computed ratio against them. The bounds hold for every such vector, whether or not it is an eigenvector.