Rayleigh Quotient

For a given Hermitian matrix (A \in \mathbb{C}^{n \times n}) and a non-zero vector (x \in \mathbb{C}^{n}), the Rayleigh quotient (R(A, x)) is defined as:

[ R(A, x) = \frac{x^* A x}{x^* x} ]

where (x^*) denotes the conjugate transpose of (x). For real symmetric matrices and real vectors, this simplifies to (\frac{x^T A x}{x^T x}). This scalar provides an estimate for an eigenvalue of (A).

A fundamental property is that the Rayleigh quotient's stationary points occur precisely at the eigenvectors of (A). This means the gradient (\nabla_x R(A, x)) is zero when (x) is an eigenvector. At such a point, the value of the quotient equals the corresponding eigenvalue:

[ R(A, v_i) = \lambda_i ]

where (A v_i = \lambda_i v_i). This makes it a crucial tool for iterative eigenvalue algorithms like Rayleigh quotient iteration.

The Courant-Fischer Min-Max Theorem provides a variational characterization of eigenvalues using the Rayleigh quotient. For a Hermitian matrix with eigenvalues (\lambda_1 \leq \lambda_2 \leq ... \leq \lambda_n), the (k)-th eigenvalue is given by:

[ \lambda_k = \min_{\dim(S)=k} \max_{x \in S, x \neq 0} R(A, x) ]

and equivalently,

[ \lambda_k = \max_{\dim(S)=n-k+1} \min_{x \in S, x \neq 0} R(A, x) ]

where (S) ranges over subspaces. This shows eigenvalues are solutions to constrained optimization problems.

The Rayleigh quotient is bounded by the smallest and largest eigenvalues of (A). For any non-zero vector (x), the following inequality holds:

[ \lambda_{\min}(A) \leq R(A, x) \leq \lambda_{\max}(A) ]

The minimum value (\lambda_{\min}) is achieved at the corresponding eigenvector, and the maximum value (\lambda_{\max}) is achieved at its eigenvector. This property is used in spectral clustering and principal component analysis (PCA), where the first principal component maximizes the Rayleigh quotient of the covariance matrix.

The Rayleigh quotient is the foundation for powerful iterative eigenvalue algorithms. The most notable is Rayleigh Quotient Iteration (RQI), a cubically convergent algorithm for finding an eigenpair.

• Process: Given an estimate (x_k), compute the shift (\sigma_k = R(A, x_k)), then solve ((A - \sigma_k I) x_{k+1} = x_k) for the next iterate.
• Convergence: With a good initial guess, RQI converges cubically to an eigenpair, making it significantly faster than the linearly convergent power iteration.
• Use Case: It is used in large-scale eigenvalue problems where only a few eigenpairs are needed, such as in low-rank matrix approximations.

The Rayleigh quotient naturally extends to the generalized eigenvalue problem (A x = \lambda B x), where (A) and (B) are Hermitian and (B) is positive definite. The generalized Rayleigh quotient is:

[ R(A, B, x) = \frac{x^* A x}{x^* B x} ]

Its stationary points give the generalized eigenvectors, and its min-max properties characterize the generalized eigenvalues. This form is essential in Fisher's Linear Discriminant Analysis (LDA), where the goal is to maximize the Rayleigh quotient of between-class to within-class scatter matrices.

An eigenvector of a square matrix is a non-zero vector that, when multiplied by the matrix, only changes by a scalar factor called the eigenvalue. For a Hermitian matrix A and eigenvector v, the relationship is Av = λv. The Rayleigh quotient R(A, x) provides an estimate for an eigenvalue λ when given an approximate eigenvector x. This is the core problem the Rayleigh quotient helps solve.

A Hermitian matrix (or self-adjoint matrix) is a complex square matrix that is equal to its own conjugate transpose (A = A*). For real matrices, this simplifies to a symmetric matrix (A = Aᵀ). Key properties include:

• All eigenvalues are real numbers.
• Eigenvectors corresponding to distinct eigenvalues are orthogonal. The Rayleigh quotient is specifically defined for Hermitian (or real symmetric) matrices, guaranteeing that its value is real and bounded by the minimum and maximum eigenvalues.

An iterative algorithm used to find an eigenvalue-eigenvector pair of a matrix. It is a refined version of the inverse iteration method that uses the Rayleigh quotient to update the eigenvalue estimate at each step.

Algorithm steps:

1. Start with an initial vector x₀.
2. Compute the Rayleigh quotient: σₖ = (xₖ A xₖ) / (xₖ xₖ)**.
3. Solve the linear system: (A - σₖI) y = xₖ for y.
4. Normalize: xₖ₊₁ = y / ||y||. This iteration exhibits cubic convergence for Hermitian matrices, making it exceptionally fast for finding isolated eigenvalues.

The spectral norm (or operator 2-norm) of a matrix is its largest singular value, which for a Hermitian matrix is the absolute value of its eigenvalue with the largest magnitude (||A||₂ = max |λ|). The Rayleigh quotient is directly related to this norm. For any vector x, the quotient |R(A, x)| ≤ ||A||₂. Minimizing or maximizing the Rayleigh quotient is a variational approach to finding the extreme eigenvalues that define the spectral norm.

A principle stating that the eigenvalues of a Hermitian matrix can be expressed as the solution to optimization problems involving the Rayleigh quotient. Specifically, for a Hermitian matrix A with eigenvalues λ₁ ≤ λ₂ ≤ ... ≤ λₙ:

• The minimum eigenvalue: λ₁ = min_{x≠0} R(A, x)
• The maximum eigenvalue: λₙ = max_{x≠0} R(A, x)
• The k-th eigenvalue (Courant-Fischer theorem): λₖ = min_{dim(S)=k} max_{x∈S, x≠0} R(A, x) This makes the Rayleigh quotient the central objective function for computing eigenvalues.

A simple iterative algorithm to find the dominant eigenvector (associated with the eigenvalue of largest magnitude) of a diagonalizable matrix. It repeatedly multiplies a vector by the matrix and normalizes: xₖ₊₁ = A xₖ / ||A xₖ||. While power iteration finds the eigenvector, it does not directly compute the eigenvalue. The Rayleigh quotient is used as a final step (or at each step) to estimate the corresponding eigenvalue from the converged vector, forming a basic Rayleigh quotient iteration.