Linear Algebra: Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors are not just abstract algebraic curiosities; they are the skeleton keys that unlock the fundamental behavior of linear transformations. Whether you're simplifying complex systems, compressing high-dimensional data, or modeling the steady state of a physical process, these concepts provide the essential language. Mastering them transforms your ability to analyze matrices from performing mere calculations to interpreting their deeper structural properties.

What Are Eigenvalues and Eigenvectors?

Consider a linear transformation represented by a square matrix $A$. An eigenvector of $A$ is a nonzero vector $\vec{v}$ that, when multiplied by $A$, does not change its direction. It may get stretched or compressed, but it stays on the same line. The scalar factor by which it is stretched or compressed is the corresponding eigenvalue, denoted by $\lambda$.

Formally, for an $n\times n$ matrix $A$, the eigenvector-eigenvalue relationship is defined by the equation: $$A\vec{v}=\lambda \vec{v}$$ This equation is deceptively simple. It tells us that applying the transformation $A$ to the eigenvector $\vec{v}$ has the exact same effect as simply scaling $\vec{v}$ by $\lambda$. The eigenvector $\vec{v}$ thus reveals an invariant direction under the transformation, and the eigenvalue $\lambda$ tells you how much that direction is scaled.

For example, consider a transformation that stretches space horizontally by a factor of 3 and leaves vertical directions unchanged. The vector $(1,0)$ is an eigenvector with eigenvalue $3$, since it gets three times longer. The vector $(0,1)$ is an eigenvector with eigenvalue $1$, as it remains unchanged. Any other vector, like $(2,2)$, will be knocked off its original line when transformed.

The Characteristic Equation and Finding Eigenvalues

To find eigenvalues, we rearrange the defining equation: $A\vec{v}=\lambda \vec{v}$ becomes $A\vec{v}-\lambda \vec{v}=\vec{0}$, or $(A-\lambda I)\vec{v}=\vec{0}$, where $I$ is the identity matrix.

This is now a homogeneous system of equations. We are looking for nonzero solutions for $\vec{v}$. Recall from linear algebra that a homogeneous system has nonzero solutions if and only if its coefficient matrix is singular (non-invertible). Therefore, the determinant of $(A-\lambda I)$ must be zero: $$\det (A-\lambda I)=0$$ This is the characteristic equation of the matrix $A$. The expression $\det (A-\lambda I)$ expands to a polynomial in $\lambda$ of degree $n$, called the characteristic polynomial. The roots of this polynomial are the eigenvalues of $A$.

Worked Example: Find the eigenvalues of $A=\left(\begin{array}{cc}2& 1\\ 1& 2\end{array}\right)$.

1. Form $A-\lambda I=\left(\begin{array}{cc}2-\lambda & 1\\ 1& 2-\lambda \end{array}\right)$.
2. Compute the determinant: $\det (A-\lambda I)=(2-\lambda )(2-\lambda )-(1)(1)={\lambda }^2-4\lambda +3$.
3. Set the determinant to zero and solve: ${\lambda }^2-4\lambda +3=0$. This factors to $(\lambda -1)(\lambda -3)=0$.
4. The eigenvalues are ${\lambda }_1=1$ and ${\lambda }_2=3$.

Determining Eigenvectors and Eigenspaces

Once you have an eigenvalue $\lambda$, you find its corresponding eigenvectors by solving the system $(A-\lambda I)\vec{v}=\vec{0}$ for the vector $\vec{v}$.

The set of all solutions to this system (including the zero vector) is called the eigenspace corresponding to $\lambda$. It is a subspace of ${\mathbb{R}}^n$ (or ${\mathbb{C}}^n$). A basis for this eigenspace is formed by the linearly independent eigenvectors associated with $\lambda$.

Continuing the Example: For $A=\left(\begin{array}{cc}2& 1\\ 1& 2\end{array}\right)$, find eigenvectors for ${\lambda }_1=1$ and ${\lambda }_2=3$.

• For ${\lambda }_1=1$: Solve $(A-1I)\vec{v}=\left(\begin{array}{cc}1& 1\\ 1& 1\end{array}\right)\vec{v}=\vec{0}$. This reduces to the single equation $v_1+v_2=0$, so $v_2=-v_1$. Eigenvectors are of the form $\vec{v}=t\left(\begin{array}{c}1\\ -1\end{array}\right)$, for any scalar $t\ne 0$. A basis for this eigenspace is $\left(\begin{array}{c}1\\ -1\end{array}\right)$.
• For ${\lambda }_2=3$: Solve $(A-3I)\vec{v}=\left(\begin{array}{cc}-1& 1\\ 1& -1\end{array}\right)\vec{v}=\vec{0}$. This reduces to $-v_1+v_2=0$, so $v_2=v_1$. Eigenvectors are of the form $\vec{v}=s\left(\begin{array}{c}1\\ 1\end{array}\right)$. A basis is $\left(\begin{array}{c}1\\ 1\end{array}\right)$.

Diagonalization and Spectral Decomposition

A square matrix $A$ is said to be diagonalizable if it can be written in the form $A=PD{P}^{-1}$, where $D$ is a diagonal matrix and $P$ is an invertible matrix. This is a powerful factorization because it simplifies matrix operations, such as computing powers: $A^k=P{D}^k{P}^{-1}$.

The diagonal entries of $D$ are precisely the eigenvalues of $A$. The columns of $P$ are the corresponding eigenvectors of $A$, placed in the same order. A matrix is diagonalizable if and only if it has $n$ linearly independent eigenvectors (which is always true if it has $n$ distinct eigenvalues, or if it is symmetric).

Spectral decomposition (or eigendecomposition) is the process of expressing a matrix in terms of its eigenvalues and eigenvectors. For a symmetric matrix ($A=A^T$), this takes an especially elegant form. Its eigenvectors can be chosen to be orthonormal, and the decomposition becomes $A=Q\Lambda Q^T$, where $Q$ is an orthogonal matrix ($Q^{-1}=Q^T$) whose columns are the orthonormal eigenvectors, and $\Lambda$ is the diagonal eigenvalue matrix.

Key Applications

The power of eigenvalues and eigenvectors is realized in their wide-ranging applications:

• Principal Component Analysis (PCA): In data science, PCA is a dimensionality reduction technique. It finds the eigenvectors (principal components) of the covariance matrix of the data. The eigenvalues indicate the amount of variance captured by each component, allowing you to project high-dimensional data onto the few most important directions.
• Systems of Differential Equations: Solutions to linear systems like $\frac{d\vec{x}}{dt}=A\vec{x}$ are built from terms like $e^{\lambda t}\vec{v}$, where $\lambda$ and $\vec{v}$ are eigenvalue-eigenvector pairs of $A$. Eigenvalues determine stability: negative real parts lead to decay, positive to growth.
• Quantum Mechanics: In the Schrödinger equation $H\psi =E\psi$, the Hamiltonian operator $H$ acts like a matrix, its eigenfunctions $\psi$ are the "state vectors," and the corresponding eigenvalues $E$ are the allowable energy levels of the system.
• Google's PageRank Algorithm: The web is modeled as a giant graph. The PageRank vector, which ranks the importance of web pages, is the dominant eigenvector (corresponding to the largest eigenvalue $\lambda =1$) of the modified adjacency matrix (the Google matrix) of the web graph.

Common Pitfalls

1. Forgetting that eigenvectors must be nonzero. The equation $A\vec{v}=\lambda \vec{v}$ explicitly requires $\vec{v}\ne \vec{0}$. The zero vector is never an eigenvector, though it is always in the eigenspace.
2. Confusing algebraic and geometric multiplicity. The algebraic multiplicity of an eigenvalue is its multiplicity as a root of the characteristic polynomial. The geometric multiplicity is the dimension of its eigenspace (the number of linearly independent eigenvectors). The geometric multiplicity is always less than or equal to the algebraic multiplicity. If they are not equal for an eigenvalue, the matrix is not diagonalizable.
3. Assuming all matrices are diagonalizable. Many are, but not all. A classic counterexample is the shear matrix $\left(\begin{array}{cc}1& 1\\ 0& 1\end{array}\right)$. Its only eigenvalue is $\lambda =1$ (with algebraic multiplicity 2), but its eigenspace is only one-dimensional (spanned by $(1,0)$). It has a defective eigenvalue and cannot be diagonalized.
4. Misapplying the characteristic equation formula. Remember, the equation is $\det (A-\lambda I)=0$, not $\det (\lambda I-A)=0$. While the latter is mathematically equivalent (it just multiplies the polynomial by $(-1{)}^n$), consistently using the former helps avoid sign errors when calculating determinants.

Summary

• Eigenvectors $\vec{v}$ are nonzero vectors whose direction is unchanged by the matrix transformation $A$, satisfying $A\vec{v}=\lambda \vec{v}$, where the scalar $\lambda$ is the corresponding eigenvalue.
• Eigenvalues are found by solving the characteristic equation $\det (A-\lambda I)=0$. Each eigenvalue's eigenvectors are found by solving the homogeneous system $(A-\lambda I)\vec{v}=\vec{0}$.
• A matrix is diagonalizable ($A=PD{P}^{-1}$) if it has a full set of linearly independent eigenvectors, which form the columns of $P$, with eigenvalues on the diagonal of $D$.
• For symmetric matrices, the spectral decomposition $A=Q\Lambda Q^T$ provides an orthogonal diagonalization with orthonormal eigenvectors.
• These concepts are foundational for techniques like Principal Component Analysis, stability analysis in differential equations, quantum state analysis, and network ranking algorithms like PageRank.