Linear Algebra: Symmetric Matrices

Symmetric matrices are not just a common structure in linear algebra; they are a cornerstone for understanding stability, optimization, and geometric transformations in engineering and data science. Their elegant properties guarantee real eigenvalues and orthogonal eigenvectors, leading to powerful factorization theorems that simplify complex problems in structural analysis, computer graphics, and machine learning. Mastering symmetric matrices provides you with the mathematical toolkit to diagonalize systems efficiently and uncover their principal directions.

Definition and Foundational Properties

A symmetric matrix is a square matrix $A$ that is equal to its own transpose: $A=A^T$. This means the entries are symmetric across the main diagonal, so $a_{ij}=a_{ji}$ for all $i$ and $j$. For example, the following matrix is symmetric: $$A=\left[\begin{array}{ccc}1& 4& -1\\ 4& 2& 5\\ -1& 5& 3\end{array}\right]$$

This simple definition leads to a suite of powerful properties. First, the sum of two symmetric matrices and any scalar multiple of a symmetric matrix are also symmetric. However, the product of two symmetric matrices is symmetric only if the matrices commute (i.e., $AB=BA$). The inverse of an invertible symmetric matrix is also symmetric. Furthermore, for any matrix $B$ (not necessarily square), the products $B^{T}B$ and $B{B}^T$ are always symmetric matrices. This last property is fundamental in statistics (covariance matrices) and data science (Gram matrices).

The Spectral Properties: Real Eigenvalues and Orthogonal Eigenvectors

The most profound properties of symmetric matrices concern their eigenvalues and eigenvectors. For a real symmetric matrix, all eigenvalues are real numbers, even if the matrix contains real numbers. This is a critical theorem because it ensures stability in physical systems—complex eigenvalues often indicate oscillatory or growing behavior. The proof relies on complex inner products and the property $A={\bar{A}}^T$.

Equally important is the orthogonality of eigenvectors corresponding to distinct eigenvalues. If ${\lambda }_1$ and ${\lambda }_2$ are distinct eigenvalues of a symmetric matrix $A$ with corresponding eigenvectors ${\mathbf{v}}_1$ and ${\mathbf{v}}_2$, then ${\mathbf{v}}_1$ and ${\mathbf{v}}_2$ are orthogonal (i.e., their dot product ${\mathbf{v}}_1\cdot {\mathbf{v}}_2=0$). Even if an eigenvalue is repeated (has algebraic multiplicity $k>1$), you can always find $k$ mutually orthogonal eigenvectors that span that eigenspace. This guarantees that an $n\times n$ symmetric matrix always has a full set of $n$ linearly independent, orthogonal eigenvectors.

The Spectral Theorem: Orthogonal Diagonalization

These spectral properties culminate in the Spectral Theorem (also called the Principal Axis Theorem). It states: For any real symmetric $n\times n$ matrix $A$, there exists an orthogonal matrix $Q$ and a diagonal matrix $D$ such that $$A=QD{Q}^T.$$

Let's unpack this. An orthogonal matrix $Q$ has orthonormal columns, meaning $Q^{T}Q=I$ and $Q^{-1}=Q^T$. The diagonal matrix $D$ contains the real eigenvalues of $A$ on its diagonal. The columns of $Q$ are the corresponding orthonormal eigenvectors. The factorization $A=QD{Q}^T$ is called an orthogonal diagonalization. This is stronger than standard diagonalization ($A=PD{P}^{-1}$) because the inverse of the eigenvector matrix is simply its transpose, which is numerically stable and trivial to compute.

Procedure for Orthogonal Diagonalization

To orthogonally diagonalize a real symmetric matrix $A$, follow this step-by-step procedure:

1. Find the Eigenvalues: Solve the characteristic equation $\det (A-\lambda I)=0$. The roots will be real.
2. Find a Basis for Each Eigenspace: For each eigenvalue $\lambda$, find a basis for the solution space of $(A-\lambda I)\mathbf{x}=0$.
3. Orthonormalize the Eigenvector Basis: Apply the Gram-Schmidt process to the basis for each eigenspace (especially if the dimension is >1) to obtain an orthonormal basis for that eigenspace.
4. Construct Matrices $Q$ and $D$: Form the orthogonal matrix $Q$ by using the orthonormal eigenvectors as columns. Place the corresponding eigenvalues in the same order on the diagonal of $D$.

Example: Diagonalize $A=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]$.

• Eigenvalues: ${\lambda }_1=3,{\lambda }_2=1$.
• Eigenvectors: For ${\lambda }_1=3$: ${\mathbf{v}}_1=(1,1)$. For ${\lambda }_2=1$: ${\mathbf{v}}_2=(-1,1)$.
• Orthonormalize: These are already orthogonal. Normalize them to length 1:

${\mathbf{q}}_1=(1/\sqrt{2},1/\sqrt{2})$, ${\mathbf{q}}_2=(-1/\sqrt{2},1/\sqrt{2})$.

• Construct:

$$Q=\left[\begin{array}{cc}1/\sqrt{2}& -1/\sqrt{2}\\ 1/\sqrt{2}& 1/\sqrt{2}\end{array}\right],D=\left[\begin{array}{cc}3& 0\\ 0& 1\end{array}\right].$$ You can verify that $A=QD{Q}^T$.

Applications to Quadratic Forms and the Principal Axis Theorem

A quadratic form is a scalar function $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$, where $A$ is symmetric. The Spectral Theorem allows us to remove cross-terms. Substituting $A=QD{Q}^T$ and letting $\mathbf{y}=Q^T\mathbf{x}$, we get: $$Q(\mathbf{x})={\mathbf{x}}^T(QD{Q}^T)\mathbf{x}=({\mathbf{x}}^{T}Q)D(Q^T\mathbf{x})={\mathbf{y}}^{T}D\mathbf{y}={\lambda }_1{y}_1^2+{\lambda }_2{y}_2^2+...+{\lambda }_n{y}_n^2.$$

This transformation is the Principal Axis Theorem in action. It states that by rotating the coordinate system to the axes defined by the eigenvectors of $A$ (the principal axes), the quadratic form becomes a sum of pure squared terms. The eigenvalues determine the shape: if all are positive, the surface is an ellipsoid; if signs are mixed, it is a hyperboloid. In engineering, this is used to find principal stresses in a stress tensor (where the eigenvectors are the principal directions and eigenvalues are the max/min normal stresses) and to analyze the vibrational modes of a mechanical system (where eigenvalues correspond to natural frequencies squared).

Common Pitfalls

1. Assuming Orthogonal Diagonalizability Implies Symmetry: While all real symmetric matrices are orthogonally diagonalizable, the converse is not true. Some non-symmetric matrices can be diagonalized by an orthogonal matrix (like orthogonal matrices themselves), but they are the exception, not the rule. Do not conclude a matrix is symmetric just because you found orthogonal eigenvectors.

1. Misapplying the Orthogonality Property: Eigenvectors for distinct eigenvalues are automatically orthogonal only for symmetric (or more generally, Hermitian) matrices. For a general diagonalizable matrix, eigenvectors from different eigenspaces are merely linearly independent, not necessarily orthogonal. Attempting to use them as an orthogonal basis without applying Gram-Schmidt will lead to errors.

1. Ignoring the Need for Orthonormalization within an Eigenspace: If an eigenspace has dimension greater than one (a repeated eigenvalue), the basis you find by solving $(A-\lambda I)\mathbf{x}=0$ is not automatically orthogonal. You must apply the Gram-Schmidt process to the basis of that specific eigenspace to obtain orthonormal vectors before constructing $Q$. Overlooking this results in a $Q$ that is not orthogonal.

1. Misinterpreting the Definiteness of a Quadratic Form: The definiteness (positive, negative, indefinite) of a quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$ is determined solely by the signs of the eigenvalues of $A$. Do not attempt to judge this by looking at the entries of $A$. A matrix with all positive entries can have a negative eigenvalue and represent an indefinite form.

Summary

• A symmetric matrix, defined by $A=A^T$, guarantees that all its eigenvalues are real and that a complete set of orthogonal eigenvectors exists.
• The Spectral Theorem provides an orthogonal diagonalization $A=QD{Q}^T$, where $Q$ is an orthogonal matrix of eigenvectors and $D$ is the diagonal eigenvalue matrix.
• The orthogonal diagonalization procedure requires finding eigenvalues, finding eigenvector bases, orthonormalizing them (using Gram-Schmidt within eigenspaces if needed), and then constructing $Q$ and $D$.
• This theory is directly applied to quadratic forms ${\mathbf{x}}^{T}A\mathbf{x}$ through the Principal Axis Theorem, which uses the eigenbasis to eliminate cross-terms, revealing the principal axes of a conic section, stress tensor, or vibrational system.
• In engineering, these concepts are essential for analyzing stability, optimizing systems, and performing coordinate transformations that decouple complex interactions into independent components.