Linear Algebra: Quadratic Forms

Quadratic forms are not just abstract algebraic expressions; they are the fundamental mathematical language for describing energy in mechanical systems, optimization landscapes in machine learning, and the shape of surfaces in computer graphics. Mastering them provides you with the tools to analyze stability, classify geometric objects, and solve complex engineering optimization problems by translating a multi-variable quadratic expression into the powerful framework of matrix algebra.

What is a Quadratic Form?

A quadratic form is a homogeneous polynomial of degree two in $n$ variables, where "homogeneous" means every term has the same total degree (two). For two variables $x$ and $y$, it looks like $Q(x,y) = ax^2 + bxy + cy^2$. The key is that there are no linear terms (like $dx$) or constant terms. In $n$ variables $x_1, x_2, \ldots, x_n$, it is a sum where each term is of the form $a_{ij}x_i x_j$.

The power of linear algebra comes from representing this polynomial compactly. Every quadratic form can be uniquely expressed as $Q(\mathbf{x}) = \mathbf{x}^T A \mathbf{x}$, where $\mathbf{x}$ is a column vector of the variables and $A$ is a symmetric matrix ($A^T = A$). For example, the form $Q(x_1, x_2) = 5x_1^2 + 3x_1x_2 + 8x_2^2$ is written with $\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$ and $A = \begin{bmatrix} 5 & 1.5 \\ 1.5 & 8 \end{bmatrix}$. Notice how the coefficient 3 of the cross-term $x_1x_2$ is split equally to make the matrix symmetric: $a_{12} = a_{21} = 1.5$. This symmetric representation is crucial for all subsequent analysis.

Classifying Quadratic Forms: Definiteness

The classification of a quadratic form is based on the sign of its output for all non-zero input vectors $\mathbf{x}$. This definiteness tells you about the "shape" of the associated surface.

• Positive Definite: $Q(\mathbf{x}) > 0$ for all $\mathbf{x} \neq \mathbf{0}$. Graphically, this corresponds to an upward-opening paraboloid or ellipsoid. Its matrix $A$ has all positive eigenvalues.
• Positive Semidefinite: $Q(\mathbf{x}) \ge 0$ for all $\mathbf{x}$, and there exists some $\mathbf{x} \neq \mathbf{0}$ where $Q(\mathbf{x}) = 0$. The surface is like a parabolic cylinder or a "bowl" that is flat along certain directions. $A$ has all non-negative eigenvalues, with at least one being zero.
• Negative Definite/Semidefinite: The opposites of the above, with all outputs negative (or non-positive). The matrix has all negative (or non-positive) eigenvalues.
• Indefinite: $Q(\mathbf{x})$ takes on both positive and negative values. This describes a hyperbolic paraboloid (a saddle surface). Its matrix $A$ has both positive and negative eigenvalues.

In engineering, positive definiteness is a hallmark of stable equilibrium in physical systems (e.g., the stiffness matrix in structural analysis), while an indefinite Hessian matrix in optimization indicates a saddle point, not a local minimum.

Diagonalization: Completing the Square and the Principal Axis Theorem

To simplify and analyze a quadratic form, we seek to eliminate the cross-terms. This process is the multivariate generalization of completing the square. Algebraically, it involves a change of variable $\mathbf{x} = P\mathbf{y}$ to transform $Q(\mathbf{x}) = \mathbf{x}^TA\mathbf{x}$ into $Q(\mathbf{y}) = \mathbf{y}^TD\mathbf{y} = \lambda_1 y_1^2 + \lambda_2 y_2^2 + ... + \lambda_n y_n^2$, where $D$ is a diagonal matrix.

This is formalized by the Principal Axis Theorem. It states that for any real symmetric matrix $A$, there exists an orthogonal matrix $P$ (whose columns are orthonormal eigenvectors of $A$) such that $P^TAP = D$, where $D$ is a diagonal matrix of the eigenvalues of $A$. The new variables $\mathbf{y} = P^T\mathbf{x}$ represent the coordinates along the principal axes of the quadratic surface. This theorem is the spectral theorem in action and is the most robust, systematic method for diagonalization.

The signature of a quadratic form is a compact summary of its defineness, given as a triple $(p, z, n)$, where $p$ is the number of positive eigenvalues, $z$ is the number of zero eigenvalues, and $n$ is the number of negative eigenvalues. For a $2 \times 2$ matrix, a signature of $(2,0,0)$ means positive definite, $(1,0,1)$ means indefinite (a saddle), and $(1,1,0)$ means positive semidefinite.

Key Applications: Optimization and Conic Sections

The diagonalization process directly enables two major applications.

First, in multivariable optimization, the Hessian matrix of second derivatives is a symmetric matrix defining a quadratic form. The definiteness of this Hessian at a critical point determines the nature of that point: a local minimum if positive definite, a local maximum if negative definite, and a saddle point if indefinite. This is the essence of the second derivative test for functions of several variables.

Second, quadratic forms are used to classify conic sections and quadric surfaces. The general equation for a conic $Ax^2 + Bxy + Cy^2 + Dx + Ey + F = 0$ has a quadratic part represented by a $2 \times 2$ matrix. By diagonalizing this matrix (rotating the axes to eliminate the $Bxy$ term via the Principal Axis Theorem), you can easily identify the conic as an ellipse, hyperbola, or parabola based on the signs of the eigenvalues (the signature). This extends to 3D for classifying surfaces like ellipsoids, hyperboloids, and paraboloids, which is vital in computer vision and geometric modeling.

Common Pitfalls

1. Forgetting to Symmetrize the Matrix: A common error is writing the matrix $A$ directly from coefficients without splitting cross-terms. For $Q = 2x^2 + 6xy + 4y^2$, the incorrect matrix is $\begin{bmatrix} 2 & 6 \\ 0 & 4 \end{bmatrix}$. The correct, symmetric matrix is $\begin{bmatrix} 2 & 3 \\ 3 & 4 \end{bmatrix}$. Only the symmetric representation guarantees real eigenvalues and orthogonal eigenvectors, which all major theorems require.

1. Confusing "Positive" with "Positive Definite": A matrix can have all positive entries yet be indefinite. Definiteness is a property of the eigenvalues, not the entries. For example, $A = \begin{bmatrix} 1 & 4 \\ 4 & 1 \end{bmatrix}$ has positive entries but eigenvalues $5$ and $-3$, making it indefinite. Always test using eigenvalues, determinants of leading principal minors, or reliable numerical methods.

1. Misapplying the 2x2 Determinant Test for Larger Matrices: For a $2\times2$ symmetric matrix $A$, if $\det(A) > 0$, it's either positive or negative definite. You must check $a_{11}$ to decide which. For larger matrices, this simple rule fails. You must check that all leading principal minors are positive for positive definiteness (Sylvester's criterion) or find the eigenvalues.

1. Ignoring the Orthogonal Transformation in the Principal Axis Theorem: The theorem states you can use an orthogonal $P$ (a rotation/reflection) to diagonalize a real symmetric matrix. This preserves the geometry (lengths and angles) while aligning the axes. Using a non-orthogonal $P$ (from other decompositions like LU) will change the shape of the quadratic surface, defeating the purpose of geometric analysis.

Summary

• A quadratic form $Q(\mathbf{x})$ is a degree-2 polynomial that can be uniquely written as $\mathbf{x}^T A \mathbf{x}$ where $A$ is a symmetric matrix.
• Forms are classified by definiteness—positive definite, semidefinite, indefinite, etc.—based on the signs of their outputs or, equivalently, the eigenvalues of $A$.
• Completing the square is generalized by the Principal Axis Theorem, which states a real symmetric matrix can be orthogonally diagonalized, eliminating cross-terms and revealing the signature $(p, z, n)$.
• These tools are essential for executing the second derivative test in multivariable optimization and for classifying conic sections and quadric surfaces by diagonalizing the associated quadratic form matrix.