Linear Algebra: Projection Matrices

Projection matrices are the computational engines that power tasks from fitting a trendline to noisy data to cleaning up a distorted signal. They provide the rigorous linear algebra framework for taking a vector and casting its shadow onto a subspace, a fundamental operation for approximation and decomposition. Mastering their properties and construction is essential for any engineer or data scientist working with models, signals, or high-dimensional data.

The Geometry and Formula for Projection

At its core, an orthogonal projection is the operation of dropping a perpendicular from a vector onto a subspace. Imagine a point in 3D space; its shadow on the flat ground, cast directly from a light source overhead, is an orthogonal projection onto the 2D plane. In linear algebra, we project a vector $\vec{b}$ onto the column space of a matrix $A$, which defines our target subspace.

The goal is to find the vector $\vec{p}$ in the column space of $A$ that is closest to $\vec{b}$. This "closest" vector is defined by the condition that the error $\vec{e}=\vec{b}-\vec{p}$ is orthogonal to the column space of $A$. Mathematically, this orthogonality condition is $A^T(\vec{b}-\vec{p})=\vec{0}$. Since $\vec{p}$ is in the column space of $A$, it can be written as $\vec{p}=A\vec{x}$ for some coefficient vector $\vec{x}$.

Substituting $\vec{p}=A\vec{x}$ into the orthogonality condition gives the normal equations: $A^{T}A\vec{x}=A^T\vec{b}$. Solving for the coefficients yields $\vec{x}=(A^{T}A{)}^{-1}{A}^T\vec{b}$. Finally, the projected vector is $\vec{p}=A\vec{x}=A(A^{T}A{)}^{-1}{A}^T\vec{b}$.

This leads us to the central object: the projection matrix $P$. We define it as the matrix that acts on $\vec{b}$ to produce $\vec{p}$: $$P=A(A^{T}A{)}^{-1}{A}^T$$ For any vector $\vec{b}$, the projection onto the column space of $A$ is $\vec{p}=P\vec{b}$. It is critical that $A$ has full column rank for $(A^{T}A)$ to be invertible; if not, a generalized inverse is required, but we assume full rank here.

Fundamental Algebraic Properties

Projection matrices defined by $P=A(A^{T}A{)}^{-1}{A}^T$ possess two defining and deeply interconnected algebraic properties.

First, a projection matrix is idempotent. A matrix is idempotent if $P^2=P$. This makes perfect geometric sense: once you project a vector onto a subspace, projecting it again changes nothing. The vector is already in the subspace. You can verify this algebraically: $$P^2=[A(A^{T}A{)}^{-1}{A}^T][A(A^{T}A{)}^{-1}{A}^T]=A(A^{T}A{)}^{-1}(A^{T}A)(A^{T}A{)}^{-1}{A}^T=A(A^{T}A{)}^{-1}{A}^T=P.$$

Second, a projection matrix for an orthogonal projection is symmetric. A matrix is symmetric if $P^T=P$. This property flows from the formula and is tied to the orthogonality of the error. You can confirm it by taking the transpose: $$P^T=(A(A^{T}A{)}^{-1}{A}^T{)}^T=A((A^{T}A{)}^{-1}{)}^T{A}^T=A(A^{T}A{)}^{-1}{A}^T=P,$$ where we use the fact that $(A^{T}A{)}^{-1}$ is symmetric. The symmetry and idempotence properties together are the hallmarks of an orthogonal projection matrix.

Rank and Complementary Projections

The rank of a projection matrix $P$ is precisely the dimension of the subspace onto which it projects. Since $P$ projects onto the column space of $A$, $\text{rank}(P)=\text{rank}(A)$. For example, if $A$ is an $m\times n$ matrix with $n$ independent columns, then $\text{rank}(P)=n$. The trace of a projection matrix also equals its rank.

For every projection matrix $P$ onto a subspace, there exists a complementary projection onto the orthogonal subspace. This is the matrix $P_{\perp }=I-P$, which projects onto the space orthogonal to the column space of $A$. It projects a vector $\vec{b}$ onto the error vector $\vec{e}=\vec{b}-\vec{p}$. This complementary matrix is also idempotent and symmetric. Crucially, their sum is the identity matrix: $P+(I-P)=I$, and their product is zero: $P(I-P)=0$, reflecting the orthogonality of the two subspaces.

Application to Least-Squares Regression

The most direct application is linear least-squares regression. In statistical modeling or curve fitting, you have an overdetermined system $A\vec{x}\approx \vec{b}$, where $A$ contains the predictor data, $\vec{x}$ are the unknown coefficients, and $\vec{b}$ is the response vector. The system typically has no exact solution. The least-squares solution finds the coefficient vector $\hat{\vec{x}}$ that minimizes the squared error $||A\vec{x}-\vec{b}|{|}^2$.

This is exactly the projection problem. The least-squares solution is given by the normal equations: $\hat{\vec{x}}=(A^{T}A{)}^{-1}{A}^T\vec{b}$. The vector of predicted or "fitted" values is: $$\vec{p}=A\hat{\vec{x}}=A(A^{T}A{)}^{-1}{A}^T\vec{b}=P\vec{b}.$$ Here, $P$ is often called the hat matrix in statistics because it puts the "hat" on $\vec{b}$ to produce $\hat{\vec{b}}=\vec{p}$. The residual vector (the errors) is $\vec{e}=\vec{b}-\vec{p}=(I-P)\vec{b}$, which is the projection onto the orthogonal complement.

Application to Signal Processing in Engineering

In signal processing, projection matrices are used for tasks like noise filtering and signal separation. Consider a received signal vector $\vec{s}$ that is a combination of a desired signal component lying in a known subspace and an unwanted noise component.

If the subspace of valid signals is the column space of a matrix $A$, then the operation ${\vec{s}}_{clean}=P\vec{s}$ extracts the component of the received signal that lies in that valid subspace. This effectively filters out any component of the noise that is orthogonal to the signal subspace. For instance, in communications, if the transmitted signals are known to be linear combinations of specific waveforms, projecting the noisy received signal onto the subspace spanned by those waveforms can significantly enhance the signal-to-noise ratio.

Conversely, the operation $(I-P)\vec{s}$ isolates the component orthogonal to the signal subspace, which can be analyzed as noise or used in anomaly detection. This decomposition of a vector into orthogonal components—one in a subspace of interest and one in its complement—is a foundational signal processing technique enabled by projection matrices.

Common Pitfalls

1. Applying the formula without checking for linear independence.

• Pitfall: Directly computing $P=A(A^{T}A{)}^{-1}{A}^T$ when the columns of $A$ are linearly dependent. This makes $A^{T}A$ singular and non-invertible.
• Correction: First confirm that $A$ has full column rank. If it does not, the subspace is still well-defined, but you must use a more robust method like the singular value decomposition (SVD) to construct the projector onto the column space.

2. Confusing the rank of the matrix with the size of the matrix.

• Pitfall: Assuming an $m\times m$ projection matrix $P$ has rank $m$. A projection matrix is often singular.
• Correction: Remember that $\text{rank}(P)=\text{dimension of the subspace}$. If you project ${\mathbb{R}}^3$ onto a plane, $P$ is a 3x3 matrix but has rank 2. Its trace will also be 2.

3. Misunderstanding the action of the complementary projector.

• Pitfall: Thinking $I-P$ projects onto "everything else" in a non-orthogonal sense.
• Correction: $I-P$ specifically projects onto the orthogonal complement of the column space of $A$. The error vector $\vec{e}=(I-P)\vec{b}$ is perpendicular to every vector in the column space of $A$.

4. Assuming all idempotent matrices are orthogonal projectors.

• Pitfall: Concluding a matrix is an orthogonal projection matrix just because $P^2=P$.
• Correction: An orthogonal projection must be both idempotent and symmetric. A matrix that is idempotent but not symmetric represents an oblique projection, where the error vector is not perpendicular to the subspace.

Summary

• The projection matrix $P=A(A^{T}A{)}^{-1}{A}^T$ orthogonally projects any vector onto the column space of a full-column-rank matrix $A$.
• Orthogonal projection matrices are idempotent ($P^2=P$) and symmetric ($P^T=P$), properties that define their behavior.
• The rank of the projection matrix equals the dimension of the target subspace, and the complementary projector $I-P$ projects onto the orthogonal complement.
• In regression, $P$ (the hat matrix) produces the least-squares fitted values, while $I-P$ produces the residuals.
• In signal processing, these matrices separate a signal into components within a desired subspace and orthogonal noise, enabling filtering and enhancement.