Linear Algebra: Projection Matrices

Projection matrices are the computational engines that turn abstract vector space concepts into concrete numerical operations, bridging geometry and algebra. Understanding them is crucial because they are not just mathematical curiosities; they are fundamental tools for solving inconsistent equations in least squares regression, filtering noise in signal processing, and compressing data by isolating essential components.

The Geometric Foundation of Projection

At its heart, a projection is the act of dropping a perpendicular from a vector onto a subspace. Imagine a point in 3D space casting a shadow onto a 2D plane under a light directly overhead—that shadow is the orthogonal projection of the point onto the plane. In linear algebra, we formalize this. Given a vector $\mathbf{b}$ and a subspace spanned by the columns of a matrix $A$, we seek the vector $\mathbf{p}$ within that subspace that is closest to $\mathbf{b}$. This "closeness" is defined by minimizing the squared error $||\mathbf{b}-\mathbf{p}|{|}^2$, which leads directly to the condition that the error vector $(\mathbf{b}-\mathbf{p})$ must be orthogonal to the column space of $A$.

This orthogonality condition is the key that unlocks the formula. If $\mathbf{p}$ is in the column space of $A$, it can be written as $A\hat{\mathbf{x}}$ for some coefficient vector $\hat{\mathbf{x}}$. The orthogonality condition states $A^T(\mathbf{b}-A\hat{\mathbf{x}})=0$. Rearranging gives the normal equations: $A^{T}A\hat{\mathbf{x}}=A^T\mathbf{b}$. Solving for the coefficients yields $\hat{\mathbf{x}}=(A^{T}A{)}^{-1}{A}^T\mathbf{b}$. Finally, since the projected vector is $\mathbf{p}=A\hat{\mathbf{x}}$, we substitute to find $\mathbf{p}=A(A^{T}A{)}^{-1}{A}^T\mathbf{b}$.

Deriving and Understanding the Projection Matrix Formula

The matrix that acts on $\mathbf{b}$ to produce $\mathbf{p}$ is the projection matrix $P$. From our derivation, we extract it directly: $$P=A(A^{T}A{)}^{-1}{A}^T$$

This is the central formula. For it to be valid, the matrix $A$ must have full column rank, ensuring $A^{T}A$ is invertible. The matrix $P$ possesses several profound and interconnected algebraic properties that stem from its geometric purpose.

First, a projection matrix is idempotent. This means $P^2=P$. Why? Projecting a vector once onto a subspace places it perfectly within that subspace. Projecting it a second time changes nothing, as the vector is already there. Applying the formula confirms this: $P^2=[A(A^{T}A{)}^{-1}{A}^T][A(A^{T}A{)}^{-1}{A}^T]=A(A^{T}A{)}^{-1}(A^{T}A)(A^{T}A{)}^{-1}{A}^T=A(A^{T}A{)}^{-1}{A}^T=P$.

Second, a projection matrix for an orthogonal projection is symmetric, meaning $P^T=P$. You can verify this by taking the transpose of the formula: $P^T=(A(A^{T}A{)}^{-1}{A}^T{)}^T=A((A^{T}A{)}^T{)}^{-1}{A}^T=A(A^{T}A{)}^{-1}{A}^T=P$. This symmetry reflects the geometric symmetry of orthogonal projection.

The Rank and Complementary Projections

The rank of a projection matrix $P$ is precisely the dimension of the subspace onto which it projects. Since $P$ projects onto the column space of $A$ ($\mathcal{C}(A)$), we have $\text{rank}(P)=\text{rank}(A)$. Intuitively, the matrix can only output vectors within that $\text{rank}(A)$-dimensional space.

If $P$ projects a vector $\mathbf{b}$ onto a subspace, then where does the remaining part, the error, go? This leads to the concept of complementary projections. The error vector is $\mathbf{e}=\mathbf{b}-\mathbf{p}=(I-P)\mathbf{b}$. The matrix $I-P$ is itself a projection matrix—it projects onto the subspace orthogonal to $\mathcal{C}(A)$. You can check that it is also idempotent and symmetric. Together, $P$ and $I-P$ decompose any vector $\mathbf{b}$ into two orthogonal components: $\mathbf{b}=P\mathbf{b}+(I-P)\mathbf{b}$, lying in complementary subspaces.

Applications in Least Squares Regression

One of the most powerful applications is in least squares regression. In data fitting, you often have an overdetermined system $A\mathbf{x}\approx \mathbf{b}$ with no exact solution. The least squares solution finds the vector $\hat{\mathbf{x}}$ that minimizes the sum of squared residuals $||A\mathbf{x}-\mathbf{b}|{|}^2$. As we derived, this is exactly the projection of the data vector $\mathbf{b}$ onto the column space of the design matrix $A$. The fitted values $\hat{\mathbf{b}}=A\hat{\mathbf{x}}$ are simply $P\mathbf{b}$. The residuals are the orthogonal error component: $\mathbf{r}=\mathbf{b}-\hat{\mathbf{b}}=(I-P)\mathbf{b}$. This geometric view elegantly shows why the residuals are orthogonal to the fitted values.

Applications in Signal Processing

In signal processing and engineering, projection matrices are used for filtering and noise reduction. A common task is to separate a signal into components of interest and noise. You can construct a subspace that contains all "clean" signals of a certain type (e.g., all band-limited signals). The projection matrix $P$ onto this subspace acts as a filter: when you apply it to a noisy signal ${\mathbf{s}}_{noisy}$, the output $P{\mathbf{s}}_{noisy}$ is the best approximation of the clean signal within the desired subspace. The complementary projection $(I-P){\mathbf{s}}_{noisy}$ captures the noise component assumed to be orthogonal to the signal subspace. This principle underpins techniques like the Gram-Schmidt process for creating orthogonal bases, which is essential for efficient signal representation and compression.

Common Pitfalls

1. Assuming All Projections are Orthogonal: The formula $P=A(A^{T}A{)}^{-1}{A}^T$ defines an orthogonal projection. Not all projections in linear algebra are orthogonal (these are called oblique projections). A matrix can be idempotent ($P^2=P$) but not symmetric; such a matrix is a projection, but not an orthogonal one. Always check for symmetry to confirm orthogonality.
2. Applying the Formula When $A$ Lacks Full Column Rank: If the columns of $A$ are linearly dependent, $A^{T}A$ is singular and not invertible. The formula $P=A(A^{T}A{)}^{-1}{A}^T$ is invalid. In such cases, you must use a more general formula involving the pseudoinverse, or first find an orthonormal basis for the column space.
3. Confusing the Projection Matrix with the Coefficient Vector: Remember that $P$ acts on the data vector $\mathbf{b}$ to give the projected vector $\mathbf{p}$. The separate coefficient vector $\hat{\mathbf{x}}=(A^{T}A{)}^{-1}{A}^T\mathbf{b}$ solves the system and tells you "how much" of each column of $A$ is used to build $\mathbf{p}$. Don't mistake $\hat{\mathbf{x}}$ for the projection itself.
4. Overlooking the Geometric Picture: It's easy to get lost in the algebra. When stuck, return to the core geometry: projection finds the closest point in a subspace, and the error is perpendicular to that subspace. This intuition can guide you through derivations and help you sanity-check results.

Summary

• The projection matrix $P=A(A^{T}A{)}^{-1}{A}^T$ provides the linear transformation that orthogonally projects any vector onto the column space of a full-column-rank matrix $A$.
• Two defining algebraic properties of an orthogonal projection matrix are idempotence ($P^2=P$) and symmetry ($P^T=P$), both direct consequences of its geometric meaning.
• The rank of the projection matrix equals the dimension of the target subspace, and the matrix $I-P$ is the complementary projection onto the perpendicular subspace.
• In least squares regression, the projection matrix directly computes the fitted values by projecting the observed data vector onto the model's column space.
• In signal processing, projection matrices serve as optimal filters, separating a signal into components that lie within a desired subspace and orthogonal noise.