Linear Algebra: Orthogonal Projections

In engineering, you constantly face problems of approximation and noise reduction. Whether you're filtering a corrupted signal, compressing an image, or fitting a line to noisy sensor data, the mathematical machinery that enables you to find the best possible approximation is the orthogonal projection. This operation allows you to decompose a complex vector into a meaningful component within a subspace of interest and an irrelevant, orthogonal "error" component. Mastering projections transforms abstract vector spaces into a practical toolkit for simplifying and solving real-world problems.

The Geometry and Derivation of the Projection Formula

At its heart, an orthogonal projection maps a vector onto a subspace by dropping a perpendicular "shadow." The simplest case is projecting a vector $\vec{b}$ onto a line spanned by another vector $\vec{a}$. The goal is to find the scalar multiple $c\vec{a}$ that is closest to $\vec{b}$. "Closest" here is defined by minimizing the squared error $||\vec{b}-c\vec{a}|{|}^2$, which is equivalent to making the error vector $(\vec{b}-c\vec{a})$ perpendicular to $\vec{a}$.

This orthogonality condition gives us the key equation: $$\vec{a}\cdot (\vec{b}-c\vec{a})=0.$$ Solving for the scalar $c$ yields the projection formula: $$c=\frac{\vec{a}\cdot \vec{b}}{\vec{a}\cdot \vec{a}}.$$ Therefore, the projection of $\vec{b}$ onto the line of $\vec{a}$ is: $${\text{proj}}_{\vec{a}}(\vec{b})=\left(\frac{\vec{a}\cdot \vec{b}}{\vec{a}\cdot \vec{a}}\right)\vec{a}.$$ The denominator $\vec{a}\cdot \vec{a}$ is the squared length $||\vec{a}|{|}^2$. The component of $\vec{b}$ orthogonal to $\vec{a}$ is simply $\vec{b}-{\text{proj}}_{\vec{a}}(\vec{b})$. This geometric derivation—minimizing distance by enforcing orthogonality—is the foundational principle that generalizes to all subspaces.

Projection onto Higher-Dimensional Subspaces and Planes

The logic extends seamlessly from lines to planes or any subspace defined by a basis. Suppose you want to project a vector $\vec{b}$ onto a plane (a 2D subspace) in ${\mathbb{R}}^3$ spanned by two independent vectors ${\vec{a}}_1$ and ${\vec{a}}_2$. The projection $\hat{\vec{b}}$ will be a linear combination $x_1{\vec{a}}_1+x_2{\vec{a}}_2$ that lies in the plane. The error vector $\vec{e}=\vec{b}-\hat{\vec{b}}$ must be perpendicular to the entire plane, meaning it is orthogonal to both ${\vec{a}}_1$ and ${\vec{a}}_2$.

This gives two orthogonality conditions: $${\vec{a}}_1\cdot (\vec{b}-x_1{\vec{a}}_1-x_2{\vec{a}}_2)=0,$$ $${\vec{a}}_2\cdot (\vec{b}-x_1{\vec{a}}_1-x_2{\vec{a}}_2)=0.$$ These equations reorganize into a $2\times 2$ system of normal equations: $$\{\begin{array}{l}({\vec{a}}_1\cdot {\vec{a}}_1)x_1+({\vec{a}}_1\cdot {\vec{a}}_2)x_2={\vec{a}}_1\cdot \vec{b},\\ ({\vec{a}}_2\cdot {\vec{a}}_1)x_1+({\vec{a}}_2\cdot {\vec{a}}_2)x_2={\vec{a}}_2\cdot \vec{b}.\end{array}$$ In matrix form, if $A=[{\vec{a}}_1{\vec{a}}_2]$, this system is $A^{T}A\vec{x}=A^T\vec{b}$. The solution vector $\vec{x}$ contains the coefficients that produce the projection: $\hat{\vec{b}}=A\vec{x}=A(A^{T}A{)}^{-1}{A}^T\vec{b}$. This framework works for projecting onto any column space of a matrix $A$.

The Orthogonal Decomposition Theorem

The previous sections implicitly used a powerful, general result: the Orthogonal Decomposition Theorem. It states that for any vector $\vec{b}$ in ${\mathbb{R}}^n$ and any subspace $W$, $\vec{b}$ can be written uniquely as a sum: $$\vec{b}=\hat{\vec{b}}+\vec{z},$$ where $\hat{\vec{b}}$ is in $W$ and $\vec{z}$ is in $W^{\perp }$ (the orthogonal complement of $W$). The component $\hat{\vec{b}}$ is precisely the orthogonal projection of $\vec{b}$ onto $W$, and it is the unique vector in $W$ closest to $\vec{b}$.

This decomposition is fundamental. The vector $\hat{\vec{b}}$ represents the best approximation to $\vec{b}$ from within the subspace $W$. The orthogonal component $\vec{z}$ represents the unavoidable error or residual of this approximation. In engineering terms, if $W$ is the space of valid signal models, $\hat{\vec{b}}$ is the cleaned signal, and $\vec{z}$ is the removed noise.

Projection Matrices and Their Key Properties

The formula $\hat{\vec{b}}=A(A^{T}A{)}^{-1}{A}^T\vec{b}$ reveals a linear transformation: $\vec{b}\to \hat{\vec{b}}$. The matrix that performs this transformation is the projection matrix $P$: $$P=A(A^{T}A{)}^{-1}{A}^T.$$ This matrix projects any vector onto the column space of $A$.

Projection matrices have two distinctive algebraic properties that you should recognize instantly. First, they are idempotent, meaning $P^2=P$. Projecting a vector that is already in the subspace does nothing. Second, they are symmetric, so $P^T=P$. This symmetry is a direct consequence of the orthogonal projection and is evident in the formula.

A special case occurs when projecting onto a line in the direction of a unit vector $\vec{u}$. Here, $A$ becomes $\vec{u}$, $A^{T}A=1$, and the projection matrix simplifies to $P=\vec{u}{\vec{u}}^T$, which is an outer product. Understanding these properties helps you verify calculations and reason about the behavior of iterative algorithms that use projections.

Engineering Applications: Signal Processing and Data Approximation

Orthogonal projections are not just abstract exercises; they are the workhorse of many engineering systems. In signal processing, a common task is to separate a signal from noise. You can model the space of "clean" signals as a subspace $W$ (e.g., signals of a certain frequency band). The received noisy signal $\vec{b}$ is then projected onto $W$ to obtain the best estimate $\hat{\vec{b}}$ of the original signal, effectively filtering out orthogonal noise components. This is the principle behind many filtering techniques, including those used in audio and image processing.

In data science and data approximation, linear regression is a direct application. Fitting a least-squares line $y=mx+c$ to a set of data points $\{(x_i,y_i)\}$ is equivalent to projecting the data vector $\vec{y}$ onto the subspace spanned by the vector of ones (for the constant term $c$) and the vector of $x$-values (for the slope term $m$). The projection $\hat{\vec{y}}$ gives the predicted values on the best-fit line, and the orthogonal error vector $\vec{e}=\vec{y}-\hat{\vec{y}}$ contains the residuals, which are minimized in length.

Common Pitfalls

1. Misapplying the Formula for Non-Orthogonal Bases: The standard projection formula $A(A^{T}A{)}^{-1}{A}^T\vec{b}$ works for any basis $A$ of the subspace. A common mistake is trying to use the simple one-vector formula $(\vec{u}\cdot \vec{b}/\vec{u}\cdot \vec{u})\vec{u}$ for each basis vector separately and adding the results. This only yields the correct projection if the basis vectors are orthogonal to each other. If they are not, you must solve the normal equations or use the matrix formula.

1. Confusing the Projection with the Component Vector: The projection $\hat{\vec{b}}$ is a vector in the original space (e.g., in ${\mathbb{R}}^3$). The coefficient vector $\vec{x}$ in $\hat{\vec{b}}=A\vec{x}$ is a different object—it lists the coordinates of the projection *with respect to the basis $A$*. Don't report $\vec{x}$ as the final answer when the problem asks for the projection vector itself.

1. Forgetting that $A^{T}A$ Must Be Invertible: The formula $P=A(A^{T}A{)}^{-1}{A}^T$ requires that $A^{T}A$ be invertible. This condition holds if and only if the columns of $A$ are linearly independent. If you construct $A$ with dependent columns, you have not defined a proper basis for the subspace, and the formula fails. Always check that your spanning vectors are independent.

1. Assuming Projection Matrices are Invertible: A projection matrix is never invertible (unless it projects onto the whole space, which is just the identity matrix). Since it squashes any orthogonal component to zero, it has a nontrivial nullspace. Remember the idempotent property $P^2=P$; a true inverse cannot exist.

Summary

• The orthogonal projection of a vector $\vec{b}$ onto a subspace is the unique vector in that subspace closest to $\vec{b}$, found by enforcing that the error vector is perpendicular to the subspace.
• The projection formula generalizes from a line $(\frac{\vec{a}\cdot \vec{b}}{\vec{a}\cdot \vec{a}}\vec{a})$ to a subspace via the matrix equation $\hat{\vec{b}}=A(A^{T}A{)}^{-1}{A}^T\vec{b}$, which solves the normal equations $A^{T}A\vec{x}=A^T\vec{b}$.
• The Orthogonal Decomposition Theorem guarantees any vector can be uniquely split into a projection (in the subspace) and an orthogonal error component, which is the foundation for best approximation.
• Projection matrices $P$ are symmetric ($P^T=P$) and idempotent ($P^2=P$). They provide an algebraic tool for repeatedly applying projections.
• Core engineering applications include least-squares data fitting (regression) and signal filtering, where projecting onto a "clean signal" subspace removes orthogonal noise.