Linear Algebra: Orthonormal Bases and Applications

In engineering and data science, complex problems are often solved by breaking them into simpler, independent parts. Orthonormal bases provide the perfect mathematical framework for this decomposition, transforming messy calculations into clean, efficient, and geometrically intuitive operations. Mastering this concept is essential for applications ranging from compressing digital images to filtering noise from signals and solving large systems of equations with numerical stability.

What is an Orthonormal Basis?

A basis for a vector space is a set of vectors that are linearly independent and span the entire space, meaning any vector in that space can be written as a unique linear combination of the basis vectors. An orthonormal basis is a special basis with two critical properties: every vector in the set has unit length, and every pair of distinct vectors is orthogonal (perpendicular).

Formally, a set of vectors $\{{\mathbf{u}}_1,{\mathbf{u}}_2,...,{\mathbf{u}}_n\}$ is orthonormal if:

• ${\mathbf{u}}_i\cdot {\mathbf{u}}_i=\Vert {\mathbf{u}}_i{\Vert }^2=1$ for all $i$ (unit length).
• ${\mathbf{u}}_i\cdot {\mathbf{u}}_j=0$ for all $i\ne j$ (orthogonality).

The standard basis in ${\mathbb{R}}^3$, $\{{\mathbf{e}}_1=(1,0,0),{\mathbf{e}}_2=(0,1,0),{\mathbf{e}}_3=(0,0,1)\}$, is the classic example. Each axis-aligned vector is length 1 and at a right angle to the others. This orthogonality eliminates cross-talk between dimensions, which is the source of its computational power.

Expressing Vectors Using Inner Products

The true utility of an orthonormal basis emerges when you want to express an arbitrary vector $\mathbf{v}$ as a combination of the basis vectors. For a general basis, finding the coefficients requires solving a system of linear equations. For an orthonormal basis, the process simplifies dramatically: each coefficient is simply the inner product (dot product) of $\mathbf{v}$ with the corresponding basis vector.

If $\{{\mathbf{u}}_1,{\mathbf{u}}_2,...,{\mathbf{u}}_n\}$ is an orthonormal basis, then any vector $\mathbf{v}$ in the space can be written as: $$\mathbf{v}=c_1{\mathbf{u}}_1+c_2{\mathbf{u}}_2+...+c_n{\mathbf{u}}_n$$ where the coefficients are given by $c_i=\mathbf{v}\cdot {\mathbf{u}}_i$.

Why does this work? The inner product $\mathbf{v}\cdot {\mathbf{u}}_i$ measures the projection of $\mathbf{v}$ onto the direction of ${\mathbf{u}}_i$. Because the basis vectors are orthogonal, projecting onto ${\mathbf{u}}_i$ is unaffected by components of $\mathbf{v}$ along other basis directions. Because they are unit length, the projection magnitude is the exact coefficient needed. This means you can find each coefficient independently, a massive simplification.

Parseval's Identity: Conserved Energy

A profound consequence of working with an orthonormal basis is Parseval's identity. This theorem states that the squared length (or "energy") of a vector is equal to the sum of the squares of its coefficients with respect to an orthonormal basis.

Mathematically, if $\mathbf{v}={\sum }_{i=1}^n{c}_i{\mathbf{u}}_i$, then: $$\Vert \mathbf{v}{\Vert }^2=\mathbf{v}\cdot \mathbf{v}=c_1^2+c_2^2+...+c_n^2.$$

This is not true for a non-orthonormal basis. Parseval's identity is a generalization of the Pythagorean theorem to $n$ dimensions. For engineers, this is often interpreted as energy conservation: the total energy of a signal (represented by $\Vert \mathbf{v}{\Vert }^2$) is perfectly preserved when you sum the energies of its individual orthogonal components. This property is crucial for ensuring that transformations (like the Fourier transform) do not artificially amplify or lose signal power.

Fourier Coefficients as Orthogonal Projections

The concepts of orthonormal bases and projections find one of their most famous applications in Fourier analysis. Consider representing a periodic function as a sum of sines and cosines of different frequencies.

The set of functions $\{1,\cos (x),\sin (x),\cos (2x),\sin (2x),...\}$ over an interval $[-\pi ,\pi ]$ forms an orthogonal basis with respect to the inner product $f\cdot g={\int }_{-\pi }^{\pi }f(x)g(x)dx$. When properly normalized, it becomes orthonormal.

The Fourier coefficients—the $a_n$ and $b_n$ in the series expansion—are computed precisely as projections: $$a_n=\frac{f\cdot \cos (nx)}{\Vert \cos (nx){\Vert }^2},b_n=\frac{f\cdot \sin (nx)}{\Vert \sin (nx){\Vert }^2}.$$ This formula is identical in structure to $c_i=\mathbf{v}\cdot {\mathbf{u}}_i$ from the vector case. Each coefficient tells you "how much" of a specific frequency is present in the original signal. This allows you to decompose a complex waveform into its pure frequency components, analyze them, and reconstruct the signal perfectly.

Computational Advantages in Signal and Image Processing

The theoretical elegance of orthonormal bases translates directly into practical computational advantages, especially in signal and image processing.

1. Efficient Compression (JPEG/MP3): Signals and images are represented as vectors of pixel or amplitude values. Transforming this data into an orthonormal basis (like the Discrete Cosine Transform used in JPEG or various wavelet bases) often concentrates the signal's "energy" into a few large coefficients. You can discard the many small coefficients with minimal perceptual loss, achieving high compression ratios. Parseval's identity helps quantify the error introduced by this truncation.

1. Noise Filtering: Noise is typically spread across all basis coefficients. In a well-chosen basis (like the Fourier basis), the true signal is concentrated in specific coefficients. By zeroing out coefficients associated primarily with noise (e.g., high-frequency components), you can filter the signal cleanly and efficiently. This is far simpler than trying to operate on the raw, noisy data directly.

1. Numerical Stability: Solving the least-squares problem $A\mathbf{x}\approx \mathbf{b}$ is a cornerstone of engineering. If the columns of matrix $A$ are orthonormal, the solution simplifies to $\mathbf{x}=A^T\mathbf{b}$, which is trivial to compute and avoids the numerically unstable process of inverting $A^{T}A$. The QR decomposition, which factorizes a matrix into an orthogonal matrix $Q$ and an upper triangular $R$, exploits this to solve systems stably.

1. Simplified Calculations: Changes of coordinates, finding projections onto subspaces, and rotations all become straightforward matrix multiplications when the underlying basis is orthonormal. A matrix $Q$ with orthonormal columns has the property that $Q^{T}Q=I$, making its inverse equal to its transpose—an extremely cheap operation.

Common Pitfalls

1. Assuming Any Set of Perpendicular Vectors is Orthonormal: Orthogonality and normality are separate conditions. The set $\{(2,0),(0,3)\}$ is orthogonal but not orthonormal because the vectors are not unit length. You can normalize an orthogonal set by dividing each vector by its norm to create an orthonormal set.

1. Confusing Coefficients for General vs. Orthonormal Bases: For a general basis $\{{\mathbf{b}}_i\}$, you cannot find the coefficient for expressing $\mathbf{v}$ by just calculating $\mathbf{v}\cdot {\mathbf{b}}_i$. You must solve a system. Only for an orthonormal basis does the coefficient equal the inner product. Applying the simple dot product method to a non-orthonormal basis will give incorrect results.

1. Misapplying Parseval's Identity: Remember, Parseval's identity $\Vert \mathbf{v}{\Vert }^2=\sum c_i^2$ holds only if the coefficients $c_i$ are calculated with respect to an orthonormal basis. If your basis is orthogonal but not normalized, you must account for the squared lengths of the basis vectors. The correct formula for an orthogonal basis is $\Vert \mathbf{v}{\Vert }^2=\sum (c_i^{\prime }{)}^2\Vert {\mathbf{u}}_i{\Vert }^2$, where $c_i^{\prime }$ is the projection coefficient before normalization.

Summary

• An orthonormal basis consists of mutually perpendicular unit vectors, providing the simplest possible coordinate system for a vector space.
• Any vector can be expressed in an orthonormal basis using coefficients calculated as simple inner products (dot products), which represent geometric projections.
• Parseval's identity guarantees that the squared length (energy) of a vector equals the sum of squares of its coefficients, a crucial property for energy conservation in signal processing.
• Fourier coefficients are prime examples of these projection coefficients, allowing the decomposition of signals into their fundamental frequency components.
• The computational benefits are immense, enabling efficient data compression, noise filtering, and numerically stable algorithms foundational to modern signal and image processing.