Linear Discriminant Analysis for Classification

Linear Discriminant Analysis (LDA) is a cornerstone of statistical pattern recognition, providing a powerful and elegant framework for supervised classification. At its core, LDA finds the optimal linear boundaries between classes by leveraging assumptions about the underlying data distribution. Understanding LDA not only gives you a robust classification tool but also deepens your insight into dimensionality reduction, Bayes' optimal decisions, and the fundamental trade-offs in model complexity.

Foundational Assumptions: Gaussian Classes with Shared Covariance

LDA is built upon a specific probabilistic model. It assumes that the data for each class $k$ is generated from a multivariate Gaussian (normal) distribution. This means the probability density function for a feature vector $\mathbf{x}$ given class $k$ is:

$$P(\mathbf{x}|Y=k)=\frac{1}{(2\pi {)}^{p/2}|\mathbf{\Sigma }{|}^{1/2}}\exp \left(-\frac{1}{2}(\mathbf{x}-{\mathbf{\mu }}_k{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-{\mathbf{\mu }}_k)\right)$$

The critical assumption here is that all classes share the same covariance matrix $\mathbf{\Sigma }$. Think of it as assuming each class forms an identically shaped and oriented "cloud" in feature space, but each cloud is centered at its own class mean ${\mathbf{\mu }}_k$. This assumption of homoscedasticity (equal covariance) is what leads to linear decision boundaries.

To classify a new point $\mathbf{x}$, LDA applies Bayes' theorem. It computes the posterior probability $P(Y=k|\mathbf{x})$—the probability of belonging to class $k$ given the observed features. The classifier then assigns $\mathbf{x}$ to the class with the highest posterior probability. After applying the Gaussian density and simplifying with logarithms, the discriminant function ${\delta }_k(\mathbf{x})$ for class $k$ becomes:

$${\delta }_k(\mathbf{x})={\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}{\mathbf{\mu }}_k-\frac{1}{2}{\mathbf{\mu }}_k^T{\mathbf{\Sigma }}^{-1}{\mathbf{\mu }}_k+\log ({\pi }_k)$$

where ${\pi }_k$ is the prior probability of class $k$. You classify $\mathbf{x}$ by calculating ${\delta }_k(\mathbf{x})$ for all classes and choosing the $k$ that yields the largest value. The function is linear in $\mathbf{x}$, hence the name Linear Discriminant Analysis. The decision boundary between any two classes is found where their discriminant functions are equal, which forms a hyperplane.

Fisher's Discriminant: The Geometric Perspective

An equivalent and highly intuitive view of LDA is provided by Fisher's Linear Discriminant. This approach frames the problem as a projection. Instead of modeling distributions, Fisher asked: "What one-dimensional projection (i.e., line) of the data maximizes the separation between the classes?"

He defined separation as the ratio of the between-class variance to the within-class variance for the projected data. You want the class means to be as far apart as possible (large between-class variance) while keeping the data from each class as tight as possible (small within-class variance). This ratio is called Fisher's criterion.

Mathematically, for a two-class problem, we seek a projection vector $\mathbf{w}$ that maximizes:

$$J(\mathbf{w})=\frac{({\mathbf{w}}^T{\mathbf{\mu }}_1-{\mathbf{w}}^T{\mathbf{\mu }}_2{)}^2}{{\mathbf{w}}^T\mathbf{\Sigma }\mathbf{w}}=\frac{\text{Between-class variance}}{\text{Within-class variance}}$$

The solution for the optimal $\mathbf{w}$ is proportional to ${\mathbf{\Sigma }}^{-1}({\mathbf{\mu }}_1-{\mathbf{\mu }}_2)$. For $K$ classes, Fisher's method finds up to $K-1$ projection directions that maximize class separation in a lower-dimensional subspace. This projection is identical to the classification rule derived from the Gaussian assumption. You project the data onto these directions and then perform classification in this reduced space, which is excellent for visualization and combating the curse of dimensionality.

Extension to Quadratic Discriminant Analysis (QDA)

What if the assumption of a shared covariance matrix is too restrictive? In many real-world datasets, one class might be more spread out than another. Quadratic Discriminant Analysis (QDA) relaxes this key LDA assumption. It models each class $k$ with its own multivariate Gaussian distribution, characterized by a unique mean ${\mathbf{\mu }}_k$ and a unique covariance matrix ${\mathbf{\Sigma }}_k$.

Following the same Bayes' rule procedure, the discriminant function for QDA becomes:

$${\delta }_k^{QDA}(\mathbf{x})=-\frac{1}{2}\log |{\mathbf{\Sigma }}_k|-\frac{1}{2}(\mathbf{x}-{\mathbf{\mu }}_k{)}^T{\mathbf{\Sigma }}_k^{-1}(\mathbf{x}-{\mathbf{\mu }}_k)+\log ({\pi }_k)$$

Notice the term $(\mathbf{x}-{\mathbf{\mu }}_k{)}^T{\mathbf{\Sigma }}_k^{-1}(\mathbf{x}-{\mathbf{\mu }}_k)$ is not linear in $\mathbf{x}$ because ${\mathbf{\Sigma }}_k$ is inside the quadratic form. This results in quadratic decision boundaries, which are more flexible than the hyperplanes from LDA. QDA can model more complex, curved shapes separating the classes. The trade-off is increased model complexity: QDA must estimate many more parameters (a separate covariance matrix for each class), which requires more data to avoid high variance in the estimates.

LDA in Comparison with Logistic Regression

Both LDA and logistic regression are linear classification methods, but they originate from different philosophical approaches. Logistic regression models the posterior probabilities $P(Y=k|\mathbf{x})$ directly using the logistic (or softmax) function. It makes no explicit assumptions about the distribution of the features $\mathbf{x}$. It is a discriminative model because it focuses on modeling the boundary between classes.

LDA, in contrast, is a generative model. It models the full joint distribution $P(\mathbf{x},Y)$ by specifying the class-conditional distributions $P(\mathbf{x}|Y)$ and the class priors $P(Y)$. Classification is performed by applying Bayes' rule to these modeled distributions.

This distinction leads to practical differences:

• When Assumptions Hold: When the Gaussian class-conditional assumption with shared covariance is approximately true, LDA tends to be more statistically efficient than logistic regression, especially with small sample sizes. It uses the data more effectively because it's based on a "correct" model of the data generation process.
• Robustness to Model Misspecification: Logistic regression is more robust. If the features are clearly non-Gaussian (e.g., binary features), logistic regression will typically outperform LDA because it doesn't rely on those distributional assumptions.
• Stability with Few Data Points: With well-separated classes or very few data points, the parameters in logistic regression can become unstable (tending toward infinity), while LDA's parameter estimates remain stable.

In practice, the performance of the two methods is often very similar. The choice can come down to the nature of your data and your comfort with the underlying assumptions.

Common Pitfalls

1. Assuming Linearity is Always Sufficient: Applying LDA to data where classes have vastly different covariance structures or are separated by a highly non-linear boundary will lead to poor performance. Always visualize your data or use techniques like cross-validation to compare LDA with QDA or non-linear classifiers. If a simple quadratic boundary (QDA) performs significantly better, your data violates LDA's core assumption.
2. Ignoring the Prior Probabilities (${\pi }_k$): The default in many software implementations is to estimate priors from the class proportions in the training set. However, if your operational environment has a different class prevalence (e.g., diagnosing a rare disease), you must explicitly set the priors ${\pi }_k$ to reflect the true base rates. Failing to do so will bias predictions toward the more common class in your training data, which may not be appropriate.
3. Using LDA with Categorical or Sparse Features: LDA's Gaussian assumption is poorly suited for binary, count, or extremely high-dimensional sparse data (like text bag-of-words). The estimates for the covariance matrix become meaningless or singular. In these cases, models like logistic regression, Naive Bayes (with appropriate distributions), or regularized classifiers are far more appropriate.
4. Confusing Classification LDA with Dimensionality Reduction LDA: It's important to distinguish between LDA as a classifier (which outputs class labels) and Fisher's LDA as a supervised dimensionality reduction technique (which outputs projections). They are derived from the same math, but the latter is often used as a preprocessing step for visualization or before applying another classifier.

Summary

• LDA is a generative, probabilistic classifier that assumes each class follows a Gaussian distribution with a class-specific mean but a common covariance matrix across all classes. This leads to optimal linear decision boundaries.
• Fisher's Discriminant provides a geometric interpretation, finding projections that maximize between-class separation relative to within-class spread. This is identical to the classification LDA solution and is a powerful tool for supervised dimensionality reduction.
• Quadratic Discriminant Analysis (QDA) relaxes the equal-covariance assumption, allowing each class its own covariance matrix. This results in more flexible quadratic decision boundaries at the cost of requiring more data to reliably estimate more parameters.
• Compared to logistic regression, LDA is often more efficient if its Gaussian assumptions are met, especially with limited data. Logistic regression is more robust when these assumptions are violated, as it makes no distributional claims about the features.
• Successful application requires diagnosing your data's structure. Use visualization and model comparison to check if the linearity and homoscedasticity assumptions are reasonable before choosing LDA as your classifier.