Explained Variance and Scree Plots

Selecting the right number of principal components is the critical bridge between performing a mathematical decomposition and extracting genuine insight from your data. The goal is to balance dimensionality reduction against information retention, a process guided by quantifying how much of your data's story each component tells. Mastery of explained variance and scree plots transforms PCA from a black box into a strategic, interpretable tool.

1. Quantifying Information: The Explained Variance Ratio

After performing PCA, you obtain a set of principal components, each associated with an eigenvalue. The eigenvalue quantifies the variance captured by its corresponding component. To understand its relative importance, you calculate the explained variance ratio.

For a given principal component $i$, with eigenvalue ${\lambda }_i$, the explained variance ratio is: $${\text{Explained Variance Ratio}}_i=\frac{{\lambda }_i}{{\sum }_{j=1}^p{\lambda }_j}$$ where $p$ is the total number of original features (and thus the total number of components). This ratio tells you the proportion of the dataset's total variance that is explained by component $i$.

For example, if your first principal component has an explained variance ratio of 0.65, it means that this single new dimension captures 65% of the total spread present in your original, potentially high-dimensional data. This is the foundational metric for all subsequent decision-making about component retention.

2. Visual Diagnostics: The Scree Plot and Cumulative Variance Plot

Numerical ratios are best interpreted visually. The primary tool for this is the scree plot.

A scree plot is a simple line plot with the principal component number (1, 2, 3,...) on the x-axis and the corresponding eigenvalue (or the explained variance ratio) on the y-axis. The plot typically shows a steep downward curve that eventually flattens into a gentle slope, resembling the loose debris or "scree" at the base of a mountain. The heuristic is to retain all components before the plot elbow—the point where the curve transitions from steep to flat. Components after the elbow are considered to explain negligible variance, often just noise.

A more precise companion is the cumulative explained variance plot. Here, the x-axis remains the component number, but the y-axis shows the cumulative sum of the explained variance ratios. This plot answers a direct practical question: "If I keep $k$ components, what total percentage of my data's variance am I preserving?" You might use this to choose a $k$ that captures, for instance, 95% of the total variance.

3. Formal Statistical Rules: Kaiser Criterion and Parallel Analysis

Visual heuristics can be subjective. Formal rules provide objective thresholds.

The Kaiser criterion is straightforward: retain any principal component with an eigenvalue greater than 1. The logic stems from standardization; if your data is standardized (mean=0, variance=1), each original feature contributes 1 unit of variance to the total. An eigenvalue > 1 indicates a component that captures more variance than a single original variable did, suggesting it represents a meaningful signal. While simple, this rule often retains too many components in datasets with many variables.

A more robust alternative is parallel analysis. This technique simulates what eigenvalues would look like in random, uncorrelated data with the same dimensions as your real dataset. You:

1. Generate many random datasets matching the size of your real data.
2. Perform PCA on each random dataset and record the eigenvalues.
3. Calculate the average eigenvalue for each component position across all simulations.
4. Retain only those components from your real analysis whose eigenvalues exceed the corresponding average eigenvalue from the random analysis.

Parallel analysis effectively filters out components that could have arisen by chance, making it a gold standard for statistical component selection.

4. Pragmatic and Numerical Criteria: Business Needs and Reconstruction Error

Ultimately, the best statistical model is the one that serves your project's goal. This often involves pragmatic criteria.

Business or application requirements frequently dictate the choice. In a real-time fraud detection system, you might need to reduce 500 features to just 10 for millisecond latency, accepting a 15% loss in variance. For a high-fidelity image compression algorithm, you might require 99.9% variance retention. The cumulative variance plot directly enables this cost-benefit analysis.

A related, more technical criterion is the reconstruction error threshold. PCA can be viewed as a compression and decompression algorithm. You compress data by projecting it onto $k$ components, then decompress (reconstruct) it by transforming it back to the original space. The difference between the original and reconstructed data is the reconstruction error. You can set a threshold for the maximum allowable mean squared reconstruction error and select the smallest $k$ that meets it. This directly links component selection to the fidelity of the data representation.

Common Pitfalls

1. Blindly Following the Kaiser Criterion: In datasets with many variables (e.g., 100+), the sum of eigenvalues is large, and the criterion $\lambda >1$ becomes too lenient, retaining components that represent trivial variance. Always cross-validate with a scree plot or parallel analysis.
2. Misidentifying the Elbow: The scree plot's elbow is not always obvious. Applying transformations like plotting on a log scale can sometimes help, but this pitfall underscores why you should never rely on a single method. Use the elbow as a guide, not a definitive answer.
3. Ignoring the Domain Context: Selecting 3 components because they explain 80% of variance is mathematically sound, but useless if your downstream clustering algorithm requires 5 dimensions to separate key customer segments. Always align the statistical outcome with the project's objective.
4. Applying Rules to Unstandardized Data: The Kaiser criterion and the interpretation of variance ratios assume your data is standardized. Applying PCA to data on wildly different scales (e.g., salary in dollars and age in years) will distort variance calculations, making the first component dominated by the highest-scale variable and rendering these selection rules invalid.

Summary

• The explained variance ratio measures the proportion of total dataset variance captured by each principal component, providing the primary metric for component importance.
• The scree plot visualizes eigenvalues to help identify an "elbow point," a heuristic for separating signal (valuable components) from noise, while the cumulative explained variance plot shows the total information retained for any number of components $k$.
• The Kaiser criterion ($\lambda >1$) offers a simple rule but often overestimates components; parallel analysis is a more robust statistical method that compares real eigenvalues to those from random data.
• The optimal number of components is ultimately a trade-off, often decided by business requirements (e.g., latency, fidelity) or a reconstruction error threshold, which quantifies the data loss from compression.