t-SNE and UMAP

In the age of big data, we often work with information that has hundreds or even thousands of features. Visualizing or analyzing data in such a high-dimensional space is practically impossible for humans. Dimensionality reduction techniques solve this by creating a faithful, lower-dimensional representation of the data, most commonly a 2D or 3D map we can see. Among these, t-Distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP) are two powerful, non-linear methods specifically crafted for visualization. While t-SNE has been the gold standard for revealing intricate local structures, UMAP builds upon its theoretical foundation to offer faster computation and improved preservation of the data's global layout, making the choice between them a critical decision in any data science workflow.

Understanding the Core Challenge: From High Dimensions to a Map

The fundamental problem is known as the curse of dimensionality. In high dimensions, data points become increasingly sparse and distant from each other, making meaningful relationships hard to discern. Linear methods like Principal Component Analysis (PCA) project data onto axes of maximal variance, which works well when the data lies on a linear subspace. However, most real-world data structures are non-linear—think of a coiled spring or nested circles. Neither t-SNE nor UMAP provides a set of components you can apply like PCA; instead, they create a non-linear embedding, a new coordinate system where the geometric relationships between points reflect their original high-dimensional similarities.

Imagine a crowded room in the dark (high-dimensional space). You can't see anyone, but you can hear how loud their voice is based on your distance from them. Your goal is to sketch a map of the room (2D space) where people are placed such that those who sounded close to you are drawn nearby, and those who sounded far are drawn far away. Both t-SNE and UMAP are algorithms for creating this map, but they use different strategies to measure "closeness" and optimize the final layout.

The Mechanics of t-SNE: Stochastic Neighbor Embedding

t-SNE operates by converting high-dimensional Euclidean distances between data points into conditional probabilities that represent similarities. The core idea is stochastic neighbor embedding: if two points are close in the original space, they have a high probability of being neighbors. t-SNE then constructs a similar probability distribution in the low-dimensional map and minimizes the difference between the two distributions.

The process has two key stages. First, for each data point $i$, t-SNE computes a conditional probability $p_{j|i}$ that point $j$ would be its neighbor under a Gaussian distribution centered at $i$. This creates a matrix of pairwise similarities in the high-dimensional space. The perplexity parameter is crucial here; it is a smooth measure of the effective number of neighbors for each point. A low perplexity (e.g., 5) focuses on very local structure, while a high perplexity (e.g., 50) considers more global relationships. There is no universally "correct" value, but it should be smaller than the number of data points. Typically, values between 5 and 50 are explored.

Second, t-SNE initializes points randomly in the low-dimensional space (e.g., 2D) and defines a second probability distribution $q_{ij}$ using a Student's t-distribution (which is where the "t" comes from). This heavy-tailed distribution helps alleviate the crowding problem—the difficulty of placing moderately distant data points in a limited 2D area. The algorithm then uses gradient descent to minimize the Kullback-Leibler (KL) divergence between the two distributions, $P$ and $Q$. The cost function is:

$$C=KL(P||Q)=\sum _i\sum _j{p}_{ij}\log \frac{p_{ij}}{q_{ij}}$$

Minimizing this cost function pulls similar points together and pushes dissimilar points apart in the final map. The result is often stunning visual clusters that reveal natural groupings in the data.

UMAP: Preserving Global and Local Structure

UMAP shares a strong theoretical foundation with t-SNE—it also begins by constructing a fuzzy topological representation of the high-dimensional data. However, its underlying mathematics and optimization choices lead to several practical advantages. Conceptually, UMAP assumes the data is uniformly distributed on a Riemannian manifold (a topological space that is locally smooth like Euclidean space) and then finds a low-dimensional representation that has the closest equivalent topological structure.

The algorithm first builds a weighted $k$-nearest neighbor graph in high dimensions. The key parameter here is n_neighbors, which balances local versus global structure preservation, analogous to perplexity in t-SNE. UMAP then applies fuzzy set theory to create a probabilistic graph. To find the low-dimensional embedding, it minimizes the cross-entropy between the high-dimensional and low-dimensional topological representations using stochastic gradient descent.

This different theoretical framing and optimization yield UMAP's celebrated benefits:

1. Faster Computation: UMAP is often significantly faster than t-SNE, especially on larger datasets, due to its more efficient optimization and the ability to use stochastic gradient descent more effectively.
2. Better Global Structure Preservation: While t-SNE excels at revealing local clusters, it can sometimes tear apart the broader manifold, placing similar clusters far apart arbitrarily. UMAP generally does a better job of maintaining the relative positions and connections between larger-scale clusters.
3. Ability to Embed New Data Points: This is a major operational difference. t-SNE is a transductive algorithm; it creates an embedding for the dataset it is trained on. To add a new point, you must rerun t-SNE on the entire combined dataset. UMAP, as part of its model, learns a transform that can be applied to new, unseen data points (out-of-sample extension), making it more practical for production pipelines.

A Practical Comparison and When to Use Each

Choosing between t-SNE and UMAP depends on your goal. For both, standard practice involves scaling your features (e.g., using StandardScaler) before application to ensure distances are meaningful.

• Use t-SNE when: Your primary goal is exploratory data visualization to discover fine-grained, local clusters within a dataset of modest size (e.g., up to tens of thousands of points). It is excellent for tasks like visualizing MNIST digit clusters or single-cell RNA-seq data, where discerning tight, separate groups is paramount. Its results can be highly sensitive to perplexity, so you must experiment with this parameter.
• Use UMAP when: You need a faster solution for larger datasets, you care about the broad relationships between clusters (global topology), or you require a reusable model to transform new data. It is increasingly becoming the default for many visualization tasks due to its speed and solid all-around performance. Its n_neighbors parameter controls the scale of structure preserved; low values focus on local detail, high values on global layout.

A side-by-side visualization on a complex dataset like Fashion-MNIST often shows t-SNE creating tighter, more separated clusters for each clothing type, while UMAP may show clearer spaces between the major categories (e.g., separating tops from shoes more distinctly).

Common Pitfalls

1. Misinterpreting Distances and Sizes: In both t-SNE and UMAP plots, the absolute distance between clusters is not meaningful. A large gap between two clusters does not necessarily mean they are more dissimilar than two clusters placed close together. The meaningful information is the relative clustering of points themselves. Similarly, the size of a cluster in the plot is arbitrary and should not be interpreted as the importance or density of that group.

1. Ignoring Parameter Sensitivity: Using default parameters for every dataset is a mistake. In t-SNE, a perplexity that is too low will find irrelevant micro-structures, while one too high will blur distinct clusters into a blob. In UMAP, a very low n_neighbors value can create artificially fragmented clusters. Always run the algorithm multiple times with different parameters and, for t-SNE, different random seeds to ensure the structure you see is robust.

1. Using the Embedding for Clustering or Feature Reduction: While the maps suggest clusters, the axes of a t-SNE or UMAP plot have no intrinsic meaning. They should not be used as input features for a downstream machine learning model (unlike PCA components). These are visualization tools. If you need features for a model, consider other non-linear methods like kernel PCA or an autoencoder.

1. Forgetting Stochasticity: t-SNE, in particular, has a random initialization. Two runs on the same data with the same parameters can produce visually different layouts, though the clustering should be similar. Use a fixed random seed (e.g., random_state=42) for reproducibility during analysis. UMAP's results are generally more stable across runs.

Summary

• t-SNE and UMAP are essential non-linear dimensionality reduction techniques designed primarily for visualizing complex, high-dimensional data in 2D or 3D.
• t-SNE works by minimizing the divergence between probability distributions in high and low dimensions, excels at revealing local cluster structure, but is computationally intensive and does not preserve global relationships well.
• The perplexity parameter in t-SNE controls the balance between attention to local and global data structure and must be tuned for each application.
• UMAP is based on manifold theory and topological data analysis, offering faster computation, better preservation of the data's global topology, and the critical ability to embed new data points using a learned transform.
• Both methods produce visualizations where cluster membership is informative, but absolute distances, cluster sizes, and axis values are not interpretable. They are tools for exploration, not feature engineering for downstream models.