High-Dimensional Data Plotting Solutions

Dimensionality Reduction (DR) simplifies complex, high-dimensional datasets into more manageable lower-dimensional forms for easier interpretation and better computational efficiency while preserving key information. Modern nonlinear DR techniques, such as t-SNE and UMAP, are popular for transforming complex datasets into simpler visual representations. However, they can produce results that are difficult to interpret due to the lack of inherent meaning in the shapes and clusters, suboptimal hyperparameters, and potential distortions. DimVis is a visualization tool developed by the authors of [1] which employs supervised EBM (Explainable Boosting Machine) models (trained on user-selected data of interest) as an interpretation assistant for DR projections. The DimVis tool facilitates high-dimensional data analysis by providing an interpretation of feature relevance in visual clusters through interactive exploration of UMAP projections. Specifically, the DimVis tool utilizes a contrastive EBM model that is trained in real-time to distinguish between data points inside and outside a cluster of interest. Leveraging the inherently explainable nature of the EBM, this model is then used to interpret the cluster through single and pairwise feature comparisons, ranked according to the EBM model’s feature importance. The applicability and effectiveness of DimVis are demonstrated through a use case and a scenario involving real-world data. Their paper [1] and the DimVis #Python code [2] are available through the links provided in the comments.

1/ That UMAP plot you’re staring at? It might be lying to you. Let’s talk about why dimensionality reduction can mislead, and how to read your single-cell data without fooling yourself. 🧵 2/ The meme says it all: same hand, different angle—completely different meaning. That’s exactly what happens when you project high-dimensional data into 2D. 3/ In single-cell RNA-seq, we deal with thousands of genes per cell. That’s high-dimensional space. You can’t visualize that directly. So we use dimensionality reduction: PCA, t-SNE, UMAP. Each method tells a different story. 4/ PCA is linear. The axes are ranked by variance. The distances between points actually mean something—how much two cells differ overall. It’s interpretable but can miss nonlinear structure. 5/ t-SNE and UMAP are nonlinear. They are designed to preserve local structure: Points (or cells) that are close together in high-dimensional space tend to remain close in the 2D or 3D embedding. Global distances are not reliable 6/ In a UMAP plot, two clusters might look far apart. But that doesn’t mean they’re biologically distinct. They might just be far in the 2D embedding, not in gene expression space. 7/ So why use UMAP or t-SNE? Because they’re great at showing local structure. Rare cell types. Developmental branches. They’re visualization tools, not metrics. They help you see, not measure. 8/ But over-interpretation is common. Don’t draw conclusions from cluster distance on a UMAP. Don’t assume biological meaning from gaps or shapes unless you’ve validated it with statistics or biology. 9/ So what should you do? Use PCA when interpretability matters Use UMAP or t-SNE for visualization Cross-validate patterns with known markers or expression patterns Never trust a plot blindly—question everything 10/ Final thought: Dimensionality reduction simplifies the data, but you still have to do the thinking. That eye-catching UMAP? It’s just the starting point. Not the conclusion. 11/ Want to learn more? Read https://lnkd.in/egZDZB7N I hope you've found this post helpful. Follow me for more. Subscribe to my FREE newsletter chatomics to learn bioinformatics https://lnkd.in/erw83Svn

A new version of Jupyter Scatter (https://lnkd.in/e2ncaxVb) is out: v0.21.0. 🥳 This version introduces fun new ways to select points using either a brush or rectangle. You can now also more easily extend or reduce a selection by holding down the meta (CMD on macOS) key or alt key respectively. The brush selection can be useful when working with temporal or sequential patterns. Huge kudos to Andres Colubri who brought up that idea and the initial implementation for regl-scatterplot (Jupyter Scatter's rendering engine https://lnkd.in/eq3HGDgN). To test the new brush selection mode, I integrated Jupyter Scatter with a novel algorithm called DimBridge (https://lnkd.in/e-t-VMEY) by Mingwei Li and Remco Chang et al. DimBridge aspires to identify key dimensions in the high-dimensional space that explain a subset of points in the lower-dimensional embedding. For instance, imagine working with a 2D UMAP or t-SNE plot of a 100 dimensional dataset and you see an interesting pattern. The idea of DimBridge is to tell you which handful of dimensions from the dataset can explain the pattern you're seeing in the embedding plot. Using a single-cell surface protein dataset from Florian Mair et al. (https://lnkd.in/eyruQcP6) that we analyzed with Ozette's Cell Discovery method (an extension of Evan Greene et al.'s FAUST method https://lnkd.in/ezbjgZK4) and embedded with t-SNE (https://lnkd.in/e-hYVixU), you can use Jupyter Scatter to select a cluster and have DimBridge compute the key protein expressions of that cluster. You can also contrast two or more selections to identify proteins that differentiate them. Or you can use the brush selection to study a sequence of selections in the dataset. To test it out yourself, head over to https://lnkd.in/ewrQVYpp, clone the repo, cd into notebooks, and run `juv run dimbridge.ipynb`. This requires you to have juv (https://lnkd.in/eswu2hXp) installed, a super handy new tool from Trevor Manz that makes Jupyter Notebooks reproducible! I highly recommend you check it out if you haven't.