Rethinking Imbalance: LLM Embeddings for Detecting Subtle Irregularities 0 like March 3, 2025Share this postAuthor(s): Elangoraj Thiruppandiaraj Originally published on Towards AI. This member-only story is on us. Upgrade to access all of Medium.Ive worked on anomaly detection problems for a while now, and one obstacle I consistently face is extreme imbalance in the data. When only a fraction of a percent of your records are anomalies, most standard methods like oversampling or undersampling just dont cut it. In my experience, these approaches either lead to overfitting (repeating the same rare examples too often) or throw away valuable data. Thats where a newer technique Ive been exploring comes in: using Large Language Model (LLM) embeddings to spot subtle irregularities.Even though embeddings are typically associated with text (thanks to tools like BERT, GPT, or other transformer models), Ive found that the same idea representing data in a dense, meaningful vector space works wonders for detecting outliers across various data types. Let me walk you through the thinking and process behind this method.Source: Mario DudjakThose experienced in anomaly or rare event detection are keenly aware of the datas extreme imbalance. In many scenarios:Less than 1% of the dataset represents rare or critical events, forming the minority class.99% or more of the dataset falls under the normal or unknown category Read the full blog for free on Medium.Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming asponsor. Published via Towards AITowards AI - Medium Share this post