towardsai.net
Author(s): Aditya Baser Originally published on Towards AI. 1. Introduction1.1. What is chunking, and why do we need it?The intuition behind chunking and how it helps in the retrieval of informationImagine you are searching for a specific piece of information in a vast library. If the books are arranged haphazardly some with irrelevant sections bound together and others with critical pages scattered across volumes youd spend a frustrating amount of time flipping through unrelated content. Now, consider a library where each book is carefully organized by topic, with coherent sections that neatly encapsulate a single idea or concept. This is the intuition behind chunking in the context of retrieval-augmented generation (RAG): its about organizing information so it can be easily retrieved and understood.RAG Workflow Our emphasis would be on understanding chunkingChunking refers to the process of dividing large bodies of text into smaller, self-contained segments called chunks. Each chunk is designed to encapsulate a coherent unit of information that can be efficiently stored, retrieved, and used for downstream tasks like search, indexing, or contextual input for an LLM.1.2. What are the different types of chunking methods?Extending the library analogy, imagine you walk into the library to find information about The Effects of Climate Change on Marine Life. The way the books are organized will determine how easily you can find the specific information youre looking for:1.2.1. Fixed-Length ChunkingEvery book in the library is arbitrarily divided into fixed-sized sections, say, 100 pages each. No matter what the content is, each section stops at the 100-page mark. As a result, a chapter about coral bleaching might be split across two sections, leaving you scrambling to piece together the full information.Fixed-Length Chunking splits the text into chunks based on a fixed token, word, or character count. While this method is simple to implement, it often causes relevant information to be split among different chunks or for the same chunk to have information on different topics, making retrieval less accurate.1.2.2. Recursive Chunking (Hierarchical)The books are structured into sections, chapters, and paragraphs following their natural hierarchy. For instance, a book on climate change might have sections on global warming, rising sea levels, and marine ecosystems. However, if a section about marine life is too large, it may remain unwieldy and difficult to search through quickly.Recursive chunking breaks text hierarchically, following natural structures such as chapters, sections, or paragraphs. While it preserves the natural structure of the document it would lead to chunks that are too large for cases when sections are lengthy and poorly organized.1.2.3. Semantic ChunkingIn this case, the books are reorganized based on meaning and topic coherence. Instead of rigidly splitting sections by length or following a strict hierarchy, every section focuses on a specific topic or concept. For example, a section might cover The Impact of Rising Temperatures on Coral Reefs in its entirety, regardless of length, ensuring all related content stays together. As a result, you can retrieve exactly what you need without having to sift through unrelated material.Semantic chunking uses meaning or context to define chunk boundaries, often leveraging embeddings or similarity measures to detect where one topic ends and another begins.2. Semantic Chunking: 101Semantic chunking involves breaking text into smaller, meaningful units (chunks) that retain context and meaning.2.1. Why Semantic Chunking is SuperiorSemantic chunking stands out among chunking methods because it optimizes the retrieval process for contextual relevance, precision, and user satisfaction. In retrieval-augmented generation (RAG), where the goal is to feed highly relevant and coherent information into a large language model (LLM), semantic chunking eliminates many pitfalls associated with fixed-length and hierarchical approaches.Lets explore the unique advantages of semantic chunking and why it is crucial for building high-performance RAG systems.2.1.1. Context PreservationSemantic chunking ensures that each chunk contains complete, self-contained information related to a single topic. This contrasts with fixed-length chunking, were arbitrary boundaries often split context, leading to incomplete or fragmented information retrieval. When feeding an LLM, context completeness is critical. Missing context forces the LLM to hallucinate or generate suboptimal answers, while semantic chunking minimizes this risk by delivering coherent inputs.2.1.2. Improved Retrieval PrecisionSemantic chunking generates chunks that are tightly focused on specific topics. This makes it easier for retrieval systems to match queries to the most relevant chunks, improving the precision of retrieval. Precise retrieval reduces the number of irrelevant chunks passed to the LLM. This saves tokens, minimizes noise, and ensures the LLM focuses only on information that directly answers the query.2.1.3. Minimized RedundancySemantic chunking reduces overlap and redundancy across chunks. While some overlap is necessary for preserving context, semantic chunking ensures this is deliberate and optimized, unlike fixed-length chunking, where overlaps are arbitrary and often wasteful. RAG pipelines often must deal with token constraints. Redundancy wastes valuable token space, while semantic chunking maximizes the information density of each chunk.3. Implementing Semantic Chunking3.1. Loading the dataset and setting up the API keyWe will use the dataset jamescalam/ai-arxiv2, which contains research papers on artificial intelligence. These papers are often long and contain distinct sections like abstracts, methodologies, experiments, and conclusions. Chunking this dataset using semantic methods ensures we preserve context within sections and facilitate efficient retrieval for downstream tasks like summarization or question answering.Snippet of the dataset "jamescalam/ai-arxiv2Semantic chunking stands out by splitting text based on meaning and context rather than arbitrary rules, ensuring each chunk is coherent and self-contained.One of the key tools for implementing semantic chunking is the semantic_router package.Among its core features, the semantic_router.splitters module is specifically designed for splitting text into meaningful chunks using cutting-edge semantic methods.The semantic_router.splitters module is central to the chunking functionality. It offers three key chunking methodsconsecutive_sim, cumulative_sim, and rolling_windoweach catering to different document structures and use cases.To use OpenAIs tools, you need an API key for authentication, which we securely load from a .env file using the dotenv library. This keeps your key safe and out of your code. The OpenAIEncoder is then initialized to convert text into embeddingsnumerical representations of meaning and context. These embeddings are crucial for semantic chunking, enabling us to measure similarity between text segments and create coherent chunks. Make sure your API key is set up in the .env file, and the encoder is configured with a model like text-embedding-ada-002 for efficient and accurate embedding generation. Below is the code for the same from datasets import load_datasetimport osfrom getpass import getpassfrom semantic_router.encoders import OpenAIEncoderfrom dotenv import load_dotenvimport sysimport openai#import the datadataset = load_dataset("jamescalam/ai-arxiv2", split= "train")#Securly loading the openai api key from a .env fileopenai.api_key = os.environ["OPENAI_API_KEY"]#The OpenAIEncoder is initialized for accurate embedding generationencoder = OpenAIEncoder(name= "text-embedding-3-small")The below code uses the RollingWindowSplitter from the semantic_router package to semantically chunk the dataset. The rolling window technique creates overlapping chunks to maintain context across boundaries, making it particularly effective for NLP tasks like retrieval-augmented generation (RAG).The rolling window splits text into chunks of a specified size (defined by window_size) with overlaps between adjacent chunks. This overlap helps preserve context from one chunk to the next, ensuring downstream models, such as large language models (LLMs), receive coherent input for processing.3.2. RollingWindowSplitter Parameter BreakdownencoderThe encoder generates embeddings for the text, representing its semantic meaning. These embeddings help measure similarity and guide chunking.dynamic_threshold = FalseWhat it Does: Disables automatic adjustment of the similarity threshold based on content. This means chunks will be determined solely by the fixed parameters (window_size, min_split_tokens, etc.).Best Practice: Use False when you have a clear idea of your thresholds or if the dataset is consistent in structure. Use True for varied or unstructured datasets.min_split_tokens = 100What it Does: Ensures each chunk contains at least 100 tokens. This prevents overly small, uninformative chunks.Best Practice: Set this based on the minimum amount of information required for your task.max_split_tokens = 500What it Does: Caps each chunk at 500 tokens to fit within token limits for downstream models (e.g., OpenAI models with token constraints).Best Practice: Match this value to your LLMs token limit, subtracting space for query tokens and prompts.window_size = 2What it Does: Specifies how many segments (e.g., sentences, paragraphs) to include in each chunk. Smaller windows produce tighter chunks; larger windows preserve more context but may include unrelated content.Best Practice: Adjust based on the granularity of your text (e.g., use 12 for short sentences, 35 for paragraphs).plot_splits = TrueWhat it Does: Visualizes the chunking process, showing how the text was divided into chunks. This is helpful for debugging and parameter tuning.enable_statistics = TrueWhat it Does: Outputs statistics about the chunking process, such as the number of chunks and their average size. This helps evaluate how well your chunking configuration performs.from semantic_router.splitters import RollingWindowSplitterfrom semantic_router.utils.logger import loggerlogger.setLevel("WARNING") # reduce logs from splitterencoder.score_threshold = config.score_threshold#Read the above description to understand best practices for parameterssplitter = RollingWindowSplitter( encoder=encoder, dynamic_threshold = False, min_split_tokens = 100, max_split_tokens = 500, window_size = 2, plot_splits = False, # set this to true to visualize chunking enable_statistics = False# to print chunking stats)splits = splitter([dataset["content"][0]])The build_chunk function combines a title and a content chunk into a formatted string, where the title is prefixed with a # (indicating a heading in markdown) followed by the content. This is useful for creating easily readable and structured outputs, particularly when chunking large datasets like research papers. In the example, the title is taken from the first document in the dataset, and the function is applied to the first three chunks from the splits. By looping through these chunks, it prints them as well-organized sections, helping users see how each chunk relates to the overall document title. This approach ensures clarity and structure, making the output more comprehensible for tasks like summarization or retrieval.def build_chunk(title: str, content: str): return f"# {title}\n{content}"# we use it like:title = dataset[0]["title"]for s in splits[:3]: print("---") print(build_chunk(title=title, content=s.content))The build_metadata function creates a structured metadata list for a document and its corresponding chunks. It starts by extracting document-level metadata like the ArXiv ID, title, and references, then iterates over the provided chunks (doc_splits) to assign each chunk its own metadata. For each chunk, it adds identifiers for the current chunk, the previous chunk (prechunk_id), and the next chunk (postchunk_id) to maintain contextual links without storing the full neighboring chunks, which helps save storage in systems like Pinecone. This metadata structure is particularly useful for indexing and retrieval tasks, as it combines chunk-level context with document-wide details for efficient querying and navigation.from semantic_router.schema import DocumentSplitdef build_metadata(doc: dict, doc_splits: list[DocumentSplit]): # get document level metadata first arxiv_id = doc["id"] title = doc["title"] refs = list(doc["references"].values()) # init split level metadata list metadata = [] for i, split in enumerate(doc_splits): # get neighboring chunks prechunk_id = "" if i == 0 else f"{arxiv_id}#{i-1}" postchunk_id = "" if i+1 == len(doc_splits) else f"{arxiv_id}#{i+1}" # create dict and append to metadata list metadata.append({ "id": f"{arxiv_id}#{i}", "title": title, "content": split.content, "prechunk_id": prechunk_id, "postchunk_id": postchunk_id, "arxiv_id": arxiv_id, "references": refs }) return metadatametadata = build_metadata( doc=dataset[0], doc_splits=splits[:3])Metadata structureThis code connects to a Pinecone instance, a vector database optimized for storing and retrieving embeddings, using an API key for authentication. It checks if the specified index (configured with config.chunk_index_name) already exists. If not, it creates a new index with a specified dimensionality (dims), which matches the size of the embeddings generated by the encoder. The index uses the dotproduct similarity metric for vector comparisons, and ServerlessSpec specifies the cloud and region (e.g., us-east-1). The code waits for the index to initialize before connecting and displaying its stats. This setup ensures that your embeddings can be stored, queried, and managed efficiently for downstream tasks like semantic search or retrieval.To use Pinecone, you first need an API key to authenticate your connection. Head over to Pinecones website and sign up for a free account. Once logged in, navigate to the API Keys section in the Pinecone dashboard. Here, youll find an automatically generated key or the option to create a new one. Copy the key and store it securely in your environment variables file (e.g., .env) as PINECONE_API_KEY. This ensures your key remains private and can be accessed by your code without hardcoding it directly, enhancing security while enabling seamless integration.from pinecone import Pinecone# initialize connection to pinecone (get API key at app.pinecone.io)api_key = os.environ["PINECONE_API_KEY"] # configure clientpc = Pinecone(api_key=api_key)from pinecone import ServerlessSpecspec = ServerlessSpec( cloud= config.chunk_cloud, region= config.chunk_region # us-east-1 is the free one. )dims = len(encoder(["some random text"])[0])import timeindex_name = config.chunk_index_name# check if index already exists (it shouldn't if this is first time)if index_name not in pc.list_indexes().names(): # if does not exist, create index pc.create_index( index_name, dimension=dims, # dimensionality of embed 3 from openai metric='dotproduct', spec=spec ) # wait for index to be initialized while not pc.describe_index(index_name).status['ready']: time.sleep(1)# connect to indexindex = pc.Index(index_name)time.sleep(1)# view index statsindex.describe_index_stats()This code processes the dataset in batches to create semantic chunks, embed them, and store the results in a Pinecone index. It first converts the dataset into a Pandas DataFrame (limited to 10,000 documents for efficiency) and prepares a list, full_dataset, to store all chunk metadata. The splitter is configured to suppress statistics and visual outputs for faster processing. For each document, the splitter generates chunks, and build_metadata adds identifiers and metadata. Batches of chunks (batch_size = 128) are then processed, with unique IDs assigned to each chunk, embeddings generated using the encoder, and all data uploaded to the Pinecone index using the upsert method. This approach ensures scalable and efficient processing, embedding, and storage for large datasets in Pinecone, suitable for retrieval-augmented generation and semantic search.from tqdm.auto import tqdm# easier to work with dataset as pandas dataframedata = dataset.to_pandas().iloc[:10000] # limit to 10k docs# store dataset *without* embeddings herefull_dataset = []batch_size = 128# adjust splitter to not display stats and visualssplitter.enable_statistics = Falsesplitter.plot_splits = Falsefor doc in tqdm(dataset): # create splits splits = splitter([doc["content"]]) # create IDs and metadata for all splits in doc metadata = build_metadata(doc=doc, doc_splits=splits) for i in range(0, len(splits), batch_size): i_end = min(len(splits), i+batch_size) # get batch of data metadata_batch = metadata[i:i_end] full_dataset.extend(metadata_batch) # generate unique ids for each chunk ids = [m["id"] for m in metadata_batch] # get text content to embed content = [ build_chunk( title=x["title"], content=x["content"] ) for x in metadata_batch ] # embed text embeds = encoder(content) # add to Pinecone index.upsert(vectors=zip(ids, embeds, metadata))The query function retrieves relevant chunks from the Pinecone index based on a user's input query (text), embedding it using the same encoder used during index creation to ensure consistency in the semantic space. The function searches the index for the top k matches, where top_k=5 retrieves the 5 most similar chunks to the query based on the similarity metric (e.g., dot product, which measures alignment in the embedding space). It includes metadata for each match (include_metadata=True), such as the title, content, and IDs for the preceding (prechunk_id) and following (postchunk_id) chunks. Neighboring chunks are fetched to provide additional context, appending up to 400 characters from their edges to the current chunk. Each result is then formatted with the document's title as a heading and enriched with context for coherence, ensuring that the query response is accurate, relevant, and easy to understand.def query(text: str): xq = encoder([text])[0] #We are using the same encoder that we used while creating the index here so that the query is plotted in the same space matches = index.query( vector=xq, top_k=5, #How many chunks to retrive include_metadata=True #Allows us to get the metadata ) chunks = [] for m in matches["matches"]: content = m["metadata"]["content"] title = m["metadata"]["title"] pre = m["metadata"]["prechunk_id"] post = m["metadata"]["postchunk_id"] other_chunks = index.fetch(ids=[pre, post])["vectors"] prechunk = other_chunks[pre]["metadata"]["content"] postchunk = other_chunks[post]["metadata"]["content"] chunk = f"""# {title} {prechunk[-400:]} {content} {postchunk[:400]}""" chunks.append(chunk) return chunksquery("what are large language models?")Retrieving 5 relevant chunks from the database to be fed into the LLM modelReferencesDesign and Evaluation of RAG Solutions: Chunk Processing. (n.d.). Retrieved from https://github.com/Azure-Samples/Design-and-evaluation-of-RAG-solutions/blob/main/3.-chunk-processing/README.mdSemantic Chunking for RAG: Better Context, Better Results. (n.d.). Retrieved from https://www.multimodal.dev/post/semantic-chunking-for-ragRetrieval-Augmented Generation with Azure AI Document Intelligence. (n.d.). Retrieved from https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept/retrieval-augmented-generationRAG Techniques: Semantic Chunking. (n.d.). Retrieved from https://github.com/NirDiamant/RAG_Techniques/blob/main/all_rag_techniques/semantic_chunking.ipynbRAG Best Practice with AI Search. (n.d.). Retrieved from https://techcommunity.microsoft.com/blog/azure-ai-services-blog/rag-best-practice-with-ai-search/4357711Chunking Strategies for Production-Grade RAG Applications. (n.d.). Retrieved from https://www.helicone.ai/blog/rag-chunking-strategiesChunking Techniques Every Developer Should Know for Enhanced RAG Applications. (n.d.). Retrieved from https://dev.to/pavanbelagatti/chunking-techniques-every-developer-should-know-for-enhanced-rag-applications-famA Deep-Dive into Chunking Strategy, Chunking Methods, and Precision in RAG Applications. (n.d.). Retrieved from https://www.superteams.ai/blog/a-deep-dive-into-chunking-strategy-chunking-methods-and-precision-in-rag-applicationsRAG Optimization: Use an LLM to Chunk Your Text Semantically. (n.d.). Retrieved from https://www.luminis.eu/blog/rag-optimisation-use-an-llm-to-chunk-your-text-semantically/Mastering RAG: Advanced Chunking Techniques for LLM Applications. (n.d.). Retrieved from https://www.galileo.ai/blog/mastering-rag-advanced-chunking-techniques-for-llm-applicationsJoin thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming asponsor. Published via Towards AI
/https://tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer_public/3b/e1/3be1dac0-0f4c-426b-86cf-ae12ff959df6/p.jpeg)
/cdn.vox-cdn.com/uploads/chorus_asset/file/25834396/spacex_starship_launch.jpeg)
