Semantic search goes beyond traditional keyword matching by understanding the contextual meaning of search queries. Instead of simply matching exact words, semantic search systems capture the intent and contextual definition of the query and return relevant results even when they dont contain the same keywords.In this tutorial, well implement a semantic search system using Sentence Transformers, a powerful library built on top of Hugging Faces Transformers that provides pre-trained models specifically optimized for generating sentence embeddings. These embeddings are numerical representations of text that capture semantic meaning, allowing us to find similar content through vector similarity. Well create a practical application: a semantic search engine for a collection of scientific abstracts that can answer research queries with relevant papers, even when the terminology differs between the query and relevant documents.First, lets install the necessary libraries in our Colab notebook:!pip install sentence-transformers faiss-cpu numpy pandas matplotlib datasetsNow, lets import the libraries well need:import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom sentence_transformers import SentenceTransformerimport faissfrom typing import List, Dict, Tupleimport timeimport reimport torchFor our demonstration, well use a collection of scientific paper abstracts. Lets create a small dataset of abstracts from various fields:abstracts = [ { "id": 1, "title": "Deep Learning for Natural Language Processing", "abstract": "This paper explores recent advances in deep learning models for natural language processing tasks. We review transformer architectures including BERT, GPT, and T5, and analyze their performance on various benchmarks including question answering, sentiment analysis, and text classification." }, { "id": 2, "title": "Climate Change Impact on Marine Ecosystems", "abstract": "Rising ocean temperatures and acidification are severely impacting coral reefs and marine biodiversity. This study presents data collected over a 10-year period, demonstrating accelerated decline in reef ecosystems and proposing conservation strategies to mitigate further damage." }, { "id": 3, "title": "Advancements in mRNA Vaccine Technology", "abstract": "The development of mRNA vaccines represents a breakthrough in immunization technology. This review discusses the mechanism of action, stability improvements, and clinical efficacy of mRNA platforms, with special attention to their rapid deployment during the COVID-19 pandemic." }, { "id": 4, "title": "Quantum Computing Algorithms for Optimization Problems", "abstract": "Quantum computing offers potential speedups for solving complex optimization problems. This paper presents quantum algorithms for combinatorial optimization and compares their theoretical performance with classical methods on problems including traveling salesman and maximum cut." }, { "id": 5, "title": "Sustainable Urban Planning Frameworks", "abstract": "This research proposes frameworks for sustainable urban development that integrate renewable energy systems, efficient public transportation networks, and green infrastructure. Case studies from five cities demonstrate reductions in carbon emissions and improvements in quality of life metrics." }, { "id": 6, "title": "Neural Networks for Computer Vision", "abstract": "Convolutional neural networks have revolutionized computer vision tasks. This paper examines recent architectural innovations including residual connections, attention mechanisms, and vision transformers, evaluating their performance on image classification, object detection, and segmentation benchmarks." }, { "id": 7, "title": "Blockchain Applications in Supply Chain Management", "abstract": "Blockchain technology enables transparent and secure tracking of goods throughout supply chains. This study analyzes implementations across food, pharmaceutical, and retail industries, quantifying improvements in traceability, reduction in counterfeit products, and enhanced consumer trust." }, { "id": 8, "title": "Genetic Factors in Autoimmune Disorders", "abstract": "This research identifies key genetic markers associated with increased susceptibility to autoimmune conditions. Through genome-wide association studies of 15,000 patients, we identified novel variants that influence immune system regulation and may serve as targets for personalized therapeutic approaches." }, { "id": 9, "title": "Reinforcement Learning for Robotic Control Systems", "abstract": "Deep reinforcement learning enables robots to learn complex manipulation tasks through trial and error. This paper presents a framework that combines model-based planning with policy gradient methods to achieve sample-efficient learning of dexterous manipulation skills." }, { "id": 10, "title": "Microplastic Pollution in Freshwater Systems", "abstract": "This study quantifies microplastic contamination across 30 freshwater lakes and rivers, identifying primary sources and transport mechanisms. Results indicate correlation between population density and contamination levels, with implications for water treatment policies and plastic waste management." }]papers_df = pd.DataFrame(abstracts)print(f"Dataset loaded with {len(papers_df)} scientific papers")papers_df[["id", "title"]]Now well load a pre-trained Sentence Transformer model from Hugging Face. Well use the all-MiniLM-L6-v2 model, which provides a good balance between performance and speed:model_name = 'all-MiniLM-L6-v2'model = SentenceTransformer(model_name)print(f"Loaded model: {model_name}")Next, well convert our text abstracts into dense vector embeddings:documents = papers_df['abstract'].tolist()document_embeddings = model.encode(documents, show_progress_bar=True)print(f"Generated {len(document_embeddings)} embeddings with dimension {document_embeddings.shape[1]}")FAISS (Facebook AI Similarity Search) is a library for efficient similarity search. Well use it to index our document embeddings:dimension = document_embeddings.shape[1] index = faiss.IndexFlatL2(dimension)index.add(np.array(document_embeddings).astype('float32'))print(f"Created FAISS index with {index.ntotal} vectors")Now lets implement a function that takes a query, converts it to an embedding, and retrieves the most similar documents:def semantic_search(query: str, top_k: int = 3) -> List[Dict]: """ Search for documents similar to query Args: query: Text to search for top_k: Number of results to return Returns: List of dictionaries containing document info and similarity score """ query_embedding = model.encode([query]) distances, indices = index.search(np.array(query_embedding).astype('float32'), top_k) results = [] for i, idx in enumerate(indices[0]): results.append({ 'id': papers_df.iloc[idx]['id'], 'title': papers_df.iloc[idx]['title'], 'abstract': papers_df.iloc[idx]['abstract'], 'similarity_score': 1 - distances[0][i] / 2 }) return resultsLets test our semantic search with various queries that demonstrate its ability to understand meaning beyond exact keywords:test_queries = [ "How do transformers work in natural language processing?", "What are the effects of global warming on ocean life?", "Tell me about COVID vaccine development", "Latest algorithms in quantum computing", "How can cities reduce their carbon footprint?"]for query in test_queries: print("\n" + "="*80) print(f"Query: {query}") print("="*80) results = semantic_search(query, top_k=3) for i, result in enumerate(results): print(f"\nResult #{i+1} (Score: {result['similarity_score']:.4f}):") print(f"Title: {result['title']}") print(f"Abstract snippet: {result['abstract'][:150]}...")Lets visualize the document embeddings to see how they cluster by topic:from sklearn.decomposition import PCApca = PCA(n_components=2)reduced_embeddings = pca.fit_transform(document_embeddings)plt.figure(figsize=(12, 8))plt.scatter(reduced_embeddings[:, 0], reduced_embeddings[:, 1], s=100, alpha=0.7)for i, (x, y) in enumerate(reduced_embeddings): plt.annotate(papers_df.iloc[i]['title'][:20] + "...", (x, y), fontsize=9, alpha=0.8)plt.title('Document Embeddings Visualization (PCA)')plt.xlabel('Component 1')plt.ylabel('Component 2')plt.grid(True, linestyle='--', alpha=0.7)plt.tight_layout()plt.show()Lets create a more interactive search interface:from IPython.display import display, HTML, clear_outputimport ipywidgets as widgetsdef run_search(query_text): clear_output(wait=True) display(HTML(f"<h3>Query: {query_text}</h3>")) start_time = time.time() results = semantic_search(query_text, top_k=5) search_time = time.time() - start_time display(HTML(f"<p>Found {len(results)} results in {search_time:.4f} seconds</p>")) for i, result in enumerate(results): html = f""" <div style="margin-bottom: 20px; padding: 15px; border: 1px solid #ddd; border-radius: 5px;"> <h4>{i+1}. {result['title']} <span style="color: #007bff;">(Score: {result['similarity_score']:.4f})</span></h4> <p>{result['abstract']}</p> </div> """ display(HTML(html))search_box = widgets.Text( value='', placeholder='Type your search query here...', description='Search:', layout=widgets.Layout(width='70%'))search_button = widgets.Button( description='Search', button_style='primary', tooltip='Click to search')def on_button_clicked(b): run_search(search_box.value)search_button.on_click(on_button_clicked)display(widgets.HBox([search_box, search_button]))In this tutorial, weve built a complete semantic search system using Sentence Transformers. This system can understand the meaning behind user queries and return relevant documents even when there isnt exact keyword matching. Weve seen how embedding-based search provides more intelligent results than traditional methods.Here is the Colab Notebook. Also,dont forget to follow us onTwitterand join ourTelegram ChannelandLinkedIn Group. Dont Forget to join our85k+ ML SubReddit. Asif RazzaqWebsite| + postsBioAsif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.Asif Razzaqhttps://www.marktechpost.com/author/6flvq/NVIDIA AI Just Open Sourced Canary 1B and 180M Flash Multilingual Speech Recognition and Translation ModelsAsif Razzaqhttps://www.marktechpost.com/author/6flvq/Microsoft AI Introduces Claimify: A Novel LLM-based Claim-Extraction Method that Outperforms Prior Solutions to Produce More Accurate, Comprehensive, and Substantiated Claims from LLM OutputsAsif Razzaqhttps://www.marktechpost.com/author/6flvq/A Coding Implementation to Build a Document Search Agent (DocSearchAgent) with Hugging Face, ChromaDB, and LangchainAsif Razzaqhttps://www.marktechpost.com/author/6flvq/NVIDIA Open-Sources cuOpt: An AI-Powered Decision Optimization EngineUnlocking Real-Time Optimization at an Unprecedented Scale