Extracting a Summary of Industry Applications from One Week of arXiv AI Papers
Extracting a Summary of Industry Applications from One Week of arXiv AI Papers9 min read·Just now--Capstone — Gen AI Intensive Course 2025Q1https://www.kaggle.com/code/helgaguerreiro/capstone-gen-ai-2025q1-1-week-arxiv-summaryGoalThis project aimed to build a practical method to get a clear overview of what AI researchers are working on, segmented by real-world industry applications.Take one week of AI papers from arXiv, classify them by Industry, and then summarize how those papers contribute to the Industry.Problem statementGetting a bird’s eye view of research in AI as a hobbyist or curious external observer is surprisingly complex. The field is drowning in hyperbole and fad trends.Communication channels are often polarized by generalist clickbait headlines and super dense academic or Industry channels specializing in specific sub-fields.The arXiv dataset, which Cornell University graciously makes available, contains more than 1 million articles from arXiv. The sheer number of AI papers is overwhelming, and the trend is that it will continue to increase over time.For this illustration, we use the first version of the date on the arXiv dataset. Each bar represents the number of papers created in 1 week.The large number of submissions makes it hard to analyze without automation of some kind.Every summary is a projection of some kind. You want to take something larger and represent it in a smaller size while retaining its general information.A good metaphor is thinking of shadows on a wall. Presume you have this dispersed set of objects, and you want to be able to tell what they are just by seeing their shadows. We think of the papers as objects whose key shapes and features we need to recognize clearly. The model and prompt are like a bright, precisely positioned light that we shine on these objects. By first organizing objects into clearly defined groups, we ensure each group’s shadows remain a good representation of the group. Then, within each group, we auto-cluster similar objects to ensure similar objects are reflected together; this helps preserve their shape in the reflection, adding definition to the diversity of shapes within the group.Methodology (Data, Classification, Embeddings, Clustering, Summarization)DataWe start with a JSON list of [ arxiv_id, title, abstract, v1_date ] from 23–03–2025 to 29–03–2025, extracted from the arXiv dataset.ClassificationFirst, we want to determine which industries are mentioned in a paper.Each paper will be assigned an industry list attribute, which can be:N/A — no industry mentionedOther — not part of our listList of industriesGiven the number of papers involved and my goal of running this pipeline regularly for personal use, cost is an important factor.Using a pro model would undoubtedly produce better results, but it would also be significantly more expensive.For that reason, I prioritized integrating flash models as much as possible throughout the pipeline.To better optimize the balance between quality and cost, it was important to understand the limitations of flash models and design acceptable trade-offs that still produce meaningful results.GroundingIn preliminary tests, simply asking the model to identify industries or industrial methods often produced hilarious — and unusable — results.For example:Paper 2402.09573 — “Changes by Butterflies: Farsighted Forecasting with Group Reservoir Transformer” — was assigned the industry “Butterflies.”This kind of noise was frequent and made the Industry lists unmanageable.I grounded the model by providing a manually curated list of generic industries to circumvent this.It’s worth noting that the list of industries is arbitrary.Any attempt at rigid classification is subject to inherent limitations because industry applications often overlap and defy strict categorization.(For instance, should satellites fall under telecommunications, or are they also tools of environmental science?)However, because the industry list is stored externally in a configuration file, it can easily be adjusted or expanded later if needed.Example prompt excerpt for clarityYou are a research assistant tasked with identifying the relevant industries for a given academic paper.Below are predefined industries you must choose from:Industry: HealthcareKeywords: doctor, hospital, medical, imaging, diagnosis, clinicalExamples: Detecting tumors from MRI scans.Industry: Finance & BankingKeywords: finance, banking, fraud, credit, insurance, investment, stockExamples: Predicting credit default risk using machine learning.[…. Other industries]Do this step by step.You will be given a title and abstract of a paper.First, carefully read the abstract to find any clear mentions of industries or application domains.Only then skim the title if needed to supplement missing context.Map the findings to one or more industries from the list above.If no category is a good fit, assign the Industry:OtherSummarize this paper based on its contributions / impact into the identified industries Finally, estimate “Relevance Score”, On a scale from 0 (not related) to 100 (highly related), how strongly is this paper connected to the [Industry] industry based on its topic and content? Provide only the numeric score.A weak match should have a Relevance Score of 50 or less.A strong match should have a Relevance Score of 80 or more.Do not infer other sectors not mentioned.Do not guess possible applications.Ignore general technologies like “Machine Learning” or “Artificial Intelligence” unless the Industry itself is the tech industry (e.g., software engineering, chip design).Title: [the title]Abstract: [the abstract]Return your answer as JSONExpose your reasoningExample response format paper matches category x because y category alfa because beta“`json[{“industry”: “Healthcare”, “relevanceScore”:80, “summary”: “relevant because it improves or impacts gama”}{“industry”: “Legal & Compliance”, “summary”: “relevant because it improves or impacts teta”, “relevanceScore”:20]` “Task Diversity in Small ModelsWe initially attempted to have the flash model detect whether a paper was industry-related and, if so, classify it into relevant industries. This approach consistently failed: even with the temperature set as low as 0.2 and explicit step-by-step instructions, the model frequently hallucinated potential applications instead of restricting itself to explicit mentions.To address this, we split the task into two stages:Stage 1: Determine if the paper explicitly mentions any industry or application domain.Stage 2: If so, classify it into the appropriate industry categories.This adjustment significantly improved performance, allowing us to rely predominantly on flash models for industry classification while preserving acceptable quality levels.Semantic limitations in small modelsSmall models, like flash models, struggle when instructions require non-trivial reasoning. The further the required judgment drifts from “explicitly stated” information, the more unreliable the model becomes.A common failure mode was loose mentions of industry or technology domains being over-interpreted as strong industrial relevance. To mitigate this, we implemented two heuristic checks:Multiple Categories: We flagged the result as suspicious if more than two industries are assigned to a paper. (While cross-industry methods exist, hallucination was more common.)Assignment of “Other” Plus Specific Categories: Assigning “Other” alongside one or more concrete industries often indicates model confusion.When these conditions were met, the classification was re-run using a Pro model, and the corrected results were stored.Example of semantic limitationsThis example highlights how the small model inferred potential applications as Industry matches, while the pro model correctly distinguished between potential use and explicit industry relevance.Paper 2503.19267 — NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic ScenariosThe flash model returned:Transportation & Logistics — ‘The paper introduces a new benchmark, NeoRL-2, for offline reinforcement learning that addresses challenges encountered in real-world applications, including those involving deployed control systems and delayed action effects, which are relevant to optimizing transportation and logistics operations like fleet management and delivery routes.Manufacturing — paper introduces a new benchmark, NeoRL-2, for offline reinforcement learning that addresses challenges encountered in real-world applications, including those involving deployed control systems and delayed action effects, which are relevant to optimizing manufacturing processes and supply chain managementEnergy & Utilities — paper introduces a new benchmark, NeoRL-2, for offline reinforcement learning that addresses challenges encountered in real-world applications, including those involving deployed control systems and delayed action effects, which are relevant to optimizing energy distribution in smart grids.Aerospace & Defense — The paper introduces a new benchmark, NeoRL-2, for offline reinforcement learning that addresses challenges encountered in real-world applications, including those involving deployed control systems and delayed action effects, which are relevant to optimizing Aerospace & Defense applications.’}]🤓 Pro model returned 🏆Other — This paper introduces NeoRL-2, a benchmark for offline reinforcement learning algorithms, focusing on general real-world challenges like conservative data and delayed actions, rather than specific industry applications. Its contribution is methodological, aiming to improve RL algorithms for potential use across various unspecified domains.Classification NumbersPapers: 1905Identified as Industry related: 968 (50.8%)Suspicious Classifications Identified: 201 (20.7% of industry-related papers)Given the relatively low rate of suspicious classifications, relying primarily on the flash model with selective fallback to a pro model remains more cost-effective than using the pro model for all classifications.EmbeddingsEmbeddings transform raw data — like text — into compact semantic vectors, making it possible to measure relationships and patterns. However, embeddings are projections: they capture some aspects of the original data while discarding others.The inherent loss of information in the process makes the chosen embedding algorithm critical — poor embedding risks losing the very properties we need for downstream tasks like clustering or semantic comparison.The quality of what we can infer later depends heavily on what we choose to preserve now.For embedding, we use a modern semantic model (text-embedding-004) to better preserve meaningful relationships in textClusteringMost clustering algorithms work on one of two principles: frequency analysis or semantic proximity. Abstracts are relatively small bodies of text; therefore, frequency analysis will be inferior because any variation will disproportionally distance papers that could be close. For this reason, we choose a semantic algorithm that can handle noise and variable density like HDBSCAN.The final problem to solve is large clusters. Even with segmentation by Industry, some clusters will be over 100 papers, ex: “medical imaging”.If you feed an LLM 100 abstracts, you’re guaranteed to collapse information into distortion. To minimize this, we first instruct the model to summarize the industry applications of a smaller cluster of papers and then run the group of summaries again for a broader summary. We understand this process can exacerbate bias present in the individual groups. Is it a poorer choice than letting the model choose what information to preserve from a more extensive collection of abstracts?SummarizationThe goal is to create 1 section per Industry.Each section contains multiple clusters and one “General Outlook” summary representing the papers that did not fit any clusters.We also ask the model to assign a representative title for each cluster.For the papers that didn’t match a cluster, we only summarize those to which the model assigned a relevance score > 90, which helps ensure that only very strong matches to the Industry are included.Results and FindingsThe compilation of summaries feels repetitive and is not proper for direct publication, but it could be a good base for human review.Future Improvements, Reflections and LimitationsInefficiencies — Duplicate records in the embeddings.Since a single paper can be assigned to multiple industries, and ChromaDB lacks the ability to filter individual values within a list stored in metadata, we chose to encode the record ID as <arXiv_id>__< Industry>.This approach enables per-industry filtering but results in duplicate embeddings for papers associated with multiple industries.Poor evaluation mechanisms.Most evaluation steps in this pipeline are manual. While intermediate outputs are saved for inspection, verifying their quality currently requires human review.There is no automated mechanism to quantify output quality. Future work could focus on developing evaluation metrics by Industry to fine-tune industry keywords and examples, improving classification accuracy.Information lossRelying solely on abstracts is a limitation, as industry applications are often better described in the introduction sections of papers.Future improvements could involve lightweight NLP techniques to extract high-relevance semantic excerpts from the first page, increasing coverage without dramatically raising token usage.Clustering OptimizationCurrent clustering relies on a single embedding model and algorithm.Future iterations could experiment with alternative embedding models and clustering algorithms to assess their impact on the quality and interpretability of the thematic groupings.ConclusionThis is a prototype; it contains many weaknesses that can be improved. The goal was to outline a structure that could be improved modularly.The basic outline of its components:A seed semantic structure to ground the model when classifying the papers into buckets — the industry listA structured outline that can be ingested into a document to be reviewed by a human and turned into a newsletter or executive summary — the section list outputOne way to further filter and sort large buckets is the relevance score, which we recognize is an arbitrary and unreliable metric as the flash model in this pipeline primarily assigns it.An automated clustering mechanism for the buckets to further subdivide large semantically similar groups.
English
Arabic
French
Spanish
Portuguese
Deutsch
Turkish
Dutch
Italiano
Russian
Romaian
Portuguese (Brazil)