Reading Time: 9 minutes In the ever-evolving landscape of customer engagement, staying ahead of the curve is not just advantageous, it’s essential. That’s why, for Chapter 7 of “The Customer Engagement Book: Adapt or Die,” we sat down with Mirela Cialai, a seasoned expert in CRM and Martech strategies at brands like Equinox. Mirela brings a wealth of knowledge in aligning technology roadmaps with business goals, shifting organizational focuses from acquisition to retention, and leveraging hyper-personalization to drive success. In this interview, Mirela dives deep into building robust customer engagement technology roadmaps. She unveils the “PAPER” framework—Plan, Audit, Prioritize, Execute, Refine—a simple yet effective strategy for marketers. You’ll gain insights into identifying gaps in your Martech stack, ensuring data accuracy, and prioritizing initiatives that deliver the greatest impact and ROI. Whether you’re navigating data silos, striving for cross-functional alignment, or aiming for seamless tech integration, Mirela’s expertise provides practical solutions and actionable takeaways.   Mirela Cialai Q&A Interview 1. How do you define the vision for a customer engagement platform roadmap in alignment with the broader business goals? Can you share any examples of successful visions from your experience? Defining the vision for the roadmap in alignment with the broader business goals involves creating a strategic framework that connects the team’s objectives with the organization’s overarching mission or primary objectives. This could be revenue growth, customer retention, market expansion, or operational efficiency. We then break down these goals into actionable areas where the team can contribute, such as improving engagement, increasing lifetime value, or driving acquisition. We articulate how the team will support business goals by defining the KPIs that link CRM outcomes — the team’s outcomes — to business goals. In a previous role, the CRM team I was leading faced significant challenges due to the lack of attribution capabilities and a reliance on surface-level metrics such as open rates and click-through rates to measure performance. This approach made it difficult to quantify the impact of our efforts on broader business objectives such as revenue growth. Recognizing this gap, I worked on defining a vision for the CRM team to address these shortcomings. Our vision was to drive measurable growth through enhanced data accuracy and improved attribution capabilities, which allowed us to deliver targeted, data-driven, and personalized customer experiences. To bring this vision to life, I developed a roadmap that focused on first improving data accuracy, building our attribution capabilities, and delivering personalization at scale. By aligning the vision with these strategic priorities, we were able to demonstrate the tangible impact of our efforts on the key business goals. 2. What steps did you take to ensure data accuracy? The data team was very diligent in ensuring that our data warehouse had accurate data. So taking that as the source of truth, we started cleaning the data in all the other platforms that were integrated with our data warehouse — our CRM platform, our attribution analytics platform, etc. That’s where we started, looking at all the different integrations and ensuring that the data flows were correct and that we had all the right flows in place. And also validating and cleaning our email database — that helped, having more accurate data. 3. How do you recommend shifting organizational focus from acquisition to retention within a customer engagement strategy? Shifting an organization’s focus from acquisition to retention requires a cultural and strategic shift, emphasizing the immense value that existing customers bring to long-term growth and profitability. I would start by quantifying the value of retention, showcasing how retaining customers is significantly more cost-effective than acquiring new ones. Research consistently shows that increasing retention rates by just 5% can boost profits by at least 25 to 95%. This data helps make a compelling case to stakeholders about the importance of prioritizing retention. Next, I would link retention to core business goals by demonstrating how enhancing customer lifetime value and loyalty can directly drive revenue growth. This involves shifting the organization’s focus to retention-specific metrics such as churn rate, repeat purchase rate, and customer LTV. These metrics provide actionable insights into customer behaviors and highlight the financial impact of retention initiatives, ensuring alignment with the broader company objectives. By framing retention as a driver of sustainable growth, the organization can see it not as a competing priority, but as a complementary strategy to acquisition, ultimately leading to a more balanced and effective customer engagement strategy. 4. What are the key steps in analyzing a brand’s current Martech stack capabilities to identify gaps and opportunities for improvement? Developing a clear understanding of the Martech stack’s current state and ensuring it aligns with a brand’s strategic needs and future goals requires a structured and strategic approach. The process begins with defining what success looks like in terms of technology capabilities such as scalability, integration, automation, and data accessibility, and linking these capabilities directly to the brand’s broader business objectives. I start by doing an inventory of all tools currently in use, including their purpose, owner, and key functionalities, assessing if these tools are being used to their full potential or if there are features that remain unused, and reviewing how well tools integrate with one another and with our core systems, the data warehouse. Also, comparing the capabilities of each tool and results against industry standards and competitor practices and looking for missing functionalities such as personalization, omnichannel orchestration, or advanced analytics, and identifying overlapping tools that could be consolidated to save costs and streamline workflows. Finally, review the costs of the current tools against their impact on business outcomes and identify technologies that could reduce costs, increase efficiency, or deliver higher ROI through enhanced capabilities. Establish a regular review cycle for the Martech stack to ensure it evolves alongside the business and the technological landscape. 5. How do you evaluate whether a company’s tech stack can support innovative customer-focused campaigns, and what red flags should marketers look out for? I recommend taking a structured approach and first ensure there is seamless integration across all tools to support a unified customer view and data sharing across the different channels. Determine if the stack can handle increasing data volumes, larger audiences, and additional channels as the campaigns grow, and check if it supports dynamic content, behavior-based triggers, and advanced segmentation and can process and act on data in real time through emerging technologies like AI/ML predictive analytics to enable marketers to launch responsive and timely campaigns. Most importantly, we need to ensure that the stack offers robust reporting tools that provide actionable insights, allowing teams to track performance and optimize campaigns. Some of the red flags are: data silos where customer data is fragmented across platforms and not easily accessible or integrated, inability to process or respond to customer behavior in real time, a reliance on manual intervention for tasks like segmentation, data extraction, campaign deployment, and poor scalability. If the stack struggles with growing data volumes or expanding to new channels, it won’t support the company’s evolving needs. 6. What role do hyper-personalization and timely communication play in a successful customer engagement strategy? How do you ensure they’re built into the technology roadmap? Hyper-personalization and timely communication are essential components of a successful customer engagement strategy because they create meaningful, relevant, and impactful experiences that deepen the relationship with customers, enhance loyalty, and drive business outcomes. Hyper-personalization leverages data to deliver tailored content that resonates with each individual based on their preferences, behavior, or past interactions, and timely communication ensures these personalized interactions occur at the most relevant moments, which ultimately increases their impact. Customers are more likely to engage with messages that feel relevant and align with their needs, and real-time triggers such as cart abandonment or post-purchase upsells capitalize on moments when customers are most likely to convert. By embedding these capabilities into the roadmap through data integration, AI-driven insights, automation, and continuous optimization, we can deliver impactful, relevant, and timely experiences that foster deeper customer relationships and drive long-term success. 7. What’s your approach to breaking down the customer engagement technology roadmap into manageable phases? How do you prioritize the initiatives? To create a manageable roadmap, we need to divide it into distinct phases, starting with building the foundation by addressing data cleanup, system integrations, and establishing metrics, which lays the groundwork for success. Next, we can focus on early wins and quick impact by launching behavior-based campaigns, automating workflows, and improving personalization to drive immediate value. Then we can move to optimization and expansion, incorporating predictive analytics, cross-channel orchestration, and refined attribution models to enhance our capabilities. Finally, prioritize innovation and scalability, leveraging AI/ML for hyper-personalization, scaling campaigns to new markets, and ensuring the system is equipped for future growth. By starting with foundational projects, delivering quick wins, and building towards scalable innovation, we can drive measurable outcomes while maintaining our agility to adapt to evolving needs. In terms of prioritizing initiatives effectively, I would focus on projects that deliver the greatest impact on business goals, on customer experience and ROI, while we consider feasibility, urgency, and resource availability. In the past, I’ve used frameworks like Impact Effort Matrix to identify the high-impact, low-effort initiatives and ensure that the most critical projects are addressed first. 8. How do you ensure cross-functional alignment around this roadmap? What processes have worked best for you? Ensuring cross-functional alignment requires clear communication, collaborative planning, and shared accountability. We need to establish a shared understanding of the roadmap’s purpose and how it ties to the company’s overall goals by clearly articulating the “why” behind the roadmap and how each team can contribute to its success. To foster buy-in and ensure the roadmap reflects diverse perspectives and needs, we need to involve all stakeholders early on during the roadmap development and clearly outline each team’s role in executing the roadmap to ensure accountability across the different teams. To keep teams informed and aligned, we use meetings such as roadmap kickoff sessions and regular check-ins to share updates, address challenges collaboratively, and celebrate milestones together. 9. If you were to outline a simple framework for marketers to follow when building a customer engagement technology roadmap, what would it look like? A simple framework for marketers to follow when building the roadmap can be summarized in five clear steps: Plan, Audit, Prioritize, Execute, and Refine. In one word: PAPER. Here’s how it breaks down. Plan: We lay the groundwork for the roadmap by defining the CRM strategy and aligning it with the business goals. Audit: We evaluate the current state of our CRM capabilities. We conduct a comprehensive assessment of our tools, our data, the processes, and team workflows to identify any potential gaps. Prioritize: initiatives based on impact, feasibility, and ROI potential. Execute: by implementing the roadmap in manageable phases. Refine: by continuously improving CRM performance and refining the roadmap. So the PAPER framework — Plan, Audit, Prioritize, Execute, and Refine — provides a structured, iterative approach allowing marketers to create a scalable and impactful customer engagement strategy. 10. What are the most common challenges marketers face in creating or executing a customer engagement strategy, and how can they address these effectively? The most critical is when the customer data is siloed across different tools and platforms, making it very difficult to get a unified view of the customer. This limits the ability to deliver personalized and consistent experiences. The solution is to invest in tools that can centralize data from all touchpoints and ensure seamless integration between different platforms to create a single source of truth. Another challenge is the lack of clear metrics and ROI measurement and the inability to connect engagement efforts to tangible business outcomes, making it very hard to justify investment or optimize strategies. The solution for that is to define clear KPIs at the outset and use attribution models to link customer interactions to revenue and other key outcomes. Overcoming internal silos is another challenge where there is misalignment between teams, which can lead to inconsistent messaging and delayed execution. A solution to this is to foster cross-functional collaboration through shared goals, regular communication, and joint planning sessions. Besides these, other challenges marketers can face are delivering personalization at scale, keeping up with changing customer expectations, resource and budget constraints, resistance to change, and others. While creating and executing a customer engagement strategy can be challenging, these obstacles can be addressed through strategic planning, leveraging the right tools, fostering collaboration, and staying adaptable to customer needs and industry trends. By tackling these challenges proactively, marketers can deliver impactful customer-centric strategies that drive long-term success. 11. What are the top takeaways or lessons that you’ve learned from building customer engagement technology roadmaps that others should keep in mind? I would say one of the most important takeaways is to ensure that the roadmap directly supports the company’s broader objectives. Whether the focus is on retention, customer lifetime value, or revenue growth, the roadmap must bridge the gap between high-level business goals and actionable initiatives. Another important lesson: The roadmap is only as effective as the data and systems it’s built upon. I’ve learned the importance of prioritizing foundational elements like data cleanup, integrations, and governance before tackling advanced initiatives like personalization or predictive analytics. Skipping this step can lead to inefficiencies or missed opportunities later on. A Customer Engagement Roadmap is a strategic tool that evolves alongside the business and its customers. So by aligning with business goals, building a solid foundation, focusing on impact, fostering collaboration, and remaining adaptable, you can create a roadmap that delivers measurable results and meaningful customer experiences.     This interview Q&A was hosted with Mirela Cialai, Director of CRM & MarTech at Equinox, for Chapter 7 of The Customer Engagement Book: Adapt or Die. Download the PDF or request a physical copy of the book here. The post Mirela Cialai Q&A: Customer Engagement Book Interview appeared first on MoEngage. #mirela #cialai #qampampa #customer #engagement

In 2024, Egypt completed its 1,155km stretch of the Cairo–Cape Town Highway, a 10,228km‑long road connecting 10 African countries – Egypt, Sudan, South Sudan, Ethiopia, Kenya, Tanzania, Zambia, Zimbabwe, Botswana and South Africa.   The imaginary of ‘Cape to Cairo’ is not new. In 1874, editor of the Daily Telegraph Edwin Arnold proposed a plan to connect the African continent by rail, a project that came to be known as the Cape to Cairo Railway project. Cecil Rhodes expressed his support for the project, seeing it as a means to connect the various ‘possessions’ of the British Empire across Africa, facilitating the movement of troops and natural resources. This railway project was never completed, and in 1970 was overlaid by a very different attempt at connecting the Cape to Cairo, as part of the Trans‑African Highway network. This 56,683km‑long system of highways – some dating from the colonial era, some built as part of the 1970s project, and some only recently built – aimed to create lines of connection across the African continent, from north to south as well as east to west.  Here, postcolonial state power invested in ‘moving the continent’s people and economies from past to future’, as architectural historians Kenny Cupers and Prita Meier write in their 2020 essay ‘Infrastructure between Statehood and Selfhood: The Trans‑African Highway’. The highways were to be built with the support of Kenya’s president Jomo Kenyatta, Ghana’s president Kwame Nkrumah and Ghana’s director of social welfare Robert Gardiner, as well as the United Nations Economic Commission for Africa. This project was part of a particular historical moment during which anticolonial ideas animated most of the African continent; alongside trade, this iteration of Cape to Cairo centred social and cultural connection between African peoples. But though largely socialist in ambition, the project nevertheless engaged modernist developmentalist logics that cemented capitalism.  Lead image: Over a century in the making, the final stretches of the Cairo–Cape Town Highway are being finished. Egypt completed the section within its borders last year and a section over the dry Merille River in Kenya was constructed in 2019. Credit: Allan Muturi / SOPA / ZUMA / Alamy. Above: The route from Cairo to Cape Town, outlined in red, belongs to the Trans‑African Highway network, which comprises nine routes, here in black The project failed to fully materialise at the time, but efforts to complete the Trans‑African Highway network have been revived in the last 20 years; large parts are now complete though some links remain unbuilt and many roads are unpaved or hazardous. The most recent attempts to realise this project coincide with a new continental free trade agreement, the agreement on African Continental Free Trade Area, established in 2019, to increase trade within the continent. The contemporary manifestation of the Cairo–Cape Town Highway – also known as Trans‑African Highway4 – is marked by deepening neoliberal politics. Represented as an opportunity to boost trade and exports, connecting Egypt to African markets that the Egyptian government view as ‘untapped’, the project invokes notions of trade steeped in extraction, reflecting the neoliberal logic underpinning contemporary Egyptian governance; today, the country’s political project, led by Abdel Fattah El Sisi, is oriented towards Egyptian dominance and extraction in relation to the rest of the continent.  Through an allusion to markets ripe for extraction, this language brings to the fore historical forms of domination that have shaped the connections between Egypt and the rest of the continent; previous iterations of connection across the continent often reproduced forms of domination stretching from the north of the African continent to the south, including the Trans‑Saharan slave trade routes across Africa that ended in various North African and Middle Eastern territories. These networks, beginning in the 8th century and lasting until the 20th, produced racialised hierarchies across the continent, shaping North Africa into a comparably privileged space proximate to ‘Arabness’. This was a racialised division based on a civilisational narrative that saw Arabs as superior, but more importantly a political economic division resulting from the slave trade routes that produced huge profits for North Africa and the Middle East. In the contemporary moment, these racialised hierarchies are bound up in political economic dependency on the Arab Gulf states, who are themselves dependent on resource extraction, land grabbing and privatisation across the entire African continent.  ‘The Cairo–Cape Town Highway connects Egypt to African markets viewed as “untapped”, invoking notions steeped in extraction’ However, this imaginary conjured by the Cairo–Cape Town Highway is countered by a network of streets scattered across Africa that traces the web of Egyptian Pan‑African solidarity across the continent. In Lusaka in Zambia, you might find yourself on Nasser Road, as you might in Mwanza in Tanzania or Luanda in Angola. In Mombasa in Kenya, you might be driving down Abdel Nasser Road; in Kampala in Uganda, you might find yourself at Nasser Road University; and in Tunis in Tunisia, you might end up on Gamal Abdel Nasser Street. These street names are a reference to Gamal Abdel Nasser, Egypt’s first postcolonial leader and president between 1956 and 1970.  Read against the contemporary Cairo–Cape Town Highway, these place names signal a different form of connection that brings to life Egyptian Pan‑Africanism, when solidarity was the hegemonic force connecting the continent, coming up against the notion of a natural or timeless ‘great divide’ within Africa. From the memoirs of Egyptian officials who were posted around Africa as conduits of solidarity, to the broadcasts of Radio Cairo that were heard across the continent, to the various conferences attended by anticolonial movements and postcolonial states, Egypt’s orientation towards Pan‑Africanism, beginning in the early 20th century and lasting until the 1970s, was both material and ideological. Figures and movements forged webs of solidarity with their African comrades, imagining an Africa that was united through shared commitments to ending colonialism and capitalist extraction.  The route between Cape Town in South Africa and Cairo in Egypt has long occupied the colonial imaginary. In 1930, Margaret Belcher and Ellen Budgell made the journey, sponsored by car brand Morris and oil company Shell Credit: Fox Photos / Getty The pair made use of the road built by British colonisers in the 19th century, and which forms the basis for the current Cairo–Cape Town Highway. The road was preceded by the 1874 Cape to Cairo Railway project, which connected the colonies of the British Empire Credit: Library of Congress, Geography and Map Division This network of eponymous streets represents attempts to inscribe anticolonial power into the materiality of the city. Street‑naming practices are one way in which the past comes into the present, ‘weaving history into the geographic fabric of everyday life’, as geographer Derek Alderman wrote in his 2002 essay ‘Street Names as Memorial Arenas’. In this vein, the renaming of streets during decolonisation marked a practice of contesting the production of colonial space. In the newly postcolonial city, renaming was a way of ‘claiming the city back’, Alderman continues. While these changes may appear discursive, it is their embedding in material spaces, through signs and maps, that make the names come to life; place names become a part of the everyday through sharing addresses or giving directions. This quality makes them powerful; consciously or unconsciously, they form part of how the spaces of the city are navigated.  These are traces that were once part of a dominant historical narrative; yet when they are encountered in the present, during a different historical moment, they no longer act as expressions of power but instead conjure up a moment that has long passed. A street in Lusaka named after an Egyptian general made more sense 60 years ago than it does today, yet contextualising it recovers a marginalised history of Egyptian Pan‑Africanism.  Markers such as street names or monuments are simultaneously markers of anticolonial struggle as well as expressions of state power – part of an attempt, by political projects such as Nasser’s, to exert their own dominance over cities, towns and villages. That such traces are expressions of both anticolonial hopes and postcolonial state power produces a sense of tension within them. For instance, Nasser’s postcolonial project in Egypt was a contradictory one; it gave life to anticolonial hopes – for instance by breaking away from European capitalism and embracing anticolonial geopolitics – while crushing many parts of the left through repression, censorship and imprisonment. Traces of Nasser found today inscribe both anticolonial promises – those that came to life and those that did not – while reproducing postcolonial power that in most instances ended in dictatorship.  Recent efforts to complete the route build on those of the post‑independence era – work on a section north of Nairobi started in 1968 Credit: Associated Press / Alamy The Trans‑African Highway network was conceived in 1970 in the spirit of Pan‑Africanism At that time, the routes did not extend into South Africa, which was in the grip of apartheid. The Trans‑African Highway initiative was motivated by a desire to improve trade and centre cultural links across the continent – an ambition that was even celebrated on postage stamps There have been long‑standing debates about the erasure of the radical anticolonial spirit from the more conservative postcolonial states that emerged; the promises and hopes of anticolonialism, not least among them socialism and a world free of white supremacy, remain largely unrealised. Instead, by the 1970s neoliberalism emerged as a new hegemonic project. The contemporary instantiation of Cape to Cairo highlights just how pervasive neoliberal logics continue to be, despite multiple global financial crises and the 2011 Egyptian revolution demanding ‘bread, freedom, social justice’.  But the network of streets named after anticolonial figures and events across the world is testament to the immense power and promise of anticolonial revolution. Most of the 20th century was characterised by anticolonial struggle, decolonisation and postcolonial nation‑building, as nations across the global south gained independence from European empire and founded their own political projects. Anticolonial traces, present in street and place names, point to the possibility of solidarity as a means of reorienting colonial geographies. They are a reminder that there have been other imaginings of Cape to Cairo, and that things can be – and have been – otherwise. 2025-06-13 Kristina Rapacki Share #cape #cairo #making #unmaking #colonial

One of the key use cases for generative AI involves answering questions over private datasets, with retrieval-augmented generation as the go-to framework. As new RAG techniques emerge, there’s a growing need to benchmark their performance across diverse datasets and metrics.  To meet this need, we’re introducing BenchmarkQED, a new suite of tools that automates RAG benchmarking at scale, available on GitHub. It includes components for query generation, evaluation, and dataset preparation, each designed to support rigorous, reproducible testing.   BenchmarkQED complements the RAG methods in our open-source GraphRAG library, enabling users to run a GraphRAG-style evaluation across models, metrics, and datasets. GraphRAG uses a large language model to generate and summarize entity-based knowledge graphs, producing more comprehensive and diverse answers than standard RAG for large-scale tasks.  In this post, we walk through the core components of BenchmarkQED that contribute to the overall benchmarking process. We also share some of the latest benchmark results comparing our LazyGraphRAG system to competing methods, including a vector-based RAG with a 1M-token context window, where the leading LazyGraphRAG configuration showed significant win rates across all combinations of quality metrics and query classes. In the paper, we distinguish between local queries, where answers are found in a small number of text regions, and sometimes even a single region, and global queries, which require reasoning over large portions of or even the entire dataset.  Conventional vector-based RAG excels at local queries because the regions containing the answer to the query resemble the query itself and can be retrieved as the nearest neighbor in the vector space of text embeddings. However, it struggles with global questions, such as, “What are the main themes of the dataset?” which require understanding dataset qualities not explicitly stated in the text.   AutoQ: Automated query synthesis This limitation motivated the development of GraphRAG a system designed to answer global queries. GraphRAG’s evaluation requirements subsequently led to the creation of AutoQ, a method for synthesizing these global queries for any dataset. AutoQ extends this approach by generating synthetic queries across the spectrum of queries, from local to global. It defines four distinct classes based on the source and scope of the queryforming a logical progression along the spectrum. Figure 1. Construction of a 2×2 design space for synthetic query generation with AutoQ, showing how the four resulting query classes map onto the local-global query spectrum.  AutoQ can be configured to generate any number and distribution of synthetic queries along these classes, enabling consistent benchmarking across datasets without requiring user customization. Figure 2 shows the synthesis process and sample queries from each class, using an AP News dataset. Figure 2. Synthesis process and example query for each of the four AutoQ query classes.  About Microsoft Research Advancing science and technology to benefit humanity View our story Opens in a new tab AutoE: Automated evaluation framework  Our evaluation of GraphRAG focused on analyzing key qualities of answers to global questions. The following qualities were used for the current evaluation: Comprehensiveness: Does the answer address all relevant aspects of the question?  Diversity: Does it present varied perspectives or insights?  Empowerment: Does it help the reader understand and make informed judgments?  Relevance: Does it address what the question is specifically asking?   The AutoE component scales evaluation of these qualities using the LLM-as-a-Judge method. It presents pairs of answers to an LLM, along with the query and target metric, in counterbalanced order. The model determines whether the first answer wins, loses, or ties with the second. Over a set of queries, whether from AutoQ or elsewhere, this produces win rates between competing methods. When ground truth is available, AutoE can also score answers on correctness, completeness, and related metrics. An illustrative evaluation is shown in Figure 3. Using a dataset of 1,397 AP News articles on health and healthcare, AutoQ generated 50 queries per class . AutoE then compared LazyGraphRAG to a competing RAG method, running six trials per query across four metrics, using GPT-4.1 as a judge. These trial-level results were aggregated using metric-based win rates, where each trial is scored 1 for a win, 0.5 for a tie, and 0 for a loss, and then averaged to calculate the overall win rate for each RAG method. Figure 3. Win rates of four LazyGraphRAG configurations across methods, broken down by the AutoQ query class and averaged across AutoE’s four metrics: comprehensiveness, diversity, empowerment, and relevance. LazyGraphRAG outperforms comparison conditions where the bar is above 50%. The four LazyGraphRAG conditionsdiffer by query budgetand chunk size. All used GPT-4o mini for relevance tests and GPT-4o for query expansionand answer generation, except for LGR_b200_c200_mini, which used GPT-4o mini throughout. Comparison systems were GraphRAG , Vector RAG with 8k- and 120k-token windows, and three published methods: LightRAG, RAPTOR, and TREX. All methods were limited to the same 8k tokens for answer generation. GraphRAG Global Search used level 2 of the community hierarchy. LazyGraphRAG outperformed every comparison condition using the same generative model, winning all 96 comparisons, with all but one reaching statistical significance. The best overall performance came from the larger budget, smaller chunk size configuration. For DataLocal queries, the smaller budgetperformed slightly better, likely because fewer chunks were relevant. For ActivityLocal queries, the larger chunk sizehad a slight edge, likely because longer chunks provide a more coherent context. Competing methods performed relatively better on the query classes for which they were designed: GraphRAG Global for global queries, Vector RAG for local queries, and GraphRAG Drift Search, which combines both strategies, posed the strongest challenge overall. Increasing Vector RAG’s context window from 8k to 120k tokens did not improve its performance compared to LazyGraphRAG. This raised the question of how LazyGraphRAG would perform against Vector RAG with 1-million token context window containing most of the dataset. Figure 4 shows the follow-up experiment comparing LazyGraphRAG to Vector RAG using GPT-4.1 that enabled this comparison. Even against the 1M-token window, LazyGraphRAG achieved higher win rates across all comparisons, failing to reach significance only for the relevance of answers to DataLocal queries. These queries tend to benefit most from Vector RAG’s ranking of directly relevant chunks, making it hard for LazyGraphRAG to generate answers that have greater relevance to the query, even though these answers may be dramatically more comprehensive, diverse, and empowering overall. Figure 4. Win rates of LazyGraphRAG  over Vector RAG across different context window sizes, broken down by the four AutoQ query classes and four AutoE metrics: comprehensiveness, diversity, empowerment, and relevance. Bars above 50% indicate that LazyGraphRAG outperformed the comparison condition.  AutoD: Automated data sampling and summarization Text datasets have an underlying topical structure, but the depth, breadth, and connectivity of that structure can vary widely. This variability makes it difficult to evaluate RAG systems consistently, as results may reflect the idiosyncrasies of the dataset rather than the system’s general capabilities. The AutoD component addresses this by sampling datasets to meet a target specification, defined by the number of topic clustersand the number of samples per cluster. This creates consistency across datasets, enabling more meaningful comparisons, as structurally aligned datasets lead to comparable AutoQ queries, which in turn support consistent AutoE evaluations. AutoD also includes tools for summarizing input or output datasets in a way that reflects their topical coverage. These summaries play an important role in the AutoQ query synthesis process, but they can also be used more broadly, such as in prompts where context space is limited. Since the release of the GraphRAG paper, we’ve received many requests to share the dataset of the Behind the Tech podcast transcripts we used in our evaluation. An updated version of this dataset is now available in the BenchmarkQED repository, alongside the AP News dataset containing 1,397 health-related articles, licensed for open release.   We hope these datasets, together with the BenchmarkQED tools, help accelerate benchmark-driven development of RAG systems and AI question-answering. We invite the community to try them on GitHub.  Opens in a new tab #benchmarkqedautomatedbenchmarking #ofrag #systems