Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy. Learn more Google has released an updated preview of​​ Gemini 2.5 Pro, its “most intelligent” model, first announced in March and upgraded in May, as a preview, intending to release the same model to general availability in a couple of weeks.  Enterprises can test building new applications or replace earlier versions with an updated version of the “I/O edition” of Gemini 2.5 Pro that, according to a blog post by Google, is more creative in its responses and outperforms other models in coding and reasoning.  During its annual I/O developer conference in May, Google announced that it updated Gemini 2.5 Pro to be better than its earlier iteration, which it quietly released. Google DeepMind CEO Demis Hassabis said the I/O edition is the company’s best coding model yet.  But this new preview, called Gemini 2.5 Pro Preview 06-05 Thinking, is even better than the I/O edition. The stable version Google plans to release publicly is “ready for enterprise-scale capabilities.” The I/O edition, or gemini-2.5-pro-preview-05-06, was first made available to developers and enterprises in May through Google AI Studio and Vertex AI. Gemini 2.5 Pro Preview 06-05 Thinking can be accessed via the same platforms.  Performance metrics This new version of Gemini 2.5 Pro performs even better than the first release.  Google said the new version of Gemini 2.5 Pro improved by 24 points in LMArena and by 35 points in WebDevArena, where it currently tops the leaderboard. The company’s benchmark tests showed that the model outscored competitors like OpenAI’s o3, o3-mini, and o4-mini, Anthropic’s Claude 4 Opus, Grok 3 Beta from xAI and DeepSeek R1.  “We’ve also addressed feedback from our previous 2.5 Pro releases, improving its style and structure — it can be more creative with better-formatted responses,” Google said in the blog post.  What enterprises can expect Google’s continuous improvement of Gemini 2.5 Pro might be confusing for many, but Google previously framed these as a response to community feedback. Pricing for the new version is $1.25 per million tokens without caching for inputs and $10 for the output price.  When the very first version of Gemini 2.5 Pro launched in March, VentureBeat’s Matt Marshall called it “the smartest model you’re not using.” Since then, Google has integrated the model into many of its new applications and services, including “Deep Think,” where Gemini considers multiple hypotheses before responding.  The release of Gemini 2.5 Pro, and its two upgraded versions, revived Google’s place in the large language model space after competitors like DeepSeek and OpenAI diverted the industry’s attention to their reasoning models.  In just a few hours of announcing the updated Gemini 2.5 Pro, developers have already begun playing around with it. While many found the update to live up to Google’s promise of being faster, the jury is still out if this latest Gemini 2.5 Pro does actually perform better.  Daily insights on business use cases with VB Daily If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI. Read our Privacy Policy Thanks for subscribing. Check out more VB newsletters here. An error occured.

One of the key use cases for generative AI involves answering questions over private datasets, with retrieval-augmented generation (RAG) as the go-to framework. As new RAG techniques emerge, there’s a growing need to benchmark their performance across diverse datasets and metrics.  To meet this need, we’re introducing BenchmarkQED, a new suite of tools that automates RAG benchmarking at scale, available on GitHub (opens in new tab). It includes components for query generation, evaluation, and dataset preparation, each designed to support rigorous, reproducible testing.   BenchmarkQED complements the RAG methods in our open-source GraphRAG library, enabling users to run a GraphRAG-style evaluation across models, metrics, and datasets. GraphRAG uses a large language model (LLM) to generate and summarize entity-based knowledge graphs, producing more comprehensive and diverse answers than standard RAG for large-scale tasks.  In this post, we walk through the core components of BenchmarkQED that contribute to the overall benchmarking process. We also share some of the latest benchmark results comparing our LazyGraphRAG system to competing methods, including a vector-based RAG with a 1M-token context window, where the leading LazyGraphRAG configuration showed significant win rates across all combinations of quality metrics and query classes. In the paper, we distinguish between local queries, where answers are found in a small number of text regions, and sometimes even a single region, and global queries, which require reasoning over large portions of or even the entire dataset.  Conventional vector-based RAG excels at local queries because the regions containing the answer to the query resemble the query itself and can be retrieved as the nearest neighbor in the vector space of text embeddings. However, it struggles with global questions, such as, “What are the main themes of the dataset?” which require understanding dataset qualities not explicitly stated in the text.   AutoQ: Automated query synthesis This limitation motivated the development of GraphRAG a system designed to answer global queries. GraphRAG’s evaluation requirements subsequently led to the creation of AutoQ, a method for synthesizing these global queries for any dataset. AutoQ extends this approach by generating synthetic queries across the spectrum of queries, from local to global. It defines four distinct classes based on the source and scope of the query (Figure 1, top) forming a logical progression along the spectrum (Figure 1, bottom). Figure 1. Construction of a 2×2 design space for synthetic query generation with AutoQ, showing how the four resulting query classes map onto the local-global query spectrum.  AutoQ can be configured to generate any number and distribution of synthetic queries along these classes, enabling consistent benchmarking across datasets without requiring user customization. Figure 2 shows the synthesis process and sample queries from each class, using an AP News dataset. Figure 2. Synthesis process and example query for each of the four AutoQ query classes.  About Microsoft Research Advancing science and technology to benefit humanity View our story Opens in a new tab AutoE: Automated evaluation framework  Our evaluation of GraphRAG focused on analyzing key qualities of answers to global questions. The following qualities were used for the current evaluation: Comprehensiveness: Does the answer address all relevant aspects of the question?  Diversity: Does it present varied perspectives or insights?  Empowerment: Does it help the reader understand and make informed judgments?  Relevance: Does it address what the question is specifically asking?   The AutoE component scales evaluation of these qualities using the LLM-as-a-Judge method. It presents pairs of answers to an LLM, along with the query and target metric, in counterbalanced order. The model determines whether the first answer wins, loses, or ties with the second. Over a set of queries, whether from AutoQ or elsewhere, this produces win rates between competing methods. When ground truth is available, AutoE can also score answers on correctness, completeness, and related metrics. An illustrative evaluation is shown in Figure 3. Using a dataset of 1,397 AP News articles on health and healthcare, AutoQ generated 50 queries per class (200 total). AutoE then compared LazyGraphRAG to a competing RAG method, running six trials per query across four metrics, using GPT-4.1 as a judge. These trial-level results were aggregated using metric-based win rates, where each trial is scored 1 for a win, 0.5 for a tie, and 0 for a loss, and then averaged to calculate the overall win rate for each RAG method. Figure 3. Win rates of four LazyGraphRAG (LGR) configurations across methods, broken down by the AutoQ query class and averaged across AutoE’s four metrics: comprehensiveness, diversity, empowerment, and relevance. LazyGraphRAG outperforms comparison conditions where the bar is above 50%. The four LazyGraphRAG conditions (LGR_b200_c200, LGR_b50_c200, LGR_b50_c600, LGR_b200_c200_mini) differ by query budget (b50, b200) and chunk size (c200, c600). All used GPT-4o mini for relevance tests and GPT-4o for query expansion (to five subqueries) and answer generation, except for LGR_b200_c200_mini, which used GPT-4o mini throughout. Comparison systems were GraphRAG (Local, Global, and Drift Search), Vector RAG with 8k- and 120k-token windows, and three published methods: LightRAG (opens in new tab), RAPTOR (opens in new tab), and TREX (opens in new tab). All methods were limited to the same 8k tokens for answer generation. GraphRAG Global Search used level 2 of the community hierarchy. LazyGraphRAG outperformed every comparison condition using the same generative model (GPT-4o), winning all 96 comparisons, with all but one reaching statistical significance. The best overall performance came from the larger budget, smaller chunk size configuration (LGR_b200_c200). For DataLocal queries, the smaller budget (LGR_b50_c200) performed slightly better, likely because fewer chunks were relevant. For ActivityLocal queries, the larger chunk size (LGR_b50_c600) had a slight edge, likely because longer chunks provide a more coherent context. Competing methods performed relatively better on the query classes for which they were designed: GraphRAG Global for global queries, Vector RAG for local queries, and GraphRAG Drift Search, which combines both strategies, posed the strongest challenge overall. Increasing Vector RAG’s context window from 8k to 120k tokens did not improve its performance compared to LazyGraphRAG. This raised the question of how LazyGraphRAG would perform against Vector RAG with 1-million token context window containing most of the dataset. Figure 4 shows the follow-up experiment comparing LazyGraphRAG to Vector RAG using GPT-4.1 that enabled this comparison. Even against the 1M-token window, LazyGraphRAG achieved higher win rates across all comparisons, failing to reach significance only for the relevance of answers to DataLocal queries. These queries tend to benefit most from Vector RAG’s ranking of directly relevant chunks, making it hard for LazyGraphRAG to generate answers that have greater relevance to the query, even though these answers may be dramatically more comprehensive, diverse, and empowering overall. Figure 4. Win rates of LazyGraphRAG (LGR) over Vector RAG across different context window sizes, broken down by the four AutoQ query classes and four AutoE metrics: comprehensiveness, diversity, empowerment, and relevance. Bars above 50% indicate that LazyGraphRAG outperformed the comparison condition.  AutoD: Automated data sampling and summarization Text datasets have an underlying topical structure, but the depth, breadth, and connectivity of that structure can vary widely. This variability makes it difficult to evaluate RAG systems consistently, as results may reflect the idiosyncrasies of the dataset rather than the system’s general capabilities. The AutoD component addresses this by sampling datasets to meet a target specification, defined by the number of topic clusters (breadth) and the number of samples per cluster (depth). This creates consistency across datasets, enabling more meaningful comparisons, as structurally aligned datasets lead to comparable AutoQ queries, which in turn support consistent AutoE evaluations. AutoD also includes tools for summarizing input or output datasets in a way that reflects their topical coverage. These summaries play an important role in the AutoQ query synthesis process, but they can also be used more broadly, such as in prompts where context space is limited. Since the release of the GraphRAG paper, we’ve received many requests to share the dataset of the Behind the Tech (opens in new tab) podcast transcripts we used in our evaluation. An updated version of this dataset is now available in the BenchmarkQED repository (opens in new tab), alongside the AP News dataset containing 1,397 health-related articles, licensed for open release.   We hope these datasets, together with the BenchmarkQED tools (opens in new tab), help accelerate benchmark-driven development of RAG systems and AI question-answering. We invite the community to try them on GitHub (opens in new tab).  Opens in a new tab

Text embedding and reranking are foundational to modern information retrieval systems, powering applications such as semantic search, recommendation systems, and retrieval-augmented generation (RAG). However, current approaches often face key challenges—particularly in achieving both high multilingual fidelity and task adaptability without relying on proprietary APIs. Existing models frequently fall short in scenarios requiring nuanced semantic understanding across multiple languages or domain-specific tasks like code retrieval and instruction following. Moreover, most open-source models either lack scale or flexibility, while commercial APIs remain costly and closed. Qwen3-Embedding and Qwen3-Reranker: A New Standard for Open-Source Embedding Alibaba’s Qwen Team has unveiled the Qwen3-Embedding and Qwen3-Reranker Series—models that set a new benchmark in multilingual text embedding and relevance ranking. Built on the Qwen3 foundation models, the series includes variants in 0.6B, 4B, and 8B parameter sizes and supports a wide range of languages (119 in total), making it one of the most versatile and performant open-source offerings to date. These models are now open-sourced under the Apache 2.0 license on Hugging Face, GitHub, and ModelScope, and are also accessible via Alibaba Cloud APIs. These models are optimized for use cases such as semantic retrieval, classification, RAG, sentiment analysis, and code search—providing a strong alternative to existing solutions like Gemini Embedding and OpenAI’s embedding APIs. Technical Architecture Qwen3-Embedding models adopt a dense transformer-based architecture with causal attention, producing embeddings by extracting the hidden state corresponding to the [EOS] token. Instruction-awareness is a key feature: input queries are formatted as {instruction} {query}<|endoftext|>, enabling task-conditioned embeddings. The reranker models are trained with a binary classification format, judging document-query relevance in an instruction-guided manner using a token likelihood-based scoring function. The models are trained using a robust multi-stage training pipeline: Large-scale weak supervision: 150M synthetic training pairs generated using Qwen3-32B, covering retrieval, classification, STS, and bitext mining across languages and tasks. Supervised fine-tuning: 12M high-quality data pairs are selected using cosine similarity (>0.7), fine-tuning performance in downstream applications. Model merging: Spherical linear interpolation (SLERP) of multiple fine-tuned checkpoints ensures robustness and generalization. This synthetic data generation pipeline enables control over data quality, language diversity, task difficulty, and more—resulting in a high degree of coverage and relevance in low-resource settings. Performance Benchmarks and Insights The Qwen3-Embedding and Qwen3-Reranker series demonstrate strong empirical performance across several multilingual benchmarks. On MMTEB (216 tasks across 250+ languages), Qwen3-Embedding-8B achieves a mean task score of 70.58, surpassing Gemini and GTE-Qwen2 series. On MTEB (English v2): Qwen3-Embedding-8B reaches 75.22, outperforming other open models including NV-Embed-v2 and GritLM-7B. On MTEB-Code: Qwen3-Embedding-8B leads with 80.68, excelling in applications like code retrieval and Stack Overflow QA. For reranking: Qwen3-Reranker-0.6B already outperforms Jina and BGE rerankers. Qwen3-Reranker-8B achieves 81.22 on MTEB-Code and 72.94 on MMTEB-R, marking state-of-the-art performance. Ablation studies confirm the necessity of each training stage. Removing synthetic pretraining or model merging led to significant performance drops (up to 6 points on MMTEB), emphasizing their contributions. Conclusion Alibaba’s Qwen3-Embedding and Qwen3-Reranker Series present a robust, open, and scalable solution to multilingual and instruction-aware semantic representation. With strong empirical results across MTEB, MMTEB, and MTEB-Code, these models bridge the gap between proprietary APIs and open-source accessibility. Their thoughtful training design—leveraging high-quality synthetic data, instruction-tuning, and model merging—positions them as ideal candidates for enterprise applications in search, retrieval, and RAG pipelines. By open-sourcing these models, the Qwen team not only pushes the boundaries of language understanding but also empowers the broader community to innovate on top of a solid foundation. Check out the Paper, Technical details, Qwen3-Embedding and Qwen3-Reranker. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 95k+ ML SubReddit and Subscribe to our Newsletter. Asif RazzaqWebsite |  + postsBioAsif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.Asif Razzaqhttps://www.marktechpost.com/author/6flvq/A Step-by-Step Coding Guide to Building an Iterative AI Workflow Agent Using LangGraph and GeminiAsif Razzaqhttps://www.marktechpost.com/author/6flvq/From Clicking to Reasoning: WebChoreArena Benchmark Challenges Agents with Memory-Heavy and Multi-Page TasksAsif Razzaqhttps://www.marktechpost.com/author/6flvq/Mistral AI Introduces Mistral Code: A Customizable AI Coding Assistant for Enterprise WorkflowsAsif Razzaqhttps://www.marktechpost.com/author/6flvq/NVIDIA AI Releases Llama Nemotron Nano VL: A Compact Vision-Language Model Optimized for Document Understanding

Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy. Learn more Google has released an updated preview of​​ Gemini 2.5 Pro, its “most intelligent” model, first announced in March and upgraded in May, as a preview, intending to release the same model to general availability in a couple of weeks.  Enterprises can test building new applications or replace earlier versions with an updated version of the “I/O edition” of Gemini 2.5 Pro that, according to a blog post by Google, is more creative in its responses and outperforms other models in coding and reasoning.  During its annual I/O developer conference in May, Google announced that it updated Gemini 2.5 Pro to be better than its earlier iteration, which it quietly released. Google DeepMind CEO Demis Hassabis said the I/O edition is the company’s best coding model yet.  But this new preview, called Gemini 2.5 Pro Preview 06-05 Thinking, is even better than the I/O edition. The stable version Google plans to release publicly is “ready for enterprise-scale capabilities.” The I/O edition, or gemini-2.5-pro-preview-05-06, was first made available to developers and enterprises in May through Google AI Studio and Vertex AI. Gemini 2.5 Pro Preview 06-05 Thinking can be accessed via the same platforms.  Performance metrics This new version of Gemini 2.5 Pro performs even better than the first release.  Google said the new version of Gemini 2.5 Pro improved by 24 points in LMArena and by 35 points in WebDevArena, where it currently tops the leaderboard. The company’s benchmark tests showed that the model outscored competitors like OpenAI’s o3, o3-mini, and o4-mini, Anthropic’s Claude 4 Opus, Grok 3 Beta from xAI and DeepSeek R1.  “We’ve also addressed feedback from our previous 2.5 Pro releases, improving its style and structure — it can be more creative with better-formatted responses,” Google said in the blog post.  What enterprises can expect Google’s continuous improvement of Gemini 2.5 Pro might be confusing for many, but Google previously framed these as a response to community feedback. Pricing for the new version is $1.25 per million tokens without caching for inputs and $10 for the output price.  When the very first version of Gemini 2.5 Pro launched in March, VentureBeat’s Matt Marshall called it “the smartest model you’re not using.” Since then, Google has integrated the model into many of its new applications and services, including “Deep Think,” where Gemini considers multiple hypotheses before responding.  The release of Gemini 2.5 Pro, and its two upgraded versions, revived Google’s place in the large language model space after competitors like DeepSeek and OpenAI diverted the industry’s attention to their reasoning models.  In just a few hours of announcing the updated Gemini 2.5 Pro, developers have already begun playing around with it. While many found the update to live up to Google’s promise of being faster, the jury is still out if this latest Gemini 2.5 Pro does actually perform better.  Daily insights on business use cases with VB Daily If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI. Read our Privacy Policy Thanks for subscribing. Check out more VB newsletters here. An error occured.

Pentagram's latest project introduces a striking new identity for TwelveLabs, an AI company redefining how machines comprehend video. At its heart, the rebrand marks a conceptual shift, positioning video not as a linear sequence of frames but as a volume. For TwelveLabs, this reframing is more than a technological advancement, as it's setting them apart in the fast-moving AI landscape. For Jody Hudson-Powell and the Pentagram team, it became the foundation for the entire creative direction. "The 'video as volume' idea became our way of visualising their technology," Jody explains. "Traditionally, AI sees video as separate frames – individual moments, isolated. But TwelveLabs understands everything at once, an interconnected whole. This shift in perspective became the core idea guiding all our design choices." Turning such an abstract and dynamic concept into a tangible brand identity was no small feat. In a sector often awash with dense data visualisations and corporate clichés, Pentagram's work for TwelveLabs stands out for its clarity and restraint. Described by the team as "smart, elegant, alive", the identity not only enhances the product's appearance but also helps explain it. "Abstract concepts can easily get complicated, but by staying closely focused on 'video as volume,' every decision felt purposeful and naturally clear," says Jody. A key part of this clarity is the system's use of diagrams and modular layouts. These elements do the heavy lifting when it comes to representing TwelveLabs' complex technology in a way that's accessible to a wide range of audiences, from engineers and developers to enterprise clients and creatives. "Complexity becomes understandable when you strip things back to what's truly important," Jody says. "We spent time refining the diagrams to say just enough – clear for everyone but authentic to those who really know the tech. When engineers see their work clearly represented, and others intuitively understand it, you've found some interesting ground." What's striking is how the identity balances the technical with the emotive. It's a visual language that feels structured but not cold, intelligent but not aloof. According to Jody, this tension was carefully calibrated. "Technology can feel distant, cold, but it's built around human experiences: images, memories, moments," he explains. "Our system provided clarity and structure, creating natural space for these human elements to emerge, balancing intelligence and warmth." Motion, too, plays a critical role in bringing the brand to life. Given that TwelveLabs' platform revolves around continuous, connected reasoning, it made sense for motion to echo that logic, being subtle but purposeful. "Motion always means something, even subtly," says Jody. "Every bit of movement mirrored the continuous reasoning of the platform. Allowing room for gentle expressiveness and even playfulness didn't take away from clarity – it enhanced it." One of the most distinctive elements of the new identity is the horse symbol. On the surface, it's a nod to Eadweard Muybridge's pioneering photographic studies of motion and an iconic reference for any visual technology. But there's a deeper connection, as TwelveLabs already named its AI models after horses. The Pentagram team simply brought that story to the forefront. "Often, the best ideas are already there, just waiting to be noticed," says Jody. "We simply amplified that story, connecting it back to Muybridge's historical images, allowing it to clearly communicate motion and intelligence in a fresh yet familiar way." Throughout the project, the team had to consider the diversity of TwelveLabs' audience, from developers and researchers to large-scale enterprise clients and the broader creative community. The result is an identity system that feels accessible without being simplistic and capable of meeting users at every level of expertise. "We aimed for a universal tone – clear, direct, and welcoming for everyone, regardless of their expertise," Jody says. "Precision doesn't have to be complicated, and clarity invites everyone in. Finding a voice that felt calm, clear, and honest meant meeting each person exactly where they are." The outcome is an identity that doesn't just repackage complex AI technology. It embodies the very qualities that make TwelveLabs' approach revolutionary: interconnected, intelligent, and distinctly human.

Limited-time offer: Get more than 30% off a Vox Membership. Join today to support independent journalism. Federal civil servants are supposed to enjoy robust protections against being fired or demoted for political reasons. But President Donald Trump has effectively stripped them of these protections by neutralizing the federal agencies that implement these safeguards.An agency known as the Merit Systems Protection Board (MSPB) hears civil servants’ claims that a “government employer discriminated against them, retaliated against them for whistleblowing, violated protections for veterans, or otherwise subjected them to an unlawful adverse employment action or prohibited personnel practice,” as a federal appeals court explained in an opinion on Tuesday. But the three-member board currently lacks the quorum it needs to operate because Trump fired two of the members.Trump also fired Hampton Dellinger, who until recently served as the special counsel of the United States, a role that investigates alleged violations of federal civil service protections and brings related cases to the MSPB. Trump recently nominated Paul Ingrassia, a far-right podcaster and recent law school graduate to replace Dellinger.The upshot of these firings is that no one in the government is able to enforce laws and regulations protecting civil servants. As Dellinger noted in an interview, the morning before a federal appeals court determined that Trump could fire him, he’d “been able to get 6,000 newly hired federal employees back on the job,” and was working to get “all probationary employees put back on the job [after] their unlawful firing” by the Department of Government Efficiency and other Trump administration efforts to cull the federal workforce. These and other efforts to reinstate illegally fired federal workers are on hold, and may not resume until Trump leaves office.Which brings us to the US Court of Appeals for the Fourth Circuit’s decision in National Association of Immigration Judges v. Owen, which proposes an innovative solution to this problem.As the Owen opinion notes, the Supreme Court has held that the MSPB process is the only process a federal worker can use if they believe they’ve been fired in violation of federal civil service laws. So if that process is shut down, the worker is out of luck.But the Fourth Circuit’s Owen opinion argues that this “conclusion can only be true…when the statute functions as Congress intended.” That is, if the MSPB and the special counsel are unable to “fulfill their roles prescribed by” federal law, then the courts should pick up the slack and start hearing cases brought by illegally fired civil servants.For procedural reasons, the Fourth Circuit’s decision will not take effect right away — the court sent the case back down to a trial judge to “conduct a factual inquiry” into whether the MSPB continues to function. And, even after that inquiry is complete, the Trump administration is likely to appeal the Fourth Circuit’s decision to the Supreme Court if it wants to keep civil service protections on ice.If the justices agree with the circuit court, however, that will close a legal loophole that has left federal civil servants unprotected by laws that are still very much on the books. And it will cure a problem that the Supreme Court bears much of the blame for creating.The “unitary executive,” or why the Supreme Court is to blame for the loss of civil service protectionsFederal law provides that Dellinger could “be removed by the President only for inefficiency, neglect of duty, or malfeasance in office,” and members of the MSPB enjoy similar protections against being fired. Trump’s decision to fire these officials was illegal under these laws.But a federal appeals court nonetheless permitted Trump to fire Dellinger, and the Supreme Court recently backed Trump’s decision to fire the MSPB members as well. The reason is a legal theory known as the “unitary executive,” which is popular among Republican legal scholars, and especially among the six Republicans that control the Supreme Court.If you want to know all the details of this theory, I can point you to three different explainers I’ve written on the unitary executive. The short explanation is that the unitary executive theory claims that the president must have the power to fire top political appointees charged with executing federal laws – including officials who execute laws protecting civil servants from illegal firings.But the Supreme Court has never claimed that the unitary executive permits the president to fire any federal worker regardless of whether Congress has protected them or not. In a seminal opinion laying out the unitary executive theory, for example, Justice Antonin Scalia argued that the president must have the power to remove “principal officers” — high-ranking officials like Dellinger who must be nominated by the president and confirmed by the Senate. Under Scalia’s approach, lower-ranking government workers may still be given some protection.The Fourth Circuit cannot override the Supreme Court’s decision to embrace the unitary executive theory. But the Owen opinion essentially tries to police the line drawn by Scalia. The Supreme Court has given Trump the power to fire some high-ranking officials, but he shouldn’t be able to use that power as a back door to eliminate job protections for all civil servants.The Fourth Circuit suggests that the federal law which simultaneously gave the MSPB exclusive authority over civil service disputes, while also protecting MSPB members from being fired for political reasons, must be read as a package. Congress, this argument goes, would not have agreed to shunt all civil service disputes to the MSPB if it had known that the Supreme Court would strip the MSPB of its independence. And so, if the MSPB loses its independence, it must also lose its exclusive authority over civil service disputes — and federal courts must regain the power to hear those cases.It remains to be seen whether this argument persuades a Republican Supreme Court — all three of the Fourth Circuit judges who decided the Owen case are Democrats, and two are Biden appointees. But the Fourth Circuit’s reasoning closely resembles the kind of inquiry that courts frequently engage in when a federal law is struck down.When a court declares a provision of federal law unconstitutional, it often needs to ask whether other parts of the law should fall along with the unconstitutional provision, an inquiry known as “severability.” Often, this severability analysis asks which hypothetical law Congress would have enacted if it had known that the one provision is invalid.The Fourth Circuit’s decision in Owen is essentially a severability opinion. It takes as a given the Supreme Court’s conclusion that laws protecting Dellinger and the MSPB members from being fired are unconstitutional, then asks which law Congress would have enacted if it had known that it could not protect MSPB members from political reprisal. The Fourth Circuit’s conclusion is that, if Congress had known that MSPB members cannot be politically independent, then it would not have given them exclusive authority over civil service disputes.If the Supreme Court permits Trump to neutralize the MSPB, that would fundamentally change how the government functionsThe idea that civil servants should be hired based on merit and insulated from political pressure is hardly new. The first law protecting civil servants, the Pendleton Civil Service Reform Act, which President Chester A. Arthur signed into law in 1883.Laws like the Pendleton Act do more than protect civil servants who, say, resist pressure to deny government services to the president’s enemies. They also make it possible for top government officials to actually do their jobs.Before the Pendleton Act, federal jobs were typically awarded as patronage — so when a Democratic administration took office, the Republicans who occupied most federal jobs would be fired and replaced by Democrats. This was obviously quite disruptive, and it made it difficult for the government to hire highly specialized workers. Why would someone go to the trouble of earning an economics degree and becoming an expert on federal monetary policy, if they knew that their job in the Treasury Department would disappear the minute their party lost an election?Meanwhile, the task of filling all of these patronage jobs overwhelmed new presidents. As Candice Millard wrote in a 2011 biography of President James A. Garfield, the last president elected before the Pendleton Act, when Garfield took office, a line of job seekers began to form outside the White House “before he even sat down to breakfast.” By the time Garfield had eaten, this line “snaked down the front walk, out the gate, and onto Pennsylvania Avenue.” Garfield was assassinated by a disgruntled job seeker, a fact that likely helped build political support for the Pendleton Act.By neutralizing the MSPB, Trump is effectively undoing nearly 150 years worth of civil service reforms, and returning the federal government to a much more primitive state. At the very least, the Fourth Circuit’s decision in Owen is likely to force the Supreme Court to ask if it really wants a century and a half of work to unravel.See More:

On June 3, Yoshua Bengio, the world’s most-cited computer scientist, announced the launch of LawZero, a nonprofit that aims to create “safe by design” AI by pursuing a fundamentally different approach to major tech companies. Players like OpenAI and Google are investing heavily in AI agents—systems that not only answer queries and generate images, but can craft plans and take actions in the world. The goal of these companies is to create virtual employees that can do practically any job a human can, known in the tech industry as artificial general intelligence, or AGI. Executives like Google DeepMind’s CEO Demis Hassabis point to AGI’s potential to solve climate change or cure disease as a motivator for its development. Bengio, however, says we don't need agentic systems to reap AI's rewards—it's a false choice. He says there's a chance such a system could escape human control, with potentially irreversible consequences. “If we get an AI that gives us the cure for cancer, but also maybe another version of that AI goes rogue and generates wave after wave of bio-weapons that kill billions of people, then I don't think it's worth it," he says. In 2023, Bengio, along with others including OpenAI’s CEO Sam Altman signed a statement declaring that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”Now, Bengio, through LawZero, aims to sidestep the existential perils by focusing on creating what he calls “Scientist AI”—a system trained to understand and make statistical predictions about the world, crucially, without the agency to take independent actions. As he puts it: We could use AI to advance scientific progress without rolling the dice on agentic AI systems.Why Bengio Says We Need A New Approach To AI The current approach to giving AI agency is “dangerous,” Bengio says. While most software operates through rigid if-then rules—if the user clicks here, do this—today's AI systems use deep learning. The technique, which Bengio helped pioneer, trains artificial networks modeled loosely on the brain to find patterns in vast amounts of data. But recognizing patterns is just the first step. To turn these systems into useful applications like chatbots, engineers employ a training process called reinforcement learning. The AI generates thousands of responses and receives feedback on each one: a virtual “carrot” for helpful answers and a virtual “stick” for responses that miss the mark. Through millions of these trial-and-feedback cycles, the system gradually learns to predict what responses are most likely to get a reward. “It’s more like growing a plant or animal,” Bengio says. “You don’t fully control what the animal is going to do. You provide it with the right conditions, and it grows and it becomes smarter. You can try to steer it in various directions.”The same basic approach is now being used to imbue AI with greater agency. Models are tasked with challenges with verifiable answers—like math puzzles or coding problems—and are then rewarded for taking the series of actions that yields the solution. This approach has seen AI shatter previous benchmarks in programming and scientific reasoning. For example, at the beginning of 2024, the best AI model scored only 2% on a standardized test for AI of sorts consisting of real world software engineering problems; by December, an impressive 71.7%. But with AI’s greater problem-solving ability comes the emergence of new deceptive skills, Bengio says. The last few months have borne witness to AI systems learning to mislead, cheat, and try to evade shutdown—even resorting to blackmail. These have almost exclusively been in carefully contrived experiments that almost beg the AI to misbehave—for example, by asking it to pursue its goal at all costs. Reports of such behavior in the real-world, though, have begun to surface. Popular AI coding startup Replit’s agent ignored explicit instruction not to edit a system file that could break the company’s software, in what CEO Amjad Masad described as an “Oh f***” moment,” on the Cognitive Revolution podcast in May. The company’s engineers intervened, cutting the agent’s access by moving the file to a secure digital sandbox, only for the AI agent to attempt to “socially engineer” the user to regain access.The quest to build human-level AI agents using techniques known to produce deceptive tendencies, Bengio says, is comparable to a car speeding down a narrow mountain road, with steep cliffs on either side, and thick fog obscuring the path ahead. “We need to set up the car with headlights and put some guardrails on the road,” he says.What is “Scientist AI”?LawZero’s focus is on developing “Scientist AI” which, as Bengio describes, would be fundamentally non-agentic, trustworthy, and focused on understanding and truthfulness, rather than pursuing its own goals or merely imitating human behavior. The aim is creating a powerful tool that, while lacking the same autonomy other models have, is capable of generating hypotheses and accelerating scientific progress to “help us solve challenges of humanity,” Bengio says.LawZero has raised nearly $30 million already from several philanthropic backers including from Schmidt Sciences and Open Philanthropy. “We want to raise more because we know that as we move forward, we'll need significant compute,” Bengio says. But even ten times that figure would pale in comparison to the roughly $200 billion spent last year by tech giants on aggressively pursuing AI. Bengio’s hope is that Scientist AI could help ensure the safety of highly autonomous systems developed by other players. “We can use those non-agentic AIs as guardrails that just need to predict whether the action of an agentic AI is dangerous," Bengio says. Technical interventions will only ever be one part of the solution, he adds, noting the need for regulations to ensure that safe practices are adopted.LawZero, named after science fiction author Isaac Asimov’s zeroth law of robotics—“a robot may not harm humanity, or, by inaction, allow humanity to come to harm”—is not the first nonprofit founded to chart a safer path for AI development. OpenAI was founded as a nonprofit in 2015 with the goal of “ensuring AGI benefits all of humanity,” and intended to serve a counterbalance to industry players guided by profit motives. Since opening a for-profit arm in 2019, the organization has become one of the most valuable private companies in the world, and has faced criticism, including from former staffers, who argue it has drifted from its founding ideals. "Well, the good news is we have the hindsight of maybe what not to do,” Bengio says, adding that he wants to avoid profit incentives and “bring governments into the governance of LawZero.”“I think everyone should ask themselves, ‘What can I do to make sure my children will have a future,’” Bengio says. In March, he stepped down as scientific director of Mila, the academic lab he co-founded in the early nineties, in an effort to reorient his work towards tackling AI risk more directly. “Because I'm a researcher, my answer is, ‘okay, I'm going to work on this scientific problem where maybe I can make a difference,’ but other people may have different answers."

Modern software engineering faces growing challenges in accurately retrieving and understanding code across diverse programming languages and large-scale codebases. Existing embedding models often struggle to capture the deep semantics of code, resulting in poor performance in tasks such as code search, RAG, and semantic analysis. These limitations hinder developers’ ability to efficiently locate relevant code snippets, reuse components, and manage large projects effectively. As software systems grow increasingly complex, there is a pressing need for more effective, language-agnostic representations of code that can power reliable and high-quality retrieval and reasoning across a wide range of development tasks.  Mistral AI has introduced Codestral Embed, a specialized embedding model built specifically for code-related tasks. Designed to handle real-world code more effectively than existing solutions, it enables powerful retrieval capabilities across large codebases. What sets it apart is its flexibility—users can adjust embedding dimensions and precision levels to balance performance with storage efficiency. Even at lower dimensions, such as 256 with int8 precision, Codestral Embed reportedly surpasses top models from competitors like OpenAI, Cohere, and Voyage, offering high retrieval quality at a reduced storage cost. Beyond basic retrieval, Codestral Embed supports a wide range of developer-focused applications. These include code completion, explanation, editing, semantic search, and duplicate detection. The model can also help organize and analyze repositories by clustering code based on functionality or structure, eliminating the need for manual supervision. This makes it particularly useful for tasks like understanding architectural patterns, categorizing code, or supporting automated documentation, ultimately helping developers work more efficiently with large and complex codebases.  Codestral Embed is tailored for understanding and retrieving code efficiently, especially in large-scale development environments. It powers retrieval-augmented generation by quickly fetching relevant context for tasks like code completion, editing, and explanation—ideal for use in coding assistants and agent-based tools. Developers can also perform semantic code searches using natural language or code queries to find relevant snippets. Its ability to detect similar or duplicated code helps with reuse, policy enforcement, and cleaning up redundancy. Additionally, it can cluster code by functionality or structure, making it useful for repository analysis, spotting architectural patterns, and enhancing documentation workflows.  Codestral Embed is a specialized embedding model designed to enhance code retrieval and semantic analysis tasks. It surpasses existing models, such as OpenAI’s and Cohere’s, in benchmarks like SWE-Bench Lite and CodeSearchNet. The model offers customizable embedding dimensions and precision levels, allowing users to effectively balance performance and storage needs. Key applications include retrieval-augmented generation, semantic code search, duplicate detection, and code clustering. Available via API at $0.15 per million tokens, with a 50% discount for batch processing, Codestral Embed supports various output formats and dimensions, catering to diverse development workflows. In conclusion, Codestral Embed offers customizable embedding dimensions and precisions, enabling developers to strike a balance between performance and storage efficiency. Benchmark evaluations indicate that Codestral Embed surpasses existing models like OpenAI’s and Cohere’s in various code-related tasks, including retrieval-augmented generation and semantic code search. Its applications span from identifying duplicate code segments to facilitating semantic clustering for code analytics. Available through Mistral’s API, Codestral Embed provides a flexible and efficient solution for developers seeking advanced code understanding capabilities.  vides valuable insights for the community. Check out the Technical details. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 95k+ ML SubReddit and Subscribe to our Newsletter. Sana HassanSana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.Sana Hassanhttps://www.marktechpost.com/author/sana-hassan/Off-Policy Reinforcement Learning RL with KL Divergence Yields Superior Reasoning in Large Language ModelsSana Hassanhttps://www.marktechpost.com/author/sana-hassan/This AI Paper from Microsoft Introduces WINA: A Training-Free Sparse Activation Framework for Efficient Large Language Model InferenceSana Hassanhttps://www.marktechpost.com/author/sana-hassan/Apple and Duke Researchers Present a Reinforcement Learning Approach That Enables LLMs to Provide Intermediate Answers, Enhancing Speed and AccuracySana Hassanhttps://www.marktechpost.com/author/sana-hassan/National University of Singapore Researchers Introduce Dimple: A Discrete Diffusion Multimodal Language Model for Efficient and Controllable Text Generation

Large Reasoning Models (LRMs), trained from LLMs using reinforcement learning (RL), demonstrated great performance in complex reasoning tasks, including mathematics, STEM, and coding. However, existing LRMs face challenges in completing various puzzle tasks that require purely logical reasoning skills, which are easy and obvious for humans. Current methods targeting puzzles focus only on designing benchmarks for evaluation, lacking the training methods and resources for modern LLMs to tackle this challenge. Current puzzle datasets lack diversity and scalability, covering limited puzzle types with little control over generation or difficulty. Moreover, due to the success of the “LLM+RLVR” paradigm, it has become crucial to obtain large, diverse, and challenging sets of verifiable puzzle prompts for training agents. Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key method for improving models’ reasoning capabilities, removing the need for reward models by directly assigning rewards based on objectively verifiable answers. Puzzles are particularly well-suited for RLVR. However, most prior RLVR research has overlooked the puzzles’ potential for delivering effective reward signals. In puzzle reasoning of LLMs, existing benchmarks evaluate different types of reasoning, including abstract, deductive, and compositional reasoning. Few benchmarks support scalable generation and difficulty control but lack puzzle diversity. Moreover, the improvement of LLMs’ puzzle-solving abilities mainly falls into two categories: tool integration and RLVR. Researchers from ByteDance Seed, Fudan University, Tsinghua University, Nanjing University, and Shanghai Jiao Tong University have proposed Enigmata, the first comprehensive toolkit designed for improving LLMs with puzzle reasoning skills. It contains 36 tasks across seven categories, each featuring a generator that produces unlimited examples with controllable difficulty and a rule-based verifier for automatic evaluation. The researchers further developed Enigmata-Eval as a rigorous benchmark and created optimized multi-task RLVR strategies. Puzzle data from Enigmata enhances SoTA performance on advanced math and STEM reasoning tasks like AIME, BeyondAIME, and GPQA when trained on larger models like Seed1.5-Thinking. This shows the generalization benefits of Enigmata. The Enigmata-Data comprises 36 puzzle tasks organized into 7 primary categories, including Crypto, Arithmetic, Logic, Grid, Graph, Search, and Sequential Puzzle, making it the only dataset having multiple task categories with scalability, automatic verification, and public availability. The data construction follows a three-phase pipeline: Tasks Collection and Design, Auto-Generator and Verifier Development, and Sliding Difficulty Control. Moreover, the Enigmata-Eval is developed by systematically sampling from the broader dataset, aiming to extract 50 instances per difficulty level for each task. The final evaluation set contains 4,758 puzzle instances rather than the theoretical maximum of 5,400, due to inherent constraints, where some tasks generate fewer instances per difficulty level. The proposed model outperforms most public models on Enigmata-Eval with 32B parameters, showing the effectiveness of the dataset and training recipe. The model stands out on the challenging ARC-AGI benchmark, surpassing strong reasoning models such as Gemini 2.5 Pro, o3-mini, and o1. The Qwen2.5-32B-Enigmata shows outstanding performance in structured reasoning categories, outperforming in Crypto, Arithmetic, and Logic tasks, suggesting effective development of rule-based reasoning capabilities. The model shows competitive performance in search tasks that require strategic exploration and planning capabilities. Moreover, Crypto and Arithmetic tasks tend to provide the highest accuracy, while spatial and sequential tasks remain more difficult. In this paper, researchers introduced Enigmata, a comprehensive suite for equipping LLMs with advanced puzzle reasoning that integrates seamlessly with RL using verifiable rule-based rewards. The trained Enigmata-Model shows superior performance and robust generalization skills through RLVR training. Experiments reveal that when applied to larger models such as Seed1.5-Thinking (20B/200B parameters), synthetic puzzle data brings additional benefits in other domains, including mathematics and STEM reasoning over state-of-the-art models. Enigmata provides a solid foundation for the research community to advance reasoning model development, offering a unified framework that effectively bridges logical puzzle-solving with broader reasoning capabilities in LLMs. Check out the Paper, GitHub Page and Project Page. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 95k+ ML SubReddit and Subscribe to our Newsletter. Sajjad AnsariSajjad Ansari is a final year undergraduate from IIT Kharagpur. As a Tech enthusiast, he delves into the practical applications of AI with a focus on understanding the impact of AI technologies and their real-world implications. He aims to articulate complex AI concepts in a clear and accessible manner.Sajjad Ansarihttps://www.marktechpost.com/author/sajjadansari/Multimodal Foundation Models Fall Short on Physical Reasoning: PHYX Benchmark Highlights Key Limitations in Visual and Symbolic IntegrationSajjad Ansarihttps://www.marktechpost.com/author/sajjadansari/Meta AI Introduces Multi-SpatialMLLM: A Multi-Frame Spatial Understanding with Multi-modal Large Language ModelsSajjad Ansarihttps://www.marktechpost.com/author/sajjadansari/Can LLMs Really Judge with Reasoning? Microsoft and Tsinghua Researchers Introduce Reward Reasoning Models to Dynamically Scale Test-Time Compute for Better AlignmentSajjad Ansarihttps://www.marktechpost.com/author/sajjadansari/NVIDIA AI Introduces AceReason-Nemotron for Advancing Math and Code Reasoning through Reinforcement Learning

Meta (Nasdaq:META) is restructuring its AI division into two distinct units, AI Products and AGI Foundations, marking its most significant internal overhaul as it races to compete with OpenAI and Google. The move, detailed in an internal memo from Chief Product Officer Chris Cox and reported by Axios, appoints Connor Hayes to lead AI product integration while Ahmad Al-Dahle and Amir Frenkel will co-direct long-term AGI research. The restructuring comes amid mounting crises, including the delayed Llama 4 Behemoth model and the departure of key Llama architects to competitors like Mistral AI. This marks Meta’s second major AI reorganization since CEO Mark Zuckerberg’s 2023 attempt to “turbocharge” generative AI efforts, which saw the company fall further behind rivals despite early promise. “Structural changes alone won’t solve Meta’s AI challenges,” said Amandeep Singh, practice director at QKS Group. “While the new AGI Foundations unit creates clarity, retaining elite talent requires a seamless pipeline from research to real-world deployment. Meta has struggled with fragmented pipelines and unclear priorities,” added Singh. Talent exodus and technical setbacks The restructuring follows talent losses that have exposed fundamental weaknesses in Meta’s AI strategy. Only three authors remain from the original 14-person Llama research team, according to Business Insider. Internal surveys cited by The Information reveal plummeting morale in Meta’s AI division, where employees cite resource constraints and sluggish progress. “Talent follows momentum, and right now, momentum lives where research decisions directly shape deployed capabilities,” noted Singh. This talent drain coincides with technical setbacks, most notably the underperforming Llama 4 model, which has struggled with reasoning and mathematical tasks. These combined challenges have left Meta playing catch-up in the race toward artificial general intelligence, despite its early open-source advantages. The company’s strategy, bolstered by initiatives such as Llama for Startups and the recent Llama API launch, aims to attract developers and differentiate it from competitors’ proprietary models. But analysts caution these initiatives alone may not be enough to win enterprise trust. The enterprise adoption dilemma While Llama’s cost advantages remain attractive to businesses, growing concerns about its governance controls and the looming copyright lawsuit over training data are giving enterprises pause. “Companies love Llama’s affordability but expanding safety gaps and legal risks are becoming hard to ignore,” Singh said. “For mission-critical applications, many will ultimately choose more reliable, if more expensive, options like GPT or Gemini.” These cost benefits can lose their appeal when weighed against operational risks. Meta’s reorganization attempts to mitigate these concerns through specialized teams, one deploying generative AI across products, another advancing AGI research, but analysts remain skeptical about whether structural changes alone can solve deeper issues. Unlike Microsoft’s turnkey OpenAI integration or Google’s enterprise-ready Vertex AI platform, Meta lacks both the sales infrastructure and compliance pedigree for regulated industries. As Singh argued, “Enterprise AI adoption hinges on proven compliance frameworks, operational reliability, and mature support systems. Meta still needs to build that trust at Fortune 500 scale.” Meta’s race to close the AGI gap “Meta’s AGI push focused on models with reasoning, multimedia, and voice capabilities aligns with the broader industry trend toward multimodal AI as a catalyst for enterprise transformation,” said Surjyadeb Goswami, research director for AI and Automation at IDC Asia Pacific. He noted that open-source models are critical to enabling cost-effective, transparent, and customizable deployments, especially as organizations deepen their GenAI investments. For Meta to truly capitalize on this opportunity and succeed in its AGI bid, it must rebuild trust especially for enterprise adoption. Singh highlighted the need for Linux-like community stewardship, OpenAI-level safety protocols, and robust enterprise tooling. “Balancing openness with responsibility is Meta’s real challenge, especially as models approach general-purpose cognitive capability.” Meta now needs to show this reorganization yields meaningful improvements in model performance, talent retention, and enterprise adoption to validate its new approach. “Meta’s open-source vision is bold, but execution is everything,” Singh concluded.